ENH: Add pipe method

[TomAugspurger]

Closes #10129

In the dev meeting, we settled on the following:

• .pipe will not include a check for __pipe_func__ on the function passed in.
• To avoid messiness with lambdas when a function takes the DataFrame other than in the first position, users can pass in a (callable, data_keyword) argument to .pipe (thanks @mwaskom)

 import statsmodels.formula.api as sm bb = pd.read_csv('data/baseball.csv', index_col='id') (bb.query('h > 0') .assign(ln_h = lambda df: np.log(df.h)) # sm.possion expects `formula, data` .pipe((sm.poisson, 'data'), 'hr ~ ln_h + year + g + C(lg)') .fit() .summary() ) ## -- End pasted text -- Optimization terminated successfully. Current function value: 2.116284 Iterations 24 Out[1]: <class 'statsmodels.iolib.summary.Summary'> """  Poisson Regression Results ============================================================================== Dep. Variable: hr No. Observations: 68 Model: Poisson Df Residuals: 63 Method: MLE Df Model: 4 Date: Tue, 02 Jun 2015 Pseudo R-squ.: 0.6878 Time: 20:57:27 Log-Likelihood: -143.91 converged: True LL-Null: -460.91  LLR p-value: 6.774e-136 ===============================================================================  coef std err z P>|z| [95.0% Conf. Int.] ------------------------------------------------------------------------------- Intercept -1267.3636 457.867 -2.768 0.006 -2164.767 -369.960 C(lg)[T.NL] -0.2057 0.101 -2.044 0.041 -0.403 -0.008 ln_h 0.9280 0.191 4.866 0.000 0.554 1.302 year 0.6301 0.228 2.762 0.006 0.183 1.077 g 0.0099 0.004 2.754 0.006 0.003 0.017 =============================================================================== """

Thanks everyone for the input in the issue.

---

This is mostly ready. What's a good name for the argument to pipe? I have func right now, @shoyer you had target in the issue thread. I've been using target as the keyword expecting the data, i.e. where the DataFrame should be pipe to.

The tests are extremely minimal... but so is the implementation. Am I missing any obvious edge-cases?

We'll see how this goes. I don't think I push .pipe as a protocol at all in the documentation, though we can change that in the future. We should be forwards-compatible if we do ever go down the __pipe_func__ route.

[jreback]

maybe use pure instead?

[TomAugspurger]

I'm being pedantic, but "pure" implies no-side effects. df.plot() is non-mutating, but not pure since it has the side-effect of drawing a plot. I didn't want some functional programming guru to call us out :)

[jreback]

sure...

[shoyer]

A couple of other things that would be nice to highlight in the docs:

1. Let's mention how this was inspired by popular pipe operator %>% from R's magrittr package, but the implementation here is explicit and Pythonic. I would encourage reading the source code -- we might even include it in the docs.
2. We should encourage just using pipe instead of monkey patching. I would consider removing the mention of monkey patching at all -- this is a far better way to go.

[jorisvandenbossche]

I think the comma's at the end of the lines are not correct here?

[TomAugspurger]

I would consider removing the mention of monkey patching at all

Any objections to removing this section from the docs? I forgot it even existed.

[TomAugspurger]

Let's mention how this was inspired by popular pipe operator %>% from R's magrittr package, but the implementation here is explicit and Pythonic. I would encourage reading the source code -- we might even include it in the docs.

The pipe method is inspired by unix pipes and more recently dplyr_ and magrittr_, which
have introduced the popular (%>%) (read pipe) operator for R_.
The implementation of pipe here is quite clean and feels right at home in python.
We encourage you to view the source code (pd.DataFrame.pipe?? in IPyhton).

---

Ok, addressed all the comments I think (thanks). I removed the section on monkey patching.

The outstanding issue I see is @shoyer's comment about maybe checking whether we're about to clobber a kwarg when the tuple-style is used.

if target in kwargs: raise ValueError('%s is both the pipe target and a keyword argument' % target)

to catch a case of df.add(1).pipe((f, 'data'), x=x, y=y, data=df). Right now we (silently) replace kwargs['data'], The closest parallel I see is writing f(a=1, a=2), which Python catches.

[shoyer]

IPyhton -> IPython

[TomAugspurger]

Thanks, I should read what I write :/

[TomAugspurger]

I put the kwarg clobbering (df.pipe((f, 'data'), x=1, data=2)) in a second commit: 968d0ae

EDIT: I also removed the monkey patching docs FYI, can reinstate them if anyone is attached to them.

[jreback]

pls rebase on master and repush had an issue with some builds

[TomAugspurger]

Travis is green. Are we good with checking whether the target is clobbered? And are we OK with raising a ValueError instead of SyntaxError like f(a=2, a=3) does?

[jreback]

this needs backticks to pick up the references

[TomAugspurger]

backticks on the "Elementwise_"? It works w/o the backticks since it's a single word. Are do you mean the method references?

[jreback]

oh, ok, then didn't know that.

[TomAugspurger]

Ok, fixed that newline in the docstring and I just link people to the pipe section from internals.rst instead of the monkeypathcing.

[jreback]

minor point, but I usually have these refer to the whatsnew itself, e.g. below (with the actual doc link from the whatsnew to the docs)

[TomAugspurger]

I think the numba one is linking to the docs (could be wrong). Want me to change that to the section in whatsnew?

[TomAugspurger]

nvm, didn't see it doesn't have a section

[jreback]

the numba one doesn't have another what's entry that's why its that way. Whereas you have a section in the whatsnew (which is good). So the link will skip to there. Then you already have the link back to the docs (at the end).

[jreback]

@TomAugspurger lgtm. merge when ready (see that minor point above though)

[shoyer]

spelling: function

[shoyer]

looks pretty much good to go for me, too, after a few quick fixes

[TomAugspurger]

Ok, I added a section header for pipe in whatsnew, linked to that from above. Fixed the (two) typos on function, and added the ... continuations in the docstring.

[TomAugspurger]

We waiting on Travis? It's felt slow today :/

[jreback]

? I think you checked in after you buildt the docs....!

[TomAugspurger]

Bah, one sec.

[jreback]

travis IS slow today (on master anyhow). not real sure why.

[TomAugspurger]

OK, fixed the accidental deletion of api.rst

[jreback]

maybe show an example of using the callable & data_keyword in the Notes? (can do later)

[TomAugspurger]

Added.

[TomAugspurger]

@shoyer @jreback merge at your leisure. Or I'll be back online later tonight to push the button :)

[shoyer]

spelling: argument

[TomAugspurger]

I'm normally not the bad at spelling.

[shoyer]

"the bad"? :)

[TomAugspurger]

Ugh. Long day.

[jreback]

http://blog.ibis-project.org/design-composability/

from @wesm