API: provide Rolling/Expanding/EWM objects for deferred rolling type calculations #10702

[jreback]

closes #10702
closes #9052
xref #4950, removal of depr

So this basically takes all of the pd.rolling_*,pd.expanding_*,pd.ewma_* routines and allows an object oriented interface, similar to groupby.

Some benefits:

• nice tab completions on the Rolling/Expanding/EWM objects
• much cleaner code internally
• complete back compat, e.g. everything just works like it did
• added a .agg/aggregate function, similar to groupby, where you can do multiple aggregations at once
• added __getitem__ accessing, e.g. df.rolling(....)['A','B'].sum() for a nicer API
• allows for much of API/ENH: master issue for pd.rolling_apply #8659 to be done very easily
• fix for coercing Timedeltas properly
• handling nuiscance (string) columns

Other:

• along with window doc rewrite, fixed doc-strings for groupby/window to provide back-refs

ToDO:

• I think that all of the doc-strings are correct, but need check
• implement .agg
• update API.rst, what's new
• deprecate the pd.expanding_*,pd.rolling_*,pd.ewma_* interface as this is polluting the top-level namespace quite a bit
• change the docs to use the new API

In [4]: df = DataFrame({'A' : range(5), 'B' : pd.timedelta_range('1 day',periods=5), 'C' : 'foo'}) In [5]: df.rolling(window=2).sum() Out[5]: A B C 0 NaN NaT foo 1 1 3 days foo 2 3 5 days foo 3 5 7 days foo 4 7 9 days foo In [6]: df.rolling(window=2)['A','C'].sum() Out[6]: A C 0 NaN foo 1 1 foo 2 3 foo 3 5 foo 4 7 foo In [2]: r = df.rolling(window=3) In [3]: r. r.A r.C r.corr r.cov r.max r.median r.name r.skew r.sum r.B r.apply r.count r.kurt r.mean r.min r.quantile r.std r.var 

do rolling/expanding/ewma ops

In [1]: s = Series(range(5)) In [2]: r = s.rolling(2) I# pd.rolling_sum In [3]: r.sum() Out[3]: 0 NaN 1 1 2 3 3 5 4 7 dtype: float64 # nicer repr In [4]: r Out[4]: Rolling [window->2,center->False,axis->0] In [5]: e = s.expanding(min_periods=2) # pd.expanding_sum In [6]: e.sum() Out[6]: 0 NaN 1 1 2 3 3 6 4 10 dtype: float64 In [7]: em = s.ewm(com=10) # pd.ewma In [8]: em.mean() Out[8]: 0 0.000000 1 0.523810 2 1.063444 3 1.618832 4 2.189874 dtype: float64 

and allow the various aggregation type of ops (similar to groupby)

In [1]: df = DataFrame({'A' : range(5), 'B' : pd.timedelta_range('1 day',periods=5), 'C' : 'foo'}) In [2]: r = df.rolling(2,min_periods=1) In [3]: r.agg([np.sum,np.mean]) Out[3]: A B C sum mean sum mean sum mean 0 0 0.0 1 days 1 days 00:00:00 foo foo 1 1 0.5 3 days 1 days 12:00:00 foo foo 2 3 1.5 5 days 2 days 12:00:00 foo foo 3 5 2.5 7 days 3 days 12:00:00 foo foo 4 7 3.5 9 days 4 days 12:00:00 foo foo In [4]: r.agg({'A' : 'sum', 'B' : 'mean'}) Out[4]: A B 0 0 1 days 00:00:00 1 1 1 days 12:00:00 2 3 2 days 12:00:00 3 5 3 days 12:00:00 4 7 4 days 12:00:00 

[shoyer]

Cc @jhamman who has been working on this for xray.

[jreback]

pushing to 0.18.0, I think __getitem__ will be a really nice add here. might as well do all of this at once.

[jreback]

ok, this is ready, MUCH bigger rabbit hole that I thought.

note on the doc-strings.

Since now we have much more like a groupby interface, e.g.

s.rolling(....).sum(), the doc-strings for Rolling.sum are minimal but have a See also back to the Series/DataFrame.rolling (We don't have the notion of a RollingSeries,RolldingDataFrame class so this would be quite tricky).

Further I did the same with groupby doc-strings (again don't have the class distinction on the See Also).

@jorisvandenbossche @shoyer @sinhrks @TomAugspurger @cpcloud

[seth-p]

One thing I've wanted to add but haven't, which may be easier using this framework, at least interface-wise, is rolling exponentially weighted functions -- i.e. add a window to all the ewm*() parameters. Obviously this would need to be implemented in Cython for performance, but perhaps interface-wise it would be simpler using the scheme proposed here.

[jreback]

yep that would be quite straightforward to do interface wise but yes would need to be added to the cython functions (but not too hard there)

[jreback]

any comments?

[shoyer]

should be GroupBy

[jreback]

done

[jreback]

so now aggregations are consistent with what you'd expect

In [3]: df = DataFrame({'A' : range(5),'B' : range(0,10,2)}) In [4]: r = df.rolling(window=3) In [5]: r.agg(['mean','sum']) Out[5]: A B mean sum mean sum 0 NaN NaN NaN NaN 1 NaN NaN NaN NaN 2 1 3 2 6 3 2 6 4 12 4 3 9 6 18 In [6]: r['A'].agg(['mean','sum']) Out[6]: mean sum 0 NaN NaN 1 NaN NaN 2 1 3 3 2 6 4 3 9 In [7]: r.agg({'A' : ['mean','sum']}) Out[7]: mean sum 0 NaN NaN 1 NaN NaN 2 1 3 3 2 6 4 3 9 

[jorisvandenbossche]

For the last one (r.agg({'A' : ['mean','sum']})), in the case of groupby you still have the 'A' in the columns:

In [49]: grouped.agg({'C': {'r':np.sum, 'r2':np.mean}}) Out[49]: C r r2 A B bar one 1.249205 1.249205 three -0.262759 -0.262759 two 1.151419 1.151419 foo one -0.518008 -0.259004 three 0.588044 0.588044 two 1.635643 0.817821 

There are some problems with the how and freq keyword. freq is still allowed but deprecated, but the accompanying how is not allowed:

this is correct, you cannot pass how to .mean nor .sum. This is simply invalid syntax. (and nor was it allowed in the original impl).

In any case, this was allowed and is in the docstring. Eg the example I gave works for 0.16.2:

In [63]: ser = pd.Series(np.random.randn(20), index=pd.date_range('1/1/2000', pe riods=20, freq='12H')) In [64]: pd.rolling_mean(ser, window=5, freq='D', how='max') Out[64]: 2000-01-01 NaN 2000-01-02 NaN 2000-01-03 NaN 2000-01-04 NaN 2000-01-05 0.314516 2000-01-06 0.511396 2000-01-07 0.343306 2000-01-08 0.561190 2000-01-09 0.772309 2000-01-10 0.266875 Freq: D, dtype: float64 In [65]: pd.rolling_mean(ser, window=5, freq='D', how='min') Out[65]: 2000-01-01 NaN 2000-01-02 NaN 2000-01-03 NaN 2000-01-04 NaN 2000-01-05 -0.439707 2000-01-06 -0.431944 2000-01-07 -0.538390 2000-01-08 -0.296774 2000-01-09 -0.112988 2000-01-10 -0.425533 Freq: D, dtype: float64 

[jreback]

I think my code was correct at the beginning.

In [4]: r.agg(['sum','mean']) Out[4]: A B sum mean sum mean 0 NaN NaN NaN NaN 1 NaN NaN NaN NaN 2 3 1 6 2 3 6 2 12 4 4 9 3 18 6 In [5]: r.agg({'A' : ['sum','mean']}) Out[5]: A sum mean 0 NaN NaN 1 NaN NaN 2 3 1 3 6 2 4 9 3 In [6]: r['A'].agg(['sum','mean']) Out[6]: sum mean 0 NaN NaN 1 NaN NaN 2 3 1 3 6 2 4 9 3 In [7]: r['A'].agg({'s' : 'sum', 'm' : 'mean' }) Out[7]: s m 0 NaN NaN 1 NaN NaN 2 3 1 3 6 2 4 9 3 In [8]: r.agg({'A' : {'s' : 'sum', 'm' : 'mean' }}) Out[8]: A s m 0 NaN NaN 1 NaN NaN 2 3 1 3 6 2 4 9 3 

[jreback]

might be still some small loose ends, but any further comments @jorisvandenbossche @shoyer

[jreback]

bombs away!

[jreback]

@jorisvandenbossche http://pandas-docs.github.io/pandas-docs-travis/computation.html#stats-moments

does the :math not render on travis doc builds?

[max-sixty]

👏