API: df.rolling(..).corr()/cov() when pairwise=True to return MI DataFrame

xref #15601

Author: jreback

doc/source/computation.rst

@@ -505,13 +505,18 @@ two ``Series`` or any combination of ``DataFrame/Series`` or
 - ``DataFrame/DataFrame``: by default compute the statistic for matching column
  names, returning a DataFrame. If the keyword argument ``pairwise=True`` is
  passed then computes the statistic for each pair of columns, returning a
- ``Panel`` whose ``items`` are the dates in question (see :ref:`the next section
+ ``MultiIndexed DataFrame`` whose ``index`` are the dates in question (see :ref:`the next section
  <stats.moments.corr_pairwise>`).
 
 For example:
 
 .. ipython:: python
 
+ df = pd.DataFrame(np.random.randn(1000, 4),
+ index=pd.date_range('1/1/2000', periods=1000),
+ columns=['A', 'B', 'C', 'D'])
+ df = df.cumsum()
+
  df2 = df[:20]
  df2.rolling(window=5).corr(df2['B'])
 
@@ -520,11 +525,16 @@ For example:
 Computing rolling pairwise covariances and correlations
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
+.. warning::
+
+ Prior to version 0.20.0 if ``pairwise=True`` was passed, a ``Panel`` would be returned.
+ This will now return a 2-level MultiIndexed DataFrame, see the whatsnew :ref:`here <whatsnew_0200.api_breaking.rolling_pairwise>`
+
 In financial data analysis and other fields it's common to compute covariance
 and correlation matrices for a collection of time series. Often one is also
 interested in moving-window covariance and correlation matrices. This can be
 done by passing the ``pairwise`` keyword argument, which in the case of
-``DataFrame`` inputs will yield a ``Panel`` whose ``items`` are the dates in
+``DataFrame`` inputs will yield a ``MultiIndexed DataFrame`` whose ``index`` are the dates in
 question. In the case of a single DataFrame argument the ``pairwise`` argument
 can even be omitted:
 
@@ -539,12 +549,12 @@ can even be omitted:
 .. ipython:: python
 
  covs = df[['B','C','D']].rolling(window=50).cov(df[['A','B','C']], pairwise=True)
- covs[df.index[-50]]
+ covs.iloc[-50].unstack()
 
 .. ipython:: python
 
  correls = df.rolling(window=50).corr()
- correls[df.index[-50]]
+ correls.iloc[-50].unstack()
 
 You can efficiently retrieve the time series of correlations between two
 columns using ``.loc`` indexing:
@@ -557,7 +567,7 @@ columns using ``.loc`` indexing:
 .. ipython:: python
 
  @savefig rolling_corr_pairwise_ex.png
- correls.loc[:, 'A', 'C'].plot()
+ correls[('A', 'C')].plot()
 
 .. _stats.aggregate:
 

doc/source/whatsnew/v0.20.0.txt

@@ -13,6 +13,8 @@ Highlights include:
 - The ``.ix`` indexer has been deprecated, see :ref:`here <whatsnew_0200.api_breaking.deprecate_ix>`
 - Switched the test framework to `pytest`_ (:issue:`13097`)
 - A new orient for JSON serialization, ``orient='table'``, that uses the Table Schema spec, see :ref:`here <whatsnew_0200.enhancements.table_schema>`
+- Window Binary Corr/Cov operations return a MultiIndex DataFrame rather than a Panel, see :ref:`here <whhatsnew_0200.api_breaking.rolling_pairwise>`
+
 
 .. _pytest: http://doc.pytest.org/en/latest/
 
@@ -715,6 +717,50 @@ New Behavior:
 
  df.groupby('A').agg([np.mean, np.std, np.min, np.max])
 
+.. _whatsnew_0200.api_breaking.rolling_pairwise:
+
+Window Binary Corr/Cov operations return a MultiIndex DataFrame
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+A binary window operation, like ``.corr()`` or ``.cov()``, when operating on a ``.rolling(..)``, ``.expanding(..)``, or ``.ewm(..)`` object,
+will now return a 2-level ``MultiIndexed DataFrame`` rather than a ``Panel``. These are equivalent in function,
+but MultiIndexed DataFrames enjoy more support in pandas.
+See the section on :ref:`Windowed Binary Operations <stats.moments.binary>` for more information. (:issue:`15677`)
+
+.. ipython:: python
+
+ np.random.seed(1234)
+ df = DataFrame(np.random.rand(100, 2),
+ columns=['A', 'B'],
+ index=pd.date_range('20160101', periods=100, freq='D'))
+ df
+
+Old Behavior:
+
+.. code-block:: ipython
+
+ In [2]: df.rolling(12).corr()
+ Out[2]:
+ <class 'pandas.core.panel.Panel'>
+ Dimensions: 100 (items) x 2 (major_axis) x 2 (minor_axis)
+ Items axis: 2016-01-01 00:00:00 to 2016-04-09 00:00:00
+ Major_axis axis: A to B
+ Minor_axis axis: A to B
+
+New Behavior:
+
+.. ipython:: python
+
+ res = df.rolling(12).corr()
+ res
+
+Retrieving a correlation matrix for a specified index
+
+.. ipython:: python
+
+ res.iloc[-1].unstack()
+
+
 .. _whatsnew_0200.api_breaking.hdfstore_where:
 
 HDFStore where string comparison

pandas/core/window.py

@@ -1652,7 +1652,8 @@ def _cov(x, y):
 
 
 def _flex_binary_moment(arg1, arg2, f, pairwise=False):
- from pandas import Series, DataFrame, Panel
+ from pandas import Series, DataFrame
+
  if not (isinstance(arg1, (np.ndarray, Series, DataFrame)) and
  isinstance(arg2, (np.ndarray, Series, DataFrame))):
  raise TypeError("arguments to moment function must be of type "
@@ -1703,12 +1704,34 @@ def dataframe_from_int_dict(data, frame_template):
  else:
  results[i][j] = f(*_prep_binary(arg1.iloc[:, i],
  arg2.iloc[:, j]))
+
+ # TODO: not the most efficient (perf-wise)
+ # though not bad code-wise
+ from pandas import Panel, MultiIndex
  p = Panel.from_dict(results).swapaxes('items', 'major')
  if len(p.major_axis) > 0:
  p.major_axis = arg1.columns[p.major_axis]
  if len(p.minor_axis) > 0:
  p.minor_axis = arg2.columns[p.minor_axis]
- return p
+
+ if len(p.items):
+ result = pd.concat(
+ [p.iloc[i].T for i in range(len(p.items))],
+ keys=p.items)
+ else:
+
+ result = DataFrame(
+ index=MultiIndex(levels=[arg1.columns, arg2.index],
+ labels=[[], []]),
+ columns=arg1.columns,
+ dtype='float64')
+
+ # reset our names
+ result.columns.name = None
+ result.index.names = ['major', 'minor']
+
+ return result
+
  else:
  raise ValueError("'pairwise' is not True/False")
  else: