ENH: Allow parameters method and min_periods in DataFrame.corrwith() #15573
Conversation
| df2 = pd.DataFrame(np.random.randn(4, 4), index=index[:4], columns=columns) | ||
| df1.corrwith(df2) | ||
| df2.corrwith(df1, axis=1) | ||
| df2.corrwith(df1, axis=1, method='kendall') |
There was a problem hiding this comment.
add versionsddes tag (and small comment here)
| | ||
| correl = num / dom | ||
| correl = Series({col: nanops.nancorr(left[col].values, | ||
| right[col].values, |
There was a problem hiding this comment.
this is going to be very slow
we need to rework nancorr to do this instead
There was a problem hiding this comment.
I think the new implementation (which calls nancorr which in turns calls numpy/scipy correlation functions) is actually significantly faster than the current implementation (manually computing Pearson correlation using DataFrame.mean(), DataFrame.sum(), and DataFrame.std())
For example:
Current implementation:
>>> import pandas as pd; import timeit >>> pd.__version__ u'0.19.2' >>> iris = pd.read_csv('https://raw.githubusercontent.com/mwaskom/seaborn-data/master/iris.csv') >>> timeit.timeit(lambda: iris.corrwith(iris), number=10000) 50.891642808914185 >>> timeit.timeit(lambda: iris.T.corrwith(iris.T), number=10000) 42.0677649974823New implementation:
>>> import pandas as pd; import timeit >>> pd.__version__ '0.19.0+539.g0b77680.dirty' >>> iris = pd.read_csv('https://raw.githubusercontent.com/mwaskom/seaborn-data/master/iris.csv') >>> timeit.timeit(lambda: iris.corrwith(iris, method='pearson'), number=10000) 28.622286081314087 >>> timeit.timeit(lambda: iris.T.corrwith(iris.T, method='pearson'), number=10000) 21.898916959762573I'm pretty new to this, so please let me know if I'm missing anything here.
There was a problem hiding this comment.
look thru the benchmarks and pls add some asv as appropriate
include wide and talk data
on wide data this will be slower
| can you update |
| can you rebase, add some benchmarks to asv and show them. |
jreback added Numeric Operations | can you rebase and update? |
| closing as stale |
2 participants
Added new keyword parameters for
DataFrame.corrwith(), which allows methods other than Pearson to be used. See #9490.git diff upstream/master | flake8 --diff