-
- Notifications
You must be signed in to change notification settings - Fork 19.3k
Description
Improve coverage of NDFrame.__finalize__
Pandas uses NDFrame.__finalize__ to propagate metadata from one NDFrame to
another. This ensures that things like self.attrs and self.flags are not
lost. In general we would like that any operation that accepts one or more
NDFrames and returns an NDFrame should propagate metadata by calling
__finalize__.
The test file at
https://github.com/pandas-dev/pandas/blob/master/pandas/tests/generic/test_finalize.py
attempts to be an exhaustive suite of tests for all these cases. However there
are many tests currently xfailing, and there are likely many APIs not covered.
This is a meta-issue to improve the use of __finalize__. Here's a hopefully
accurate list of methods that don't currently call finalize.
Some general comments around finalize
- We don't have a good sense for what should happen to
attrswhen there are
multiple NDFrames involved with differing attrs (e.g. in concat). The safest
approach is to probably drop the attrs when they don't match, but this will
need some thought. - We need to be mindful of performance.
__finalize__can be somewhat expensive
so we'd like to call it exactly once per user-facing method. This can be tricky
for things likeDataFrame.applywhich is sometimes used internally. We may need
to refactor some methods to have a user-facingDataFrame.applythat calls an internal
DataFrame._apply. The internal method would not call__finalize__, just the
user-facingDataFrame.applywould.
If you're interested in working on this please post a comment indicating which method
you're working on. Un-xfail the test, then update the method to pass the test. Some of these
will be much more difficult to work on than others (e.g. groupby is going to be difficult). If you're
unsure whether a particular method is likely to be difficult, ask first.
-
DataFrame.__getitem__with a scalar -
DataFrame.evalwithengine="numexpr" -
DataFrame.duplicated -
DataFrame.add,mul, etc. (at least for most things; some work to do on conflicts / overlapping attrs in binops) -
DataFrame.combine,DataFrame.combine_first -
DataFrame.update -
DataFrame.pivot,pivot_table -
DataFrame.stack -
DataFrame.unstack -
DataFrame.explodeBUG: added finalize to explode, GH28283 #46629 -
DataFrame.meltBUG/TST/DOC: added finalize to melt, GH28283 #46648 -
DataFrame.diff -
DataFrame.applymap -
DataFrame.append -
DataFrame.merge -
DataFrame.cov -
DataFrame.corrwith -
DataFrame.count -
DataFrame.nunique -
DataFrame.idxmax,idxmin -
DataFrame.mode -
DataFrame.quantile(scalar and list of quantiles) -
DataFrame.isin -
DataFrame.pop -
DataFrame.squeeze -
Series.abs -
DataFrame.get -
DataFrame.round -
DataFrame.convert_dtypes -
DataFrame.pct_change -
DataFrame.transform -
DataFrame.apply -
DataFrame.any,sum,std,mean, etdc. -
Series.str.operations returning a Series / DataFrame -
Series.dt.operations returning a Series / DataFrame -
Series.cat.operations returning a Series / DataFrame - All groupby operations (at least some work)
-
.iloc/.locAdd missing __finalize__ calls in indexers/iterators #46101