Skip to content

BUG: RollingGroupBy.quantile ignores interpolation keyword argument #29567

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Nov 18, 2019
Merged

BUG: RollingGroupBy.quantile ignores interpolation keyword argument #29567

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
352445c
BUG: `RollingGroupBy.quantile` ignores `interpolation` keyword argume…
pwsiegel Nov 12, 2019
8a5054d
Merge remote-tracking branch 'upstream/master' into rolling-groupby-q…
pwsiegel Nov 17, 2019
ed7aa3f
Create new test `test_rolling_quantile`
pwsiegel Nov 17, 2019
48cb4e5
Parametrize test_rolling_quantile with interpolation parameters
pwsiegel Nov 17, 2019
fde54ce
add parametrized test `test_expanding_quantile` and removed quantile
pwsiegel Nov 17, 2019
61c5142
Change format of method reference in bug description in whatsnew to be
pwsiegel Nov 17, 2019
File filter

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/source/whatsnew/v1.0.0.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -427,6 +427,7 @@ Groupby/resample/rolling
- Bug in :meth:`DataFrame.groupby` not offering selection by column name when ``axis=1`` (:issue:`27614`)
- Bug in :meth:`DataFrameGroupby.agg` not able to use lambda function with named aggregation (:issue:`27519`)
- Bug in :meth:`DataFrame.groupby` losing column name information when grouping by a categorical column (:issue:`28787`)
- Bug in :meth:`RollingGroupby.quantile` ignoring ``interpolation`` keyword argument (:issue:`28779`)
Copy link
Contributor

@jreback jreback Nov 17, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i don’t think this will render, can u see how we reference rolling in other notes
Copy link
Contributor Author

@pwsiegel pwsiegel Nov 17, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, I'll find another example in the notes and imitate it.
Copy link
Contributor Author

@pwsiegel pwsiegel Nov 17, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed the reference to DataFrameGroupBy.rolling().quantile(), similarly to the reference to DataFrameGroupBy.rolling().apply() here: https://pandas.pydata.org/pandas-docs/version/0.23.4/whatsnew.html#groupby-resample-rolling. Hopefully that will work better?

Reshaping
^^^^^^^^^
Expand Down
4 changes: 3 additions & 1 deletion pandas/core/window/rolling.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1499,7 +1499,9 @@ def f(arg, *args, **kwargs):
interpolation,
)

return self._apply(f, "quantile", quantile=quantile, **kwargs)
return self._apply(
f, "quantile", quantile=quantile, interpolation=interpolation, **kwargs
)

_shared_docs[
"cov"
Expand Down
15 changes: 8 additions & 7 deletions pandas/tests/window/test_grouper.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,6 @@ def test_rolling(self):
r = g.rolling(window=4)

for f in ["sum", "mean", "min", "max", "count", "kurt", "skew"]:

result = getattr(r, f)()
expected = g.apply(lambda x: getattr(x.rolling(4), f)())
tm.assert_frame_equal(result, expected)
Expand All @@ -69,9 +68,10 @@ def test_rolling(self):
expected = g.apply(lambda x: getattr(x.rolling(4), f)(ddof=1))
tm.assert_frame_equal(result, expected)

result = r.quantile(0.5)
expected = g.apply(lambda x: x.rolling(4).quantile(0.5))
tm.assert_frame_equal(result, expected)
for f in ["linear", "lower", "higher", "midpoint", "nearest"]:
Copy link
Member

@WillAyd WillAyd Nov 17, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Either here or in a follow up could you parametrize this test with these values rather than the loop within?
Copy link
Contributor

@jreback jreback Nov 17, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can u make this a separate test and parmetrize
Copy link
Contributor Author

@pwsiegel pwsiegel Nov 17, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think I can handle that here - working on it now.
Copy link
Contributor Author

@pwsiegel pwsiegel Nov 17, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wrote a new test called test_rolling_quantile and parametrized the interpolation argument as requested, but unfortunately it looks as though most of the tests in this file have broken since I merged in master. To test, I checked out the upstream master branch, rebuilt my conda environment from scratch, and ran the tests there. 10 of the tests failed, including test_rolling (stack trace below). Is this a known issue?

self = <pandas.tests.window.test_grouper.TestGrouperGrouping object at 0x116432208> def test_rolling(self): g = self.frame.groupby("A") r = g.rolling(window=4) for f in ["sum", "mean", "min", "max", "count", "kurt", "skew"]: > result = getattr(r, f)() test_grouper.py:64: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ../../core/window/rolling.py:1870: in sum return super().sum(*args, **kwargs) ../../core/window/rolling.py:1245: in sum return self._apply("roll_sum", "sum", **kwargs) ../../core/window/common.py:81: in _apply return self._groupby.apply(f) ../../core/groupby/groupby.py:718: in apply result = self._python_apply_general(f) ../../core/groupby/groupby.py:734: in _python_apply_general keys, values, mutated = self.grouper.apply(f, self._selected_obj, self.axis) ../../core/groupby/ops.py:171: in apply result_values, mutated = splitter.fast_apply(f, group_keys) ../../core/groupby/ops.py:904: in fast_apply return libreduction.apply_frame_axis0(sdata, f, names, starts, ends) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > ??? E AttributeError: 'Int64Index' object has no attribute '_has_complex_internals' pandas/_libs/reduction.pyx:519: AttributeError
Copy link
Contributor Author

@pwsiegel pwsiegel Nov 17, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, @WillAyd's suggestion to rebuild the Cython extensions was correct. The tests are now passing; I'm going to push my changes shortly.
Copy link
Contributor Author

@pwsiegel pwsiegel Nov 17, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change is pushed - let me know if that's different from what you had in mind!
result = r.quantile(0.4, interpolation=f)
expected = g.apply(lambda x: x.rolling(4).quantile(0.4, interpolation=f))
tm.assert_frame_equal(result, expected)

def test_rolling_corr_cov(self):
g = self.frame.groupby("A")
Expand Down Expand Up @@ -141,9 +141,10 @@ def test_expanding(self):
expected = g.apply(lambda x: getattr(x.expanding(), f)(ddof=0))
tm.assert_frame_equal(result, expected)

result = r.quantile(0.5)
expected = g.apply(lambda x: x.expanding().quantile(0.5))
tm.assert_frame_equal(result, expected)
for f in ["linear", "lower", "higher", "midpoint", "nearest"]:
result = r.quantile(0.4, interpolation=f)
expected = g.apply(lambda x: x.expanding().quantile(0.4, interpolation=f))
tm.assert_frame_equal(result, expected)

def test_expanding_corr_cov(self):
g = self.frame.groupby("A")
Expand Down