Skip to content

BUG: Rolling .apply() with method='table' ignores min_periods #58868

@kasparthommen

Description

@kasparthommen

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd df = pd.DataFrame([[1, 2], [3, 4], [5, 6], [7, 8], [9, 10]], columns=['A', 'B']) print(df) # prints this: # A B # 0 1 2 # 1 3 4 # 2 5 6 # 3 7 8 # 4 9 10 df.rolling(3, min_periods=3).apply(lambda x: bool(print(x, '\n'))) # prints this, so as expected, only batches of 3 are being evaluated # 0 1.0 # 1 3.0 # 2 5.0 # dtype: float64  #  # 1 3.0 # 2 5.0 # 3 7.0 # dtype: float64  #  # 2 5.0 # 3 7.0 # 4 9.0 # dtype: float64  #  # 0 2.0 # 1 4.0 # 2 6.0 # dtype: float64  #  # 1 4.0 # 2 6.0 # 3 8.0 # dtype: float64  #  # 2 6.0 # 3 8.0 # 4 10.0 # dtype: float64  #  # Out[33]:  # A B # 0 NaN NaN # 1 NaN NaN # 2 0.0 0.0 # 3 0.0 0.0 # 4 0.0 0.0 df.rolling(3, method='table', min_periods=3).apply(lambda x: bool(print(x, '\n')), raw=True, engine='numba') # prints this, where we see that the first two batches of sizes 1 and 2 are also being evaluated # [[1. 2.]] <-- batch of size 1, illegal #  # [[1. 2.] <-- batch of size 2, illegal # [3. 4.]]  #  # [[1. 2.] # [3. 4.] # [5. 6.]]  #  # [[3. 4.] # [5. 6.] # [7. 8.]]  #  # [[ 5. 6.] # [ 7. 8.] # [ 9. 10.]]  # Out[34]:  # A B # 0 NaN NaN # 1 NaN NaN # 2 0.0 0.0 # 3 0.0 0.0 # 4 0.0 0.0

Issue Description

See example above, which violates the contract imposed by the min_periods parameter.

Expected Behavior

The min_periods parameter should be respected.

Installed Versions

INSTALLED VERSIONS

commit : d9cdd2e
python : 3.12.3.final.0
python-bits : 64
OS : Windows
OS-release : 11
Version : 10.0.22631
machine : AMD64
processor : Intel64 Family 6 Model 165 Stepping 2, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : German_Switzerland.utf8
pandas : 2.2.2
numpy : 1.26.4
pytz : 2024.1
dateutil : 2.9.0
setuptools : 69.5.1
pip : 24.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 5.2.2
html5lib : 1.1
pymysql : None
psycopg2 : None
jinja2 : 3.1.4
IPython : 8.24.0
pandas_datareader : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.12.3
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : 3.8.4
numba : 0.59.1
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 16.0.0
pyreadstat : None
python-calamine : None
pyxlsb : None
s3fs : None
scipy : 1.13.0
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
zstandard : None
tzdata : 2024.1
qtpy : None
pyqt5 : None

Metadata

Metadata

Assignees

No one assigned

    Labels

    ApplyApply, Aggregate, Transform, MapBugWindowrolling, ewma, expandingnumbanumba-accelerated operations

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions