Skip to content

BUG: inconsistent DataFrame.agg behavoir when passing as kwargs numeric_only=True #49352

@arnaudlegout

Description

@arnaudlegout

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

Python 3.11.0 | packaged by conda-forge | (main, Oct 25 2022, 06:12:32) [MSC v.1929 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import pandas as pd >>> pd.__version__ '1.5.1' >>> df = pd.DataFrame({'a': [1, 2, 3], 'b': list('abc')}) >>> df.agg('mean', numeric_only=True) a 2.0 dtype: float64 >>> df.agg(['mean', 'std'], numeric_only=True) <stdin>:1: FutureWarning: ['b'] did not aggregate successfully. If any error is raised this will raise in a future version of pandas. Drop these columns/ops to avoid this warning. a mean 2.0 std 1.0 >>> df.agg(pd.DataFrame.mean, numeric_only=True) <stdin>:1: FutureWarning: Calling Series.mean with numeric_only=True and dtype object will raise a TypeError in the future Traceback (most recent call last): File "<stdin>", line 1, in <module> File "C:\Users\alegout\Miniconda3\envs\pandas1.5\Lib\site-packages\pandas\core\frame.py", line 9329, in aggregate result = op.agg() ^^^^^^^^ File "C:\Users\alegout\Miniconda3\envs\pandas1.5\Lib\site-packages\pandas\core\apply.py", line 773, in agg result = self.obj.apply(self.orig_f, axis, args=self.args, **self.kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\alegout\Miniconda3\envs\pandas1.5\Lib\site-packages\pandas\core\frame.py", line 9555, in apply return op.apply().__finalize__(self, method="apply") ^^^^^^^^^^ File "C:\Users\alegout\Miniconda3\envs\pandas1.5\Lib\site-packages\pandas\core\apply.py", line 746, in apply return self.apply_standard() ^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\alegout\Miniconda3\envs\pandas1.5\Lib\site-packages\pandas\core\apply.py", line 873, in apply_standard results, res_index = self.apply_series_generator() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\alegout\Miniconda3\envs\pandas1.5\Lib\site-packages\pandas\core\apply.py", line 889, in apply_series_generator results[i] = self.f(v) ^^^^^^^^^ File "C:\Users\alegout\Miniconda3\envs\pandas1.5\Lib\site-packages\pandas\core\apply.py", line 139, in f return func(x, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\alegout\Miniconda3\envs\pandas1.5\Lib\site-packages\pandas\core\generic.py", line 11847, in mean return NDFrame.mean(self, axis, skipna, level, numeric_only, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\alegout\Miniconda3\envs\pandas1.5\Lib\site-packages\pandas\core\generic.py", line 11401, in mean return self._stat_function( ^^^^^^^^^^^^^^^^^^^^ File "C:\Users\alegout\Miniconda3\envs\pandas1.5\Lib\site-packages\pandas\core\generic.py", line 11353, in _stat_function return self._reduce( ^^^^^^^^^^^^^ File "C:\Users\alegout\Miniconda3\envs\pandas1.5\Lib\site-packages\pandas\core\series.py", line 4812, in _reduce raise NotImplementedError( NotImplementedError: Series.mean does not implement numeric_only. >>>

Issue Description

I expect DataFrame.agg to accept kwargs arguments passed to the aggregation functions.
It works in a seemingly inconsistent way.

df.agg('mean', numeric_only=True) works as expected.

But passing multiple functions as string in a list does not work as expected as we still get a FutureWarning showing that the argument numeric_only=True was not correctly passed to either mean, or std, or both

df.agg(['mean', 'std'], numeric_only=True) >>> <stdin>:1: FutureWarning: ['b'] did not aggregate successfully. If any error is raised this will raise in a future version of pandas. Drop these columns/ops to avoid this warning. a mean 2.0 std 1.0 

Even more surprising, using the function pd.DataFrame.mean raises a NotImplementedError for Series.mean

>>> df.agg(pd.DataFrame.mean, numeric_only=True) <stdin>:1: FutureWarning: Calling Series.mean with numeric_only=True and dtype object will raise a TypeError in the future Traceback (most recent call last): File "<stdin>", line 1, in <module> File "C:\Users\alegout\Miniconda3\envs\pandas1.5\Lib\site-packages\pandas\core\frame.py", line 9329, in aggregate result = op.agg() ^^^^^^^^ File "C:\Users\alegout\Miniconda3\envs\pandas1.5\Lib\site-packages\pandas\core\apply.py", line 773, in agg result = self.obj.apply(self.orig_f, axis, args=self.args, **self.kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\alegout\Miniconda3\envs\pandas1.5\Lib\site-packages\pandas\core\frame.py", line 9555, in apply return op.apply().__finalize__(self, method="apply") ^^^^^^^^^^ File "C:\Users\alegout\Miniconda3\envs\pandas1.5\Lib\site-packages\pandas\core\apply.py", line 746, in apply return self.apply_standard() ^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\alegout\Miniconda3\envs\pandas1.5\Lib\site-packages\pandas\core\apply.py", line 873, in apply_standard results, res_index = self.apply_series_generator() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\alegout\Miniconda3\envs\pandas1.5\Lib\site-packages\pandas\core\apply.py", line 889, in apply_series_generator results[i] = self.f(v) ^^^^^^^^^ File "C:\Users\alegout\Miniconda3\envs\pandas1.5\Lib\site-packages\pandas\core\apply.py", line 139, in f return func(x, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\alegout\Miniconda3\envs\pandas1.5\Lib\site-packages\pandas\core\generic.py", line 11847, in mean return NDFrame.mean(self, axis, skipna, level, numeric_only, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\alegout\Miniconda3\envs\pandas1.5\Lib\site-packages\pandas\core\generic.py", line 11401, in mean return self._stat_function( ^^^^^^^^^^^^^^^^^^^^ File "C:\Users\alegout\Miniconda3\envs\pandas1.5\Lib\site-packages\pandas\core\generic.py", line 11353, in _stat_function return self._reduce( ^^^^^^^^^^^^^ File "C:\Users\alegout\Miniconda3\envs\pandas1.5\Lib\site-packages\pandas\core\series.py", line 4812, in _reduce raise NotImplementedError( NotImplementedError: Series.mean does not implement numeric_only. 

Expected Behavior

Using kwargs arguments with DataFrame.agg must be correcly passed to all aggregation function, whatever the way they are given (string name or method reference)

Installed Versions

pd.show_versions()
INSTALLED VERSIONS


commit : 91111fd
python : 3.11.0.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19044
machine : AMD64
processor : Intel64 Family 6 Model 140 Stepping 1, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : fr_FR.cp1252

pandas : 1.5.1
numpy : 1.23.4
pytz : 2022.5
dateutil : 2.8.2
setuptools : 65.5.0
pip : 22.3
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
brotli : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : None
snappy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
zstandard : None
tzdata : None

Metadata

Metadata

Labels

BugClosing CandidateMay be closeable, needs more eyeballsRegressionFunctionality that used to work in a prior pandas versionWarningsWarnings that appear or should be added to pandas

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions