How do I handle these pandas error messages?

Question

Often when using pandas I get UserWarning and PerformanceWarning messages like these:

C:\Users\User\Anaconda3\lib\site-packages\pandas\core\reshape\merge.py:558: UserWarning: merging between different levels can give an unintended result (2 levels on the left, 1 on the right) warnings.warn(msg, UserWarning) C:\Users\User\Anaconda3\lib\site-packages\pandas\core\reshape\merge.py:558: UserWarning: merging between different levels can give an unintended result (1 levels on the left, 2 on the right) warnings.warn(msg, UserWarning) C:\Users\User\Anaconda3\lib\site-packages\pandas\core\generic.py:2530: PerformanceWarning: dropping on a non-lexsorted multi-index without a level parameter may impact performance. obj = obj._drop_axis(labels, axis, level=level, errors=errors)

When writing large scripts, I find difficult to know what in my code is the source of the warning.

So how can I figure out what line of my source code generates the warning messages?

Keith Dowd · Accepted Answer · 2018-02-20 17:47:37Z

One approach that I often use is to configure the filterwarnings() method in the warnings package to filter the warnings to raise which will enable you to debug them (e.g., using pdb). To do this you just need to import the warnings package and then set the filterwarnings() method on warnings to watch and raise specific warnings, like this:

import warnings warnings.filterwarnings('error', category=UserWarning) warnings.filterwarnings('error', category=PerformanceWarning)

You can also just configure warnings to raise any warning, like this:

import warnings warnings.filterwarnings('error')

You can learn more about using pdb here: https://docs.python.org/3/library/pdb.html

Initially I used the second option, as PerformanceWarning is not defined in the Warning classes of the package. Since the PerformanceWarning bothers me most, I've looked for a way to filter just that, and found this tip from one of pandas contribuitors: warnings.filterwarnings('error',category=pandas.io.pytables.PerformanceWarning) where I found: github.com/pandas-dev/pandas/issues/3622
Couldn't edit the comment so just removed and added another, sorry for the mass and thank you very much for the help!

M. Neish · Accepted Answer · 2018-10-03 20:24:26Z

If your intention is to resolve the underlying issues, you can use the -W option on the command-line to turn the warning into an error, which will give you the full stack trace.

From the command-line documentation:

-W arg : warning control; arg is action:message:category:module:lineno also PYTHONWARNINGS=arg

E.g., to trap the warnings that start with "merging between different levels...", you could run:

python -Werror:merging myscript.py

This is a good tip and a better answer. It helped me identify the specific line of code which triggered the PerformanceWarning mentioned in the question. I changed the word merging to dropping to find it. python -Werror:dropping myscript.py

Collectives™ on Stack Overflow

How do I handle these pandas error messages?

2 Answers 2

4 Comments

1 Comment

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

1 Comment

Related