My Python library just changed it's main module name from foo.bar to foobar. For backward compat, foo.bar still exists, but importing it raises a few warnings. Now, it seems some example program still imports from the old module, but not directly.
I'd like to find the erroneous import statement. Is there any tool that allows me to trace imports and find the culprit without wading through all of the code?
Start the python interpreter with -v:
$ python -v -m /usr/lib/python2.6/timeit.py # installing zipimport hook import zipimport # builtin # installed zipimport hook # /usr/lib/python2.6/site.pyc matches /usr/lib/python2.6/site.py import site # precompiled from /usr/lib/python2.6/site.pyc # /usr/lib/python2.6/os.pyc matches /usr/lib/python2.6/os.py import os # precompiled from /usr/lib/python2.6/os.pyc import errno # builtin import posix # builtin # /usr/lib/python2.6/posixpath.pyc matches /usr/lib/python2.6/posixpath.py import posixpath # precompiled from /usr/lib/python2.6/posixpath.pyc # /usr/lib/python2.6/stat.pyc matches /usr/lib/python2.6/stat.py import stat # precompiled from /usr/lib/python2.6/stat.pyc # /usr/lib/python2.6/genericpath.pyc matches /usr/lib/python2.6/genericpath.py import genericpath # precompiled from /usr/lib/python2.6/genericpath.pyc # /usr/lib/python2.6/warnings.pyc matches /usr/lib/python2.6/warnings.py import warnings # precompiled from /usr/lib/python2.6/warnings.pyc # /usr/lib/python2.6/linecache.pyc matches /usr/lib/python2.6/linecache.py import linecache # precompiled from /usr/lib/python2.6/linecache.pyc # /usr/lib/python2.6/types.pyc matches /usr/lib/python2.6/types.py import types # precompiled from /usr/lib/python2.6/types.pyc # /usr/lib/python2.6/UserDict.pyc matches /usr/lib/python2.6/UserDict.py ... Then just grep for your old module.
PYTHONVERBOSE give more info. If 1 only tells where things were picked from, a higher number will result in messages showing what paths were tried. Helps a lot when figuring out why a package was not found.
import site? Was it timeit.py? Or was it zipimport? In a much more complicated environment I need to know the trace, but cannot find one.edit foo.bar module, add following code:
import pdb pdb.set_trace() when foo.bar be imported, program will stop at pdb.set_trace() in pdb mode, where you can debug your code. For example, you can use "w" command to print the full calling stack.
I will first introduce the general solution and then show how it can be adapted for this particular use case.
Nowadays there are easier ways to debug such things utilizing the new functionalities of the python import system. Basically just add your own module finders (MetaPathFinder) to sys.meta_path. Here is an example for a script importing pandas and listing all the succesful imports in imported and unsuccesful imports in could_not_be_imported:
import sys imported = [] could_not_be_imported = [] class FirstFinder: def find_spec(self, modulename, path=None, target=None): imported.append(modulename) class LastFinder: def find_spec(self, modulename, path=None, target=None): imported.remove(modulename) could_not_be_imported.append(modulename) # setup ("start recording imports") sys.meta_path.insert(0, FirstFinder()) sys.meta_path.append(LastFinder()) import pandas # import anything here # cleanup ("stop recording") sys.meta_path = [ x for x in sys.meta_path if not isinstance(x, (FirstFinder, LastFinder)) ] The list containing the succesful imports is then imported:
>>> imported ['pandas', 'numpy', 'numpy._globals', # ... 'pandas.io.sql', 'pandas.io.stata', 'pandas.io.xml', 'pandas.util._tester', 'pandas._version'] and could_not_be_imported:
>>> could_not_be_imported ['pickle5', 'org', 'fcntl', 'backports_abc', 'six.moves', 'backports_abc', 'backports_abc', # ... 'cloudpickle', 'numexpr', 'bottleneck', 'org', 'backports_abc', 'backports_abc', 'backports_abc'] Note: There can be duplicates in either of the lists and all the modules are listed in order of imports. Modify as fits to your needs.
Modify the LastFinder to be:
class LastFinder: def find_spec(self, modulename, path=None, target=None): imported.remove(modulename) print( "\nMissing module! ", f"Tried to import: {modulename}\n", "Last few last imports:\n\t", "\n\t ".join(imported[-10:]), ) could_not_be_imported.append(modulename) This will then print, with every missing module, a list of the few last succesfully imported modules and the module you were trying to import. For example:
Missing module! Tried to import: bottleneck Last few last imports: pandas.core.ops.common pandas.core.ops.docstrings pandas.core.ops.mask_ops pandas.core.ops.methods pandas.core.missing pandas.core.array_algos.quantile pandas.core.sorting pandas.core.arrays.boolean pandas.core.arrays.masked pandas.core.nanops In this case this would mean that the pandas.core.nanops was the last succcesful import before the unsuccesful import. Therefore, it would be quite easy to check from pandas.core.nanops.__file__ where the broken import is. (look for "bottleneck" in the file). In my case, I found:
# pandas\core\nanops.py # ... bn = import_optional_dependency("bottleneck", errors="warn") The python import system goes through the sys.meta_path to look for a spec finder which will have find_spec method which returns something else than None. If None is returned, next finder in sys.meta_path is used for the import.
