Is there a straightforward way to find all the modules that are part of a python package? I've found this old discussion, which is not really conclusive, but I'd love to have a definite answer before I roll out my own solution based on os.listdir().
- 6@S.Lott: There are more general solutions available, python packages are not always in directories in the filesystem, but can also be inside zips.u0b34a0f6ae– u0b34a0f6ae2009-11-10 13:43:50 +00:00Commented Nov 10, 2009 at 13:43
- 4why reinvent the wheel? If python acquires hypermodules in Python 4, pkgutil and updated with that, my code will still work. I like to use abstractions that are available. Use the obvious method provided, it is tested and known to work. Reimplementing that.. now you have to find and work around every corner case yourself.u0b34a0f6ae– u0b34a0f6ae2009-11-10 15:48:53 +00:00Commented Nov 10, 2009 at 15:48
- 1@S.Lott: So everytime the application starts, it will unzip its own egg if installed inside one just to check this? Please submit a patch against my project to reinvent the wheel in this function: git.gnome.org/cgit/kupfer/tree/kupfer/plugins.py#n17. Please consider both eggs and normal directories, do not exceed 20 lines.u0b34a0f6ae– u0b34a0f6ae2009-11-10 17:24:59 +00:00Commented Nov 10, 2009 at 17:24
- 1@S.Lott: Why you don't understand that it is relevant is something you can't understand. Discovering this programmatically is about that the application takes interest in the content of a package, not the user.u0b34a0f6ae– u0b34a0f6ae2009-11-10 19:52:49 +00:00Commented Nov 10, 2009 at 19:52
- 3Of course I mean programmatically! Otherwise I wouldn't have mentioned "rolling out my own solution with os.listdir()"static_rtti– static_rtti2009-11-12 19:52:04 +00:00Commented Nov 12, 2009 at 19:52
7 Answers
Yes, you want something based on pkgutil or similar -- this way you can treat all packages alike regardless if they are in eggs or zips or so (where os.listdir won't help).
import pkgutil # this is the package we are inspecting -- for example 'email' from stdlib import email package = email for importer, modname, ispkg in pkgutil.iter_modules(package.__path__): print "Found submodule %s (is a package: %s)" % (modname, ispkg) How to import them too? You can just use __import__ as normal:
import pkgutil # this is the package we are inspecting -- for example 'email' from stdlib import email package = email prefix = package.__name__ + "." for importer, modname, ispkg in pkgutil.iter_modules(package.__path__, prefix): print "Found submodule %s (is a package: %s)" % (modname, ispkg) module = __import__(modname, fromlist="dummy") print "Imported", module 8 Comments
importer returned by pkgutil.iter_modules? Can I use it to import a module instead of using this seemly "hackish" __import__(modname, fromlist="dummy") ?m = importer.find_module(modname).load_module(modname) and then m is the module, so for example: m.myfunc()_path_). There should be two on either side, for a total of four (ie __path__).The right tool for this job is pkgutil.walk_packages.
To list all the modules on your system:
import pkgutil for importer, modname, ispkg in pkgutil.walk_packages(path=None, onerror=lambda x: None): print(modname) Be aware that walk_packages imports all subpackages, but not submodules.
If you wish to list all submodules of a certain package then you can use something like this:
import pkgutil import scipy package=scipy for importer, modname, ispkg in pkgutil.walk_packages(path=package.__path__, prefix=package.__name__+'.', onerror=lambda x: None): print(modname) iter_modules only lists the modules which are one-level deep. walk_packages gets all the submodules. In the case of scipy, for example, walk_packages returns
scipy.stats.stats while iter_modules only returns
scipy.stats The documentation on pkgutil (http://docs.python.org/library/pkgutil.html) does not list all the interesting functions defined in /usr/lib/python2.6/pkgutil.py.
Perhaps this means the functions are not part of the "public" interface and are subject to change.
However, at least as of Python 2.6 (and perhaps earlier versions?) pkgutil comes with a walk_packages method which recursively walks through all the modules available.
6 Comments
walk_packages is now in the documentation: docs.python.org/library/pkgutil.html#pkgutil.walk_packages_) before and after path -- that is, use package.__path__ rather than package._path_. It might be easier to try cutting & pasting the code rather than re-typing it.package is pointing to a package, not a module. Modules are files whereas packages are directories. All packages have the __path__ attribute (... unless someone deleted the attribute for some reason.)This works for me:
import types for key, obj in nltk.__dict__.iteritems(): if type(obj) is types.ModuleType: print key 1 Comment
Thanks to all previous answers, I've just merged them all into one function, which can be easily used to retrieve submodules:
def list_submodules(module) -> list[str]: """ Args: module: The module to list submodules from. """ # We first respect __all__ attribute if it already defined. submodules = getattr(module, "__all__", None) if submodules: return submodules # Then, we respect module object itself to get imported submodules. # Warning: Initially, the module object will respect the `__init__.py` # file, if its not exists, the object can partially load submoudles # by coda, so can lead `inspect` to return incomplete submodules list. import inspect submodules = [o[0] for o in inspect.getmembers(module) if inspect.ismodule(o[1])] if submodules: return submodules # Finally we can just scan for submodules via pkgutil. import pkgutil # pkgutill will invoke `importlib.machinery.all_suffixes()` # to determine whether a file is a module, so if you get any # submoudles that are unexpected to get, you need to check # this function to do the confirmation. # If you want to retrive a directory as a submoudle, you will # need to clarify this by putting a `__init__.py` file in the # folder, even for Python3. return [x.name for x in pkgutil.iter_modules(module.__path__)] Then you can just call it like:
import module print(list_submodules(module)) path = ... module = importlib.import_module(path) print(list_submodules(module)) Comments
In case you are not only interested in listing module names, but you also want to get a reference to the module objects, this answer is for you:
To list modules, use either pkgutil.iter_modules if you need just the direct children of a module, or pkgutil.walk_packages if you need all descendants of a module. Both return ModuleInfo tuples.
To import modules, there are various suggestions in the existing answers, most of which are not great choices:
__import__works if you import a top level module__import__('foo'), but__import__('foo.bar')will also return thefoomodule, notfoo.bar! You can work around this restriction, but it is cumbersome.MetaPathFinder.find_module: has been deprecated since Python 3.4 and was removed in 3.12MetaPathFinder.find_specreplacesfind_module, you can use it by accessing theModuleInfo.module_finderattribute, but it's a bit verbose:
import pkgutil submodules = [ module_info.module_finder.find_spec( f"{my_module.__name__}.{module_info.name}" ).loader.load_module() for module_info in pkgutil.iter_modules(my_module.__path__) ] My preferred method is to use importlib.import_module in combination with pkgutil.iter_modules:
import importlib import pkgutil from types import ModuleType def get_submodules(module: ModuleType) -> list[ModuleType]: return [ importlib.import_module(f"{module.__name__}.{module_info.name}") for module_info in pkgutil.iter_modules(module.__path__) ] a few notes on this solution:
- you can replace
pkgutil.iter_moduleswithpkgutil.walk_packagesif needed importlib.import_modulereturns the module specified by the path, not the module at the root of the path, like__import__- with
f"{module.__name__}.{module_info.name}"we make sure that all modules are referenced by an absolute path (modules can be loaded with shorter paths if the parent module has been imported before, but this can cause issues if you want to filter or compare modules)
Comments
I was looking for a way to reload all submodules that I'm editing live in my package. It is a combination of the answers/comments above, so I've decided to post it here as an answer rather than a comment.
package=yourPackageName import importlib import pkgutil for importer, modname, ispkg in pkgutil.walk_packages(path=package.__path__, prefix=package.__name__+'.', onerror=lambda x: None): try: modulesource = importlib.import_module(modname) reload(modulesource) print("reloaded: {}".format(modname)) except Exception as e: print('Could not load {} {}'.format(modname, e)) Comments
Here's one way, off the top of my head:
>>> import os >>> filter(lambda i: type(i) == type(os), [getattr(os, j) for j in dir(os)]) [<module 'UserDict' from '/usr/lib/python2.5/UserDict.pyc'>, <module 'copy_reg' from '/usr/lib/python2.5/copy_reg.pyc'>, <module 'errno' (built-in)>, <module 'posixpath' from '/usr/lib/python2.5/posixpath.pyc'>, <module 'sys' (built-in)>] It could certainly be cleaned up and improved.
EDIT: Here's a slightly nicer version:
>>> [m[1] for m in filter(lambda a: type(a[1]) == type(os), os.__dict__.items())] [<module 'copy_reg' from '/usr/lib/python2.5/copy_reg.pyc'>, <module 'UserDict' from '/usr/lib/python2.5/UserDict.pyc'>, <module 'posixpath' from '/usr/lib/python2.5/posixpath.pyc'>, <module 'errno' (built-in)>, <module 'sys' (built-in)>] >>> [m[0] for m in filter(lambda a: type(a[1]) == type(os), os.__dict__.items())] ['_copy_reg', 'UserDict', 'path', 'errno', 'sys'] NOTE: This will also find modules that might not necessarily be located in a subdirectory of the package, if they're pulled in in its __init__.py file, so it depends on what you mean by "part of" a package.