Could someone provide me with a good way of importing a whole directory of modules?
I have a structure like this:
/Foo bar.py spam.py eggs.py I tried just converting it to a package by adding __init__.py and doing from Foo import * but it didn't work the way I had hoped.
List all python (.py) files in the current folder and put them as __all__ variable in __init__.py
from os.path import dirname, basename, isfile, join import glob modules = glob.glob(join(dirname(__file__), "*.py")) __all__ = [ basename(f)[:-3] for f in modules if isfile(f) and not f.endswith('__init__.py')] 
if not os.path.basename(f).startswith('_') or at the very least if not f.endswith('__init__.py') to the end of the list comprehension
os.path.isfile(f) is True. That would filter out broken symlinks and directories like somedir.py/ (corner-case, I admit, but still...)
from . import * after setting __all__ if you want submodules to be available using . (e.g. as module.submodule1, module.submodule2, etc.).Add the __all__ Variable to __init__.py containing:
__all__ = ["bar", "spam", "eggs"] 
moduleName.varName ref. stackoverflow.com/a/710603/248616Update in 2017: you probably want to use importlib instead.
Make the Foo directory a package by adding an __init__.py. In that __init__.py add:
import bar import eggs import spam Since you want it dynamic (which may or may not be a good idea), list all py-files with list dir and import them with something like this:
import os for module in os.listdir(os.path.dirname(__file__)): if module == '__init__.py' or module[-3:] != '.py': continue __import__(module[:-3], locals(), globals()) del module Then, from your code do this:
import Foo You can now access the modules with
Foo.bar Foo.eggs Foo.spam etc. from Foo import * is not a good idea for several reasons, including name clashes and making it hard to analyze the code.
__import__ hackish, i think it would be better to add the names to __all__ and then put from . import * at the bottom of the script
__import__ is not for general uses, it used by interpreter, use importlib.import_module() instead.
from . import eggs etc. in the __init__.py before Python could import. With only import eggs I get ModuleNotFoundError: No module named 'eggs' when trying to import Foo in the main.py in the directory above.For newbies who just can't get it to work who need their hands held.
Make a folder /home/el/foo and make a file main.py under /home/el/foo Put this code in there:
from hellokitty import * spam.spamfunc() ham.hamfunc() Make a directory /home/el/foo/hellokitty
Make a file __init__.py under /home/el/foo/hellokitty and put this code in there:
__all__ = ["spam", "ham"] Make two python files: spam.py and ham.py under /home/el/foo/hellokitty
Define a function inside spam.py:
def spamfunc(): print("Spammity spam") Define a function inside ham.py:
def hamfunc(): print("Upgrade from baloney") Run it:
el@apollo:/home/el/foo$ python main.py spammity spam Upgrade from baloney 
import * is considered bad Python coding practice. How do you do this without that?
spam.spamfunc(),). spamfunc and hamfunc aren't in the global namespace, so there's no potential for overlap.Expanding on Mihail's answer, I believe the non-hackish way (as in, not handling the file paths directly) is the following:
__init__.py file under Foo/import pkgutil import sys def load_all_modules_from_dir(dirname): for importer, package_name, _ in pkgutil.iter_modules([dirname]): full_package_name = '%s.%s' % (dirname, package_name) if full_package_name not in sys.modules: module = importer.find_module(package_name ).load_module(full_package_name) print module load_all_modules_from_dir('Foo') You'll get:
<module 'Foo.bar' from '/home/.../Foo/bar.pyc'> <module 'Foo.spam' from '/home/.../Foo/spam.pyc'> 
RuntimeWarning messages can also be avoided by not using full_package_name at all: importer.find_module(package_name).load_module(package_name).
RuntimeWarning errors can also be avoided (in an arguably ugly way) by importing the parent (AKA dirname). One way to do that is - if dirname not in sys.modules: pkgutil.find_loader(dirname).load_module(dirname). Of course, that only works if dirname is a single-component relative path; no slashes. Personally, I prefer @Artfunkel's approach of using the base package_name instead.I know I'm updating a quite old post, and I tried using automodinit, but found out it's setup process is broken for python3. So, based on Luca's answer, I came up with a simpler answer - which might not work with .zip - to this issue, so I figured I should share it here:
within the __init__.py module from yourpackage:
#!/usr/bin/env python import os, pkgutil __all__ = list(module for _, module, _ in pkgutil.iter_modules([os.path.dirname(__file__)])) and within another package below yourpackage:
from yourpackage import * Then you'll have all the modules that are placed within the package loaded, and if you write a new module, it'll be automagically imported as well. Of course, use that kind of things with care, with great powers comes great responsibilities.
I got tired of this problem myself, so I wrote a package called automodinit to fix it. You can get it from http://pypi.python.org/pypi/automodinit/.
Usage is like this:
automodinit package into your setup.py dependencies.__all__ = ["I will get rewritten"] # Don't modify the line above, or this line! import automodinit automodinit.automodinit(__name__, __file__, globals()) del automodinit # Anything else you want can go after here, it won't get modified.
That's it! From now on importing a module will set __all__ to a list of .py[co] files in the module and will also import each of those files as though you had typed:
for x in __all__: import x Therefore the effect of "from M import *" matches exactly "import M".
automodinit is happy running from inside ZIP archives and is therefore ZIP safe.
Niall
import pkgutil __path__ = pkgutil.extend_path(__path__, __name__) for imp, module, ispackage in pkgutil.walk_packages(path=__path__, prefix=__name__+'.'): __import__(module) When from . import * isn't good enough, this is an improvement over the answer by ted. Specifically, the use of __all__ is not necessary with this approach.
"""Import all modules that exist in the current directory.""" # Ref https://stackoverflow.com/a/60861023/ from importlib import import_module from pathlib import Path for f in Path(__file__).parent.glob("*.py"): module_name = f.stem if (not module_name.startswith("_")) and (module_name not in globals()): import_module(f".{module_name}", __package__) del f, module_name del import_module, Path Note that module_name not in globals() is intended to avoid reimporting the module if it's already imported, as this can risk cyclic imports.
I have also encountered this problem and this was my solution:
import os def loadImports(path): files = os.listdir(path) imps = [] for i in range(len(files)): name = files[i].split('.') if len(name) > 1: if name[1] == 'py' and name[0] != '__init__': name = name[0] imps.append(name) file = open(path+'__init__.py','w') toWrite = '__all__ = '+str(imps) file.write(toWrite) file.close() This function creates a file (in the provided folder) named __init__.py, which contains an __all__ variable that holds every module in the folder.
For example, I have a folder named Test which contains:
Foo.py Bar.py So in the script I want the modules to be imported into I will write:
loadImports('Test/') from Test import * This will import everything from Test and the __init__.py file in Test will now contain:
__all__ = ['Foo','Bar'] This is the best way i've found so far:
from os.path import dirname, join, isdir, abspath, basename from glob import glob pwd = dirname(__file__) for x in glob(join(pwd, '*.py')): if not x.startswith('__'): __import__(basename(x)[:-3], globals(), locals()) 
abspath and isdir is unused ;)Anurag Uniyal answer with suggested improvements!
#!/usr/bin/python # -*- encoding: utf-8 -*- import os import glob all_list = list() for f in glob.glob(os.path.dirname(__file__)+"/*.py"): if os.path.isfile(f) and not os.path.basename(f).startswith('_'): all_list.append(os.path.basename(f)[:-3]) __all__ = all_list Using importlib the only thing you've got to add is
from importlib import import_module from pathlib import Path __all__ = [ import_module(f".{f.stem}", __package__) for f in Path(__file__).parent.glob("*.py") if "__" not in f.stem ] del import_module, Path 
error: Type of __all__ must be "Sequence[str]", not "List[Module]". Defining __all__ is not required if this import_module based approach is used.I'd like to add to Anurag Uniyal's answer. You can make it even simpler and get rid of a lot of the imports. Contents of the __init__.py file:
from os import listdir from os.path import dirname __all__ = [i[:-3] for i in listdir(dirname(__file__)) if not i.startswith('__') and i.endswith('.py')] I've created a module for that, which doesn't rely on __init__.py (or any other auxiliary file) and makes me type only the following two lines:
import importdir importdir.do("Foo", globals()) Feel free to re-use or contribute: http://gitlab.com/aurelien-lourot/importdir
Anurag's example with a couple of corrections:
import os, glob modules = glob.glob(os.path.join(os.path.dirname(__file__), "*.py")) __all__ = [os.path.basename(f)[:-3] for f in modules if not f.endswith("__init__.py")] See that your __init__.py defines __all__. The modules - packages doc says
The
__init__.pyfiles are required to make Python treat the directories as containing packages; this is done to prevent directories with a common name, such as string, from unintentionally hiding valid modules that occur later on the module search path. In the simplest case,__init__.pycan just be an empty file, but it can also execute initialization code for the package or set the__all__variable, described later....
The only solution is for the package author to provide an explicit index of the package. The import statement uses the following convention: if a package’s
__init__.pycode defines a list named__all__, it is taken to be the list of module names that should be imported when from package import * is encountered. It is up to the package author to keep this list up-to-date when a new version of the package is released. Package authors may also decide not to support it, if they don’t see a use for importing * from their package. For example, the filesounds/effects/__init__.pycould contain the following code:
__all__ = ["echo", "surround", "reverse"]This would mean that
from sound.effects import *would import the three named submodules of the sound package.
None of the solutions was working for me in Python 3.9.5, Flask 2.2.2, the module being a directory 2 levels down the cwd. This is my solution:
import importlib import pathlib import re path = pathlib.Path(__file__).parent.absolute() names = [x.name[:-3] for x in path.iterdir() if x.is_file() and re.search("^[a-z]*\.py$", x.name)] for name in names: importlib.import_module(f".{name}", __name__) Look at the pkgutil module from the standard library. It will let you do exactly what you want as long as you have an __init__.py file in the directory. The __init__.py file can be empty.
Just import them by importlib and add them to __all__ (add action is optional) in recurse in the __init__.py of package.
/Foo bar.py spam.py eggs.py __init__.py # __init__.py import os import importlib pyfile_extes = ['py', ] __all__ = [importlib.import_module('.%s' % filename, __package__) for filename in [os.path.splitext(i)[0] for i in os.listdir(os.path.dirname(__file__)) if os.path.splitext(i)[1] in pyfile_extes] if not filename.startswith('__')] del os, importlib, pyfile_extes Here is a solution, with which you do not have to write the file name. Just add this code snippet to your __init__.py
from inspect import isclass from pkgutil import iter_modules from pathlib import Path from importlib import import_module # iterate through the modules in the current package package_dir = Path(__file__).resolve().parent for (_, module_name, _) in iter_modules([package_dir]): # import the module and iterate through its attributes module = import_module(f"{__name__}.{module_name}") for attribute_name in dir(module): attribute = getattr(module, attribute_name) if isclass(attribute): # Add the class to this package's variables globals()[attribute_name] = attribute I had a nested directory structure i.e. I had multiple directories inside the main directory that contained the python modules.
I added the following script to my __init__.py file to import all the modules
import glob, re, os module_parent_directory = "path/to/the/directory/containing/__init__.py/file" owd = os.getcwd() if not owd.endswith(module_parent_directory): os.chdir(module_parent_directory) module_paths = glob.glob("**/*.py", recursive = True) for module_path in module_paths: if not re.match( ".*__init__.py$", module_path): import_path = module_path[:-3] import_path = import_path.replace("/", ".") exec(f"from .{import_path} import *") os.chdir(owd) Probably not the best way to achieve this, but I couldn't make anything else work for me.
If you want use import all with pyinstaller, you'll need the following extra processing.
get_all_pyfile that support .pycutil/get_all_pyfile
import glob import re from os.path import join, basename, isfile from typing import AnyStr def get_all_pyfile(pathname: AnyStr): modules = glob.glob(join(pathname, "*.py*")) return [ re.sub(r"\.pyc?", "", basename(f)) for f in modules if isfile(f) and not re.search(r"__init__.pyc?$", f) ] get_all_pyfile in __init__.pyentity/__init__.py
from os.path import dirname from util.get_all_pyfile import get_all_pyfile __all__ = get_all_pyfile(dirname(__file__)) hidden imports in pyinstaller spec file/main.spec
# -*- mode: python ; coding: utf-8 -*- import os import sys from PyInstaller.building.api import PYZ, EXE, COLLECT from PyInstaller.building.build_main import Analysis from PyInstaller.utils.hooks import collect_submodules venv_path = os.path.dirname(os.path.realpath("__file__")) sys.path.append(os.path.join(venv_path, "../")) hidden_imports_entity = collect_submodules("entity") print(hidden_imports_entity) a = Analysis( [".\\main.py"], pathex=[], binaries=[], datas=[], hiddenimports=hidden_imports_entity, hookspath=[], hooksconfig={}, runtime_hooks=[], excludes=[], noarchive=True, ) ... This isn't the same thing exactly, but is a potentially related solution.
If you're importing for future utilization (instead of importing for side-effects), you can use PEP562 to automatically import on use:
def __getattr__(name): import importlib return importlib.import_module(f".{name}", __name__) This might be useful for plugins or web frameworks where you need to pull a consistent object out of a pile of submodules.
