Importing dynamically all modules from a folder

Question

I know there is this question, but not only they're not working, it's not exactly what I want. I'm developing a racing game and want to load all tracks from a folder dynamically (They're stored as .py instead of .json). I don't want to know the names of the tracks, since users can mod/add them at will. I just want to import their data. So, for example:

>tracks >>track0.py >>track1.py >>track2.py >>track3.py >>track4.py

Inside each track, I have data like this:

track_ground_data = [ [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1], [1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1], [1, 1, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 1, 1], [1, 1, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 1, 1], [1, 1, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 1, 1], [1, 1, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 1, 1], [1, 1, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 1, 1], [1, 1, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 1, 1], [1, 1, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 1, 1], [1, 1, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 1, 1], [1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1], [1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] ]

I need to import each track module like this:

loaded_tracks = [t for t in tracks] # Where tracks is the folder.

And then access a given track_ground_data like this:

loaded_tracks[0].track_ground_data

If I knew Python was going to be so harsh with its imports, I'd have used json instead .py.

Yes. I've placed that code inside a __init__.py file and when I try to import a track (import tracks): tracks.track0, I get an AttributeError. — Ericson Willians
– Ericson Willians, Commented Oct 27, 2017 at 17:43
You're right, you probably should have used a file format designed for storing data instead of executable .py files... — martineau
– martineau, Commented Oct 27, 2017 at 17:50
You could do this by adapting my answer to a related question about packages. — martineau
– martineau, Commented Oct 27, 2017 at 17:52
Using python files for tracks is useless if they should only contain static data. However they do allow to put arbitrary code, which may be a feature... for example some track could define some functions that provide specific logic for that particular track, this is way harder to do with a simple json. If you don't need the flexibility just use json files (btw: you could still import them as python modules by using a custom module loader...) — Bakuriu
– Bakuriu, Commented Oct 27, 2017 at 17:56

Bakuriu · Accepted Answer · 2024-12-23 18:41:10Z

Python does not automatically import submodules contained in a package. Hence import tracks only loads tracks/__init__.py.

However you can put code inside the __init__.py file that imports all the modules it finds in that directory.

For example putting something like this in the __init__.py:

import os import importlib for file in os.listdir(os.path.dirname(__file__)): mod_name = file.removesuffix(".py") if mod_name in ("__init__", "__pycache__"): continue importlib.import_module('.' + mod_name, package=__name__)

Should make your submodules available as tracks.trackX when importing only tracks.

Or you could use exec:

import os import importlib for file in os.listdir(os.path.dirname(__file__)): mod_name = file.removesuffix(".py") if mod_name in ("__init__", "__pycache__"): continue exec('import .' + mod_name)

A cleaner approach would be to use import hooks or implement your own custom module importer. There are multiple ways to do this using importlib see also sys.path_hooks

You'd also want to filter out __init__.py, so only do the last statement if mod_name doesn't match the pattern ^__. Something like if not re.match(r'^__', file):.
I'm years late and this has possibly changed with the new python versions, but what's the purpose of putting the module object inside globals()? Importing "tracks" and then using "tracks.track0" works even without adding the module to _globals (at least on my machine). Am I missing something ?
@digEmAll I don't know, it has been quite a few years and the code I wrote was a non-working example to give an idea on how you could dynamically import modules. Setting the globals seems useless since import_module does already set the attribute to the module object yes.

Ericson Willians · Accepted Answer · 2017-10-27 19:36:15Z

Just for the sake of the good wellfare of future pythoners, I'm posting how I've solved. A friend helped me through it. Coudln't make Bakuriu's solution work, because the modules came empty. Inside __init__.py I've put:

import os dir = os.path.dirname(os.path.abspath(__file__)) modules = [os.path.splitext(_file)[0] for _file in os.listdir(dir) if not _file.startswith('__')] tracks = [] for mod in modules: exec('from tracks import {}; tracks.append({})'.format(mod, mod))

And then, on the main file, I've loaded it as:

dir = os.path.dirname(os.path.abspath(__file__)) sys.path.append(dir) from tracks import tracks

And then:

loaded_tracks = [t for t in tracks]

That actually solved it quite well. I was almost switching to JSON / giving up.

Guilherme Marthe · Accepted Answer · 2017-10-27 17:51:47Z

The problem of dynamically importing modules is faced usually when frameworks have a plug-in or add on system for the community to contribute. Each plug-in or add-on is a module containing classes and functions compliant with the framework's architecture and api.

With that in mind, the solution for "joining the dots" between the framework code and arbitrarily many add-ons is through the importlib present in the python standard library. You seem to face the same structural problem.

Here is a stackoverflow question that was answered with importlib. And the documentation.

Collectives™ on Stack Overflow

Importing dynamically all modules from a folder

3 Answers 3

4 Comments

Comments

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

4 Comments

Comments

Comments

Linked

Related