Python dependency analyzer library

Question

I need a way to find the dependencies for each of my Python package's sub-modules at runtime so I can initialize them in a proper order (see my current [EDIT: former] solution here, which doesn't work to well), so at first I used the standard Python module modulefinder, but that was way too slow (~1-2 seconds per module).

My next choice was to analyze all the globals of each module, and find from those globals which sub-module each sub-module depends upon. (This is my current solution EDIT: I have a better solution now - see my answer). This algorithm is much faster than modulefinder (it takes <200ms per module), but it only works for relative imports, instead of the fully-qualified import style, which is unacceptable.

So, what I need is either:

A quicker alternative to modulefinder
An alternative algorithm

NOTE: I call my dependency analyzer at the start of each module, like so:

# File my_package/module3.py import my_package.module1 # Some misc. module import my_package.module2 # Some other misc. module import my_package.dependency_analyzer my_package.dependency_analyzer.gendeps()

(Just in case it helps you any.)

Thank you!

EDIT: I have a solution now - see my answer.

Why does this have to happen in run time? Do these dependencies change at run time? — Assaf Lavie
– Assaf Lavie, Commented Jun 25, 2011 at 5:08
Not normally (there are some exceptions), but I don't want to add an extra dependency map file for each of my modules (which sounds like what you're suggesting), so I decided to analyze each module's dependencies at run-time. — DangerOnTheRanger
– DangerOnTheRanger, Commented Jun 25, 2011 at 5:20
Or you could implement your packages with some lazy initialization, so that their initialization order doesn't matter. — rafalotufo
– rafalotufo, Commented Jun 25, 2011 at 17:13
If you mean like each module having an init method, I already have that. It's just that some modules/sub-packages must be initialized after others. For example, my 3D graphics plugin must be initialized after my configuration module. — DangerOnTheRanger
– DangerOnTheRanger, Commented Jun 25, 2011 at 19:17

DangerOnTheRanger · Accepted Answer · 2011-06-27 14:50:29Z

I think I have a solution to my own question :)

Here's what would go into the dependency_analyzer module talked about above:

import sys from sys import _getframe as getframe import atexit examined_modules = [] def gendeps(): """Adds the calling module to the initialization queue.""" # Get the calling module's name, and add it to the intialization queue calling_module_name = getframe(1).f_globals['__name__'] examined_modules.append(calling_module_name) def init(): """Initializes all examined modules in the correct order.""" for module in examined_modules: module = sys.modules[module] if hasattr(module, 'init'): module.init() if hasattr(module, 'deinit'): # So modules get de-initialized in the correct order, # as well atexit.register(module.deinit)

Now, at the start of each module (after all the import statements - this is crucial), a call to gendeps is placed. This algorithm works because each time a module is imported, that call to gendeps is executed. However, since all of the import statements are placed before the call to gendeps in your own module, the least-dependent modules are placed in the initialization queue first, and the most-dependent modules are placed in the initialization queue last.

Collectives™ on Stack Overflow

Python dependency analyzer library

1 Answer 1

Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Related