4

I have a pure Python module and I want to rewrite some of submodules using Cython. Then I would like to add the new Cython submodules to the original Python module and make them available only as an option, meaning that cythoning the module is not compulsory (in which case the 'old' pure Python module should be used).

Here is an example:

my_module - __init__.py - a.py - b.py - setup.py

where a.py contains import b.

I want to write b.py in Cython. The idea would be to add a folder containing the .pyx file, for example:

my_module - __init_.py - a.py - b.py - setup.py cython -b.pyx

setup.py would contain the direction to compile b.pyx and to install the module. However, I would like that if someone runs python setup.py install then the pure Python code is installed, whereas if an option is added then the Cython code is compiled and installed.

Any idea how to do that?

Also, how should the file a.py be modified in order to import the correct module?

Improve this question

2 Answers 2

Reset to default
2

I am not sure about your setup.py requirement (I don’t know why you would need that) but as for the runtime import issue, I wrote a decorator to do just that:

from __future__ import print_function from importlib import import_module from functools import wraps import inspect import sys MAKE_NOISE = False def external(f): """ Decorator that looks for an external version of the decorated function -- if one is found and imported, it replaces the decorated function in-place (and thus transparently, to would-be users of the code). """ f.__external__ = 0 # Mark func as non-native function_name = hasattr(f, 'func_name') and f.func_name or f.__name__ module_name = inspect.getmodule(f).__name__ # Always return the straight decoratee func, # whenever something goes awry. if not function_name or not module_name: MAKE_NOISE and print("Bad function or module name (respectively, %s and %s)" % ( function_name, module_name), file=sys.stderr) return f # This function is `pylire.process.external()`. # It is used to decorate functions in `pylire.process.*`, # each of which possibly has a native (Cython) accelerated # version waiting to be imported in `pylire.process.ext.*` # … for example: if in `pylire/process/my_module.py` you did this: # # @external # def my_function(*args, **kwargs): # """ The slow, pure-Python implementation """ # pass # # … and you had a Cython version of `my_function()` set up # in `pylire/process/ext/my_module.pyx` – you would get the fast # function version, automatically at runtime, without changing code. # # TL,DR: you'll want to change the `pylire.process.ext` string (below) # to match whatever your packages' structure looks like. module_file_name = module_name.split('.')[-1] module_name = "pylire.process.ext.%s" % module_file_name # Import the 'ext' version of process try: module = import_module(module_name) except ImportError: MAKE_NOISE and print("Error importing module (%s)" % ( module_name,), file=sys.stderr) return f MAKE_NOISE and print("Using ext module: %s" % ( module_name,), file=sys.stderr) # Get the external function with a name that # matches that of the decoratee. try: ext_function = getattr(module, function_name) except AttributeError: # no matching function in the ext module MAKE_NOISE and print("Ext function not found with name (%s)" % ( function_name,), file=sys.stderr) return f except TypeError: # function_name was probably shit MAKE_NOISE and print("Bad name given for ext_function lookup (%s)" % ( function_name,), file=sys.stderr) return f # Try to set telltale/convenience attributes # on the new external function -- this doesn't # always work, for more heavily encythoned # and cdef'd function examples. try: setattr(ext_function, '__external__', 1) setattr(ext_function, 'orig', f) except AttributeError: MAKE_NOISE and print("Bailing, failed setting ext_function attributes (%s)" % ( function_name,), file=sys.stderr) return ext_function return wraps(f)(ext_function)

… this lets you decorate functions as @external – and they are replaced at runtime automatically with the Cython-optimized versions you’ve provided.

If you wanted to extend this idea to replacing entire Cythonized classes, it’d be straightforward to use the same logic in the __new__ method of a metaclass (e.g. opportunistic find-and-replace in the optimized module).

Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

I accepted your answer because it looks good! :-) Although I didn't have a chance to try it yet. I used an easier -- but somewhat conceptually similar -- solution. You can find it in my answer.
Glad you like it, thanks! – I just edited it to add the from __future__ line, incedentally, for Python 2.x users.
2

My solution was to set up the module like this:

my_module - __init_.py - a.py - b.py - setup.py cython_my_module - __init_.py - b.pyx

The setup.py would contain something similar to this:

from distutils.core import setup from Cython.Build import cythonize import numpy setup( name='My_module', ext_modules=cythonize(["cython_my_module/b.pyx",]), include_dirs=[numpy.get_include()], )

and the file a.py would contain the following lines in the header:

try: import cython_my_module.b except ImportError: import b

The way it works is very simple: if you don't do anything (i.e. if you don't compile the cython files) then module a.py imports module b.py; however, if you run python setup.py build_ext --inplace then the compiled cython files will appear inside cython_my_module and the next time you run a.py it will automatically import the cython module b.pyx (actually it will import the compiled library b.so).

So far it seems to work and requires almost no effort. Hope it helps.

fish2000 solution seems more generic but I haven't tried it yet.

Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.