32

Is it possible to create a Python 2.7 package using __init__.pyx (compiled to __init__.so)? If so how? I haven't had any luck getting it to work.

Here is what I have tried:

  • setup.py:

    #!/usr/bin/env python from distutils.core import setup from distutils.extension import Extension from Cython.Distutils import build_ext foo = Extension(name='foo.__init__', sources=['foo/__init__.pyx']) bar = Extension(name='foo.bar', sources=['foo/bar.pyx']) setup(name='foo', packages = ['foo'], cmdclass={'build_ext':build_ext}, ext_modules = [foo, bar])

  • foo/__init__.pyx:

    import foo.bar cpdef hello_world(): print "hello world" foo.bar.blah()

  • foo/bar.pyx:

    cpdef blah(): print "blah"

The above has the following behavior:

$ python -c 'import foo; foo.hello_world()' Traceback (most recent call last): File "<string>", line 1, in <module> ImportError: No module named foo

I saw Python issue #15576 which was fixed by this Hg commit. Looking at the equivalent Git commit in the Git mirror of the Python Hg repository, I see that the commit is reachable from the Python v2.7.5 tag (as well as all subsequent v2.7.x versions). Was there a regression?

Improve this question
1
  • Just out of curiousity: Why should you do that? Commented Aug 18, 2015 at 14:13

2 Answers 2

Reset to default
16
+50

According to this really old mailing list post it works if you also have an __init__.py file (the __init__.py file is not used, but seems to be necessary for the directory to be treated as a module, and hence the __init__.so file to be loaded).

If I add __init__.py:

# an exception just to confirm that the .so file is loaded instead of the .py file raise ImportError("__init__.py loaded when __init__.so should have been loaded")

then your example works on Linux Python 2.7.3:

$ python -c 'import foo; foo.hello_world()' hello world blah

This has all the signs of a buggy corner case so probably isn't recommended. Note that on Windows this doesn't seem to work for me giving

ImportError: DLL load failed: %1 is not a valid Win32 application.

Addendum (for a little extra context):

This behaviour doesn't seem to be explicitly documented. In the original description of packages from around Python 1.5 era they say:

without the __init__.py, a directory is not recognized as a package

and

Tip: the search order is determined by the list of suffixes returned by the function imp.get_suffixes(). Usually the suffixes are searched in the following order: ".so", "module.so", ".py", ".pyc". Directories don't explicitly occur in this list, but precede all entries in it.

The observed behaviour is certainly consistent with this — __init__.py needed to treat a directory as a package, but .so file is loaded in preference to .py file — but it's hardly unambiguous.

From a Cython point of view this behaviour seems to be been used to compile the standard library (in which case __init__.py would always have been present), or in the testcases given https://github.com/cython/cython/blob/master/tests/build/package_compilation.srctree (and a few other examples too). In these the "srctree" file looks to be expanded into a variety of folders containing __init__.py (and other files) then compiled. It's possible that only having __init__.so was simply never tested.

Improve this answer
Sign up to request clarification or add additional context in comments.

4 Comments

Is this trick mentioned in the official documentation? (it apparently wasn't at the time of that mailing list thread, but maybe things have changed since then)
I don't think so. (I found it by accident and then came across the newsgroup post slightly after and realised what I'd done.) I'll have a little look and update my answer if I can find a better source. It also works with an __init__.pyc file too, it seems.
Suggestions for improvement: Rather than assert False, raising an ImportError might be better. Or maybe there's some low-level stuff that can be done with the imp module as a fall-back in case this trick stops working in a future version of Python.
@RichardHansen I've updated it to raise ImportError (I agree - it's clearly better!) and added a link to the little documentation I can find that supports it. I think it's basically undocumented
-3

Try using a relative import.

in __init__:

from . import bar

Might also be from . import foo. Haven't used python 2 cython in a while.

Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.