How to compile __init__.py file using cython on Windows

Maybe this behaviour might be viewed as a small bug in distutils-package (as pointed out by @DavidW there is this open issue: https://bugs.python.org/issue35893). However, it also shows, that cythonizing/compiling __init__.py isn't very popular and uses some undocumented implementation details which might change in the future, so it could be wiser to refrain from meddling with __init__.py.

But if you must...

When a package is imported explicitly, e.g.

import ctest 

or implicitly, e.g.

import ctest.something 

The FileFinder will see that a package, and not a module, is imported and will try to load ctest/__init__.py instead of ctest.py (which most likely doesn't exists):

 # Check if the module is the name of a directory (and thus a package). if cache_module in cache: base_path = _path_join(self.path, tail_module) for suffix, loader_class in self._loaders: init_filename = '__init__' + suffix full_path = _path_join(base_path, init_filename) if _path_isfile(full_path): return self._get_spec(loader_class, fullname, full_path, [base_path], target) 

Used suffix, loader_class are for loading __init__.so, __init__.py and __init__.pyc in this order (see also this SO-post). This means, __init__.so will be loaded instead of __init__.py if we manage to create one.

While __init__.py is executed, The property __name__ is the name of the package, i.e. ctest in your case, and not __init__ as one might think. Thus, the name of the init-function, Python-interpreter will call when loading the extension __init__.so is PyInit_ctest in your case (and not PyInit___init__ as one might think).

The above explains, why it all works on Linux out-of-the-box. What about Windows?

The loader can only use symbols from a so/dll which aren't hidden. Per default all symbols built with gcc are visible, but not for VisualStudio on Windows - where all symbols are hidden per default (see e.g. this SO-post).

However, the init-function of a C-extension must be visible (and only the init-function) so it can be called with help of the loader - the solution is to export this symbol (i.e. PyInit_ctest) while linking, in your case it is the wrong /EXPORT:PyInit___init__-option for the linker.

The problem can be found in distutils, or more precise in build_ext-class:

def get_export_symbols(self, ext): """Return the list of symbols that a shared extension has to export. This either uses 'ext.export_symbols' or, if it's not provided, "PyInit_" + module_name. Only relevant on Windows, where the .pyd file (DLL) must export the module "PyInit_" function. """ initfunc_name = "PyInit_" + ext.name.split('.')[-1] if initfunc_name not in ext.export_symbols: ext.export_symbols.append(initfunc_name) return ext.export_symbols 

Here, sadly ext.name has __init__ in it.

From here, one possible solution is easy : to override get_export_symbols, i.e. to add the following to your setup.py-file (read on for a even simpler version):

... from distutils.command.build_ext import build_ext def get_export_symbols_fixed(self, ext): names = ext.name.split('.') if names[-1] != "__init__": initfunc_name = "PyInit_" + names[-1] else: # take name of the package if it is an __init__-file initfunc_name = "PyInit_" + names[-2] if initfunc_name not in ext.export_symbols: ext.export_symbols.append(initfunc_name) return ext.export_symbols # replace wrong version with the fixed: build_ext.get_export_symbols = get_export_symbols_fixed ... 

Calling python setup.py build_ext -i should be enough now (because __init__.so will be loaded rather than __init__.py).

However, as @DawidW has pointed out, Cython uses macro PyMODINIT_FUNC, which is defined as

#define PyMODINIT_FUNC Py_EXPORTED_SYMBOL PyObject* 

with Py_EXPORTED_SYMBOL being marked as visible/exported on Windows:

#define Py_EXPORTED_SYMBOL __declspec(dllexport) 

Thus, there is no need to mark the symbol as visible at the command line. Even worse, this is the reason for the warning LNK4197:

__init__.obj : warning LNK4197: export 'PyInit_ctest' specified multiple times; using first specification

as PyInit_test is marked as __declspec(dllexport) and exported via option /EXPORT: at the same time.

/EXPORT:-option will be skipped by distutils, if export_symbols is empty, we can use even a simpler version of command.build_ext:

... from distutils.command.build_ext import build_ext def get_export_symbols_fixed(self, ext): pass # return [] also does the job! # replace wrong version with the fixed: build_ext.get_export_symbols = get_export_symbols_fixed ... 

This is even better than the first version, as it also fixes warning LNK4197!