Building a package referencing a package in a parent directory in setup.py

Question

I am trying to build a pip package from source code in a Git repository that has multiple packages that share a common package. (I am not allowed to change the structure of this Git repository.)

The structure is:

├── common │ ├── __init__.py │ ├── run_helpers │ │ ├── __init__.py │ │ ├── aws.py │ │ └── s3.py └── components └── redshift_unload ├── redshift_unload │ ├── __init__.py │ └── run.py └── setup.py

My setup.py is as follows:

from setuptools import setup, find_packages setup( ... packages=find_packages(), package_dir={"": "."}, entry_points={ "console_scripts": ["redshift_unload=redshift_unload.run:main"] } )

Looking at other answers here, things I have tried so far include:

Specifying the actual package names in the packages= line instead of using find_packages().
Passing where="../../" to find_packages()
Using find_packages() + find_packages(where="../../") in the packages=` line.
Everything I can think of in the packages_dir line.

When I run pip install . I get, the package installs fine, but then when I run the installed python script I get:

# redshift_unload Traceback (most recent call last): File "/usr/local/bin/redshift_unload", line 5, in <module> from redshift_unload.run import main File "/usr/local/lib/python3.8/site-packages/redshift_unload/run.py", line 9, in <module> from common._run_helpers.aws import get_boto3_session ModuleNotFoundError: No module named 'common'

What did work:

If I moved the common directory to components/redshift_unload, then it works fine. But I can't do this. I also tried placing a symlink there in its place, but seems like that doesn't work either.

Is there a way to make this work?

Peter Mortensen · Accepted Answer · 2022-04-20 19:34:06Z

I believe I have found the best solution to this.

Based on this comment here, I concluded that what I am trying to do is not intended or supported.

However, I found a workaround as follows works fine:

from pathlib import Path from shutil import rmtree, copytree from setuptools import setup, find_packages src_path = Path(os.environ["PWD"], "../../common") dst_path = Path("./common") copytree(src_path, dst_path) setup( ... packages=find_packages(), package_dir={"": "."}, entry_points={ "console_scripts": ["redshift_unload=redshift_unload.run:main"] } ) rmtree(dst_path)

The key insight here is that, while packaging occurs in a temporary directory, the value of os.environ["PWD"] is available to the process, such that the common directory can be copied temporarily and then cleaned up again (using shutil functions copytree and rmtree) into a location that will be found by find_packages() before the setup() function is called.

Cool workaround. To be safe it's better to use tempfile and os.chdir() there though. Otherwise, python setup.py sdist will delete your code

Collectives™ on Stack Overflow

Building a package referencing a package in a parent directory in setup.py

1 Answer 1

1 Comment

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Related