4

Here is my directory tree:

prediction_model ├─ prediction_model | ├─ __init__.py | ├─ data | | ├─ SAAA.csv | | └─ VDFF.csv | ├─ models.py | ├─ preprocess.py | ├─ README.md | └─ tests └─ setup.py

Here is my ‘setup.py’:

from setuptools import find_packages, setup setup( name='prediction_model', version='0.7', url='https://project.org/', author='JL', author_email='[email protected]', packages=find_packages(), scripts=['models.py', 'preprocess.py'] )

Here is my ‘__init__.py’:

from prediction_model import models from prediction_model import preprocess

‘models.py’ has a function main and ‘preprocess.py’ has a function run that I want to use.

I install the project using:

python -m pip install --user .

Then I run the following code in the Python interpreter but it raises an exception AttributeError:

>>> import prediction_model >>> prediction_model.src.main() Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: module 'prediction_model' has no attribute 'src' >>> prediction_model.src.run() Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: module 'prediction_model' has no attribute 'src' 
>>> import prediction_model >>> prediction_model.main() Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: module 'prediction_model' has no attribute 'main' >>> prediction_model.run() Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: module 'prediction_model' has no attribute 'run'

What am I doing wrong?

Environment: Python 3.7, MacOS.

Improve this question
3
  • After installation, when you do import prediction_model in your client code, it runs implicitly the __init__.py file. Since in it you do from prediction_model import preprocess, models, the preprocess and models modules will be in your namespace. Then instead of doing prediction_model.src.main() in your client code which raises an AttributeError since there is no src object in your namespace, you should just do prediction_model.preprocess.run() and prediction_model.models.main(). Commented Nov 22, 2019 at 22:12
  • If instead you wanted to do prediction_model.run() and prediction_model.main() in your client code, you should do from prediction_model.preprocess import run; from prediction_model.models import main in your __init__.py file. Commented Nov 22, 2019 at 22:12
  • Another thing, your scripts argument should be ['prediction_model/models.py', 'prediction_model/preprocess.py'] instead of ['models.py', 'preprocess.py'] (since relative paths are resolved against the directory of setup.py). In addition the models.py and preprocess.py files need a shebang line at their top or the installation will fail: #!/usr/bin/env python. But since they are Python scripts and are part of the prediction_model package, you should use the entry_points argument instead of the scripts argument and you won’t need this shebang line. Commented Nov 22, 2019 at 23:16

2 Answers 2

Reset to default
4

Set up

The best practice is to use the src layout (cf. https://blog.ionelmc.ro/2014/05/25/python-packaging/), so I suggest that you organize your directories like this (since you should not package the ‘tests’ directory and ‘README.rst’ file for built distribution, they are left outside the ‘src’ directory):

project ├─ src | └─ package | ├─ __init__.py | ├─ __main__.py | ├─ data.tsv | └─ module.py ├─ tests | └─ test_module.py ├─ MANIFEST.in ├─ README.rst └─ setup.py

with the following ‘setup.py’ file contents:

import setuptools setuptools.setup( name="project name", version="project version", url="project URI", author="your name", author_email="your email", package_dir={"": "src"}, packages=setuptools.find_namespace_packages(where="src"), include_package_data=True )

and the following ‘MANIFEST.in’ file contents:

graft src graft tests global-exclude *.py[cod]

Remark. — If you don't want a top-level ‘package’ directory in your ‘src’ directory while keeping the ‘src’ directory (like before your post edit):

project ├─ src | ├─ __init__.py | ├─ __main__.py | ├─ data.tsv | └─ module.py ├─ tests | └─ test_module.py ├─ MANIFEST.in ├─ README.rst └─ setup.py

then map "package" to "src" in the package_dir argument and list "package" explicitly in the packages argument of the setuptools.setup function in the setup.py file (cf. https://docs.python.org/3/distutils/examples.html#pure-python-distribution-by-package):

import setuptools setuptools.setup( name="project name", version="project version", url="project URI", author="your name", author_email="your email", package_dir={"package": "src"}, packages=["package"], include_package_data=True )

Package

Now you can package the project into a source distribution with this command:

python setup.py sdist

or a built distribution with this command (this is an extension command to setuptools requiring that wheel be installed in your Python environment—so run pip install wheel before):

python setup.py bdist_wheel

or both with this command:

python setup.py sdist bdist_wheel

which create the following files in a new dist directory:

  • ‘{project name}-{project version}.tar.gz’ for a source distribution;
  • ‘{project name}-{project version}-{compatibility tags}.whl’ for a built distribution.

Install

Now you can install the project from the source distribution with this command:

pip install dist/{project name}-{project version}.tar.gz

or from the built distribution with this command:

pip install dist/{project name}-{project version}-{compatibility tags}.whl

or from the project tree with this command (no need to package the project for this one):

pip install .

Run

Eventually you can import your project like this:

import package.module package.module.function()
Improve this answer
Sign up to request clarification or add additional context in comments.

9 Comments

It's not really a src layout. They just named their top level package src. And I believe they expect it to be importable under the name of the project prediction_model, which of course doesn't work.
That's a good edit. I removed my downvote. Why the find_namespace_packages instead of find_packages? Isn't it a bit misleading?
Thanks. find_namespace_packages is the modern version of find_packages (which should not be used anymore) since it supports implicit namespace packages introduced in Python 3.3 (cf. setuptools.readthedocs.io/en/latest/…).
From my point of view, too many unrelated details now (MANIFEST.in, include_package_data), wheels, etc.). Why not just focus on the original issue: wrong name of the top level package. A link to a well established reference for the general packaging guidelines would have been enough for the extra details. Even though everything seems like good advice.
@sinoroc Actually without MANIFEST.in and include_package_data=True non .py files (like the asker's .csv files) will not be included in the packaged/installed project, so they are necessary (cf. setuptools.readthedocs.io/en/latest/…). An alternative is to use the package_data argument but it is only used for built distributions so you still have to use MANIFEST.in for source distributions; since it duplicates information it is discouraged (cf. blog.ionelmc.ro/presentations/packaging/#slide:15).
|
0

usual project structure: (Long story short you WANT a directory with your project name for setup.py to get it, this is the usual way, take a look at a couple repos, e.g. https://github.com/boto/boto3 ).

-- setup.py -- README.md --- prediction_model |-- __init__.py |-- data | |-- VDFF.csv | +-- SAAA.csv |-- models.py |-- preprocess.py |-- tests +-- _data.csv

Then you want to do something like:

virtualenv myvirtualenv source myvirtualenv/bin/activate python setup.py install

Your setup looks fine, now prediction_model is a package you can also change:

from . import preprocess from . import models

To

from prediction_model import preprocess from prediction_model import models

Normal setup.py

from setuptools import setup, find_packages setup( name='cost_estimation', version='0.0.1', packages=find_packages(), url='http://github.com/<WHATEVER/WHATEVER>', install_requires=[ 'numpy', ], tests_require=[ 'nosetests', ], test_suite='nose.collector', license='', dependency_links=[ 'http://YOURPRIVATEPYPISERVER/packages' ], keywords='models, prediction', author='your name', description='Model prediction cool stuff' )
Improve this answer

7 Comments

python-packaging-tutorial.readthedocs.io/en/latest/… pip install . equivalent to python -m pip install --user ., develop mode. I for some reason need to use root authority for install.
plus..ImportError: No module named setuptools always returns although I can import this module when I load python. I just don't get this.
Using the same name, doesn't confuse the packaging tools? This setuptools is useless anway, doesn't seem to find packages through find_packages()
Fro my point of view there are some superfluous (and possibly misleading details) in this answer, but basically, yes, that looks like a correct answer. In short, rename your src directory to the name you want to be able to import i.e. prediction_model. Note that it doesn't have to be the same name as the name of the project (the one in setup.py).
But thank you for guiding me in the right direction @sinoroc and @E.Serra. Trying smth, at least got a start. now the scripts are called. prediction_model.preprocess.run(), prediction_model.models.main() like this. with many other error.
|