114

I have my own package in python and I am using it very often. what is the most elegant or conventional directory where i should put my package so it is going to be imported without playing with PYTHONPATH or sys.path?

What about site-packages for example? /usr/lib/python2.7/site-packages.
Is it common in python to copy and paste the package there ?

3
  • 1
    make a setup.py that tells dependencies and what not and use that is better practice and will be more portable ... but yeah typically copy/paste will work fine ... although I usually make an extra folder called "user/lib/" or something and put them all there and just make sure its on my PYTHONPATH Commented Apr 24, 2013 at 15:45
  • 2
    REAL answer: I don't like other answers either for personal simple scripts/non-distributed packages, like @JoranBeasley: Make some folder to keep your scripts in (or links to them), and add that folder to PYTHONPATH from within your bash runcom tree (e.g. put scripts/links in ~/foo/bar/, add export PYTHONPATH=~/foo/bar # fields separated by ":" to your ~/.bashrc file). See man python, this manual path precedes the default python search path. Careful of name collisions. Commented Jul 13, 2016 at 15:31
  • @JoranBeasley Could you please give an example of a setup.py and how to set his up? I would like to learn these good practices. Commented Aug 11, 2022 at 23:11

6 Answers 6

74

I usually put the stuff i want to have ready to import in the user site directory:

~/.local/lib/pythonX.X/site-packages 

To show the right directory for your platform, you can use python -m site --user-site


edit: it will show up in sys.path once you create it:

mkdir -p "`python -m site --user-site`" 
Sign up to request clarification or add additional context in comments.

6 Comments

when I open python from a terminal, this is what sys.path gives. notice there is no local ['', '/usr/lib64/python27.zip', '/usr/lib64/python2.7', '/usr/lib64/python2.7/plat-linux2', '/usr/lib64/python2.7/lib-tk', '/usr/lib64/python2.7/lib-old', '/usr/lib64/python2.7/lib-dynload', '/usr/lib64/python2.7/site-packages', '/usr/lib64/python2.7/site-packages/PIL', '/usr/lib64/python2.7/site-packages/gst-0.10', '/usr/lib64/python2.7/site-packages/gtk-2.0', '/usr/lib/python2.7/site-packages', '/usr/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg-info']
when executing a script this is what it gives ['/home/username/Desktop/mypackage/Examples', '/usr/lib64/python27.zip', '/usr/lib64/python2.7', '/usr/lib64/python2.7/plat-linux2', '/usr/lib64/python2.7/lib-tk', '/usr/lib64/python2.7/lib-old', '/usr/lib64/python2.7/lib-dynload', '/usr/lib64/python2.7/site-packages', '/usr/lib64/python2.7/site-packages/PIL', '/usr/lib64/python2.7/site-packages/gst-0.10', '/usr/lib64/python2.7/site-packages/gtk-2.0', '/usr/lib/python2.7/site-packages', '/usr/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg-info']
python -m site --user-site /home/username/.local/lib/python2.7/site-packages but this is not among sys.path !!
@Cobry python -m site --user-site will tell you what directory you should create. It won't appear in sys.modules until you've actually created it.
@mata You should probably change mkdir to mkdir -p to create the intermediate directories. I didn't even have a ~/.local. ;-)
|
32

So if your a novice like myself and your directories are not very well organized you may want to try this method.

Open your python terminal. Import a module that you know works such as numpy in my case and do the following. Import numpy

numpy.__file__

which results in

'/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site- packages/numpy/__init__.py'

The result of numpy.__file__ is the location you should put the python file with your module (excluding the numpy/__init__.py) so for me that would be

/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site- packages

To do this just go to your terminal and type

mv "location of your module" "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site- packages"

Now you should be able to import your module.

4 Comments

How come at my Python, the "._file_" doesnt work? >>> import re >>> re._file_ Traceback (most recent call last): File "<pyshell#10>", line 1, in <module> re._file_ AttributeError: module 're' has no attribute 'file'
@Urso Are you sure you are using two underscores? The command re.__file__ worked for me.
Thanks a lot, @arie64 for such a clean solution. Worked well for me as well !!
Thanks this worked for me as well. But weirdly I cannot find the directory/folder I sent my module to in File explorer. It seems like the folder does not exist. Any Idea why the folder doesn't show up where it should? (I have enabled "show hidden items" so it's not simply that...)
10

This is something that works for me (I have to frequently create python packages that are uploaded to a private pip repository). elaborating on the comment by @joran on the question.

  1. create a "build directory" which is used as a workspace to create packages. any directory of your choice will do
  2. Copy your python package dir there, and create a setup.py file. this should help in creating the setup.py correctly.
  3. create a virtualenv for the project you are working on. virtualenvs have a bunch of other benefits, I am not going into the details here.
  4. create a local dist package python setup.py sdist --format=tar. the package created should ideally be in the dist folder.
  5. Install the package on your virtualenv (after activating it). pip install <yourpackage>.tar

you can use pip install --force-reinstall if you need to play around with the libraries more and re-create the dist packages.

I've found that this method works great for me. If you do not need to package the modules for use of other systems instead of just your local, this method might be an overkill

Happy hacking.

Comments

3

If you're developing a module I would recommend to follow this.

import sys sys.path.append("/home/mylinux/python-packages") 

Now any module you keep in python-packages is importable by Python-interpreter.

1 Comment

I am using a zipped python distribution and not an installation to the OS. Python exe is not in the environment variables as well. I am calling python executable to run the python file from nodejs. So I do not have the option to do sys.append(path). Any other method for my question here? stackoverflow.com/questions/72060668/…
1

On my Mac, I did a sudo find / -name "site-packages". That gave me a few paths like /Library/Python/2.6/site-packages, /Library/Python/2.7/site-packages, and /opt/X11/lib/python2.6/site-packages.

So, I knew where to put my modules if I was using v2.7 or v2.6.

Hope it helps.

Comments

1

import folders could be extracted by adding following source code:

import sys print sys.path 

automatic symlink generation example would be:

ln -s \`pip show em | grep "Location"| cut -d " " -f2\` \`python -m site --user-site\` 

instead of "em" you may use other package you've "just installed but the python can't see it"

below I'll explain in more details as being requested in the comment.

suppose you've installed python module em or pyserial with the following command (examples are for ubuntu):

sudo pip install pyserial 

and the output is like this:

Collecting pyserial Downloading pyserial-3.3-py2.py3-none-any.whl (189kB) 100% |████████████████████████████████| 194kB 2.3MB/s Installing collected packages: pyserial Successfully installed pyserial-3.3 

the question would be following - python can't see the module pyserial, why? because the location where the module has been installed isn't the one python is looking at for your particular user account.

solution - we have to create symlink from the path where pyserial arrived to the path where your python is looking for.

symlink creation command would be:

ln -s <what_to_link> <where_to_link> 

instead of typing exact location we are asking pip to tell us where it stored modules by executing command:

pip show pyserial | grep "Location"| cut -d " " -f2 

instead of typing exact location we are asking python to tell us where it looks for the modules being installed by executing command:

python -m site --user-site 

both commands has to be escaped with "`" character (usually on the left of your 1 button for the US keyboards)

in result following command will be provided for ln and the missing symlink would be created:

ln -s /usr/local/lib/python2.7/dist-packages /home/<your_username>/.local/lib/python2.7/site-packages 

or something similar, depending on your distro and python/pip defaults.

2 Comments

Looks like a powerful one-liner... can you explain a bit for us linux newbies? People might be wary to use something they don't understand :-)
@oleg-kokorin I am using a zipped python distribution and not an installation to the OS. Python exe is not in the environment variables as well. I am calling python executable to run the python file from nodejs. So I do not have the option to do sys.append(path). Any other method for my question here? stackoverflow.com/questions/72060668/…

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.