117

I'm going through and writing a setup doc for other developers at work for a python project and I've been reading up on the PYTHONPATH environment variable. I'm looking at my current development system and think I have a few things set wrong that is causing my IDE (IntelliJ) to behave incorrectly when looking up the python libraries.

I've looked at the documentation here and here and I'm still unsure of what should actually be in the PYTHONPATH environment variable.

I have PYTHONHOME pointed to C:\Python27.

My current PYTHONPATH is set to PYTHONHOME. Should I also add the directories from sys.path?

UPDATE:

Based on the below information, PYTHONPATH does not need to be set unless there are non-standard libraries that you want python to be able to find by default. For instance, when I install wxPython from the installer it will add its libraries to PYTHONPATH. I do set PYTHONHOME to the root of the python installation so that I can add it to my system PATH environment variable so that I can run python from any where.

Improve this question

4 Answers 4

Reset to default
78

You don't have to set either of them. PYTHONPATH can be set to point to additional directories with private libraries in them. PYTHONHOME sets the location of default libraries.

Documentation:

PYTHONHOME
Change the location of the standard Python libraries. By default, the libraries are searched in prefix/lib/pythonversion and exec_prefix/lib/pythonversion, where prefix and exec_prefix are installation-dependent directories, both defaulting to /usr/local.

When PYTHONHOME is set to a single directory, its value replaces both prefix and exec_prefix. To specify different values for these, set PYTHONHOME to prefix:exec_prefix.

PYTHONPATH
Augment the default search path for module files. The format is the same as the shell’s PATH: one or more directory pathnames separated by os.pathsep (e.g. colons on Unix or semicolons on Windows). Non-existent directories are silently ignored.

In addition to normal directories, individual PYTHONPATH entries may refer to zipfiles containing pure Python modules (in either source or compiled form). Extension modules cannot be imported from zipfiles.

The default search path is installation dependent, but generally begins with prefix/lib/pythonversion (see PYTHONHOME above). It is always appended to PYTHONPATH.

An additional directory will be inserted in the search path in front of PYTHONPATH as described above under Interface options. The search path can be manipulated from within a Python program as the variable sys.path.

Improve this answer
Sign up to request clarification or add additional context in comments.

5 Comments

PYTHONHOME actually points to the directory of the standard library by default (e.g. /usr/local/lib/pythonXX).
@Ferdinand Not on Windows. It isn't set. But the point is the user usually doesn't have to mess with either of them unless they have a private directory of libraries different that the defaults.
Of course it is not set -- Python never sets environment variables. But there is an internal equivalent to PYTHONHOME that can be overridden using the environment variable. I'm talking about the default value of this internal variable.
@"Ferdinand Beyer" "Of course it is not set -- Python never sets environment variables.", Well the latest python 3.3 at least can add itself to the PATH variable. Worthy of note.
@twobob python 3.3 cannot add itself to the PATH variable, it is the Python 3.3 installer that does so, and that installer is not executing any Python code when it does.
31

For most installations, you should not set these variables since they are not needed for Python to run. Python knows where to find its standard library.

The only reason to set PYTHONPATH is to maintain directories of custom Python libraries that you do not want to install in the global default location (i.e., the site-packages directory).

Make sure to read: http://docs.python.org/using/cmdline.html#environment-variables

Improve this answer

2 Comments

ok so you should not. But what SHOULD you do?! You want to run modules in the path, and don't want the source code holding paths (not relative and definitely not static paths).
For the most part, you should be using virtual environments, the creation of which takes care of modifying PYTHONPATH appropriately.
7

Here is what I learned: PYTHONPATH is a directory to add to the Python import search path "sys.path", which is made up of current dir. CWD, PYTHONPATH, standard and shared library, and customer library. For example: 

% python3 -c "import sys;print(sys.path)" ['', '/home/username/Documents/DjangoTutorial/mySite', '/usr/lib/python3.6', '/usr/lib/python3.6/lib-dynload', '/usr/local/lib/python3.6/dist-packages', '/usr/lib/python3/dist-packages']

where the first path '' denotes the current dir., the 2nd path is via 

%export PYTHONPATH=/home/username/Documents/DjangoTutorial/mySite

which can be added to ~/.bashrc to make it permanent, and the rest are Python standard and dynamic shared library plus third-party library such as django.

As said not to mess with PYTHONHOME, even setting it to '' or 'None' will cause python3 shell to stop working:

% export PYTHONHOME='' % python3 Fatal Python error: Py_Initialize: Unable to get the locale encoding ModuleNotFoundError: No module named 'encodings' Current thread 0x00007f18a44ff740 (most recent call first): Aborted (core dumped)

Note that if you start a Python script, the CWD will be the script's directory. For example:

username@bud:~/Documents/DjangoTutorial% python3 mySite/manage.py runserver ==== Printing sys.path ==== /home/username/Documents/DjangoTutorial/mySite # CWD is where manage.py resides /usr/lib/python3.6 /usr/lib/python3.6/lib-dynload /usr/local/lib/python3.6/dist-packages /usr/lib/python3/dist-packages

You can also append a path to sys.path at run-time: Suppose you have a file Fibonacci.py in ~/Documents/Python directory:

username@bud:~/Documents/DjangoTutorial% python3 >>> sys.path.append("/home/username/Documents") >>> print(sys.path) ['', '/usr/lib/python3.6', '/usr/lib/python3.6/lib-dynload', '/usr/local/lib/python3.6/dist-packages', '/usr/lib/python3/dist-packages', '/home/username/Documents'] >>> from Python import Fibonacci as fibo

or via

% PYTHONPATH=/home/username/Documents:$PYTHONPATH % python3 >>> print(sys.path) ['', '/home/username/Documents', '/home/username/Documents/DjangoTutorial/mySite', '/usr/lib/python3.6', '/usr/lib/python3.6/lib-dynload', '/usr/local/lib/python3.6/dist-packages', '/usr/lib/python3/dist-packages'] >>> from Python import Fibonacci as fibo
Improve this answer

Comments

2

A little bit less 'ad hoc' solution than directly manipulating the PYTHONPATH exists with the use of the flag -e with the pip command, and allows to seamlessly install local libraries that can be imported, and re-imported to reflect changes made.

In order to be able to import mypackage the same way you do with any other module, the correct approach is to use pip locally:

python -m pip install -e /path_to_package/mypackage/

  • python -m ensures you are using the pip package from the same python installation you are currently using.

  • -e makes it editable, i/e import mypackage will reload after you make some changes, instead of using the cached one.

mypackage must be an installable package, i/e contain an __init__.py file, and a basic setup.py (or pyproject.toml file for pipenv)

minimal setup.py

from setuptools import find_packages, setup setup( name='mypackage', # Required version='0.0.1', # Required packages=find_packages(), # Required )

the package structure must be like this:

mypackage/ setup.py mypackage/ <----- this is a folder inside the other `mypackage/` folder __init__.py

or as a tree:

└── python_perso folder └── mypackage folder ├── mypackage folder │ └── __init__.py └── setup.py

[edit] after installation, the directory will look like this:
(for a package named mypackage)

└── python_perso └── mypackage ├── mypackage │ ├── __init__.py │ └── __pycache__ │ └── __init__.cpython-38.pyc ├── mypackage.egg-info │ ├── PKG-INFO │ ├── SOURCES.txt │ ├── dependency_links.txt │ └── top_level.txt └── setup.py 5 directories, 7 files
Improve this answer

4 Comments

The -e option doesn't do any auto-reload! It just means that your package's files will be used from the place where they are, instead of copying them over to site-packages directory.
What do you mean by "restarting kernel"? When you restart your script, the newly launched Python process will import modules from their declared paths. If you pip installed your module without -e option then your files are copied from wherever they were to site-packages directory and loaded from there. If you edited your files then site-packages copy won't be edited until you re-install again with pip install.
Now, when you use pip install -e (aka --editable), pip will create a special file in site-packages named MYMODULE.egg-link with a full path to your module's original location (like /home/user/work/mymodule), and Python will load module files from there. So if you edit your module files then any subsequent fresh import will use the updated files, without you having to pip install again for updating the site-packages copy. It should be noted though that imports are cached, so they won't be "auto-updated" until you restart Python process or use importlib.reload.
Thank you for these important precisions, you've explained it better than I.