2

I am using subprocess.Popen to call an external Python script from my installed Application, bundled using PyInstaller. The format of this command is something like this:

subprocess.Popen(["/usr/bin/python", "/path/to/exe/SDK.py"], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=False)

I have successfully tested this on both Windows and MacOS - both can run the external script. However, on other Posix OS' I get the following error:

Could not find platform independent libraries <prefix> Could not find platform dependent libraries <exec_prefix> Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>] ImportError: No module named site

Now, I realize that this question has been asked before, and generally it is down to PYTHONHOME having an incorrect value. However, if I run the command in a format similar to:

PYTHONHOME=/usr /usr/bin/python /path/to/exe/SDK.py

I get no errors in the logs, but the script SDK.py doesn't execute.

Therefore to prove that this isn't an issue with my script; I installed my own version of Python onto the machine - after doing this, the script executed successfully. I tested with both /usr/bin/python and /home/vagrant/Python-2.7.15/python, without needing to specify a PYTHONHOME in the subprocess command.

However, I still need to allow users to execute the SDK.py script using the built in, OS version of Python.

So to do this I have tried these other things:

  1. Using sys.executable in the subprocess.Popen call to execute SDK.py.

(The value of which was: /opt/program_name/lib/program_name)

  1. Forcing PYTHONPATH & PYTHONHOME to be empty by explicitly setting the environment:

    import os env = os.environ.copy() env['PYTHONHOME'] = '' env['PYTHONPATH'] = '' subprocess.Popen(["/usr/bin/python", "/path/to/exe/SDK.py"], env=env)

Can anyone explain how I can call the system version of Python from an external subprocess like this?

EDIT: Output when calling Python in verbose mode (from the CLI):

>>> from sys import executable >>> executable '/usr/bin/python' >>> import _csv # trying _csv.so # trying _csvmodule.so # trying _csv.py # trying _csv.pyc # trying /usr/lib64/python2.7/_csv.so # trying /usr/lib64/python2.7/_csvmodule.so # trying /usr/lib64/python2.7/_csv.py # trying /usr/lib64/python2.7/_csv.pyc # trying /usr/lib64/python2.7/plat-linux2/_csv.so # trying /usr/lib64/python2.7/plat-linux2/_csvmodule.so # trying /usr/lib64/python2.7/plat-linux2/_csv.py # trying /usr/lib64/python2.7/plat-linux2/_csv.pyc # trying /usr/lib64/python2.7/lib-dynload/_csv.so dlopen("/usr/lib64/python2.7/lib-dynload/_csv.so", 2); import _csv # dynamically loaded from /usr/lib64/python2.7/lib-dynload/_csv.so
Improve this question
7
  • What is the value of your PYTHONHOME initially? And the path you tried isn't correct, the path components need to be separated by a colon, not a blank. Commented Sep 16, 2018 at 16:27
  • @markus Initially, on a fresh vm, whenever I did echo $PYTHONHOME it gives an empty, blank line. So I think that its not set. Commented Sep 16, 2018 at 16:31
  • As for the path being incorrect, do you mean the one where I specify the PYTHONHOME=/usr/? Commented Sep 16, 2018 at 16:35
  • Yes, that one. But I just realized that I misread that line. But just /usr for the value of PYTHONHOME is most likely not correct. Commented Sep 16, 2018 at 16:40
  • I think I was using this SO Comment as a pointer for setting PYTHONHOME to to /usr/: stackoverflow.com/a/36173084 Commented Sep 16, 2018 at 17:03

2 Answers 2

Reset to default
1

This is not a full answer, but for the sake of completeness I will document here the two things that worked - and which approach we went with in the end.

  1. For some strange reason, when my colleague did this:

env = os.environ.copy() env['LD_LIBRARY_PATH'] = env['PATH']

It allowed us to call the subprocess when we passed this env into the Popen() call like so:

self.process = subprocess.Popen(sdk_executable_command, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=False, env=env)

We did not investigate this further because we went with the second solution described below. However, it would lead me to believe that these issues may have been down to the predefined setting of LD_LIBRARY_PATH.

  1. When we created a virtualenv for python using the pip package of the same name, and we called the SDK using this python executable - the SDK would successfully run. Reason for this is still unknown, but because this is an SDK: we decided it was appropriate for the eventual SDK Developer to run this in an environment separate from the 'out of the box' Python in Linux.
Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

0

I'll try and walk you trough the process of checking you Python installation on CentOS 7.

First of all, open some terminal window and verify the installation of the python RPMs:

$ rpm -V python python-libs

If you get no output, everything is fine. Next check the environment variables:

$ env | grep PYTHON

There should be no output as well. If there is some mention of PYTHONHOME or PYTHONPATH, unset them. Now check what Python thinks about its default path:

$ /usr/bin/python Python 2.7.5 (default, Jul 13 2018, 13:06:57) [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from sys import path >>> print path ['', '/usr/lib64/python27.zip', '/usr/lib64/python2.7', '/usr/lib64/python2.7/plat-linux2', '/usr/lib64/python2.7/lib-tk', '/usr/lib64/python2.7/lib-old', '/usr/lib64/python2.7/lib-dynload', '/usr/lib64/python2.7/site-packages', '/usr/lib64/python2.7/site-packages/gtk-2.0', '/usr/lib/python2.7/site-packages']

You should also be able to import the site module without any problems:

>>> import site >>> print site <module 'site' from '/usr/lib64/python2.7/site.pyc'>

If everything is OK, try to run your script from the very same command line where you performed the tests.

Improve this answer

11 Comments

Hi Markus, everything is as you say here except for one thing: /usr/lib64/python2.7/site-packages/gtk-2.0 is missing from sys.path Would this be relevant?
@tmccaffrey Don't think so, unless you somewhere need the GTK Python packages, of course. Do you still get the same error?
@tmccaffrey Just to avoid any misunderstanding... You don't try to set any of the PYTHON* variables to an empty string? By unsetting, I mean unset PYTHONHOME. And don't fiddle with those variables in your script.
And if you create a new (copy) of the environ dict you pass to your process, like in your second example, unsetting the variable would be done by means of del(env['PYTHONHOME]) instead of env['PYTHONHOME] = '' which sets it to empty string.
Thanks for your comments lads, at the moment I get a KeyError if I try to print env['PYTHONHOME']. This should mean that it is already unset right?
|

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.