96

I need to install several Python modules on a RHEL where I don't have root access. At least one of the modules also needs access to Python.h.

In this case I find that the best thing is to install python and it dependencies in ~/local. It usually just works, but this time Python fails to build the SSL module (see details below). Here's the trace of what I'm doing.

So I downloaded python 6 source and off I went:

./configure --prefix=/home/fds/rms/local make >& make.log

An inspection to log reveals that ssl module has not been compiled, but there is no mention of the cause (no other occurence of ssl in make or configure):

Failed to find the necessary bits to build these modules: _bsddb _curses _curses_panel _hashlib _sqlite3 _ssl <----------

So I figure, python is not finding any ssl library at all (which is strange, but hey...). So I download openssl-0.9.8r and

./config --prefix=/home/fds/rms/local shared make make install

Now back to Python, I ./configure and make again. It fails, but this time it's different:

Failed to build these modules: _hashlib _ssl

A closer inspection to the log file reveals this:

gcc -pthread -shared build/temp.linux-x86_64-2.6/home/fds/rms/installers/Python-2.6.6/Modules/_ssl.o -L/home/fds/rms/local/lib -L/usr/local/lib -lssl -lcrypto -o build/lib.linux-x86_64-2.6/_ssl.so *** WARNING: renaming "_ssl" since importing it failed: libssl.so.0.9.8: cannot open shared object file: No such file or directory

So now it's picking up the library but not quite getting it right (the file is there where is should be):

$ find /home/fds/rms/local -iname libssl.so.0.9.8 /home/fds/rms/local/lib/libssl.so.0.9.8

Next thing is tracing make and see where is it looking for the file:

$ strace -f make 2>&1 | grep libssl.so.0.9.8 [pid 5584] open("/lib/libssl.so.0.9.8", O_RDONLY) = -1 ENOENT (No such file or directory) [pid 5584] open("/usr/lib/libssl.so.0.9.8", O_RDONLY) = -1 ENOENT (No such file or directory) [pid 5584] open("/lib64/tls/x86_64/libssl.so.0.9.8", O_RDONLY) = -1 ENOENT (No such file or directory) [pid 5584] open("/lib64/tls/libssl.so.0.9.8", O_RDONLY) = -1 ENOENT (No such file or directory) [pid 5584] open("/lib64/x86_64/libssl.so.0.9.8", O_RDONLY) = -1 ENOENT (No such file or directory) [pid 5584] open("/lib64/libssl.so.0.9.8", O_RDONLY) = -1 ENOENT (No such file or directory) [pid 5584] open("/usr/lib64/tls/x86_64/libssl.so.0.9.8", O_RDONLY) = -1 ENOENT (No such file or directory) [pid 5584] open("/usr/lib64/tls/libssl.so.0.9.8", O_RDONLY) = -1 ENOENT (No such file or directory) [pid 5584] open("/usr/lib64/x86_64/libssl.so.0.9.8", O_RDONLY) = -1 ENOENT (No such file or directory) [pid 5584] open("/usr/lib64/libssl.so.0.9.8", O_RDONLY) = -1 ENOENT (No such file or directory) [pid 5584] write(1, "*** WARNING: renaming \"_ssl\" sin"..., 131*** WARNING: renaming "_ssl" since importing it failed: libssl.so.0.9.8: cannot open shared object file: No such file or directory [pid 5584] open("/lib/libssl.so.0.9.8", O_RDONLY) = -1 ENOENT (No such file or directory) [pid 5584] open("/usr/lib/libssl.so.0.9.8", O_RDONLY) = -1 ENOENT (No such file or directory) [pid 5584] open("/lib64/libssl.so.0.9.8", O_RDONLY) = -1 ENOENT (No such file or directory) [pid 5584] open("/usr/lib64/tls/libssl.so.0.9.8", O_RDONLY) = -1 ENOENT (No such file or directory) [pid 5584] open("/usr/lib64/libssl.so.0.9.8", O_RDONLY) = -1 ENOENT (No such file or directory) [pid 5584] write(1, "*** WARNING: renaming \"_hashlib\""..., 135*** WARNING: renaming "_hashlib" since importing it failed: libssl.so.0.9.8: cannot open shared object file: No such file or directory

Mhhh, it's looking in all the wrong places. I try to give a hint:

CPPFLAGS="-I/home/fds/rms/local/include -I/home/fds/rms/local/include/openssl" LDFLAGS="-L/home/fds/rms/local/lib" ./configure --prefix=/home/fds/rms/local

But nothing changes, and make does not seem to try /home/fds/rms/local/lib at all.

I haven't done this in years, so maybe I'm overlooking something. Can anyone help with the problem?

Improve this question
4

13 Answers 13

Reset to default
74

You need to edit Modules/Setup.dist to specify the location of OpenSSL if it is not in the standard location. From Getting SSL Support in Python 2.5.1:

If you find yourself on a linux box needing ssl support in python (to use a client in things like httplib.HTTPSConnection or imaplib.IMAP4_SSL), then let me save you a couple of hours of hunting around the web (of course if you have found this then that means you've done some level hunting already!).

You'll know if you need ssl support compiled into your python installation if you get the following exception message: AttributeError: 'module' object has no attribute 'ssl'

In order to make that go away so you can continue happily slinging python code, you'll need to first make sure you have OpenSSL installed. By default it is installed from source at: /usr/local/ssl

If that directory doesn't exist, then grab the source package.

Do the standard:

tar zxf openssl-0.9.8g.tar.gz cd openssl-0.9.8g ./config make make install

Then grab the python sources for 2.5.1 and: tar zxf Python-2.5.1.tgz && cd Python-2.5.1

Then you need to edit the Modules/Setup.dist:

204:# Socket module helper for SSL support; you must comment out the other 205:# socket line above, and possibly edit the SSL variable: 206:SSL=/usr/local/ssl 207:_ssl _ssl.c \ 208: -DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \ 209: -L$(SSL)/lib -lssl -lcrypto

If you installed OpenSSL in the default locations you can just uncomment lines 206-209, then:

./configure make make install

Then verify your installation with:

python /usr/local/lib/python2.5/test/test_socket_ssl.py test_rude_shutdown ... test_basic ... test_timeout ...

Make sure the changes to Modules/Setup.dist get picked up by cleaning the source root (e.g. make distclean) and run configure and make again.

Improve this answer
Sign up to request clarification or add additional context in comments.

7 Comments

On RHEL, where is the standard location? I have installed openssl and openssl-devel rpm with yum
Works for python3.6 too ^^
not work on python3.7 when openssl not installed in standard location
This worked for installing Python 3.7.4 on RHEL. With default openssl install, I had to change SSL line: SSL=/etc/pki/tls, uncomment out the next three lines and I had to run sudo yum install openssl-devel
Setup.dist only needed editing up to Python 2.5. This is no longer needed in any more recent versions.
|
34

@PeterVanGalen has a good method, and is relevant as of 2021, but I disagree with some details.

His method of modifying the Modules/Setup file results in the libssl.so and libcrypto.so libraries becoming dynamic dependencies of the python binary itself. This is not as intended -- those are supposed to be dependencies of python importable .so's, like _ssl.cpython-39-x86_64-linux-gnu.so.

His method also doesn't include a solution for how libssl.so and libcrypto.so will be found at runtime, rather than build time. This matters if they are in some unusual, custom path, or else python will either not run at all or be unable to import ssl. You could solve it in lots of ways (LD_LIBRARY_PATH and ld.so.conf come to mind) but I opted to use rpath, so that the .so files can always be found in their custom location when I use this python, but will otherwise stay out of the way.

This is my modification of @PeterValGalen's approach:

# First we need openssl installed in a custom location... tar zxf openssl-1.1.1j.tar.gz cd openssl-1.1.1j ./config --prefix=/my/weird/path --openssldir=/my/weird/path/ssl make make install # Add `sudo` if needed for permission to /my/weird/path. cd .. # Now the python part... wget https://www.python.org/ftp/python/3.9.2/Python-3.9.2.tgz tar xf Python-3.9.2.tgz cd Python-3.9.2 LDFLAGS="${LDFLAGS} -Wl,-rpath=/my/weird/path/lib" ./configure --with-openssl=/my/weird/path make make install # Add `sudo` if needed for permissions.

When done this way, the openssl libs are not dependencies of the python binary, but are dependencies of the _ssl*.so

$ ldd ./python | grep weird # [Nothing] $ ldd build/lib.linux-x86_64-3.9/_ssl.cpython-39-x86_64-linux-gnu.so | grep weird libssl.so.1.1 => /my/weird/path/lib/libssl.so.1.1 (0x00007f733ee73000) libcrypto.so.1.1 => /my/weird/path/lib/libcrypto.so.1.1 (0x00007f733e9ab000)

And it works:

$ ./python -c "import ssl; print('yay')" yay $ ./python Lib/test/test_ssl.py # ... lots of good stuff... OK (skipped=10)
Improve this answer

5 Comments

I use python 3.6.8, it show configure: WARNING: unrecognized options: --with-openssl
This configure command-line option was added in CPython 3.7.
Fantastic! This should be the accepted answer for current versions of Python.
It works, but I had to make a symbolic link in my openssl install named lib pointing to lib64 just next to it. In case it can help somebody.
@Welgriv thank you! I just want to let you know that you helped me. Even though I have been caught by this lib64 and lib thing in the past, cuda did it to me and I forgot about it.
25

On Linux Red Hat 7.7 x86_64, the following worked to install openssl-1.1.1d and Python-3.8.1 in my home directory (/home/unix/vangalen):

Install OpenSSL source1 source2

cd ~ wget https://www.openssl.org/source/openssl-1.1.1d.tar.gz tar -xzf openssl-1.1.1d.tar.gz cd /home/unix/vangalen/openssl-1.1.1d ./config --prefix=/home/unix/vangalen/openssl --openssldir=/home/unix/vangalen/openssl make make test make install

Install Python source2 source3 source4

cd ~ wget https://www.python.org/ftp/python/3.8.1/Python-3.8.1.tgz tar xf Python-3.8.1.tgz

Modify Python-3.8.1/Modules/Setup in a text editor. If this file doesn't exist you may need to go through a failed run first. Uncomment lines and adjust the alias for SSL in lines 206 to 213::

_socket socketmodule.c # Socket module helper for SSL support; you must comment out the other # socket line above, and possibly edit the SSL variable: SSL=/home/unix/vangalen/openssl _ssl _ssl.c \ -DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \ -L$(SSL)/lib -lssl -lcrypto 
cd ~/Python-3.8.1 export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/home/unix/vangalen/openssl/lib ./configure --prefix=/home/unix/vangalen/py-381 --with-openssl=/home/unix/vangalen/openssl make make test make install
Improve this answer

Comments

9

in the Bourne shell (/bin/sh or /bin/bash):

$ LD_LIBRARY_PATH=/usr/local/lib $ export LD_LIBRARY_PATH $ make

in the C-shell (/bin/csh or /bin/tcsh):

% setenv LD_LIBRARY_PATH /usr/local/lib % make
Improve this answer

2 Comments

This, in combination with a local install of OpenSSL, did the trick for me.
This seems to work only at compile time, when running python later without the export the library is not found anymore!
4

For me editing Modules/Setup was not enough as _hashlib module still ended up using the wrong OpenSSL version; and LD_LIBRARY_PATH was not taken into account at runtime on my SLES system.

What worked was to statically link the local OpenSSL to both _ssl and _hashlib by editing setup.py as per GitHub patch: eddy-geek/ python_custom_openssl.diff, and then make clean && make.

More details on why I used static links on Stack Overflow at Coredump when compiling python with a custom openssl version.

Improve this answer

Comments

3

Here is the complete process I used with Python 2.7.11.

In top level Python2.7.11 source dir:

  1. Change Modules/Setup.dist, Modules/Setup : Uncomment _ssl section, comment out _socket (no-op if it's already commented out), uncomment and set SSL appropriately (path to your new ssl lib/includes etc.)

    Note: Modules/Setup file does not exist initially but after first run it get's content from Modules/Setup.dist i believe. Make sure changes are reflected in here before every run.

  2. Apply patch: http://gist.github.com/eddy-geek/9604982 (make distclean if previously ran make)

    ./configure --prefix=/root/.local/aks/py-ssl/python2.7 --enable-shared # modify: Makefile -> set svnversion to "" make make altinstall
Improve this answer

2 Comments

Confirmed that this approach works, and don't forget to change Modules/Setup file after the 1st failed run.
I've uncommented the lines in Modules/Setup.dist, then: ./configure --prefix='my build dir' --enable-shared make which fails: dyld: Symbol not found: _GENERAL_NAME_free Referenced from: my_source_dir/libpython2.7.dylib Expected in: dynamic lookup /bin/sh: line 1: 63299 Abort trap: 6 DYLD_LIBRARY_PATH=my_source_dir ./python.exe -E -S -m sysconfig --generate-posix-vars generate-posix-vars failed make only fails when I include --enable-shared with configure. Conversely, make builds correctly with --enable-shared if I don't uncomment ssl lines in Setup.dist.
3

I have a set of a couple of patches for static openssl and static libintl for 2 and 3 below.

For openssl (first patch) you have to set the env var OPENSSL_ROOT.

This is based on the patch from http://gist.github.com/eddy-geek/9604982 .

For Python 2.7.14:

https://gist.github.com/rkitover/2d9e5baff1f1cc4f2618dee53083bd35

https://gist.github.com/rkitover/afab7ed3ac7ce1860c43a258571c8ae1

For Python 3.6.3:

https://gist.github.com/rkitover/93d89a679705875c59275fb0a8f22b45

https://gist.github.com/rkitover/b18f19eafda3775a9652cc9cdf3ec914

Improve this answer

2 Comments

Is the second patch for 2.7 and 3.6 respectively for the iconv check necessary?
The second patch I use because I use a static gettext, you may not need this.
3

OpenSSL

cd /opt sudo wget https://www.openssl.org/source/openssl-1.1.1q.tar.gz --no-check-certificate sudo mkdir /opt/openssl sudo tar xfvz openssl-1.1.1q.tar.gz --directory /opt/openssl export LD_LIBRARY_PATH=/opt/openssl/lib cd /opt/openssl/openssl-1.1.1q/ sudo ./config --prefix=/opt/openssl --openssldir=/opt/openssl/ssl sudo make sudo make test sudo make install sudo ln -s /usr/local/bin/openssl /usr/bin/openssl

Python 3.9.13

cd /usr/src sudo wget https://www.python.org/ftp/python/3.9.13/Python-3.9.13.tgz sudo tar xzf Python-3.9.13.tgz cd /usr/src/Python-3.9.13 sudo CFLAGS="-I/opt/openssl/include/" LDFLAGS="${LDFLAGS} -Wl,-rpath=$LD_LIBRARY_PATH" ./configure --enable-optimizations --with-openssl=/opt/openssl sudo make altinstall -j6 sudo ln -s /usr/local/bin/python3.9 /usr/bin/python3

pip3

sudo ln -s /usr/local/bin/pip3.9 /usr/bin/pip3 pip3 install --upgrade pip

Tested on Ubuntu 14.04.06

Improve this answer

1 Comment

Perfect, still works with "openssl-1.1.1w" and "Python-3.12.2" in March 2024 Thank you!
2

I was getting the same result until I got back to the logs for openssl. There I saw that you need to use -fPIC when building openssl: building '_ssl' extension:

gcc -pthread -fPIC -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/local/ssl/include -I. -IInclude -I./Include -I/usr/local/include -I/home/feramos/Python-2.7.3/Include -I/home/feramos/Python-2.7.3 -c /home/feramos/Python-2.7.3/Modules/_ssl.c -o build/temp.linux-x86_64-2.7/home/feramos/Python-2.7.3/Modules/_ssl.o gcc -pthread -shared build/temp.linux-x86_64-2.7/home/feramos/Python-2.7.3/Modules/_ssl.o -L/usr/local/ssl/lib -L/usr/local/lib -lssl -lcrypto -o build/lib.linux-x86_64-2.7/_ssl.so /usr/bin/ld: /usr/local/ssl/lib/libcrypto.a(x86_64cpuid.o): relocation R_X86_64_PC32 against `OPENSSL_cpuid_setup' can not be used when making a shared object; recompile with -fPIC openssl-0.9.8g]# .config -fPIC

then, make, make install for openssl and then build Python again.

Improve this answer

Comments

2

For MAC OS HIGH Sierra, and Python-3.5.6 In the above answer the openssl installation is done using source,but if you install using brew it tells where the installed package is, so if you install openssl using brew

brew install openssl

This will install openssl at /usr/local/Cellar/openssl/1.0.2o_2/ this path needs to be updated in Modules/Setup.dist at

Follow this answer which is mentioned above where the location of openssl installation is not mentioned to update the Modules/Setup.dist

In the above lines update SSL value to

SSL=/usr/local/Cellar/openssl/1.0.2o_2/

and uncomment the lines, do a CMM and your python will get compiled with openssl.

Improve this answer

2 Comments

That's not sufficiently different from stackoverflow.com/a/5939170/1797006 to make it a separate answer.
@KarlRichter... I was trying to make it simpler by using brew as brew tells the location of the installation that was the sole reason I wrote it separately. I have updated the answer as it was overlapping with the mentioned answer.
1

I'm building Python2.7.13 and I see this same issue. For 2.7.13 you have to use "openssl1.0.0e" or above to make it work. I tried openssl-0.9.8g and it doesn't work. And somehow I can't make it work just modifying Modules/Setup.dist so I have to manually compile that _ssl.o. I guess this is because openssl-0.9.8g I provided is not working and it searched for system default libssl.so.10 which doesn't work either.

Improve this answer

1 Comment

Just to add one thing, edit Modules/Setup.dist doesn't work for me. I have to edit Modules/Setup to make it work
0

Try adding -Wl,-rpath,/home/fds/rms/local/lib to LDPATH.

Improve this answer

2 Comments

Thanks par, but make is ignoring everything and only looking for the lib in the "standard" locations.
-Wl, passes everything after the , to the linker executable that is invoked by the compiler driver to specify where the binary that the linker is creating is statically supposed to look for runtime libraries. A runtime option is not going to help a buildtime problem. See gcc.gnu.org/onlinedocs/gcc-3.4.2/gcc/Link-Options.html
0

I found a really good and complete explanation bellow.

  • Title: How to install a shared Python for multiple accounts without SSL issues

https://medium.com/@enahwe/how-to-06bc8a042345

Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.