21

I'm trying to compile python on RHEL because my current python is using an old 1.0.2k ssl version.

(test_env) [brad@reason tlscheck]$ python3 --version Python 3.9.3 (test_env) [brad@reason tlscheck]$ python3 -c "import ssl; print(ssl.OPENSSL_VERSION)" OpenSSL 1.0.2k-fips 26 Jan 2017 (test_env) [brad@reason tlscheck]$ openssl version OpenSSL 1.1.1l 24 Aug 2021

I think the issue is that when I compiled 3.9.3, I had not updated my OpenSSL version. I have since updated my OpenSSL and need to use it with python. So I have downloaded the newest python 3.10, but in the make stage I get an error that it will not make with ssl. I the following message:

Following modules built successfully but were removed because they could not be imported: _hashlib _ssl Could not build the ssl module! Python requires a OpenSSL 1.1.1 or newer

This is the full log of trying to compile: https://pastebin.com/36EntpFz

When I use the configure options that @tony-yip mentioned, I get the following in my configure.

checking for openssl/ssl.h in /etc/ssl... no checking whether compiling and linking against OpenSSL works... no

I'm determining my openssl location with:

[brad@reason Python-3.10.0]$ openssl version -d OPENSSLDIR: "/etc/ssl"

To configure, I'm using:

./configure --with-openssl="/etc/ssl"

When I look for ssl.h, I find it in /usr/include/openssl. So I linked this directory to lib in /etc/ssl, but it was no help.

[brad@reason Python-3.10.0]$ ls -l /etc/ssl total 40 lrwxrwxrwx 1 root root 16 Jul 16 2020 certs -> ../pki/tls/certs -rw-r--r-- 1 root root 412 Oct 12 02:53 ct_log_list.cnf -rw-r--r-- 1 root root 412 Oct 12 02:53 ct_log_list.cnf.dist lrwxrwxrwx 1 root root 20 Oct 18 10:22 lib -> /usr/include/openssl drwxr-xr-x 2 root root 4096 Oct 12 02:53 misc -rw-r--r-- 1 root root 10909 Oct 12 02:53 openssl.cnf -rw-r--r-- 1 root root 10909 Oct 12 02:53 openssl.cnf.dist drwxr-xr-x 2 root root 4096 Oct 12 02:53 private [brad@reason Python-3.10.0]$ sudo find / -name ssl.h | grep include find: ‘/tmp/.mount_jetbraAJFEnl’: Permission denied /home/brad/Downloads/freerdp-2.0.0-rc4/winpr/include/winpr/ssl.h /home/brad/Downloads/FreeRDP/winpr/include/winpr/ssl.h /home/brad/Development/tlscheck/openssl-1.1.1l/include/openssl/ssl.h /usr/include/openssl/ssl.h /var/lib/docker/overlay2/23e6f3c164ec8939352891c99393669df4ed6e66da1e04ce84616073f08c6051/diff/usr/include/openssl/ssl.h /var/lib/flatpak/runtime/org.freedesktop.Sdk/x86_64/18.08/c8075e929daaffcbe5c78c9e87c0f0463d75e90d2b59c92355fa486e79c7d0e3/files/include/nss/ssl.h /var/lib/flatpak/runtime/org.freedesktop.Sdk/x86_64/18.08/c8075e929daaffcbe5c78c9e87c0f0463d75e90d2b59c92355fa486e79c7d0e3/files/include/openssl/ssl.h find: ‘/run/user/1000/gvfs’: Permission denied

This may be extraneous information, but my libssl.so is here:

[brad@reason Python-3.10.0]$ ls /usr/lib64 | grep ssl libevent_openssl-2.0.so.5 libevent_openssl-2.0.so.5.1.9 libssl3.so libssl.so libssl.so.10 libssl.so.1.0.2k openssl

Any thoughts on why make isn't able to include ssl, please let me know. Thanks.

Improve this question
1

5 Answers 5

Reset to default
22

This was very helpful, thanks! What kicked me onto this was the inability to run/update pip in a venv, so there wasn't much of an alternative to figuring out how to get OpenSSL to work.

As of this week (21 December 2021), in my environment (CentOS Linux release 7.9.2009 (Core)), I was able to shortcut parts of this.

From EPEL, installing openssl11, (yes, also openssl11-devel); to set up Python 3.10 properly I needed to create that "OpenSSL Directory" which I did by:

mkdir /usr/local/openssl11

Then setting symlinks for the (newly-installed) requirements:

cd /usr/local/openssl11 ln -s /usr/lib64/openssl11 lib ln -s /usr/include/openssl11 include

And, finally, feeding this location to the configure script:

./configure --with-openssl=/usr/local/openssl11

Thanks for the really helpful explanations of how and why this was all necessary. Hope this helps others...

Improve this answer
Sign up to request clarification or add additional context in comments.

5 Comments

This was the only way I could get pip to work (so that I could get TurboGears working.).
This method helped me too
Thanks man! you saved my day! tested everything but this does the trick!
This worked for me, too, building python3.10 on RHEL 7.9. Thank you!
worked for centos 7, simplest solution, thank you
18

Had a very similar problem, with openssl not working and giving the same errors with python 3.10 on centos 7. Download openssl unpack then go to that directory

./config --prefix=/usr/local/custom-openssl --openssldir=/etc/ssl make -j1 depend make -j8 make install_sw

Then go to the python source unpack it and run in the directory

./configure -C --with-openssl=/usr/local/custom-openssl --with-openssl-rpath=auto --prefix=/usr/local/python-3.version make -j8 make altinstall

See also Custom OpenSSL on https://docs.python.org/3/using/unix.html.

Improve this answer

2 Comments

This was it for me, I have been trying to fix this for days
When I used this --with-openssl-rpath=auto, to my surprise it actually added a RUNPATH in the .so, not an rpath. Was no problem for me, but they do behave differently, so that's something to watch out for if you care about the differences.
6

As the output suggest, you need to point to openssl11. The easiest way I found to do it is to change the configure file in the python 3.10 source code directory:

$ sed -i 's/PKG_CONFIG openssl /PKG_CONFIG openssl11 /g' configure $ sudo ./configure --enable-optimizations . . checking whether compiling and linking against OpenSSL works... yes .

Then you can do as @CristiFati suggested in order to check before installing that it is configured properly:

$ ./python -c "import sys, ssl; print(\"{:s}\n{:s}\".format(sys.version, ssl.OPENSSL_VERSION))" 3.10.0 (default, Nov 29 2021, 17:48:34) [GCC 7.3.1 20180712 (Red Hat 7.3.1-13)] OpenSSL 1.1.1g FIPS 21 Apr 2020
Improve this answer

Comments

2

1. Intro

Some references that might be (more or less) useful:

Working on a CentOS 7 image from Docker Hub (as it's closest to RHEL 7 (that you're having))

[root@cfati-5510-0:/work/q069539286]> uname -a Linux cfati-5510-0 5.11.0-37-generic #41~20.04.2-Ubuntu SMP Fri Sep 24 09:06:38 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux [root@cfati-5510-0:/work/q069539286]> cat /etc/redhat-release CentOS Linux release 7.9.2009 (Core) [root@cfati-5510-0:/work/q069539286]> [root@cfati-5510-0:/work/q069539286]> rpm -qa | grep openssl openssl-libs-1.0.2k-22.el7_9.x86_64 openssl-1.0.2k-22.el7_9.x86_64 openssl-devel-1.0.2k-22.el7_9.x86_64 [root@cfati-5510-0:/work/q069539286]> [root@cfati-5510-0:/work/q069539286]> openssl version OpenSSL 1.0.2k-fips 26 Jan 2017 [root@cfati-5510-0:/work/q069539286]> which openssl /usr/bin/openssl [root@cfati-5510-0:/work/q069539286]> ldd /usr/bin/openssl linux-vdso.so.1 => (0x00007ffe101b7000) libssl.so.10 => /lib64/libssl.so.10 (0x00007f9483b8a000) libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f948393d000) libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f9483654000) libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f9483450000) libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f948321d000) libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007f9482dba000) libdl.so.2 => /lib64/libdl.so.2 (0x00007f9482bb6000) libz.so.1 => /lib64/libz.so.1 (0x00007f94829a0000) libc.so.6 => /lib64/libc.so.6 (0x00007f94825d2000) libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f94823c2000) libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f94821be000) libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f9481fa4000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f9481d88000) /lib64/ld-linux-x86-64.so.2 (0x00007f9483dfc000) libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f9481b61000) libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f94818ff000)

2. OpenSSL considerations

  • As seen (also present in previous snippet), CentOS 7 (and I think RHEL 7 as well) comes with OpenSSL 1.0.2*. Since OpenSSL 1.1.1* is a minor (but kind of major) release, they are not (API / ABI) compatible, and therefore not interchangeable

  • v1.0.2* is de facto (main) version for CentOS 7 (RHEL 7 too), everything is built against it, changing it would break things. But it is possible to install newer versions (I used the EPEL (more on [RedHat]: What's EPEL, and how do I use it?) repo ([Pkgs.CentOS]: EPEL x86_64) which provides v1.1.1g). But they will only work around the main version. For example, almost every (important) file name in the newer version has a suffix at the end (the .exe is called openssl11) in order to avoid name clashes with main version

    • Since v1.1.1g installs in the (pretty much) same location as the main version (with changed names), it won't get picked up (by default) by any build system. So I had to unpack the rpms (via RPM2Cpio) in a custom dir

    • Not sure how you got v1.1.1l - probably built it yourself (or found it somewhere), so some of the next steps might not be required.
      As a side note, I built (an in house modified) OpenSSL version on a variety of OSes / CPU architecutes (RHEL 7 included) many times, but since the rpms were available, I didn't bother to do it again

  • Even if openssl11 (and its dependent openssl11-libs) is "up and running", openssl11-devel is also required by the Python build

[root@cfati-5510-0:/work/q069539286]> mkdir -p openssl-1.1.1g && cd openssl-1.1.1g [root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> ls [root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> for g in /var/cache/yum/x86_64/7/epel/packages/openssl11-libs-1.1.1g-3.el7.x86_64.rpm /var/cache/yum/x86_64/7/epel/packages/openssl11-1.1.1g-3.el7.x86_64.rpm /var/cache/yum/x86_64/7/epel/packages/openssl11-devel-1.1.1g-3.el7.x86_64.rpm; do rpm2cpio ${g} | cpio -idm; done 7292 blocks 2123 blocks 8352 blocks [root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> ls etc usr [root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> ll ./usr/bin/ total 637 -rwxrwxrwx 1 root root 610 Mar 29 2021 make-dummy-cert -rwxrwxrwx 1 root root 644424 Mar 29 2021 openssl11 -rwxrwxrwx 1 root root 725 Mar 29 2021 renew-dummy-cert [root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> ./usr/bin/openssl11 ./usr/bin/openssl11: error while loading shared libraries: libssl.so.1.1: cannot open shared object file: No such file or directory [root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> ll ./usr/lib64/ total 3605 drwxrwxrwx 1 root root 0 Oct 21 23:55 engines-1.1 lrwxrwxrwx 1 root root 19 Oct 21 23:55 libcrypto.so.1.1 -> libcrypto.so.1.1.1g -rwxrwxrwx 1 root root 3082216 Mar 29 2021 libcrypto.so.1.1.1g lrwxrwxrwx 1 root root 16 Oct 21 23:55 libssl.so.1.1 -> libssl.so.1.1.1g -rwxrwxrwx 1 root root 603568 Mar 29 2021 libssl.so.1.1.1g drwxrwxrwx 1 root root 0 Oct 21 23:55 openssl11 drwxrwxrwx 1 root root 0 Oct 21 23:55 pkgconfig [root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> ldd ./usr/bin/openssl11 linux-vdso.so.1 => (0x00007ffca03f5000) libssl.so.1.1 => not found libcrypto.so.1.1 => not found libz.so.1 => /lib64/libz.so.1 (0x00007f3a61e56000) libdl.so.2 => /lib64/libdl.so.2 (0x00007f3a61c52000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f3a61a36000) libc.so.6 => /lib64/libc.so.6 (0x00007f3a61668000) /lib64/ld-linux-x86-64.so.2 (0x00007f3a6206c000) [root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> # Set LD_LIBRARY_PATH [root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> LD_LIBRARY_PATH=./usr/lib64:${LD_LIBRARY_PATH} ldd ./usr/bin/openssl11 linux-vdso.so.1 => (0x00007ffdf7fce000) libssl.so.1.1 => ./usr/lib64/libssl.so.1.1 (0x00007fe00f7a4000) libcrypto.so.1.1 => ./usr/lib64/libcrypto.so.1.1 (0x00007fe00f2c1000) libz.so.1 => /lib64/libz.so.1 (0x00007fe00f0ab000) libdl.so.2 => /lib64/libdl.so.2 (0x00007fe00eea7000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fe00ec8b000) libc.so.6 => /lib64/libc.so.6 (0x00007fe00e8bd000) /lib64/ld-linux-x86-64.so.2 (0x00007fe00fa34000) [root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> LD_LIBRARY_PATH=./usr/lib64:${LD_LIBRARY_PATH} ./usr/bin/openssl11 version OpenSSL 1.1.1g FIPS 21 Apr 2020 [root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> LD_LIBRARY_PATH=./usr/lib64:${LD_LIBRARY_PATH} ./usr/bin/openssl11 version -d OPENSSLDIR: "/etc/pki/tls"

So, here's a working OpenSSL 1.1.1g FIPS version.

3. Python build

OpenSSL is fine from its PoV, but it needs some "small" adjustments in order for Python build to pick it up:

  • include and lib dirs must have the same parent (they reside lower in the dir tree, symlink them in the OpenSSL root dir)
  • Libraries with the 11 suffix (from earlier) must also be present, symlink them as well
[root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> # OpenSSL working. Prepare for Python build. [root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> # "include" and "lib" dirs must be at the same level - symlink them [root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> ln -s ./usr/include/openssl11 ./include [root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> ln -s ./usr/lib64/openssl11 ./lib [root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> ls . ./lib .: etc include lib usr ./lib: libcrypto.so libssl.so [root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> # "lib" dir must also contain $(LIBNAME).1.1 besides what's already in there ($(LIBNAME)) [root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> ln -s libcrypto.so ./lib/libcrypto.so.1.1 [root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> ln -s libssl.so ./lib/libssl.so.1.1 [root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> ls ./lib libcrypto.so libcrypto.so.1.1 libssl.so libssl.so.1.1

Python (configure and make):

root@cfati-5510-0:/work/q069539286]> tar -zxf Python-3.10.0.tgz [root@cfati-5510-0:/work/q069539286]> cd Python-3.10.0 [root@cfati-5510-0:/work/q069539286/Python-3.10.0]> ./configure --with-openssl=$(pwd)/../openssl-1.1.1g --with-openssl-rpath=auto --prefix=$(pwd)/../python-3.10>../cfg.txt 2>&1 [root@cfati-5510-0:/work/q069539286/Python-3.10.0]> echo $? 0 [root@cfati-5510-0:/work/q069539286/Python-3.10.0]> LD_LIBRARY_PATH=$(pwd)/../openssl-1.1.1g/lib:${LD_LIBRARY_PATH} make -j8>../mk.txt 2>&1 [root@cfati-5510-0:/work/q069539286/Python-3.10.0]> echo $? 0

4. Test

Build was successful.
Quick tests (without make install):

[root@cfati-5510-0:/work/q069539286/Python-3.10.0]> # Quick test [root@cfati-5510-0:/work/q069539286/Python-3.10.0]> # -------------------------------- [root@cfati-5510-0:/work/q069539286/Python-3.10.0]> ./python -c "import sys, ssl; print(\"{:s}\n{:s}\".format(sys.version, ssl.OPENSSL_VERSION))" 3.10.0 (default, Oct 22 2021, 01:12:00) [GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] OpenSSL 1.1.1g FIPS 21 Apr 2020 [root@cfati-5510-0:/work/q069539286/Python-3.10.0]> # -------------------------------- [root@cfati-5510-0:/work/q069539286/Python-3.10.0]> # As opposed to "regular" Python [root@cfati-5510-0:/work/q069539286/Python-3.10.0]> python -c "import sys, ssl; print(\"{:s}\n{:s}\".format(sys.version, ssl.OPENSSL_VERSION))" 2.7.5 (default, Nov 16 2020, 22:23:17) [GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] OpenSSL 1.0.2k-fips 26 Jan 2017 [root@cfati-5510-0:/work/q069539286/Python-3.10.0]> ./python -c "import ssl; print(ssl._ssl)" <module '_ssl' from '/work/q069539286/Python-3.10.0/build/lib.linux-x86_64-3.10/_ssl.cpython-310-x86_64-linux-gnu.so'> [root@cfati-5510-0:/work/q069539286/Python-3.10.0]> ldd /work/q069539286/Python-3.10.0/build/lib.linux-x86_64-3.10/_ssl.cpython-310-x86_64-linux-gnu.so linux-vdso.so.1 => (0x00007ffef6b7b000) libssl.so.1.1 => /work/q069539286/Python-3.10.0/../openssl-1.1.1g/lib/libssl.so.1.1 (0x00007f0ab9745000) libcrypto.so.1.1 => /work/q069539286/Python-3.10.0/../openssl-1.1.1g/lib/libcrypto.so.1.1 (0x00007f0ab9262000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f0ab9046000) libc.so.6 => /lib64/libc.so.6 (0x00007f0ab8c78000) libz.so.1 => /lib64/libz.so.1 (0x00007f0ab8a62000) libdl.so.2 => /lib64/libdl.so.2 (0x00007f0ab885e000) /lib64/ld-linux-x86-64.so.2 (0x00007f0ab9c05000) [root@cfati-5510-0:/work/q069539286/Python-3.10.0]> readelf -d /work/q069539286/Python-3.10.0/build/lib.linux-x86_64-3.10/_ssl.cpython-310-x86_64-linux-gnu.so | grep ssl 0x0000000000000001 (NEEDED) Shared library: [libssl.so.1.1] 0x000000000000001d (RUNPATH) Library runpath: [/work/q069539286/Python-3.10.0/../openssl-1.1.1g/lib]
Improve this answer

5 Comments

What did you get for readelf -d _ssl.cpython-310-x86_64-linux-gnu.so ?
@wim: unfortunately I don't have container and image anymore (on Docker Desktop (Win)). There's a small chance that I could still have it on my Linux system (dual-boot) next time I'll work on it will check. But anyway since it worked with no LD_LIBRARY_PATH, I'm expecting it to have DT_RUNPATH (DT_RPATH) flag set (also because of --with-openssl-rpath=auto).
I got the RUNPATH which surprised me because the configure flag suggests its putting RPATH. And RPATH is what I wanted, not RUNPATH, because I don't want env LD_LIBRARY_PATH to have any say here.
I studied some years ago this topic. In many cases I see them used interchangeably. Anyway, I think that option translates to linker's -rpath.
@wim: I found the container (on my other dual boot OS). It's indeed DT_RUNPATH (but DT_RPATH is deprecated). Added to the answer.
0

Python will use the bundled ssl (witch is outdated sometimes). In order to then use OpenSSL, add flag for OpenSSL when running ./configure. For detail, run ./configure --help for more options.

 --with-openssl=DIR root of the OpenSSL directory --with-openssl-rpath=[DIR|auto|no] Set runtime library directory (rpath) for OpenSSL libraries, no (default): don't set rpath, auto: auto-detect rpath from --with-openssl and pkg-config, DIR: set an explicit rpath --with-ssl-default-suites=[python|openssl|STRING] override default cipher suites string, python: use Python's preferred selection (default), openssl: leave OpenSSL's defaults untouched, STRING: use a custom string, python and STRING also set TLS 1.2 as minimum TLS version
Improve this answer

1 Comment

Python will not "use it own bundled ssl suite" (there's no such thing). It will rely on the default OpenSSL installation (if any).

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.