1

I am trying to build Python3.12 from source as a non-root user on Linux. I am aware that it need openssl (for pip) and the libffi headers, which I have attempted to build from source as well. Doing this I end up with the following errors in the make log:

[ERROR] _ctypes failed to import: libffi.so.8: cannot open shared object file: No such file or directory [ERROR] _hashlib failed to import: libcrypto.so.3: cannot open shared object file: No such file or directory [ERROR] _ssl failed to import: libssl.so.3: cannot open shared object file: No such file or directory The following modules are *disabled* in configure script: _sqlite3 The necessary bits to build these optional modules were not found: _curses _curses_panel _dbm _gdbm _lzma _tkinter nis readline To find the necessary bits, look in configure.ac and config.log. Following modules built successfully but were removed because they could not be imported: _ctypes _hashlib _ssl Could not build the ssl module! Python requires a OpenSSL 1.1.1 or newer Checked 111 modules (31 built-in, 67 shared, 1 n/a on linux-x86_64, 1 disabled, 8 missing, 3 failed on import)

I would like to be able to (among other) use the ctype module, so this is sub-optimal.

The steps I took

Step 1: build libffi from source. I followed these instructions and included --prefix=$HOME/.local (since that is where I would like to install everything locally) when configuring the make file. (./configure --prefix=$HOME/.local).

Step 2: build openssl from source. I (more or less) followed these instructions. I again included --prefix=$HOME/.local during configuration.

By now, my .local folder looks something like this:

├── bin │   ├── c_rehash │   └── openssl ├── include │   ├── ffi.h │   ├── ffitarget.h │   └── openssl │   └── <loads of header files>.h ├── lib │   ├── libcrypto.so │   ├── libffi.so │   ├── libssl.so │   └── pkgconfig │   └── libffi.pc ├── lib64 │   ├── cmake │   │   └── OpenSSL │   │   ├── OpenSSLConfig.cmake │   │   └── OpenSSLConfigVersion.cmake │   ├── engines-3 │   │   ├── afalg.so │   │   ├── capi.so │   │   ├── loader_attic.so │   │   └── padlock.so │   ├── libcrypto.a │   ├── libcrypto.so │   ├── libcrypto.so.3 │   ├── libffi.a │   ├── libffi.la │   ├── libffi.so │   ├── libffi.so.8 │   ├── libffi.so.8.1.3 │   ├── libssl.a │   ├── libssl.so │   ├── libssl.so.3 │   ├── ossl-modules │   │   └── legacy.so │   └── pkgconfig │   ├── libcrypto.pc │   ├── libssl.pc │   └── openssl.pc ├── share │   ├── ... ...

Step 3: build Python3.12 from source. Here I also included the take-aways from this post and tried to include all the correct flag etc. My suspicion is that it goes wrong here.

To configure:

./configure --enable-optimizations --with-lto --with-openssl=$HOME/.local \ --prefix=$HOME/.local/ \ LIBFFI_INCLUDEDIR=$HOME/.local/include \ LDFLAGS="-L$HOME/.local/lib/../lib64 -L$HOME/.local/lib64" \ CFLAGS="-I$HOME/.local/include"

This resulted (after lots of tries) in some positive outputs for the checks to libiff and openssl files.

To build (for some reason repeating the include dir, see earlier link):

make LIBFFI_INCLUDEDIR=$HOME/.local/include

This executing of make then results in the errors show at the start of the post.

(Note: New here, not a Linux-wizard)

Improve this question
4
  • Have you considered using a package manager that runs builds from source for you? Nix, for example. (Granted, root access is needed to install the Nix daemon, but not to use it past that point; it's safe for unprivileged users to ask Nix to build things on their behalf, because those builds are run in sandboxed accounts and their output lives in hash-addressed locations such that only someone having the instructions leading to that particular build will reproduce the hash where the output is stored) Commented Apr 23, 2024 at 15:22
  • @CharlesDuffy, thanks. But I am working with a machine that I do not have admin-powers for. Commented Apr 23, 2024 at 15:25
  • See man ld.so, you need to tell the dynamic loader where to find the .so file, so set LD_LIBRARY_PATH. Commented Apr 23, 2024 at 16:47
  • @UlrichEckhardt, thank you very much. This was indeed a step towards the solution. (Retracing my steps I, found that when installing libffi, the output already tells you to do this, oops) Commented Apr 24, 2024 at 7:27

1 Answer 1

Reset to default
1

Dynamic libraries can be tricky, because there are so many different steps that all need to find them. You are on the right track with the -L flag, which tells the compile-time linker where to find libffi.so, but when the Python interpreter tries to load _ctypes, the run-time linker fails to find it:

[ERROR] _ctypes failed to import: libffi.so.8: cannot open shared object file: No such file or directory

The _ctypes module is actually implemented as a .so library itself, which is an ELF (Executable and Linkable Format) file. There is a section of an ELF file that stores dynamic linking information. I was able to reproduce your _ctypes issue, and I used readelf -d to show the contents of this file; I suspect yours looks similar:

$ readelf -d ./Modules/_ctypes_failed.cpython-312-x86_64-linux-gnu.so Dynamic section at offset 0x3aca0 contains 28 entries: Tag Type Name/Value 0x0000000000000001 (NEEDED) Shared library: [libpthread.so.0] 0x0000000000000001 (NEEDED) Shared library: [libc.so.6] 0x0000000000000001 (NEEDED) Shared library: [ld-linux-x86-64.so.2] 0x0000000000000001 (NEEDED) Shared library: [libffi.so.8] 0x0000000000000001 (NEEDED) Shared library: [libdl.so.2] 0x000000000000000c (INIT) 0xe000 0x000000000000000d (FINI) 0x32b4c 0x0000000000000019 (INIT_ARRAY) 0x3bc80 0x000000000000001b (INIT_ARRAYSZ) 16 (bytes) 0x000000000000001a (FINI_ARRAY) 0x3bc90 0x000000000000001c (FINI_ARRAYSZ) 16 (bytes) 0x000000006ffffef5 (GNU_HASH) 0x328 0x0000000000000005 (STRTAB) 0x1948 0x0000000000000006 (SYMTAB) 0x370 0x000000000000000a (STRSZ) 3975 (bytes) 0x000000000000000b (SYMENT) 24 (bytes) 0x0000000000000003 (PLTGOT) 0x3c000 0x0000000000000002 (PLTRELSZ) 4488 (bytes) 0x0000000000000014 (PLTREL) RELA 0x0000000000000017 (JMPREL) 0xc1d0 0x0000000000000007 (RELA) 0x2b88 0x0000000000000008 (RELASZ) 38472 (bytes) 0x0000000000000009 (RELAENT) 24 (bytes) 0x000000006ffffffe (VERNEED) 0x2aa8 0x000000006fffffff (VERNEEDNUM) 5 0x000000006ffffff0 (VERSYM) 0x28d0 0x000000006ffffff9 (RELACOUNT) 1558 0x0000000000000000 (NULL) 0x0

You can see from the NEEDED entries that this library depends on several other libraries, including libffi.so.8. However, since that file's not in the standard location, the run-time linker does not find it. One way to solve this is to use the LD_LIBRARY_PATH environment variable (as Ulrich Eckhardt suggested in the comments). I prefer a different approach, however, which doesn't require you to reconfigure your environment.

The dynamic section of an ELF file supports a RUNPATH entry type, which allows you to hard-code additional search paths for the NEEDED libraries. You can add a RUNPATH entry with the -rpath linker flag (wrapped in a -Wl flag, to tell gcc to pass the argument to the real linker). Add it to the LDFLAGS for your configure command like this:

LDFLAGS="-L$HOME/.local/lib/../lib64 -L$HOME/.local/lib64 -Wl,-rpath,$HOME/.local/lib64"

I tried this myself (on my machine it was /lib and not /lib64) and it fixed the _ctypes error. Here you can see why:

$ readelf -d ./Modules/_ctypes.cpython-312-x86_64-linux-gnu.so Dynamic section at offset 0x21cd8 contains 28 entries: Tag Type Name/Value 0x0000000000000001 (NEEDED) Shared library: [libffi.so.8] 0x0000000000000001 (NEEDED) Shared library: [libdl.so.2] 0x0000000000000001 (NEEDED) Shared library: [libpthread.so.0] 0x0000000000000001 (NEEDED) Shared library: [libc.so.6] 0x000000000000001d (RUNPATH) Library runpath: [/home/chuck/.local/lib] 0x000000000000000c (INIT) 0x6000 0x000000000000000d (FINI) 0x1b508 0x0000000000000019 (INIT_ARRAY) 0x22cc8 0x000000000000001b (INIT_ARRAYSZ) 8 (bytes) 0x000000000000001a (FINI_ARRAY) 0x22cd0 0x000000000000001c (FINI_ARRAYSZ) 8 (bytes) 0x000000006ffffef5 (GNU_HASH) 0x2f0 0x0000000000000005 (STRTAB) 0x15f0 0x0000000000000006 (SYMTAB) 0x318 0x000000000000000a (STRSZ) 3663 (bytes) 0x000000000000000b (SYMENT) 24 (bytes) 0x0000000000000003 (PLTGOT) 0x23000 0x0000000000000002 (PLTRELSZ) 3864 (bytes) 0x0000000000000014 (PLTREL) RELA 0x0000000000000017 (JMPREL) 0x4ea0 0x0000000000000007 (RELA) 0x2698 0x0000000000000008 (RELASZ) 10248 (bytes) 0x0000000000000009 (RELAENT) 24 (bytes) 0x000000006ffffffe (VERNEED) 0x25d8 0x000000006fffffff (VERNEEDNUM) 4 0x000000006ffffff0 (VERSYM) 0x2440 0x000000006ffffff9 (RELACOUNT) 389 0x0000000000000000 (NULL) 0x0

On a side note, when I followed your steps, I also saw errors with OpenSSL 3, so I installed OpenSSL 1.1.1w instead, and that fixed the _hashlib and _ssl failures.

Improve this answer
Sign up to request clarification or add additional context in comments.

7 Comments

Thanks. About your last line: I also encountered those errors, but was (indeed) able to fix it by using the already represent version of OpenSSL on the machine that I was using.
The way of working you suggest seems to work to resolve the error, however, it will not satisfy the requirements for building the _ctypes module. (At least not for me.) Let me know if it does build _ctypes at your end, then I'll try a bit hard.
with the -Wl,-rpath,$HOME/.local/lib flag it should build _ctypes correctly
Right. I think my problem was that libffi builds to $HOME/.local/lib/../lib64 on my machine. I have now configured and build with ./configure --enable-optimizations --with-lto --prefix="$HOME/.local" LIBFFI_INCLUDEDIR="$HOME/.local/include" LDFLAGS="-L$HOME/.local/lib/../lib64 -L$HOME/.local/lib64 -Wl,-rpath,$HOME/.local/lib/../lib64" CFLAGS="-I$HOME/.local/include" and make (without additions) which seems to also build _ctypes without a problem. If everything works now, should I post an answer myself with all the steps I took? (I don't know the stack overflow etiquettes well)
Also: I am writing some documentation about this. Is there a way to include both the lib and lib64 path? for example: LDFLAGS="-L$HOME/.local/lib/../lib64 -L$HOME/.local/lib64 -Wl,-rpath,$HOME/.local/lib/../lib64 -Wl,-rpath,$HOME/.local/lib" or does the second rpath overwrite the first here?
|

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.