How to Test a Python Distribution

Introduction

Your code is good. Really good.

You enforced consistent style with black, isort, ruff, and pydocstyle.

You checked for a wide range of performance, correctness, and readability issues with flake8, mypy, pylint, and ruff.

You ran extensive tests with pytest and hypothesis.

You even scanned for legal and security issues with tools like pip-audit, pip-licenses, and safety.

So finally, FINALLY, it’s time to package it up into a tarball and upload it to PyPI.

python -m build .

... right?

Hopefully! But let’s check.

  • Are those distributions valid zip or tar files?

  • Are they small enough to fit on PyPI?

  • Are they as small as possible, to be kind to package repositories and users with weak internet connections?

  • Are they free from filepaths and file names that some operating systems will struggle with?

  • Do they have correctly-formatted platform tags?

  • Will their READMEs look pretty when rendered on the PyPI homepage?

You checked those things, right? And in continuous integration, with open-source tools, not with manual steps and random tar incantations copied from Stack Overflow?

If yes, great! I’m proud of you.

If you’re feeling like you could use some help with that… read on.

Quickstart

After building at least one wheel and sdist…

python -m build --outdir dist .

Run the following on those distributions to catch a wide range of packaging issues.

# are all the sdists valid gzipped tar files? gunzip -tv dist/*.tar.gz # are all the wheels valid zip files? zip -T dist/*.whl # is the package metadata well-formatted? pyroma --min=10 dist/*.tar.gz twine check --strict dist/* # (INFO-level) print some details to logs pkginfo --json dist/*.tar.gz pkginfo --json dist/*.whl # (DEBUG-level) print even more details to logs wheel2json dist/*.whl # is the distribution properly structured and portable? check-wheel-contents dist/*.whl pydistcheck --inspect dist/*

Some of those tools can also dump package data to machine-readable formats, that could then be passed through your own custom scripts.

# simple list of filepaths in the wheel unzip -l dist/*.whl > filepaths.txt # JSON representation of the wheel metadata pkginfo --json dist/*.whl > pkginfo.json # this has most of what pkginfo has, plus things like # md5 and sha256 checksums for every file in the distribution wheel2json dist/*.whl > wheel-inspect.json

Consider using pre-commit (https://pre-commit.com/), with at least the following configuration to catch portability-related issues before they even make it into distributions.

repos: - repo: https://github.com/pre-commit/pre-commit-hooks  rev: v6.0.0  hooks:  # large files checked into source control  - id: check-added-large-files  args: ['--maxkb=512']  # files whose names only differ by case  - id: check-case-conflict  # symlinks that don't point to anything  - id: check-symlinks  # symlinks changed to regular files with content of a path  - id: destroyed-symlinks  # ensure all files end in a newline  - id: end-of-file-fixer  # mixed line endings  - id: mixed-line-ending  # superfluous whitespace  - id: trailing-whitespace - repo: https://github.com/shellcheck-py/shellcheck-py  rev: v0.11.0.1  hooks:  - id: shellcheck

If you use GitHub Actions for continuous integration, consider adding a step with the hynek/build-and-inspect-python-package action (https://github.com/hynek/build-and-inspect-python-package) running prior to publishing packages.

List of Tools

The following open-source tools take packages as input, and can be used to detect (and in some cases repair) a wide range of Python packaging issues.

  • abi3audit (link) = detect ABI incompatibilities in wheels with CPython extensions

  • auditwheel (link) = detect and repair issues in Linux wheels that link to external shared libraries

  • auditwheel-emscripten (link) = like auditwheel, but focused on Python-in-a-web-browser applications (e.g. pyodide auditwheel)

  • auditwheel-symbols (link) = detect which symbols in a Linux wheel’s shared library are causing auditwheel to suggest a more recent manylinux tag

  • check-wheel-contents (link) = detect unnecessary files, import issues, portability problems in wheels

  • conda-verify (link) = detect portability and correctness problems in conda packages

  • cph (link) = work with conda packages (e.g. cph extract to decompress them, cph list to list their contents)

  • delocate (link) = detect and repair issues in macOS wheels that link to external shared libraries

  • delvewheel (link) = detect and repair issues in Windows wheels that link to external shared libraries

  • pkginfo (link) = print sdist and wheel metadata

  • pydistcheck (link) = detect portability problems in conda packages, wheels, and sdists

  • pyroma (link) = detect incomplete or malformed metadata in sdists

  • repairwheel (link) = repair issues in Linux, macOS, and Windows wheels (wraps auditwheel, delocate, and delvewheel)

  • twine (link) = detect issues in package metadata (via twine check)

  • wheel-inspect (link) = dump summary information about wheels into machine-readable formats

These take a source tree as input and find problems in the files uses to create packages.

  • check-manifest (link) = check that sdists contain all the files you expect them to, based on what you’ve checked into version control

  • pre-commit/pre-commit-hooks (link) = many checks, including the following related to portability:

    • check-added-large-files = check for too-large files in source control

    • check-case-conflict = check for files whose names only differ by case

    • check-symlinks = check for symlinks that don’t point to real files

    • destroyed-symlinks = check for symlinks replaced with the content of the regular file they previously pointed to

  • validate-pyproject (link) = check that pyproject.toml files are valid

These take compiled files like ELF (.so), Mach-O (.dylib), or PE (.dll, .exe) files as input and can be used to test and modify them.

  • bloaty (link) = analyze contents of ELF, Mach-O, PE, and other binary files. Answers “why is this file so big?”.

  • cubloaty (link) = like bloaty, but for CUDA binaries (ELF and cubin)

  • dsymutil (link) = dump DWARF debug symbols or link debug information into an executable

  • dumpbin (link) = describe COFF files (including PE files)

  • install_name_tool (link) = modify Mach-O files (shipped with macOS developer tools)

  • machomangler (link) = patch Mach-O and PE files

  • ldd (link) = display dependencies of shared objects

  • listdlls (link) = list all of the DLLs loaded into a process, useful for tracing DLL dependencies (similar to ldd, but for Windows)

  • llvm-install-name-tool (link) = install_name_tool re-implementation from the LLVM project

  • llvm-objdump (link) = objdump re-implementation from the LLVM project

  • llvm-otool (link) = otool re-implementation from the LLVM project

  • llvm-readelf (link) = readelf re-implementation from the LLVM project

  • objdump (link) = display information about object files, including ELF files

  • otool (link) = display information about Mach-O files (shipped with macOS developer tools)

  • patchelf (link) = modify the RPATH of ELF files

  • pecheck (link) = single-file Python module that can be used to print some information about PE files

  • peframe (link) = find malware in PE files

  • pyelftools (link) = Python library for inspecting and modifying ELF files, including a Python re-implementation of readelf

  • readelf (link) = display information about ELF files (multiple implementations exist)