oborchers/packaging-distribution

Ship Wheels for Every Platform Your Users Run

A package without pre-built wheels forces users to have a compiler toolchain, correct system headers, and patience. Pydantic-core ships 30+ platform-specific wheels so pip install pydantic takes seconds, not minutes. Pure Python packages get this for free with a single py3-none-any wheel. Compiled extensions require deliberate CI investment -- but cibuildwheel and maturin-action make it tractable.

Pure Python vs Compiled Extensions

Default to pure Python. Only add compiled extensions when profiling proves Python is the bottleneck and the speedup is 10x or greater.

| Situation | Choice | Examples | |-----------|--------|----------| | I/O-bound (network, disk) | Pure Python | FastAPI, httpx, Rich, Click | | CPU-bound tight loops, parsing, serialization | Compiled extension | Pydantic-core, orjson, Polars | | Wrapping existing C/C++/Rust library | Compiled extension | cryptography, lxml |

Extension Technology Decision

| Technology | Language | Build Backend | Used By | |-----------|----------|---------------|---------| | PyO3 + maturin | Rust | maturin | Pydantic-core, Polars, Ruff, uv, orjson | | Cython | Cython/C | setuptools | uvloop, lxml, scikit-learn | | pybind11 | C++ | scikit-build-core | SciPy (partial), Open3D | | scikit-build-core | C/C++/Fortran | scikit_build_core.build | CMake-dependent projects | | cffi | C | hatchling/setuptools | cryptography (backend) |

Rust + PyO3 + maturin is the dominant choice for new compiled extensions (2023-2025). Prefer it for greenfield performance-critical work.

Wheel Format and Platform Tags

A wheel filename encodes compatibility: {name}-{version}-{python}-{abi}-{platform}.whl

httpx-0.28.0-py3-none-any.whl                           # Pure Python
pydantic_core-2.27.0-cp312-cp312-manylinux_2_17_x86_64.whl  # CPython 3.12, Linux
cryptography-44.0.0-cp39-abi3-manylinux_2_28_x86_64.whl     # Stable ABI, cp39+

Minimum Wheel Matrix for Compiled Extensions

| Priority | Platform | Tag | Covers | |----------|----------|-----|--------| | Must | Linux x86_64 | manylinux_2_28_x86_64 | Most servers, CI, Docker | | Must | macOS ARM | macosx_11_0_arm64 | Modern Mac (M1-M4) | | Must | Windows x64 | win_amd64 | Windows users | | Should | macOS Intel | macosx_10_12_x86_64 | Older Macs | | Should | Linux ARM64 | manylinux_2_28_aarch64 | AWS Graviton, RPi 4+ | | Should | Linux musl x64 | musllinux_1_2_x86_64 | Alpine Docker images |

The abi3 Stable ABI

Build one wheel per platform instead of one per Python-version-per-platform. For a package supporting cp39-cp313 across 3 platforms, abi3 reduces 15 wheels to 3.

# Cargo.toml (PyO3)
[dependencies]
pyo3 = { version = "0.22", features = ["abi3-py39"] }

Used by: cryptography (cp37-abi3), bcrypt, PyYAML.

sdist vs Wheel Contents

| | sdist (source) | wheel (built) | |---|---|---| | Include | All source (.py, .rs, .c, .pyx), pyproject.toml, Cargo.toml, Cargo.lock, LICENSE, README.md | Installed packages, compiled .so/.pyd, py.typed, dist-info/ | | Exclude | .git/, CI configs, pre-built binaries, __pycache__/ | Tests, docs, build scripts, source files for extensions |

# hatchling
[tool.hatch.build.targets.wheel]
packages = ["src/my_library"]

[tool.hatch.build.targets.sdist]
include = ["src/", "tests/", "pyproject.toml", "README.md", "LICENSE"]

Never create a MANIFEST.in for new projects. Modern backends manage inclusion in pyproject.toml.

maturin: Rust + Python Packaging

Use the mixed Python/Rust layout for packages with both Python and Rust code.

[build-system]
requires = ["maturin>=1.7,<2.0"]
build-backend = "maturin"

[tool.maturin]
python-source = "python"
module-name = "my_library._core"
features = ["pyo3/extension-module"]
strip = true
include = ["Cargo.lock"]

Ship .pyi stub files for the Rust module to enable full type-checking support (as Pydantic-core does with core_schema.pyi).

Pydantic's Dual-Package Strategy

Split into two packages when the compiled core and Python API have different release cadences, contributor pools, and CI complexity. Keep them together (Polars, orjson) when they are tightly coupled.

| | pydantic | pydantic-core | |---|---|---| | Language | Pure Python | Rust (PyO3 + maturin) | | Build backend | hatchling | maturin | | Wheel type | py3-none-any | 30+ platform wheels | | CI time | 5-10 minutes | 30-60 minutes |

Pin the core dependency to an exact version: "pydantic-core==2.27.0".

Multi-Platform Builds with cibuildwheel

# .github/workflows/wheels.yml
jobs:
  build-wheels:
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: [ubuntu-latest, macos-14, windows-latest]
    steps:
      - uses: actions/checkout@v4
      - uses: pypa/[email protected]
        env:
          CIBW_BUILD: "cp310-* cp311-* cp312-* cp313-*"
          CIBW_SKIP: "*-win32 *-manylinux_i686"
          CIBW_MANYLINUX_X86_64_IMAGE: manylinux_2_28
          CIBW_MANYLINUX_AARCH64_IMAGE: manylinux_2_28
          CIBW_TEST_REQUIRES: pytest
          CIBW_TEST_COMMAND: pytest {project}/tests -x
          CIBW_ARCHS_MACOS: "x86_64 arm64"
      - uses: actions/upload-artifact@v4
        with:
          name: wheels-${{ matrix.os }}
          path: wheelhouse/*.whl

For Rust projects, use PyO3/maturin-action@v1 instead -- it handles cross-compilation natively and is faster than QEMU emulation for aarch64.

Package Size Optimization

| Technique | Impact | |-----------|--------| | Exclude tests/docs from wheel | High -- often 2-5x reduction | | Strip debug symbols (strip = true) | High -- 5-10x for compiled extensions | | Enable LTO (lto = "fat" in Cargo.toml) | Medium -- smaller and faster binaries | | Set codegen-units = 1 in release profile | Medium -- better optimization | | Store large data externally | High -- avoid bundling datasets |

Audit wheel contents: unzip -l my_package.whl | tail

Conditional Dependencies and Environment Markers

Use PEP 508 markers for platform-specific and version-specific dependencies:

dependencies = [
    "tomli>=2.0; python_version < '3.11'",
    "uvloop>=0.20; sys_platform != 'win32'",
    "typing-extensions>=4.12; python_version < '3.13'",
]

Use minimum version constraints for libraries, not exact pins: "requests>=2.28", not "requests==2.31.0".

PyPI Metadata

Provide complete, accurate metadata in [project]:

[project.urls]
Homepage = "https://github.com/you/my-library"
Documentation = "https://my-library.readthedocs.io"
Repository = "https://github.com/you/my-library"
Issues = "https://github.com/you/my-library/issues"
Changelog = "https://github.com/you/my-library/blob/main/CHANGELOG.md"

Only claim Python version classifiers you actually test in CI. Include "Typing :: Typed" if you ship py.typed. Preview README rendering with twine check dist/* before publishing.

TestPyPI Workflow

1. Push a pre-release tag (v1.0.0rc1)
2. CI builds wheels and sdist
3. CI publishes to TestPyPI
4. Verify the TestPyPI page and test installation
5. Create the release tag (v1.0.0)
6. CI publishes to production PyPI

Test installation with: pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ my-library

Review Checklist

When reviewing code for packaging and distribution:

• [ ] Pure Python packages produce a single py3-none-any wheel
• [ ] Compiled extensions ship wheels for Linux x86_64, macOS ARM, and Windows x64 at minimum
• [ ] sdist includes all source files needed to build from scratch
• [ ] Wheel excludes tests, docs, benchmarks, and build scripts
• [ ] Debug symbols are stripped from compiled extensions (strip = true)
• [ ] MANIFEST.in is not used (configure inclusion in pyproject.toml)
• [ ] Rust extensions use maturin with the mixed Python/Rust layout
• [ ] .pyi stubs exist for compiled extension modules
• [ ] Environment markers handle backport dependencies (tomli, exceptiongroup)
• [ ] Library dependencies use minimum version bounds, not exact pins
• [ ] PyPI classifiers match versions actually tested in CI
• [ ] [project.urls] includes Repository, Issues, and Changelog
• [ ] Built wheel is installed and smoke-tested in CI before publishing
• [ ] Both wheels and sdist are published to PyPI