PyPI

Mostly referring to this user guide to upload packages onto PyPI.

Create an account, and reserve a package namespace. Create an API token for uploading.

For the project, minimally prepare a pyproject.toml with the [build-system] and [project] tables populated.

Install the build package that provides functionality to prepare the source tarball ("source distribution") and build into a wheel. Then finally upload it using twine onto PyPI.

user:~$ python -m build
user:~$ python -m twine upload dist/*
user:~$ python -m twine upload --repository testpypi dist/*  # TestPyPI

Previous workflow involved manually building, and uploading to PyPI with twine using an API token. The continuous integration method does a couple things different:

• Build is triggered with every commit push, using a series of checkout + install Python & build + upload into build environment steps
• Publishing involves retrieving the build and publishing it directly to PyPI

Authentication is done on the PyPI via OIDC: account on PyPI specifies from which source should a distribution push be accepted, and PyPI authenticates directly with the source itself (that acts as an identity provider).

PyPI screenshot when configuring OIDC

Additional security measures taken include:

• Separating the build task from the publish task, to avoid potential compromise of authentication tokens
• Github environments are used to provide additional trigger requirements (e.g. extra reviewers, push only from protected branches).

To test installation from TestPyPI, add the extra repository url. As usual, be cautious when doing this in production, since supply chain attacks can occur when a private package name is suddenly resolvable to a package of the same name on the public repository.

pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple [PACKAGE]

  publish-pypi:
    name: Push wheels to PyPI
    needs: build-wheels
    runs-on: ubuntu-latest
    environment:
      name: release
      url: https://test.pypi.org/p/s15lib
    permissions:
      id-token: write
 
    steps:
      - uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131  # v7.0.0
        with:
          pattern: cibw-*
          path: dist
          merge-multiple: true
 
      - uses: pypa/gh-action-pypi-publish@ed0c53931b1dc9bd32cbe73a98c7f6766f8a527e  # v1.13.0
        with:
          repository-url: https://test.pypi.org/legacy/
          verbose: true

Different possible build backends:

• Poetry: fpfind
• Hatchling: kochen, inst-efficiency
• setuptools: pyS15

Some articles on choosing a backend:

• Distribution of build systems:

Building PyPI-compatible wheels with C extensions require building with manylinux (see PEP600), so that the wheels will have compatible ABI. This typically involves spinning up an old OS with a sufficiently dated glibc for building. There is the cibuildwheel package that scripts this, with Docker as a dependency to pull the relevant image bases.

Some references:

• PyPI supported platform tags: https://packaging.python.org/en/latest/specifications/platform-compatibility-tags/#python-tag
• cibuildwheel configuration: https://cibuildwheel.pypa.io/en/stable/configuration/
• Recent list of glibc versions to OS: https://gist.github.com/richardlau/6a01d7829cc33ddab35269dacc127680
• glibc version release history: https://sourceware.org/glibc/wiki/Glibc%20Timeline
• numpy implementation: https://github.com/numpy/numpy/blob/main/.github/workflows/wheels.yml
• Check onboard glibc version: ldd --version

user:~$ uv pip install cibuildwheel
user:~$ python -m cibuildwheel --output-dir wheelhouse  # sudo for docker

Custom platform build can be specified in pyproject.toml:

[tool.cibuildwheel]
build = "cp38-manylinux_x86_64"