Extensions

Code that run at near-native level (think C) can be compiled for Python such that it can be imported transparently as an extension module.

The basic approach to creating such an extension follows these steps:

1. Writing the extension:
   • The code typically has the extension .pyx, with Python-like syntax and C semantics.
   • This will undergo a first pass by Cython using cythonize <FILE> to compile into C code, together with the appropriate Python bindings.
   • This step is usually performed during package build, or out-of-band by maintainers and committed into the codebase.
2. Compiling the extension:
   • The resulting C code will be compiled and linked to generate a shared library, typically with the C compiler onboard.
   • The compilation command can be dynamically called by setuptools during package build, or manually by users.
   • This step can be optionally bypassed using pyximport, but incurs a compilation step during runtime.

A basic setup might look like the following, given the file myextmodule.pyx and with additional optional Numpy headers:

setup.py

import setuptools
 
package = setuptools.Extension(
    name="myextmodule",
    sources=["myextmodule.c"],
)
setuptools.setup(ext_modules=[package])

Running the following will plop the file right beside the original file. Note the last copy operation because the build was not performed in-place.

user:~$ cythonize -3a myextmodule.pyx
user:~$ python setup.py build_ext
user:~$ cp build/lib.linux-x86_64-cpython-313/myextmodule* myextmodule.so

We note the following observations before porting this into the build step of package installation:

• setup.py has to be run to build the module, and the setuptools.setup function magically reads build_ext from the command line to trigger the extension build.
  • This is distinct from the build command that builds the package instead.
• setuptools mostly ported from distutils, but there is strong back-compatibility (with setuptools being replaced in-place).
• The build and build_ext processes can be overridden (see source code here), noting that these are actually classes being passed into setuptools.setup:

import setuptools
from setuptools.command.build import build
from setuptools.command.build_ext import build_ext
 
class _build_ext(build_ext):
 
    def initialize_options(self):
        super().initialize_options()
 
    def run(self):
        super().run()
 
    def build_extension(self, ext):
        super().build_extension(ext)
 
setuptools.setup(
    cmdclass={"build": build, "build_ext": _build_ext},
)

• The available setuptools.Extension() options can be found in the documentation. For setuptools.setup(), this is probably sitting somewhere in the source code.
• Building a library in a subdirectory therefore extends to something like:

# Trigger with 'python -m build' or 'python build.py'
import numpy as np
import setuptools
 
ext_modules=[
    setuptools.Extension(
        name="src.physicsutils.apps.clocksync.libcostream",
        sources=["src/physicsutils/apps/clocksync/libcostream.c"],
        include_dirs=[np.get_include()],
        define_macros=[("NPY_NO_DEPRECATED_API", "NPY_1_7_API_VERSION")],
    )
]
 
setuptools.setup(
    script_args=["build_ext"],
    options={"build_ext": {"inplace": True}},
    ext_modules=ext_modules,
)

These include other compilers (pybind11 for C++, rustc for Rust), other build integrations (scikit-build/setuptools for C/C++, for Maturing/setuptools-rust for Rust).