Building manylinux Python wheels

There are many operating systems built on the GNU/Linux code base; these are known as "distributions." Users fondly (or sometimes not so fondly) talk about "distro wars": loud arguments, usually on the internet, about which distribution is best.

Each distro's community makes its own choices about which library versions to include and how long to support them. This is a challenge for someone who just wants to distribute a Python library that contains native code. It would be a lot of work to build separate binaries for Red Hat, SUSE, Ubuntu, and Debian—and even more work building separate binaries for each supported version!

Fortunately, there is a way to make a binary compatible with most (though not all) Linux distros. It relies on the fact that most distributions (including all of those named above) use the GNU C library. The GNU C library uses a special method of binary compatibility by internally keeping all versions of a symbol inside the ELF libc.so dynamic library.

Python's manylinux approach takes advantage of this by intentionally building the binary redistributable package, aka the wheel, on an old version of a distribution. In order to achieve maximum compatibility, it uses the longest-supported freely distributable version of Linux: CentOS.

Currently, manylinux builds on CentOS 5, which means it supports every non-end-of-life version of the major distributions. It does so by using a specialized Docker image, the manylinux1 image. Separate binaries are still needed for minor versions of Python: for most libraries, this means 3.5, 3.6, and 3.7. In theory, we also need separate binaries for different CPU word sizes, but in reality, having 64-bit-only support is probably fine. Again, in theory, different binaries are needed for different ways of building Unicode support; but in practice, major Linux distributions always build Python in "wide Unicode."

The Docker image is available at quay.io/pypa/manylinux1_x86_64. It has versions of Python in /opt/python. The versions with wide Unicode are those whose names end with mu: for example, /opt/python/cp36-36mu. Those versions of Python already have pip installed with wheel building support. This means if code is mounted or copied into the Docker container at /src,

mkdir /output
/opt/python/cp36-36mu/pip wheel /src -w output

will produce a binary wheel in /output. However, this will still not be a manylinux wheel, since it is possible to build wheels that accidentally depend on other libraries.

The auditwheel tool will take that wheel, audit it, and copy it to a manylinux name:

auditwheel repair /output/mylibrary*whl -w /output

The repair mode will also bundle any other libraries into the wheel to make sure it is portable to any needed distribution. Now it is safe to copy /output/mylibrary*manylinux*whl out of the Docker container and upload it to PyPI. Note that PyPI will reject distribution-specific binary wheels but will happily allow uploads of manylinux wheels. The same is probably true if you are using a private index, although you might want to check with your administrator.