How to set up DevPI, a PyPI-compatible Python development server

The first time I used DevPI, I was getting ready for a camping trip with my wife and kids. By "getting ready" I do not mean practicing my s'mores-making skills. I mean that I knew my kids would be entertained by camp staff some of the time, and I planned to fix a few bugs in the Twisted package. I also knew I would not have internet on the campgrounds, so I needed to be able to develop without connecting to the internet.

A naive person would prepare virtual environments; however, virtual environments should be disposable, not precious. Many tools will discard and recreate virtual environments whenever the configuration changes. I needed to know that all my packages would be available. That was my introduction to DevPI.

DevPI is a PyPI-compatible server you can run locally. It will not, and does not try, to scale to PyPI-like levels. In return, running it locally is simple and no frills.

DevPi is made up of three parts. The most important one is devpi-server. For many uses, this is the only part that needs to run. The server serves, first and foremost, as a caching proxy to PyPI. It takes advantage of the fact that packages on PyPI are immutable: once you have a package, it can never change.

There is also a web server, which allows you to search in the local package directory. Because a lot of uses do not even involve searching on the PyPI website, this is optional. Finally, there is a client command-line tool that allows you to configure various parameters on the running instance. The client is most useful in more esoteric use cases.

Installing and running DevPI is straightforward. In a virtual environment, simply run:

(devpi)$ pip install devpi-server
(devpi)$ devpi-server --start --init

The pip tool, by default, goes to pypi.org. For some basic testing of DevPI, you can create a new virtual environment or playground and run:

(playground)$ pip install \
-i http://localhost:3141/root/pypi/+simple/ \
httpie glom
(playground)$ http --body https://httpbin.org/get | glom ’{"url":"url"}’
{
"url": "https://httpbin.org/get"
}

Naturally, having to specify the -i … argument to pip every time would be annoying. After checking that everything works correctly, you can put the configuration in an environment variable:

$ export PIP_INDEX_URL=http://localhost:3141/root/pypi/+simple/

Or, to make things more permanent:

$ mkdir -p ~/.pip && cat > ~/.pip/pip.conf << EOF
[global]
index-url = http://localhost:3141/root/pypi/+simple/
[search]
index = http://localhost:3141/root/pypi/

The above file location works for Unix operating systems. On MacOS, the configuration file is $HOME/Library/Application Support/pip/pip.conf. On Windows, the configuration file is %APPDATA%\pip\pip.ini.

To "warm up" the DevPI cache (i.e., make sure it contains all needed packages), use pip to install them. The way I chose to do it, after configuring DevPI and pip, was to git clone the Twisted repository and run tox. Since tox goes through test environments, including the ones with a lot of packages, it would download all the needed packages.

A good practice also is to pre-install in a disposable virtual environment any requirements.txt files you have; however, DevPI's usefulness is not limited to disconnected operations. If you configure one inside your build cluster and point the build cluster at it, you completely avoid the risk of a "leftpad incident," where a package you rely on is removed from PyPI by the author. It might also make builds faster and will definitely cut out a lot of outgoing traffic.

Another use for DevPI is to test uploads before uploading them to PyPI. Assuming devpi-server is already running on the default port, you can run:

(devpi)$ pip install devpi-client twine
(devpi)$ devpi use http://localhost:3141
(devpi)$ devpi user -c testuser password=123
(devpi)$ devpi login testuser --password=123
(devpi)$ devpi index -c dev bases=root/pypi
(devpi)$ devpi use testuser/dev
(devpi)$ twine upload --repository http://localhost:3141/testuser/dev \
-u testuser -p 123 my-package-18.6.0.tar.gz
(devpi)$ pip install -i http://localhost:3141/testuser/dev my-package

Note that this allows uploading to an index that's used only explicitly, so you are not shadowing my-package for all environments that are not using it explicitly.

For an even more advanced use case, you can do:

(devpi)$ devpi index root/pypi mirror_url=https://ourdevpi.local

This will make the DevPI server a mirror of a local, "upstream" DevPI server. This allows uploading private packages to the "central" DevPI server to share them with your team. In those cases, the upstream DevPI server will often need to be run behind a proxy, and you need some tools to properly manage user access. Those details, however, are beyond the scope of this article.