Pipenv, the "Python Development Workflow for Humans" created by Kenneth Reitz a little more than a year ago, has become the official Python-recommended resource for managing package dependencies. But there is still confusion about what problems it solves and how it's more useful than the standard workflow using
pip and a
requirements.txt file. In this month's Python column, we'll fill in the gaps.
A brief history of Python package installation
To understand the problems that Pipenv solves, it's useful to show how Python package management has evolved.
Take yourself back to the first Python iteration. We had Python, but there was no clean way to install packages.
Then came Easy Install, a package that installs other Python packages with relative ease. But it came with a catch: it wasn't easy to uninstall packages that were no longer needed.
Enter pip, which most Python users are familiar with.
pip lets us install and uninstall packages. We could specify versions, run
pip freeze > requirements.txt to output a list of installed packages to a text file, and use that same text file to install everything an app needed with
pip install -r requirements.txt.
pip didn't include a way to isolate packages from each other. We might work on apps that use different versions of the same libraries, so we needed a way to enable that. Along came virtual environments, which enabled us to create small, isolated environments for each app we worked on. We've seen many tools for managing virtual environments: virtualenv, venv, virtualenvwrapper, pyenv, pyenv-virtualenv, pyenv-virtualenvwrapper, and even more. They all play well with
The new kid: Pipenv
Pipenv aims to solve several problems.
First, the problem of needing the
pip library for package installation, plus a library for creating a virtual environment, plus a library for managing virtual environments, plus all the commands associated with those libraries. That's a lot to manage. Pipenv ships with package management and virtual environment support, so you can use one tool to install, uninstall, track, and document your dependencies and to create, use, and organize your virtual environments. When you start a project with it, Pipenv will automatically create a virtual environment for that project if you aren't already using one.
Pipenv accomplishes this dependency management by abandoning the
requirements.txt norm and trading it for a new document called a Pipfile. When you install a library with Pipenv, a
Pipfile for your project is automatically updated with the details of that installation, including version information and possibly the Git repository location, file path, and other information.
Second, Pipenv wants to make it easier to manage complex interdependencies. Your app might depend on a specific version of a library, and that library might depend on a specific version of another library, and it's just dependencies and turtles all the way down. When two libraries your app uses have conflicting dependencies, your life can become hard. Pipenv wants to ease that pain by keeping track of a tree of your app's interdependencies in a file called
Pipfile.lock also verifies that the right versions of dependencies are used in production.
Also, Pipenv is handy when multiple developers are working on a project. With a
pip workflow, Casey might install a library and spend two days implementing a new feature using that library. When Casey commits the changes, they might forget to run
pip freeze to update the requirements file. The next day, Jamie pulls down Casey's changes, and suddenly tests are failing. It takes time to realize that the problem is libraries missing from the requirements file that Jamie doesn't have installed in the virtual environment.
Because Pipenv auto-documents dependencies as you install them, if Jamie and Casey had been using Pipenv, the
Pipfile would have been automatically updated and included in Casey's commit. Jamie and Casey would have saved time and shipped their product faster.
Finally, using Pipenv signals to other people who work on your project that it ships with a standardized way to install project dependencies and development and testing requirements. Using a workflow with
pip and requirements files means that you may have one single
requirements.txt file, or several requirements files for different environments. It might not be clear to your colleagues whether they should run
local.txt when they're running the project on their laptops, for example. It can also create confusion when two similar requirements files get wildly out of sync with each other: Is
local.txt out of date, or is it really supposed to be that different from
dev.txt? Multiple requirements files require more context and documentation to enable others to install the dependencies properly and as expected. This workflow has the potential to confuse colleagues and increase your maintenance burden.
Using Pipenv, which gives you
Pipfile, lets you avoid these problems by managing dependencies for different environments for you. This command will install the main project dependencies:
--dev tag will install the dev/testing requirements:
pipenv install --dev
There are other benefits to using Pipenv: It has better security features, graphs your dependencies in an easier-to-understand format, seamlessly handles
.env files, and can automatically handle differing dependencies for development versus production environments in one file. You can read more in the documentation.
Pipenv in action
The basics of using Pipenv are detailed in the Managing Application Dependencies section of the official Python packaging tutorial. To install Pipenv, use
pip install pipenv
To install packages to use in your project, change into the directory for your project. Then to install a package (we'll use Django as an example), run:
pipenv install django
You will see some output that indicates that Pipenv is creating a
Pipfile for your project.
If you aren't already using a virtual environment, you will also see some output from Pipenv saying it is creating a virtual environment for you.
Then, you will see the output you are used to seeing when you install packages.
To generate a
Pipfile.lock file, run:
You can also run Python scripts with Pipenv. To run a top-level Python script called
pipenv run python hello.py
And you will see your expected result in the console.
To start a shell, run:
If you would like to convert a project that currently uses a
requirements.txt file to use Pipenv, install Pipenv and run:
pipenv install requirements.txt
This will create a Pipfile and install the specified requirements. Consider your project upgraded!
Check out the Pipenv documentation, particularly Basic Usage of Pipenv, to take you further. Pipenv creator Kenneth Reitz gave a talk on Pipenv, "The Future of Python Dependency Management," at a recent PyTennessee event. The talk wasn't recorded, but his slides are helpful in understanding what Pipenv does and the problems it solves.