Access Python package index JSON APIs with requests

PyPI's JSON API is a machine-readable source of the same kind of data you can access while browsing the website.
90 readers like this.
Python programming language logo with question marks

Opensource.com

PyPI, the Python package index, provides a JSON API for information about its packages. This is essentially a machine-readable source of the same kind of data you can access while browsing the website. For example, as a human, I can head to the NumPy project page in my browser, click around, and see which versions there are, what files are available, and things like release dates and which Python versions are supported:

But if I want to write a program to access this data, I can use the JSON API instead of having to scrape and parse the HTML on these pages.

As an aside: On the old PyPI website, when it was hosted at pypi.python.org, the NumPy project page was at pypi.python.org/pypi/numpy, and accessing the JSON was a simple matter of adding a /json on the end, hence https://pypi.org/pypi/numpy/json. Now the PyPI website is hosted at pypi.org, and NumPy's project page is at pypi.org/project/numpy. The new site doesn't include rendering the JSON, but it still runs as it was before. So now, rather than adding /json to the URL, you have to remember the URL where they are.

You can open up the JSON for NumPy in your browser by heading to its URL. Firefox renders it nicely like this:

You can open info, releases, and urls to inspect the contents within. Or you can load it into a Python shell. Here are a few lines to get started:

import requests
url = "https://pypi.org/pypi/numpy/json"
r = requests.get(url)
data = r.json()

Once you have the data (calling .json() provides a dictionary of the data), you can inspect it:

Open releases, and inspect the keys inside it:

This shows that releases is a dictionary with version numbers as keys. Pick one (say, the latest one) and inspect that:

Each release is a list, and this one contains 24 items. But what is each item? Since it's a list, you can index the first one and take a look:

This item is a dictionary containing details about a particular file. So each of the 24 items in the list relates to a file associated with this particular version number, i.e., the 24 files listed at https://pypi.org/project/numpy/1.20.1/#files.

You could write a script that looks for something within the available data. For example, the following loop looks for versions with sdist (source distribution) files that specify a requires_python attribute and prints them:

for version, files in data['releases'].items():
    for f in files:
        if f.get('packagetype') == 'sdist' and f.get('requires_python'):
            print(version, f['requires_python'])

piwheels

Last year I implemented a similar API on the piwheels website. piwheels.org is a Python package index that provides wheels (precompiled binary packages) for the Raspberry Pi architecture. It's essentially a mirror of the package set on PyPI, but with Arm wheels instead of files uploaded to PyPI by package maintainers.

Since piwheels mimics the URL structure of PyPI, you can change the pypi.org part of a project page's URL to piwheels.org. It'll show you a similar kind of project page with details about which versions we have built and which files are available. Since I liked how the old site allowed you to add /json to the end of the URL, I made ours work that way, so NumPy's project page on PyPI is pypi.org/project/numpy. On piwheels, it is piwheels.org/project/numpy, and the JSON is at piwheels.org/project/numpy/json.

There's no need to duplicate the contents of PyPI's API, so we provide information about what's available on piwheels and include a list of all known releases, some basic information, and a list of files we have:

Similar to the previous PyPI example, you could create a script to analyze the API contents, for example, to show the number of files piwheels has for each version of NumPy:

import requests

url = "https://www.piwheels.org/project/numpy/json"
package = requests.get(url).json()

for version, info in package['releases'].items():
    if info['files']:
        print('{}: {} files'.format(version, len(info['files'])))
    else:
        print('{}: No files'.format(version))

Also, each file contains some metadata:

One handy thing is the apt_dependencies field, which lists the Apt packages needed to use the library. In the case of this NumPy file, as well as installing NumPy with pip, you'll also need to install libatlas3-base and libgfortran using Debian's Apt package manager.

Here is an example script that shows the Apt dependencies for a package:

import requests

def get_install(package, abi):
    url = 'https://piwheels.org/project/{}/json'.format(package)
    r = requests.get(url)
    data = r.json()
    for version, release in sorted(data['releases'].items(), reverse=True):
        for filename, file in release['files'].items():
            if abi in filename:
                deps = ' '.join(file['apt_dependencies'])
                print("sudo apt install {}".format(deps))
                print("sudo pip3 install {}=={}".format(package, version))
                return

get_install('opencv-python', 'cp37m')
get_install('opencv-python', 'cp35m')
get_install('opencv-python-headless', 'cp37m')
get_install('opencv-python-headless', 'cp35m')

We also provide a general API endpoint for the list of packages, which includes download stats for each package:

import requests

url = "https://www.piwheels.org/packages.json"
packages = requests.get(url).json()
packages = {
    pkg: (d_month, d_all)
    for pkg, d_month, d_all, *_ in packages
}

package = 'numpy'
d_month, d_all = packages[package]

print(package, "has had", d_month, "downloads in the last month")
print(package, "has had", d_all, "downloads in total")

Since pip search is currently disabled due to its XMLRPC interface being overloaded, people have been looking for alternatives. You can use the piwheels JSON API to search for package names instead since the set of packages is the same:

#!/usr/bin/python3
import sys

import requests

PIWHEELS_URL = 'https://www.piwheels.org/packages.json'

r = requests.get(PIWHEELS_URL)
packages = {p[0] for p in r.json()}

def search(term):
    for pkg in packages:
        if term in pkg:
            yield pkg

if __name__ == '__main__':
    if len(sys.argv) == 2:
        results = search(sys.argv[1].lower())
        for res in results:
            print(res)
    else:
        print("Usage: pip_search TERM")

For more information, see the piwheels JSON API documentation.


This article originally appeared on Ben Nuttall's Tooling Tuesday blog and is reused with permission.

What to read next
Tags
User profile image.
Ben is a software engineer for BBC News Labs, and formerly Raspberry Pi's Community Manager. He's into Linux, Python and all things open source! Follow Ben on Twitter @ben_nuttall.

1 Comment

Well, this is supremely useful. Thank you!

Creative Commons LicenseThis work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License.