Why we use tests on OpenStack package builds in RDO

Making sure individual units of code work as expected is critical to reducing errors and unexpected behavior.

Image by:

Opensource.com

Unit tests are used to verify that individual units of source code work according to a defined specification (spec). While this may sound complicated to understand, in short it means that we try to verify that each part of our source code works as expected, without having to run the full program they belong to.

All OpenStack projects come with their own set of unit tests, for example, this is the unit test folder for the oslo.config project. Those tests are executed when a new patch is proposed for review, to ensure that existing (or new) functionality is not broken with the new code. For example, if you check this review, you can see that one of the continuous integration jobs executed is “openstack-tox-py27”, which runs unit tests using Python 2.7.

RDO units

How does this translate into the packaging world? As part of a spec file, we can define a %check section, where we add scripts to test the installed code. While this is not a mandatory section in the Fedora packaging guidelines, it is highly recommended, since it provides a good assurance that the code packaged is correct.

In many cases, RDO packages include this %check section in their specs, and the project’s unit tests are executed when the package is built. This is an example of the unit tests executed for the python-oslo-utils package.

"But why are these tests executed again when packaging?" you may ask. After all, these same tests are executed by the Zuul gate before being merged. Well, there are quite a few reasons for this:

Those unit tests were run with a specific operating system version and a specific package set. Those are probably different from the ones used by RDO, so we need to ensure the project compatibility with those components.
The project dependencies are installed in the OpenStack gate using pip, and some versions may differ. This is because OpenStack projects support a range of versions for each dependency, but usually only test with one version. We have seen cases where a project stated support for version x.0 of a library, but then added code that required version x.1. This change would not be noticed by the OpenStack gate, but it would make unit tests fail while packaging.
They also allow us to detect issues before they happen in the upstream gate. OpenStack projects use the requirements project to decide which version of their own libraries should be used by other projects. This allows for some inter-dependency issues, where a change in an Oslo library may uncover a bug in another project, but it is not noticed until the requirements project is updated with a new version of the Oslo library. In the RDO case, we run an RDO trunk builder using code from the master branch in all projects, which allows us to notify in advance, like in this example bug.
They give us an early warning when new dependencies have been added to a project, but they are not in the package spec yet. Since unit tests exercise most of the code, any missing dependency should make them fail.

Due to the way unit tests are executed during a package build, there are some details to keep in mind when defining them. If you as a developer follow them, you will make packagers’ life easier:

Do not create unit tests that depend on resources available from the Internet. Most packaging environments do not allow Internet access while the package is being built, so a unit test that depends on resolving an IP address via DNS will fail.
Try to keep unit test runtime within reasonable limits. If unit tests for a project take 1 hour to complete, it is likely they will not be executed during packaging, such as in this example.
Do not assume that unit tests will always be executed on a machine with 8 fast cores. We have seen cases of unit tests failing when run in a limited environment or when it takes them more than a certain time to finish.

Now that you know the importance of unit tests for RDO packaging, you can go ahead and make sure we use it on every package. Happy hacking!

Want to learn more about packaging and installing OpenStack? OpenStack Summit is taking place in Vancouver from May 21-24, and is a great opportunity to learn even more about this topic. This article was also published on the RDO Project blog.