How to create a Tempest plugin for OpenStack

Tempest is the OpenStack official test suite. Its purpose is to run tests for OpenStack API validation in an OpenStack cluster, in order to know how healthy our cloud is. It is also used as a gate for validating commits into the OpenStack core projects—it will avoid breaking them while merging changes. For more information about Tempest, see the developer documentation and the source code repository.

Thanks to a new feature proposed in the community, we can create independent plugins for Tempest's code to run custom test cases over OpenStack. This method is called the Tempest Test Plugin Interface.

With this interface, any project can run a set of external tests as part of a Tempest run. Now we only have to worry about creating the plugin to group our test cases together, and we can forget about the complexity of integrating custom test cases into the structure of the Tempest's code. This means more independence and flexibility!

Plugin structure

A Tempest plugin is basically a Python package that contains your custom test cases. It needs to be installed on the local or virtual environment (venv) where Tempest is located. All of your installed plugins will be discovered by Tempest when it runs without you needing to do anything extra.

Directory structure

setup.cfg
setup.py
README.rst
plugin_dir/
	__init__.py
 	config.py
  	plugin.py
 	tests/
    		__init__.py
    		api/
      			__init__.py
      			base.py
     	 		test_hello_world.py
    		scenario/
      			__init__.py

• setup.cfg: Contains the package metadata. For example: name, version, summary, author, etc. For the full list, see: Metadata docs.

  In this file the most important thing is to specify the entry point, which is the location of the project's main class that will allow Tempest to find the plugin.

  Consider the following example, where hello-world-tempest-plugin is the plugin's name, hello_world_tempest_plugin is the name of the plugin's folder in the file structure and MyPlugin is the main class name. So, the entry point will be:

  [entry_points]
  tempest.test_plugins =
      hello-world-tempest-plugin = hello_world_tempest_plugin.plugin:MyPlugin
        

• setup.py: The global plugin's requirements are specified here. In this case, only pbr is necessary.
• README.rst Information about the project.
• plugin_dir: Main folder, usually has the name of the plugin. Example: hello_world_tempest_plugin.
• config.py: Contains configuration variables needed for the tests in the plugin.
• plugin.py: Plugin's main class, entry point that contains all the configuration that Tempests needs to run the plugin. You just have to implement the methods from the abstract class.
• tests: Folder where all the tests have to be.
• api: All test cases related to API testing.
• scenario: All test cases related to scenario testing.

You can review hello-world-tempest-plugin as a reference to build your own plugin. This repository is a basic plugin that runs a hello world test case.

For the next steps I will use that plugin as example in the commands.

Installation

When Tempest runs, it will automatically discover the installed plugins. So we just need to install the Python packages that contains the plugin.

Upgrade pip and setuptools:

$ pip install -U pip

Install the package from the plugin root directory:

$ cd hello-world-tempest-plugin
$ sudo pip install -e .

Using virtual environments

If you run Tempest inside a virtual environment you have to ensure that the Python package containing the plugin, is installed in the venv too.

E.g, installing the plugin in a Tempest (through Rally) venv:

$ . ~/.rally/tempest/for-deployment-x-x-x-x-x/.venv/bin/activate
$ ~/.rally/tempest/for-deployment-x-x-x-x-x/.venv/bin/pip install -e ~/hello-world-tempest-plugin/


Validate that the plugin was installed correctly:

$ ~/.rally/tempest/for-deployment-x-x-x-x-x/.venv/bin/pip list

How to run the tests

1. To validate that Tempest discovered the test in the plugin, you can run:

   $ testr list-tests | grep hello_world_tempest_plugin

   This command will show your complete list of test cases inside the plugin.

2. You can run the test cases by name or running the set names that they used as decorator through testr (Tempest) or Rally:

   $ testr run hello_world_tempest_plugin.tests.api.test_hello_world.TestHelloWorld.test_hello_world
$ testr run --subunit smoke | subunit-2to1 | ./tools/colorizer.py
$ rally verify start --set smoke

Resources:

• http://docs.openstack.org/developer/tempest/plugin.html
• http://docs.openstack.org/developer/tempest/
• https://github.com/MBonell/hello-world-tempest-plugin
• https://wiki.openstack.org/wiki/Testr
• https://github.com/openstack/rally
• https://python-packaging-user-guide.readthedocs.org/en/latest/distributing/

This article is part of the Speaker Interview Series for OpenStack Summit Tokyo, a four-day conference for developers, users, and administrators of OpenStack cloud software.