governance/reference/project-testing-interface.rst

Consistent Testing Interface

OpenStack has a lot of projects. For each project, the OpenStack CI system needs to be able to perform a lot of tasks. If each project has a slightly different way to accomplish those tasks, it makes the management of a consistent testing infrastructure very difficult to deal with. Additionally, because of the high volume of development changes and testing, the testing infrastructure has to be able to pre-cache artifacts that are normally fetched over the internet. To that end, each project should support a consistent interface for driving tests and other necessary tasks.

End results needed

Each python project must be able to do:

• Unit tests for python2.7
• Codestyle checks
• Testing Coverage Report
• Source Tarball Generation
• Translations import/export and merge for translated projects
• Documentation generation

Projects which are compatible with Python 3 must also be able to do:

• Unit tests for python3.4

Projects may optionally also support:

• Constrained unit tests for python2.7
• Constrained unit tests for python3.4

Specific commands

To drive the above tasks, the following commands should be supported in a clean tree:

• tox -epy27
• tox -epep8
• tox -ecover
• tox -evenv python setup.py sdist
• tox -evenv python setup.py build_sphinx

Projects that are translated should also support:

• tox -evenv python setup.py extract_messages
• tox -evenv python setup.py update_catalog

Projects which are compatible with Python 3 must also be able to do:

• tox -epy34

Projects desiring to support constrained unit tests must support:

• tox -epy27-constraints
• tox -epy34-constraints

Requirements Listing

Each project should list its operations dependencies in requirements.txt and additional dependencies required for testing in test-requirements.txt. If there are requirements that are specific to python3 or pypy support, those may be listed in requirements.txt or test-requirements.txt using environment makers.

Virtual Environment Management

To support sensible testing across multiple python versions, we use tox config files in the projects.

unittest running

OpenStack uses testrepository as its test runner, which supports a number of things, most importantly to the expanded project is the subunit output stream collection. This is useful for aggregating and displaying test output. In support of that, the oslotest library is built on top of testtools, testscenarios and fixtures.

Constraints

The requirements project maintains a set of constraints with packages pinned to specific package versions that are known to be working. The goal is to ease the diagnosis of breakage caused by projects upstream to OpenStack and to provide a set of packages known to work together. Running constraints enabled jobs in addition to the main tests is optional and left up to individual projects to decide.

Project Configuration

All OpenStack projects use pbr for consistent operation of setuptools. To accomplish this, all setup.py files only contain a simple setup function that setup_requires on an unversioned pbr, and a directive to pass processing to the pbr library. Actual project configuration is then handled in setup.cfg.

Generated Files

ChangeLog and AUTHORS files are generated at setup.py sdist time. This is handled by pbr.

.mailmap files should exist where a developer has more than one email address or identity, and should map to the developer's canonical identity.

Translations

To support translations processing, projects should have a valid babel config. There should be a locale package inside of the top project module, and in that dir should be the $project.pot file. For instance, for nova, there should be nova/locale/nova.pot. Babel commands should be configured out output their .mo files in to $project/locale as well.

Documentation

Developer docs are generated from Sphinx sources in the tree. Additionally, there are end user docs and API docs which are maintained outside of the context of a project's repo. To support documentation generation, projects should have sphinx documentation source in doc/source and build_sphinx should output the documentation to doc/build.