openstack/charm-guide
Code Issues Proposed changes

Update testing section

Browse Source 
Change-Id: I2eb6ff576da8e55fdc1d0a1821a0c92e27c5b0d0
This commit is contained in:
Ryan Beisner
2016-11-23 08:24:23 -06:00
parent 5b8da47d88
commit a03f607d72
1 changed files with 79 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

86
doc/source/testing.rst
View File

@@ -17,7 +17,6 @@ The OpenStack Charms are compliant with the OpenStack
`Consistent Testing Interface <https://governance.openstack.org/reference/cti/python_cti.html>`__;
take a read on how this works to understand in full.
Lint
====
@@ -33,7 +32,8 @@ Ensure that any non-compliance is corrected prior to raising/updating a review.
Unit Testing
============
Execute the unit tests for a charm using the tox py27 environment:
Execute the synthetic code checks (unit tests) for a charm using the tox py27
environment:
.. code:: bash
@@ -62,15 +62,87 @@ testing of different OpenStack release combinations; typically each charm will
test the OpenStack and Ubuntu release combinations currently supported by
Ubuntu.
Execute of Amulet tests currently requires use of the Makefile:
The OpenStack Charms Amulet tests in their current form may be specific to
execution within a tenant on an OpenStack cloud, via the Juju OpenStack
provider, and that is how the third-party-CI executes them. Future functional
test enhancements include the ability run the tests against the Juju OpenStack
provider (a cloud) or the Juju LXD provider (all on one machine).
:Full Amulet: Executes all Amulet gate tests (may take several hours). The
full Amulet test set does not run automatically on each proposed change.
After the lower-cost lint, unit, charm-single and Amulet-smoke tests have
completed, reviewers can conduct code reviews then optionally trigger the
full set of Amulet tests (see Rechecking).
To manually trigger execution of all Amulet tests on your locally-defined
cloud:
.. code:: bash
make functional_test
tox -e func27
This will execute the full suite of Amulet tests under the ``tests/`` folder.
:Amulet Smoke: Executes a subset (generaly one) of the Amulet deployment test
sets. The Amulet smoke test set does run automatically on every proposed
patchset.
To manually trigger execution of the Amulet smoke test on your
locally-defined cloud:
The ``tests/README`` file in each charm will contain more details on how to
name tests and how to execute them individually.
.. code:: bash
tox -e func27-smoke
:No-Op: Builds a Python virtualenv per definitions in ``tox.ini``,
which can be useful in test authoring.
To manually trigger a build of the virtualenv on your local machine, but
execute no tests:
.. code:: bash
tox -e func27-noop
Test methods are called in lexical sort order, as with most test runners.
However, each individual test method should be idempotent and expected
to pass regardless of run order or Ubuntu:OpenStack combo. When writing
or modifying tests, ensure that every individual test is not dependent
on another test method.
Some tests may need to download files from the Internet, such as glance
images. If a web proxy server is required in the environment, the
``AMULET_HTTP_PROXY`` environment variable must be set. This is unrelated
to Juju's http-proxy settings.
See ``tox.ini`` to determine specifically which test targets will be executed by
each tox target. Amulet tests reside in the ``tests/`` directory for classic
charms, and in the ``src/tests/`` directory for layered source charms.
Rechecking
==========
*BEFORE issuing a recheck of any kind, please inspect the CI results and
log artifacts to understand the failure reason.*
*Rechecks should only be used in the event of a system failure (not for
race conditions or problems introduced by the proposed code changes).*
*Developers are expected to have executed tests prior to submitting patches.*
Tests can be retriggered, or additional tests can be requested, simply by
replying on the Gerrit review with one of the recognized magic phrases below.
``recheck``
Re-triggers events as if a new patchset had been submitted, including
all defined OpenStack Infra tests AND third-party-CI tests.
``charm-recheck``
Re-triggers only the default set of OpenStack Charms third-party-ci tests,
but not the OpenStack Infra tests. *Depending on system load and which
charm is under test, this will typically take 30 to 60 minutes.*
``charm-recheck-full``
Triggers a full set of OpenStack Charms third-party-ci tests, but not the
OpenStack Infra tests. *This will take several hours.*
.. _Amulet: https://jujucharms.com/docs/devel/tools-amulet