openstack
/
keystone
Code Issues Proposed changes

Update keystone testing documentation 

There were a lot of stale bits in our testing document. This commit
attempts to update those bits of information.

Change-Id: Ie99256a6189a7f00623f29c5c5cd49b046f181fe
Closes-Bug: 1662623
This commit is contained in:
Lance Bragstad 2017-11-28 20:16:47 +00:00
parent 62d2504237
commit 227d38e4a1
1 changed files with 49 additions and 70 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

119
doc/source/contributor/testing-keystone.rst
View File

@ -35,7 +35,7 @@ environment (in addition to the other external dependencies in
python virtualenv, or prefix the above command with ``sudo``, depending on python virtualenv, or prefix the above command with ``sudo``, depending on
your preference. your preference.
To execute the full suite of tests maintained within Keystone, simply run: To execute the full suite of tests maintained within keystone, simply run:
.. code-block:: bash .. code-block:: bash
@ -59,57 +59,40 @@ common configuration of Python 2.7 and PEP-8), list the environments with the
$ tox -e py27,pep8 $ tox -e py27,pep8
See ``tox.ini`` for the full list of available test environments. Use ``tox --listenvs`` to list all testing environments specified in keystone's
``tox.ini`` file.
Running with PDB Interactive debugging
~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~
Using PDB breakpoints with tox and testr normally doesn't work since the tests Using ``pdb`` breakpoints with ``tox`` and ``testr`` normally doesn't work
just fail with a BdbQuit exception rather than stopping at the breakpoint. since the tests just fail with a ``BdbQuit`` exception rather than stopping at
the breakpoint.
To run with PDB breakpoints during testing, use the ``debug`` tox environment To capture breakpoints while running tests, use the ``debug`` environment. The
rather than ``py27``. Here's an example, passing the name of a test since following example uses the environment while invoking a specific test run.
you'll normally only want to run the test that hits your breakpoint:
.. code-block:: bash .. code-block:: bash
$ tox -e debug keystone.tests.unit.test_auth.AuthWithToken.test_belongs_to $ tox -e debug keystone.tests.unit.test_module.TestClass.test_case
For reference, the ``debug`` tox environment implements the instructions For reference, the ``debug`` environment implements the instructions here:
here: https://wiki.openstack.org/wiki/Testr#Debugging_.28pdb.29_Tests https://wiki.openstack.org/wiki/Testr#Debugging_.28pdb.29_Tests
Disabling Stream Capture
~~~~~~~~~~~~~~~~~~~~~~~~
The stdout, stderr and log messages generated during a test are captured and
in the event of a test failure those streams will be printed to the terminal
along with the traceback. The data is discarded for passing tests.
Each stream has an environment variable that can be used to force captured
data to be discarded even if the test fails: `OS_STDOUT_CAPTURE` for stdout,
`OS_STDERR_CAPTURE` for stderr and `OS_LOG_CAPTURE` for logging. If the value
of the environment variable is not one of (True, true, 1, yes) the stream will
be discarded. All three variables default to 1.
For example, to discard logging data during a test run:
.. code-block:: bash
$ OS_LOG_CAPTURE=0 tox -e py27
Building the Documentation Building the Documentation
-------------------------- --------------------------
The documentation is generated with Sphinx using the tox command. To create HTML The ``docs`` and ``api-ref`` environments will automatically generate
docs and man pages: documentation and the API reference respectively. The results are written to
``doc/`` and ``api-ref/``.
For example, use the following command to render all documentation and manual
pages:
.. code-block:: bash .. code-block:: bash
$ tox -e docs $ tox -e docs
The results are in the ``doc/build/html`` and ``doc/build/man`` directories
respectively.
Tests Structure Tests Structure
--------------- ---------------
@ -129,27 +112,22 @@ test class in ``test_backend.py``.
.. NOTE:: .. NOTE::
The structure of backend testing is in transition, migrating from having The structure of backend testing is in transition, migrating from having
all classes in a single file (test_backend.py) to one where there is a all classes in a single file (``test_backend.py``) to one where there is a
directory structure to reduce the size of the test files. See: directory structure to reduce the size of the test files. See:
- :mod:`keystone.tests.unit.backend.role` - :mod:`keystone.tests.unit.backend.role`
- :mod:`keystone.tests.unit.backend.domain_config` - :mod:`keystone.tests.unit.backend.domain_config`
To add new drivers, subclass the base class at ``test_backend.py`` (look To add new drivers, subclass the base class at ``test_backend.py`` (look at
towards ``test_backend_sql.py`` for examples) and update the configuration of ``test_backend_sql.py`` for examples) and update the configuration of the test
the test class in ``setUp()``. class in ``setUp()``.
For example, ``test_backend.py`` has a sequence of tests under the class For example, ``test_backend.py`` has a sequence of tests under the class
:class:`~keystone.tests.unit.test_backend.IdentityTests` that will work with :class:`keystone.tests.unit.test_backend.IdentityTests` that will work with the
the default drivers as configured in this project's etc/ directory. default drivers. The ``test_backend_sql.py`` module subclasses those tests,
``test_backend_sql.py`` subclasses those tests, changing the configuration by changing the configuration by overriding with configuration files stored in the
overriding with configuration files stored in the ``tests/unit/config_files`` ``tests/unit/config_files`` directory aimed at enabling the SQL backend for the
directory aimed at enabling the SQL backend for the Identity module. Identity module.
:class:`keystone.tests.unit.test_v2_keystoneclient.ClientDrivenTestCase`
uses the installed python-keystoneclient, verifying it against a temporarily
running local keystone instance to explicitly verify basic functional testing
across the API.
Testing Schema Migrations Testing Schema Migrations
------------------------- -------------------------
@ -162,8 +140,8 @@ built-in test runner, one migration at a time.
This may leave your database in an inconsistent state; attempt this in This may leave your database in an inconsistent state; attempt this in
non-production environments only! non-production environments only!
This is useful for testing the *next* migration in sequence (both forward & This is useful for testing the *next* migration in sequence in a database under
backward) in a database under version control: version control:
.. code-block:: bash .. code-block:: bash
@ -178,7 +156,7 @@ of your data during migration.
LDAP Tests LDAP Tests
---------- ----------
LDAP has a fake backend that performs rudimentary operations. If you LDAP has a fake backend that performs rudimentary operations. If you
are building more significant LDAP functionality, you should test against are building more significant LDAP functionality, you should test against
a live LDAP server. Devstack has an option to set up a directory server for a live LDAP server. Devstack has an option to set up a directory server for
Keystone to use. Add ldap to the ``ENABLED_SERVICES`` environment variable, Keystone to use. Add ldap to the ``ENABLED_SERVICES`` environment variable,
@ -203,14 +181,14 @@ password.
Work in progress (WIP) tests are very useful in a variety of situations Work in progress (WIP) tests are very useful in a variety of situations
including: including:
* During a TDD process they can be used to add tests to a review while * While doing test-driven-development they can be used to add tests to a review
they are not yet working and will not cause test failures. (They should while they are not yet working and will not cause test failures. They can be
be removed before the final merge.) removed when the functionality is fixed in a later patch set.
* Often bug reports include small snippets of code to show broken * A common practice is to recreate bugs by exposing the broken behavior in a
behaviors. Some of these can be converted into WIP tests that can later functional or unit test. To encapsulate the correct behavior in the test, the
be worked on by a developer. This allows us to take code that can be test will usually assert the correct outcome, which will break without a fix.
used to catch bug regressions and commit it before any code is Marking the test as WIP gives us the ability to capture the broken behavior
written. in code if a fix isn't ready yet.
The :func:`keystone.tests.unit.utils.wip` decorator can be used to mark a test The :func:`keystone.tests.unit.utils.wip` decorator can be used to mark a test
as WIP. A WIP test will always be run. If the test fails then a TestSkipped as WIP. A WIP test will always be run. If the test fails then a TestSkipped
@ -220,14 +198,14 @@ successfully run tests. If the test passes an AssertionError exception is
raised so that the developer knows they made the test pass. This is a raised so that the developer knows they made the test pass. This is a
reminder to remove the decorator. reminder to remove the decorator.
The :func:`~keystone.tests.unit.utils.wip` decorator requires that the author The :func:`keystone.tests.unit.utils.wip` decorator requires that the author
provides a message. This message is important because it will tell other provides a message. This message is important because it will tell other
developers why this test is marked as a work in progress. Reviewers will developers why this test is marked as a work in progress. Reviewers will
require that these messages are descriptive and accurate. require that these messages are descriptive and accurate.
.. NOTE:: .. NOTE::
The :func:`~keystone.tests.unit.utils.wip` decorator is not a replacement for The :func:`keystone.tests.unit.utils.wip` decorator is not a replacement
skipping tests. for skipping tests.
.. code-block:: python .. code-block:: python
@ -243,11 +221,11 @@ require that these messages are descriptive and accurate.
API & Scenario Tests API & Scenario Tests
-------------------- --------------------
Keystone provides API and scenario tests via a `tempest plugin`_ located at Keystone provides API and scenario tests via a `tempest plugin`_ which is
:func:`~keystone.keystone_tempest_plugin`. This tempest plugin is mainly located in a separate `repository`_. This tempest plugin is mainly intended for
intended for specific scenarios that require a special deployment, such as specific scenarios that require a special deployment, such as the tests for the
the tests for the ``Federated Identity`` feature. For the deployment of these ``Federated Identity`` feature or live testing against LDAP. For the deployment
scenarios, keystone also provides a `devstack plugin`_. of these scenarios, keystone also provides a `devstack plugin`_.
For example, to setup a working federated environment, add the following lines For example, to setup a working federated environment, add the following lines
in your `devstack` `local.conf`` file: in your `devstack` `local.conf`` file:
@ -274,6 +252,7 @@ from the tempest directory:
.. _devstack plugin: https://docs.openstack.org/devstack/latest/plugins.html .. _devstack plugin: https://docs.openstack.org/devstack/latest/plugins.html
.. _tempest: https://git.openstack.org/cgit/openstack/tempest .. _tempest: https://git.openstack.org/cgit/openstack/tempest
.. _tempest plugin: https://docs.openstack.org/tempest/latest/plugin.html .. _tempest plugin: https://docs.openstack.org/tempest/latest/plugin.html
.. _repository: http://git.openstack.org/cgit/openstack/keystone-tempest-plugin
Writing new API & Scenario Tests Writing new API & Scenario Tests
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@ -366,7 +345,7 @@ method.
self._setup_mapping() self._setup_mapping()
self._setup_protocol() self._setup_protocol()
Finally, the tests perform the complete workflow of the feature, asserting its Finally, the tests perform the complete workflow of the feature, asserting
correctness in each step: correctness in each step:
.. code-block:: python .. code-block:: python