Browse Source

Writing API & Scenario Tests docs

It is nice to see a real example of a real test when trying to write a
new one. This patch adds a doc that gives an overview of writing new API
and Scenario tests for the keystone tempest plugin.

This also fixes a typo in the IdentityProdiversTest class name.

Change-Id: Icd631124b451f17df3d4de184e73f6223fdb7de3
Rodrigo Duarte Sousa 5 years ago
  1. 139
  2. 2


@ -474,6 +474,145 @@ from the tempest directory:
.. _tempest:
.. _tempest plugin:
Writing new API & Scenario Tests
When writing tests for the keystone tempest plugin, we should follow the
official tempest guidelines, details about the guidelines can be found at the
`tempest coding guide`_. There are also specific guides for the API and
scenario tests: `Tempest Field Guide to API tests`_ and
`Tempest Field Guide to Scenario tests`_.
The keystone tempest plugin also provides a base class. For most cases, the
tests should inherit from it:
:class:`keystone_tempest_plugin.tests.base.BaseIdentityTest`. This class
already setups the identity API version and is the container of all API
services clients.
New API services clients :mod:``
(which are used to communicate with the REST API from
the services) should also be added to this class. For example, below we have a
snippet from the tests at
.. code-block:: python
class IdentityProvidersTest(base.BaseIdentityTest):
def _create_idp(self, idp_id, idp_ref):
idp = self.idps_client.create_identity_provider(
idp_id, **idp_ref)['identity_provider']
self.idps_client.delete_identity_provider, idp_id)
return idp
def test_identity_provider_create(self):
idp_id = data_utils.rand_uuid_hex()
idp_ref = fixtures.idp_ref()
idp = self._create_idp(idp_id, idp_ref)
# The identity provider is disabled by default
idp_ref['enabled'] = False
# The remote_ids attribute should be set to an empty list by default
idp_ref['remote_ids'] = []
self._assert_identity_provider_attributes(idp, idp_id, idp_ref)
The test class extends
:class:`keystone_tempest_plugin.tests.base.BaseIdentityTest`. Also, the
``_create_idp`` method calls keystone's API using the ``idps_client``,
which is an instance from.
Additionally, to illustrate the construction of a new test class, below we have
a snippet from the scenario test that checks the complete federated
authentication workflow (
In the test setup, all of the needed resources are created using the API
service clients. Since it is a scenario test, it is common to need some
customized settings that will come from the environment (in this case, from
the devstack plugin) - these settings are collected in the ``_setup_settings``
.. code-block:: python
class TestSaml2EcpFederatedAuthentication(base.BaseIdentityTest):
def _setup_settings(self):
self.idp_id = CONF.fed_scenario.idp_id
self.idp_url = CONF.fed_scenario.idp_ecp_url
self.keystone_v3_endpoint = CONF.identity.uri_v3
self.password = CONF.fed_scenario.idp_password
self.protocol_id = CONF.fed_scenario.protocol_id
self.username = CONF.fed_scenario.idp_username
def setUp(self):
super(TestSaml2EcpFederatedAuthentication, self).setUp()
# Reset client's session to avoid getting garbage from another runs
# Setup identity provider, mapping and protocol
Finally, the tests perform the complete workflow of the feature, asserting its
correctness in each step:
.. code-block:: python
def _request_unscoped_token(self):
resp = self.saml2_client.send_service_provider_request(
self.keystone_v3_endpoint, self.idp_id, self.protocol_id)
self.assertEqual(http_client.OK, resp.status_code)
saml2_authn_request = etree.XML(resp.content)
relay_state = self._str_from_xml(
saml2_authn_request, self.ECP_RELAY_STATE)
sp_consumer_url = self._str_from_xml(
saml2_authn_request, self.ECP_SERVICE_PROVIDER_CONSUMER_URL)
# Perform the authn request to the identity provider
resp = self.saml2_client.send_identity_provider_authn_request(
saml2_authn_request, self.idp_url, self.username, self.password)
self.assertEqual(http_client.OK, resp.status_code)
saml2_idp_authn_response = etree.XML(resp.content)
idp_consumer_url = self._str_from_xml(
saml2_idp_authn_response, self.ECP_IDP_CONSUMER_URL)
# Assert that both saml2_authn_request and saml2_idp_authn_response
# have the same consumer URL.
self.assertEqual(sp_consumer_url, idp_consumer_url)
"Federated Identity feature not enabled")
def test_request_unscoped_token(self):
Notice that the ``test_request_unscoped_token`` test only executes if the the
``federation`` feature flag is enabled.
.. NOTE::
For each patch submitted upstream, all of the tests from the keystone
tempest plugin are executed in the
``gate-keystone-dsvm-functional-v3-only-*`` job.
.. _Tempest Field Guide to Scenario tests:
.. _Tempest Field Guide to API tests:
.. _tempest coding guide:
Developing ``doctor`` checks


@ -20,7 +20,7 @@ from keystone_tempest_plugin.tests.api.identity.v3 import fixtures
from keystone_tempest_plugin.tests import base
class IndentityProvidersTest(base.BaseIdentityTest):
class IdentityProvidersTest(base.BaseIdentityTest):
def _assert_identity_provider_attributes(self, idp, idp_id, idp_ref=None):
self.assertIn('id', idp)