Writing API & Scenario Tests docs

It is nice to see a real example of a real test when trying to write a
new one. This patch adds a doc that gives an overview of writing new API
and Scenario tests for the keystone tempest plugin.

This also fixes a typo in the IdentityProdiversTest class name.

Change-Id: Icd631124b451f17df3d4de184e73f6223fdb7de3

doc/source/devref/development_best_practices.rst

@@ -474,6 +474,145 @@ from the tempest directory:
 .. _tempest: https://git.openstack.org/cgit/openstack/tempest
 .. _tempest plugin: https://docs.openstack.org/developer/tempest/plugin.html
 
+Writing new API & Scenario Tests
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+When writing tests for the keystone tempest plugin, we should follow the
+official tempest guidelines, details about the guidelines can be found at the
+`tempest coding guide`_. There are also specific guides for the API and
+scenario tests: `Tempest Field Guide to API tests`_ and
+`Tempest Field Guide to Scenario tests`_.
+
+The keystone tempest plugin also provides a base class. For most cases, the
+tests should inherit from it:
+:class:`keystone_tempest_plugin.tests.base.BaseIdentityTest`. This class
+already setups the identity API version and is the container of all API
+services clients.
+New API services clients :mod:`keystone_tempest_plugin.services`
+(which are used to communicate with the REST API from
+the services) should also be added to this class. For example, below we have a
+snippet from the tests at
+:py:mod:`keystone_tempest_plugin.tests.api.identity.v3.test_identity_providers.py`.
+
+.. code-block:: python
+
+    class IdentityProvidersTest(base.BaseIdentityTest):
+
+    ...
+
+    def _create_idp(self, idp_id, idp_ref):
+        idp = self.idps_client.create_identity_provider(
+            idp_id, **idp_ref)['identity_provider']
+        self.addCleanup(
+            self.idps_client.delete_identity_provider, idp_id)
+        return idp
+
+    @decorators.idempotent_id('09450910-b816-4150-8513-a2fd4628a0c3')
+    def test_identity_provider_create(self):
+        idp_id = data_utils.rand_uuid_hex()
+        idp_ref = fixtures.idp_ref()
+        idp = self._create_idp(idp_id, idp_ref)
+
+        # The identity provider is disabled by default
+        idp_ref['enabled'] = False
+
+        # The remote_ids attribute should be set to an empty list by default
+        idp_ref['remote_ids'] = []
+
+        self._assert_identity_provider_attributes(idp, idp_id, idp_ref)
+
+The test class extends
+:class:`keystone_tempest_plugin.tests.base.BaseIdentityTest`. Also, the
+``_create_idp`` method calls keystone's API using the ``idps_client``,
+which is an instance from.
+:class:`keystone_tempest_plugin.tests.services.identity.v3.identity_providers_client.IdentityProvidersClient`.
+
+Additionally, to illustrate the construction of a new test class, below we have
+a snippet from the scenario test that checks the complete federated
+authentication workflow (
+:py:mod:`keystone_tempest_plugin.tests.scenario.test_federated_authentication.py`).
+In the test setup, all of the needed resources are created using the API
+service clients. Since it is a scenario test, it is common to need some
+customized settings that will come from the environment (in this case, from
+the devstack plugin) - these settings are collected in the ``_setup_settings``
+method.
+
+.. code-block:: python
+
+    class TestSaml2EcpFederatedAuthentication(base.BaseIdentityTest):
+
+    ...
+
+    def _setup_settings(self):
+        self.idp_id = CONF.fed_scenario.idp_id
+        self.idp_url = CONF.fed_scenario.idp_ecp_url
+        self.keystone_v3_endpoint = CONF.identity.uri_v3
+        self.password = CONF.fed_scenario.idp_password
+        self.protocol_id = CONF.fed_scenario.protocol_id
+        self.username = CONF.fed_scenario.idp_username
+
+    ...
+
+    def setUp(self):
+        super(TestSaml2EcpFederatedAuthentication, self).setUp()
+        self._setup_settings()
+
+        # Reset client's session to avoid getting garbage from another runs
+        self.saml2_client.reset_session()
+
+        # Setup identity provider, mapping and protocol
+        self._setup_idp()
+        self._setup_mapping()
+        self._setup_protocol()
+
+Finally, the tests perform the complete workflow of the feature, asserting its
+correctness in each step:
+
+.. code-block:: python
+
+    def _request_unscoped_token(self):
+        resp = self.saml2_client.send_service_provider_request(
+            self.keystone_v3_endpoint, self.idp_id, self.protocol_id)
+        self.assertEqual(http_client.OK, resp.status_code)
+        saml2_authn_request = etree.XML(resp.content)
+
+        relay_state = self._str_from_xml(
+            saml2_authn_request, self.ECP_RELAY_STATE)
+        sp_consumer_url = self._str_from_xml(
+            saml2_authn_request, self.ECP_SERVICE_PROVIDER_CONSUMER_URL)
+
+        # Perform the authn request to the identity provider
+        resp = self.saml2_client.send_identity_provider_authn_request(
+            saml2_authn_request, self.idp_url, self.username, self.password)
+        self.assertEqual(http_client.OK, resp.status_code)
+        saml2_idp_authn_response = etree.XML(resp.content)
+
+        idp_consumer_url = self._str_from_xml(
+            saml2_idp_authn_response, self.ECP_IDP_CONSUMER_URL)
+
+        # Assert that both saml2_authn_request and saml2_idp_authn_response
+        # have the same consumer URL.
+        self.assertEqual(sp_consumer_url, idp_consumer_url)
+
+        ...
+
+
+    @testtools.skipUnless(CONF.identity_feature_enabled.federation,
+                          "Federated Identity feature not enabled")
+    def test_request_unscoped_token(self):
+        self._request_unscoped_token()
+
+Notice that the ``test_request_unscoped_token`` test only executes if the the
+``federation`` feature flag is enabled.
+
+.. NOTE::
+   For each patch submitted upstream, all of the tests from the keystone
+   tempest plugin are executed in the
+   ``gate-keystone-dsvm-functional-v3-only-*`` job.
+
+.. _Tempest Field Guide to Scenario tests: https://docs.openstack.org/developer/tempest/field_guide/scenario.html
+.. _Tempest Field Guide to API tests: https://docs.openstack.org/developer/tempest/field_guide/api.html
+.. _tempest coding guide: https://docs.openstack.org/developer/tempest/HACKING.html
 
 Developing ``doctor`` checks
 ============================

keystone_tempest_plugin/tests/api/identity/v3/test_identity_providers.py

@@ -20,7 +20,7 @@ from keystone_tempest_plugin.tests.api.identity.v3 import fixtures
 from keystone_tempest_plugin.tests import base
 
 
-class IndentityProvidersTest(base.BaseIdentityTest):
+class IdentityProvidersTest(base.BaseIdentityTest):
 
     def _assert_identity_provider_attributes(self, idp, idp_id, idp_ref=None):
         self.assertIn('id', idp)