From 11d89450c12f47ff7d55f6eed18a5e7f91d7cf0a Mon Sep 17 00:00:00 2001 From: Stephen Finucane Date: Wed, 22 Dec 2021 13:22:02 +0000 Subject: [PATCH] doc: Update testing documentation The information on how to run functional tests was badly out of date. Correct it. Change-Id: I7e77e2a96ab15f7dede01344cb03b0e61c8589bd Signed-off-by: Stephen Finucane --- create_yaml.sh | 31 ---------- doc/source/contributor/clouds.yaml | 44 ++++++++++---- doc/source/contributor/testing.rst | 96 ++++++++++++++++++------------ tox.ini | 2 +- 4 files changed, 90 insertions(+), 83 deletions(-) delete mode 100755 create_yaml.sh diff --git a/create_yaml.sh b/create_yaml.sh deleted file mode 100755 index 4443986ee..000000000 --- a/create_yaml.sh +++ /dev/null @@ -1,31 +0,0 @@ -#!/bin/bash -# -# NOTE(thowe): There are some issues with OCC envvars that force us to do -# this for now. -# -mkdir -p ~/.config/openstack/ -FILE=~/.config/openstack/clouds.yaml -export OS_IDENTITY_API_VERSION=3 # force v3 identity -echo 'clouds:' >$FILE -echo ' test_cloud:' >>$FILE -env | grep OS_ | tr '=' ' ' | while read k v -do - k=$(echo $k | sed -e 's/OS_//') - k=$(echo $k | tr '[A-Z]' '[a-z]') - case "$k" in - region_name|*_api_version) - echo " $k: $v" >>$FILE - esac -done -echo " auth:" >>$FILE -env | grep OS_ | tr '=' ' ' | while read k v -do - k=$(echo $k | sed -e 's/OS_//') - k=$(echo $k | tr '[A-Z]' '[a-z]') - case "$k" in - region_name|*_api_version) - ;; - *) - echo " $k: $v" >>$FILE - esac -done diff --git a/doc/source/contributor/clouds.yaml b/doc/source/contributor/clouds.yaml index b92ba85b7..2c141c5e5 100644 --- a/doc/source/contributor/clouds.yaml +++ b/doc/source/contributor/clouds.yaml @@ -1,19 +1,37 @@ clouds: - test_cloud: - region_name: RegionOne + devstack: auth: - auth_url: http://xxx.xxx.xxx.xxx:5000/v2.0/ - username: demo - password: secrete + auth_url: http://xxx.xxx.xxx.xxx/identity + password: password + project_domain_id: default project_name: demo - rackspace: - cloud: rackspace + user_domain_id: default + username: demo + identity_api_version: '3' + region_name: RegionOne + volume_api_version: '3' + devstack-admin: auth: - username: joe - password: joes-password - project_name: 123123 - region_name: IAD + auth_url: http://xxx.xxx.xxx.xxx/identity + password: password + project_domain_id: default + project_name: admin + user_domain_id: default + username: admin + identity_api_version: '3' + region_name: RegionOne + volume_api_version: '3' + devstack-alt: + auth: + auth_url: http://xxx.xxx.xxx.xxx/identity + password: password + project_domain_id: default + project_name: alt_demo + user_domain_id: default + username: alt_demo + identity_api_version: '3' + region_name: RegionOne + volume_api_version: '3' example: - image_name: fedora-20.x86_64 + image_name: cirros-0.5.2-x86_64-disk flavor_name: m1.small - network_name: private diff --git a/doc/source/contributor/testing.rst b/doc/source/contributor/testing.rst index 991b604ad..c3261df16 100644 --- a/doc/source/contributor/testing.rst +++ b/doc/source/contributor/testing.rst @@ -3,25 +3,27 @@ Testing The tests are run with `tox `_ and configured in ``tox.ini``. The test results are tracked by -`testr `_ and configured -in ``.testr.conf``. +`stestr `_ and configured +in ``.stestr.conf`` and via command line options passed to the ``stestr`` +executable when it's called by ``tox``. + Unit Tests ---------- -Run -*** +Running tests +~~~~~~~~~~~~~ In order to run the entire unit test suite, simply run the ``tox`` command inside of your source checkout. This will attempt to run every test command -listed inside of ``tox.ini``, which includes Python 3.8, and a PEP 8 check. +listed inside of ``tox.ini``, which includes Python 3.x, and a PEP 8 check. You should run the full test suite on all versions before submitting changes for review in order to avoid unexpected failures in the continuous integration system.:: - (sdk3)$ tox + $ tox ... - py38: commands succeeded + py3: commands succeeded pep8: commands succeeded congratulations :) @@ -29,8 +31,13 @@ During development, it may be more convenient to run a subset of the tests to keep test time to a minimum. You can choose to run the tests only on one version. A step further is to run only the tests you are working on.:: - (sdk3)$ tox -e py38 # Run run the tests on Python 3.8 - (sdk3)$ tox -e py38 TestContainer # Run only the TestContainer tests on 3.8 + # Run run the tests on Python 3.9 + $ tox -e py39 + # Run only the compute unit tests on Python 3.9 + $ tox -e py39 openstack.tests.unit.compute + # Run only the tests in a specific file on Python 3.9 + $ tox -e py39 -- -n openstack/tests/unit/compute/test_version.py + Functional Tests ---------------- @@ -41,17 +48,43 @@ public clouds but first and foremost they must be run against OpenStack. In practice, this means that the tests should initially be run against a stable branch of `DevStack `_. -os-client-config -**************** +Configuration +~~~~~~~~~~~~~ -To connect the functional tests to an OpenStack cloud we use -`os-client-config `_. -To setup os-client-config create a ``clouds.yaml`` file in the root of your -source checkout. +To connect the functional tests to an OpenStack cloud we require a +``clouds.yaml`` file, as discussed in :doc:`/user/config/configuration`. +You can place this ``clouds.yaml`` file in the root of your source checkout or +in one of the other standard locations, ``$HOME/.config/openstack`` or +``/etc/openstack``. + +There must be at least three clouds configured, or rather three accounts +configured for the one cloud. These accounts are: + +- An admin account, which defaults to ``devstack-admin`` but is configurable + via the ``OPENSTACKSDK_OPERATOR_CLOUD`` environment variable, +- A user account, which defaults to ``devstack`` but is configurable + via the ``OPENSTACKSDK_DEMO_CLOUD`` environment variable, and +- An alternate user account, which defaults to ``devstack-demo`` but is + configurable via the ``OPENSTACKSDK_DEMO_CLOUD_ALT`` environment variable + +In addition, you must indicate the names of the flavor and image that should be +used for tests. These can be configured via ``functional.flavor_name`` and +``functional.image_name`` settings in the ``clouds.yaml`` file. + +Finally, you can configure the timeout for tests using the +``OPENSTACKSDK_FUNC_TEST_TIMEOUT`` environment variable (defaults to 300 +seconds). Some test modules take specific timeout values. For example, all +tests in ``openstack.tests.functional.compute`` will check for the +``OPENSTACKSDK_FUNC_TEST_TIMEOUT_COMPUTE`` environment variable before checking +for ``OPENSTACKSDK_FUNC_TEST_TIMEOUT``. + +.. note:: + + Recent versions of DevStack will configure a suitable ``clouds.yaml`` file + for you, which will be placed at ``/etc/openstack/clouds.yaml``. This is an example of a minimal configuration for a ``clouds.yaml`` that -connects the functional tests to a DevStack instance. Note that one cloud -under ``clouds`` must be named ``test_cloud``. +connects the functional tests to a DevStack instance. .. literalinclude:: clouds.yaml :language: yaml @@ -59,29 +92,16 @@ under ``clouds`` must be named ``test_cloud``. Replace ``xxx.xxx.xxx.xxx`` with the IP address or FQDN of your DevStack instance. -You can also create a ``~/.config/openstack/clouds.yaml`` file for your -DevStack cloud environment using the following commands. Replace -``DEVSTACK_SOURCE`` with your DevStack source checkout.:: +Running tests +~~~~~~~~~~~~~ - (sdk3)$ source DEVSTACK_SOURCE/accrc/admin/admin - (sdk3)$ ./create_yaml.sh +Functional tests are also run against multiple Python versions. In order to run +the entire functional test suite against the default Python 3 version in your +environment, run the ``tox -e functional`` command inside of your source +checkout. This will attempt to run every tests in the +``openstack/tests/functional`` directory. For example:: -Run -*** - -Functional tests are run against both Python 2 and 3. In order to run the -entire functional test suite, run the ``tox -e functional`` and -``tox -e functional3`` command inside of your source checkout. This will -attempt to run every test command under ``/openstack/tests/functional/`` -in the source tree. You should run the full functional test suite before -submitting changes for review in order to avoid unexpected failures in -the continuous integration system.:: - - (sdk3)$ tox -e functional + $ tox -e functional ... functional: commands succeeded congratulations :) - (sdk3)$ tox -e functional3 - ... - functional3: commands succeeded - congratulations :) diff --git a/tox.ini b/tox.ini index e6f59015b..69e3869ec 100644 --- a/tox.ini +++ b/tox.ini @@ -25,7 +25,7 @@ commands = stestr run {posargs} stestr slowest -[testenv:functional] +[testenv:functional{,-py36,-py37,-py38,-py39}] # Some jobs (especially heat) takes longer, therefore increase default timeout # This timeout should not be smaller, than the longest individual timeout setenv =