diff --git a/HACKING.rst b/HACKING.rst index 57f0409ba8..a3e9c266f7 100644 --- a/HACKING.rst +++ b/HACKING.rst @@ -121,38 +121,38 @@ however they do not need to be tagged as ``compute``. Test fixtures and resources --------------------------- Test level resources should be cleaned-up after the test execution. Clean-up -is best scheduled using `addCleanup` which ensures that the resource cleanup +is best scheduled using ``addCleanup`` which ensures that the resource cleanup code is always invoked, and in reverse order with respect to the creation order. -Test class level resources should be defined in the `resource_setup` method of -the test class, except for any credential obtained from the credentials -provider, which should be set-up in the `setup_credentials` method. -Cleanup is best scheduled using `addClassResourceCleanup` which ensures that +Test class level resources should be defined in the ``resource_setup`` method +of the test class, except for any credential obtained from the credentials +provider, which should be set-up in the ``setup_credentials`` method. +Cleanup is best scheduled using ``addClassResourceCleanup`` which ensures that the cleanup code is always invoked, and in reverse order with respect to the creation order. In both cases - test level and class level cleanups - a wait loop should be scheduled before the actual delete of resources with an asynchronous delete. -The test base class `BaseTestCase` defines Tempest framework for class level -fixtures. `setUpClass` and `tearDownClass` are defined here and cannot be +The test base class ``BaseTestCase`` defines Tempest framework for class level +fixtures. ``setUpClass`` and ``tearDownClass`` are defined here and cannot be overwritten by subclasses (enforced via hacking rule T105). Set-up is split in a series of steps (setup stages), which can be overwritten by test classes. Set-up stages are: -- `skip_checks` -- `setup_credentials` -- `setup_clients` -- `resource_setup` +- ``skip_checks`` +- ``setup_credentials`` +- ``setup_clients`` +- ``resource_setup`` Tear-down is also split in a series of steps (teardown stages), which are stacked for execution only if the corresponding setup stage had been reached during the setup phase. Tear-down stages are: -- `clear_credentials` (defined in the base test class) -- `resource_cleanup` +- ``clear_credentials`` (defined in the base test class) +- ``resource_cleanup`` Skipping Tests -------------- diff --git a/doc/source/test_removal.rst b/doc/source/test_removal.rst index b57e98f893..ddae6e2ab7 100644 --- a/doc/source/test_removal.rst +++ b/doc/source/test_removal.rst @@ -62,9 +62,9 @@ and failure rates on the bottom graph. For example, something like `this URL`_. The Old Way using subunit2sql directly """""""""""""""""""""""""""""""""""""" -SELECT * from tests where test_id like "%test_id%"; -(where $test_id is the full test_id, but truncated to the class because of -setUpClass or tearDownClass failures) +``SELECT * from tests where test_id like "%test_id%";`` +(where ``$test_id`` is the full test_id, but truncated to the class because of +``setUpClass`` or ``tearDownClass`` failures) You can access the infra mysql subunit2sql db w/ read-only permissions with: @@ -74,15 +74,20 @@ You can access the infra mysql subunit2sql db w/ read-only permissions with: * db_name: subunit2sql For example if you were trying to remove the test with the id: -tempest.api.compute.admin.test_flavors_negative.FlavorsAdminNegativeTestJSON.test_get_flavor_details_for_deleted_flavor +``tempest.api.compute.admin.test_flavors_negative.FlavorsAdminNegativeTestJSON.test_get_flavor_details_for_deleted_flavor`` you would run the following: -#. run: "mysql -u query -p -h logstash.openstack.org subunit2sql" to connect - to the subunit2sql db -#. run the query: MySQL [subunit2sql]> select * from tests where test_id like - "tempest.api.compute.admin.test_flavors_negative.FlavorsAdminNegativeTestJSON%"; +#. run the command: ``mysql -u query -p -h logstash.openstack.org subunit2sql`` + to connect to the subunit2sql db +#. run the query: + + .. code-block:: console + + MySQL [subunit2sql]> select * from tests where test_id like \ + "tempest.api.compute.admin.test_flavors_negative.FlavorsAdminNegativeTestJSON%"; + which will return a table of all the tests in the class (but it will also - catch failures in setUpClass and tearDownClass) + catch failures in ``setUpClass`` and ``tearDownClass``) #. paste the output table with numbers and the mysql command you ran to generate it into the etherpad. diff --git a/tempest/cmd/account_generator.py b/tempest/cmd/account_generator.py index c2f8627ec7..92eae027b2 100755 --- a/tempest/cmd/account_generator.py +++ b/tempest/cmd/account_generator.py @@ -15,18 +15,18 @@ # under the License. """ -Utility for creating **accounts.yaml** file for concurrent test runs. +Utility for creating ``accounts.yaml`` file for concurrent test runs. Creates one primary user, one alt user, one swift admin, one stack owner and one admin (optionally) for each concurrent thread. The utility creates -user for each tenant. The **accounts.yaml** file will be valid and contain +user for each tenant. The ``accounts.yaml`` file will be valid and contain credentials for created users, so each user will be in separate tenant and have the username, tenant_name, password and roles. -**Usage:** ``tempest account-generator [-h] [OPTIONS] accounts_file.yaml``. +**Usage:** ``tempest account-generator [-h] [OPTIONS] accounts_file.yaml`` Positional Arguments -------------------- -**accounts_file.yaml** (Required) Provide an output accounts yaml file. Utility +``accounts_file.yaml`` (Required) Provide an output accounts yaml file. Utility creates a .yaml file in the directory where the command is ran. The appropriate name for the file is *accounts.yaml* and it should be placed in *tempest/etc* directory. @@ -40,62 +40,62 @@ through CLI options or environment variables. You're probably familiar with these, but just to remind: -======== ======================== ==================== -Param CLI Environment Variable -======== ======================== ==================== -Username --os-username OS_USERNAME -Password --os-password OS_PASSWORD -Project --os-project-name OS_PROJECT_NAME -Tenant --os-tenant-name (depr.) OS_TENANT_NAME -Domain --os-domain-name OS_DOMAIN_NAME -======== ======================== ==================== +======== ============================ ==================== +Param CLI Environment Variable +======== ============================ ==================== +Username ``--os-username`` OS_USERNAME +Password ``--os-password`` OS_PASSWORD +Project ``--os-project-name`` OS_PROJECT_NAME +Tenant ``--os-tenant-name`` (depr.) OS_TENANT_NAME +Domain ``--os-domain-name`` OS_DOMAIN_NAME +======== ============================ ==================== Optional Arguments ------------------ -**-h**, **--help** (Optional) Shows help message with the description of -utility and its arguments, and exits. +* ``-h, --help`` (Optional) Shows help message with the description of + utility and its arguments, and exits. -**-c /etc/tempest.conf**, **--config-file /etc/tempest.conf** (Optional) Path -to tempest config file. If not specified, it searches for tempest.conf in these -locations: +* ``-c, --config-file /etc/tempest.conf`` (Optional) Path + to tempest config file. If not specified, it searches for tempest.conf in + these locations: -- ./etc/ -- /etc/tempest -- ~/.tempest/ -- ~/ -- /etc/ + - ./etc/ + - /etc/tempest + - ~/.tempest/ + - ~/ + - /etc/ -**--os-username ** (Optional) Name used for authentication with -the OpenStack Identity service. Defaults to env[OS_USERNAME]. Note: User should -have permissions to create new user accounts and tenants. +* ``--os-username `` (Optional) Name used for authentication + with the OpenStack Identity service. Defaults to env[OS_USERNAME]. Note: User + should have permissions to create new user accounts and tenants. -**--os-password ** (Optional) Password used for authentication -with the OpenStack Identity service. Defaults to env[OS_PASSWORD]. +* ``--os-password `` (Optional) Password used for authentication + with the OpenStack Identity service. Defaults to env[OS_PASSWORD]. -**--os-project-name ** (Optional) Project to request -authorization on. Defaults to env[OS_PROJECT_NAME]. +* ``--os-project-name `` (Optional) Project to request + authorization on. Defaults to env[OS_PROJECT_NAME]. -**--os-tenant-name ** (Optional, deprecated) Tenant to -request authorization on. Defaults to env[OS_TENANT_NAME]. +* ``--os-tenant-name `` (Optional, deprecated) Tenant to + request authorization on. Defaults to env[OS_TENANT_NAME]. -**--os-domain-name ** (Optional) Domain the user and project -belong to. Defaults to env[OS_DOMAIN_NAME]. +* ``--os-domain-name `` (Optional) Domain the user and + project belong to. Defaults to env[OS_DOMAIN_NAME]. -**--tag TAG** (Optional) Resources tag. Each created resource (user, project) -will have the prefix with the given TAG in its name. Using tag is recommended -for the further using, cleaning resources. +* ``--tag TAG`` (Optional) Resources tag. Each created resource (user, project) + will have the prefix with the given TAG in its name. Using tag is recommended + for the further using, cleaning resources. -**-r CONCURRENCY**, **--concurrency CONCURRENCY** (Optional) Concurrency count -(default: 1). The number of accounts required can be estimated as -CONCURRENCY x 2. Each user provided in *accounts.yaml* file will be in -a different tenant. This is required to provide isolation between test for -running in parallel. +* ``-r, --concurrency CONCURRENCY`` (Optional) Concurrency count + (default: 1). The number of accounts required can be estimated as + CONCURRENCY x 2. Each user provided in *accounts.yaml* file will be in + a different tenant. This is required to provide isolation between test for + running in parallel. -**--with-admin** (Optional) Creates admin for each concurrent group -(default: False). +* ``--with-admin`` (Optional) Creates admin for each concurrent group + (default: False). -**-i VERSION**, **--identity-version VERSION** (Optional) Provisions accounts -using the specified version of the identity API. (default: '3'). +* ``-i, --identity-version VERSION`` (Optional) Provisions accounts + using the specified version of the identity API. (default: '3'). To see help on specific argument, please do: ``tempest account-generator [OPTIONS] -h``. diff --git a/tempest/cmd/cleanup.py b/tempest/cmd/cleanup.py index d0aa7dcddd..29abd49098 100644 --- a/tempest/cmd/cleanup.py +++ b/tempest/cmd/cleanup.py @@ -28,45 +28,48 @@ specified in ``tempest.conf`` is never deleted. Example Run ----------- -**WARNING: If step 1 is skipped in the example below, the cleanup procedure -may delete resources that existed in the cloud before the test run. This -may cause an unwanted destruction of cloud resources, so use caution with -this command.** +.. warning:: -``$ tempest cleanup --init-saved-state`` + If step 1 is skipped in the example below, the cleanup procedure + may delete resources that existed in the cloud before the test run. This + may cause an unwanted destruction of cloud resources, so use caution with + this command. -``$ # Actual running of Tempest tests`` + Examples:: -``$ tempest cleanup`` + $ tempest cleanup --init-saved-state + $ # Actual running of Tempest tests + $ tempest cleanup Runtime Arguments ----------------- -**--init-saved-state**: Initializes the saved state of the OpenStack deployment -and will output a ``saved_state.json`` file containing resources from your -deployment that will be preserved from the cleanup command. This should be -done prior to running Tempest tests. +* ``--init-saved-state``: Initializes the saved state of the OpenStack + deployment and will output a ``saved_state.json`` file containing resources + from your deployment that will be preserved from the cleanup command. This + should be done prior to running Tempest tests. -**--delete-tempest-conf-objects**: If option is present, then the command will -delete the admin project in addition to the resources associated with them on -clean up. If option is not present, the command will delete the resources -associated with the Tempest and alternate Tempest users and projects but will -not delete the projects themselves. +* ``--delete-tempest-conf-objects``: If option is present, then the command + will delete the admin project in addition to the resources associated with + them on clean up. If option is not present, the command will delete the + resources associated with the Tempest and alternate Tempest users and + projects but will not delete the projects themselves. -**--dry-run**: Creates a report (``./dry_run.json``) of the projects that will -be cleaned up (in the ``_projects_to_clean`` dictionary [1]_) and the global -objects that will be removed (domains, flavors, images, roles, projects, -and users). Once the cleanup command is executed (e.g. run without -parameters), running it again with **--dry-run** should yield an empty report. +* ``--dry-run``: Creates a report (``./dry_run.json``) of the projects that + will be cleaned up (in the ``_projects_to_clean`` dictionary [1]_) and the + global objects that will be removed (domains, flavors, images, roles, + projects, and users). Once the cleanup command is executed (e.g. run without + parameters), running it again with ``--dry-run`` should yield an empty + report. -**--help**: Print the help text for the command and parameters. +* ``--help``: Print the help text for the command and parameters. .. [1] The ``_projects_to_clean`` dictionary in ``dry_run.json`` lists the projects that ``tempest cleanup`` will loop through to delete child objects, but the command will, by default, not delete the projects themselves. This may differ from the ``projects`` list as you can clean the Tempest and alternate Tempest users and projects but they will not be - deleted unless the **--delete-tempest-conf-objects** flag is used to + deleted unless the ``--delete-tempest-conf-objects`` flag is used to force their deletion. """ diff --git a/tempest/cmd/run.py b/tempest/cmd/run.py index f07f197c27..0d847bd218 100644 --- a/tempest/cmd/run.py +++ b/tempest/cmd/run.py @@ -19,11 +19,11 @@ Test Selection ============== Tempest run has several options: - * **--regex/-r**: This is a selection regex like what testr uses. It will run - any tests that match on re.match() with the regex - * **--smoke/-s**: Run all the tests tagged as smoke +* ``--regex, -r``: This is a selection regex like what testr uses. It will run + any tests that match on re.match() with the regex +* ``--smoke, -s``: Run all the tests tagged as smoke -There are also the **--blacklist-file** and **--whitelist-file** options that +There are also the ``--blacklist-file`` and ``--whitelist-file`` options that let you pass a filepath to tempest run with the file format being a line separated regex, with '#' used to signify the start of a comment on a line. For example:: @@ -44,21 +44,21 @@ something like:: When combined with a whitelist file all the regexes from the file and the CLI regexes will be ORed. -You can also use the **--list-tests** option in conjunction with selection +You can also use the ``--list-tests`` option in conjunction with selection arguments to list which tests will be run. -You can also use the **--load-list** option that lets you pass a filepath to +You can also use the ``--load-list`` option that lets you pass a filepath to tempest run with the file format being in a non-regex format, similar to the -tests generated by the **--list-tests** option. You can specify target tests +tests generated by the ``--list-tests`` option. You can specify target tests by removing unnecessary tests from a list file which is generated from -**--list-tests** option. +``--list-tests`` option. Test Execution ============== There are several options to control how the tests are executed. By default tempest will run in parallel with a worker for each CPU present on the machine. -If you want to adjust the number of workers use the **--concurrency** option -and if you want to run tests serially use **--serial/-t** +If you want to adjust the number of workers use the ``--concurrency`` option +and if you want to run tests serially use ``--serial/-t`` Running with Workspaces ----------------------- @@ -82,7 +82,7 @@ Test Output =========== By default tempest run's output to STDOUT will be generated using the subunit-trace output filter. But, if you would prefer a subunit v2 stream be -output to STDOUT use the **--subunit** flag +output to STDOUT use the ``--subunit`` flag Combining Runs ============== @@ -90,7 +90,7 @@ Combining Runs There are certain situations in which you want to split a single run of tempest across 2 executions of tempest run. (for example to run part of the tests serially and others in parallel) To accomplish this but still treat the results -as a single run you can leverage the **--combine** option which will append +as a single run you can leverage the ``--combine`` option which will append the current run's results with the previous runs. """ diff --git a/tempest/cmd/subunit_describe_calls.py b/tempest/cmd/subunit_describe_calls.py index f9ebe20309..f0ade7ea10 100644 --- a/tempest/cmd/subunit_describe_calls.py +++ b/tempest/cmd/subunit_describe_calls.py @@ -21,17 +21,14 @@ API calls are made inside of a test and in what order they are called. Runtime Arguments ----------------- -**--subunit, -s**: (Optional) The path to the subunit file being parsed, -defaults to stdin - -**--non-subunit-name, -n**: (Optional) The file_name that the logs are being -stored in - -**--output-file, -o**: (Optional) The path where the JSON output will be -written to. This contains more information than is present in stdout. - -**--ports, -p**: (Optional) The path to a JSON file describing the ports being -used by different services +* ``--subunit, -s``: (Optional) The path to the subunit file being parsed, + defaults to stdin +* ``--non-subunit-name, -n``: (Optional) The file_name that the logs are being + stored in +* ``--output-file, -o``: (Optional) The path where the JSON output will be + written to. This contains more information than is present in stdout. +* ``--ports, -p``: (Optional) The path to a JSON file describing the ports + being used by different services Usage ----- diff --git a/tempest/cmd/workspace.py b/tempest/cmd/workspace.py index 8166b4f8ca..929a584d62 100644 --- a/tempest/cmd/workspace.py +++ b/tempest/cmd/workspace.py @@ -26,28 +26,28 @@ Outputs the name and path of all known tempest workspaces register -------- -Registers a new tempest workspace via a given --name and --path +Registers a new tempest workspace via a given ``--name`` and ``--path`` rename ------ -Renames a tempest workspace from --old-name to --new-name +Renames a tempest workspace from ``--old-name`` to ``--new-name`` move ---- -Changes the path of a given tempest workspace --name to --path +Changes the path of a given tempest workspace ``--name`` to ``--path`` remove ------ -Deletes the entry for a given tempest workspace --name +Deletes the entry for a given tempest workspace ``--name`` ---rmdir Deletes the given tempest workspace directory +``--rmdir`` Deletes the given tempest workspace directory General Options =============== - **--workspace_path**: Allows the user to specify a different location for the - workspace.yaml file containing the workspace definitions - instead of ~/.tempest/workspace.yaml +* ``--workspace_path``: Allows the user to specify a different location for the + workspace.yaml file containing the workspace definitions instead of + ``~/.tempest/workspace.yaml`` """ import os diff --git a/tempest/scenario/README.rst b/tempest/scenario/README.rst index ad300c2266..efcd1394eb 100644 --- a/tempest/scenario/README.rst +++ b/tempest/scenario/README.rst @@ -15,6 +15,7 @@ multiple OpenStack services to exercise the touch points between them. Any scenario test should have a real-life use case. An example would be: - "As operator I want to start with a blank environment": + 1. upload a glance image 2. deploy a vm from it 3. ssh to the guest