openstack/tempest
Code Issues Proposed changes

Merge "Fix docs markup consistency"

Browse Source
This commit is contained in:
Zuul 2017-11-30 23:30:00 +00:00 committed by Gerrit Code Review
commit dba764c8f3
8 changed files with 128 additions and 122 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

26
HACKING.rst
View File

@ -121,38 +121,38 @@ however they do not need to be tagged as ``compute``.
Test fixtures and resources Test fixtures and resources
--------------------------- ---------------------------
Test level resources should be cleaned-up after the test execution. Clean-up Test level resources should be cleaned-up after the test execution. Clean-up
is best scheduled using `addCleanup` which ensures that the resource cleanup is best scheduled using ``addCleanup`` which ensures that the resource cleanup
code is always invoked, and in reverse order with respect to the creation code is always invoked, and in reverse order with respect to the creation
order. order.
Test class level resources should be defined in the `resource_setup` method of Test class level resources should be defined in the ``resource_setup`` method
the test class, except for any credential obtained from the credentials of the test class, except for any credential obtained from the credentials
provider, which should be set-up in the `setup_credentials` method. provider, which should be set-up in the ``setup_credentials`` method.
Cleanup is best scheduled using `addClassResourceCleanup` which ensures that Cleanup is best scheduled using ``addClassResourceCleanup`` which ensures that
the cleanup code is always invoked, and in reverse order with respect to the the cleanup code is always invoked, and in reverse order with respect to the
creation order. creation order.
In both cases - test level and class level cleanups - a wait loop should be In both cases - test level and class level cleanups - a wait loop should be
scheduled before the actual delete of resources with an asynchronous delete. scheduled before the actual delete of resources with an asynchronous delete.
The test base class `BaseTestCase` defines Tempest framework for class level The test base class ``BaseTestCase`` defines Tempest framework for class level
fixtures. `setUpClass` and `tearDownClass` are defined here and cannot be fixtures. ``setUpClass`` and ``tearDownClass`` are defined here and cannot be
overwritten by subclasses (enforced via hacking rule T105). overwritten by subclasses (enforced via hacking rule T105).
Set-up is split in a series of steps (setup stages), which can be overwritten Set-up is split in a series of steps (setup stages), which can be overwritten
by test classes. Set-up stages are: by test classes. Set-up stages are:
- `skip_checks` - ``skip_checks``
- `setup_credentials` - ``setup_credentials``
- `setup_clients` - ``setup_clients``
- `resource_setup` - ``resource_setup``
Tear-down is also split in a series of steps (teardown stages), which are Tear-down is also split in a series of steps (teardown stages), which are
stacked for execution only if the corresponding setup stage had been stacked for execution only if the corresponding setup stage had been
reached during the setup phase. Tear-down stages are: reached during the setup phase. Tear-down stages are:
- `clear_credentials` (defined in the base test class) - ``clear_credentials`` (defined in the base test class)
- `resource_cleanup` - ``resource_cleanup``
Skipping Tests Skipping Tests
-------------- --------------

23
doc/source/test_removal.rst
View File

@ -62,9 +62,9 @@ and failure rates on the bottom graph. For example, something like `this URL`_.
The Old Way using subunit2sql directly The Old Way using subunit2sql directly
"""""""""""""""""""""""""""""""""""""" """"""""""""""""""""""""""""""""""""""
SELECT * from tests where test_id like "%test_id%"; ``SELECT * from tests where test_id like "%test_id%";``
(where $test_id is the full test_id, but truncated to the class because of (where ``$test_id`` is the full test_id, but truncated to the class because of
setUpClass or tearDownClass failures) ``setUpClass`` or ``tearDownClass`` failures)
You can access the infra mysql subunit2sql db w/ read-only permissions with: You can access the infra mysql subunit2sql db w/ read-only permissions with:
@ -74,15 +74,20 @@ You can access the infra mysql subunit2sql db w/ read-only permissions with:
* db_name: subunit2sql * db_name: subunit2sql
For example if you were trying to remove the test with the id: For example if you were trying to remove the test with the id:
tempest.api.compute.admin.test_flavors_negative.FlavorsAdminNegativeTestJSON.test_get_flavor_details_for_deleted_flavor ``tempest.api.compute.admin.test_flavors_negative.FlavorsAdminNegativeTestJSON.test_get_flavor_details_for_deleted_flavor``
you would run the following: you would run the following:
#. run: "mysql -u query -p -h logstash.openstack.org subunit2sql" to connect #. run the command: ``mysql -u query -p -h logstash.openstack.org subunit2sql``
to the subunit2sql db to connect to the subunit2sql db
#. run the query: MySQL [subunit2sql]> select * from tests where test_id like #. run the query:
"tempest.api.compute.admin.test_flavors_negative.FlavorsAdminNegativeTestJSON%";
.. code-block:: console
MySQL [subunit2sql]> select * from tests where test_id like \
"tempest.api.compute.admin.test_flavors_negative.FlavorsAdminNegativeTestJSON%";
which will return a table of all the tests in the class (but it will also which will return a table of all the tests in the class (but it will also
catch failures in setUpClass and tearDownClass) catch failures in ``setUpClass`` and ``tearDownClass``)
#. paste the output table with numbers and the mysql command you ran to #. paste the output table with numbers and the mysql command you ran to
generate it into the etherpad. generate it into the etherpad.

92
tempest/cmd/account_generator.py
View File

@ -15,18 +15,18 @@
# under the License. # under the License.
""" """
Utility for creating **accounts.yaml** file for concurrent test runs. Utility for creating ``accounts.yaml`` file for concurrent test runs.
Creates one primary user, one alt user, one swift admin, one stack owner Creates one primary user, one alt user, one swift admin, one stack owner
and one admin (optionally) for each concurrent thread. The utility creates and one admin (optionally) for each concurrent thread. The utility creates
user for each tenant. The **accounts.yaml** file will be valid and contain user for each tenant. The ``accounts.yaml`` file will be valid and contain
credentials for created users, so each user will be in separate tenant and credentials for created users, so each user will be in separate tenant and
have the username, tenant_name, password and roles. have the username, tenant_name, password and roles.
**Usage:** ``tempest account-generator [-h] [OPTIONS] accounts_file.yaml``. **Usage:** ``tempest account-generator [-h] [OPTIONS] accounts_file.yaml``
Positional Arguments Positional Arguments
-------------------- --------------------
**accounts_file.yaml** (Required) Provide an output accounts yaml file. Utility ``accounts_file.yaml`` (Required) Provide an output accounts yaml file. Utility
creates a .yaml file in the directory where the command is ran. The appropriate creates a .yaml file in the directory where the command is ran. The appropriate
name for the file is *accounts.yaml* and it should be placed in *tempest/etc* name for the file is *accounts.yaml* and it should be placed in *tempest/etc*
directory. directory.
@ -40,62 +40,62 @@ through CLI options or environment variables.
You're probably familiar with these, but just to remind: You're probably familiar with these, but just to remind:
======== ======================== ==================== ======== ============================ ====================
Param CLI Environment Variable Param CLI Environment Variable
======== ======================== ==================== ======== ============================ ====================
Username --os-username OS_USERNAME Username ``--os-username`` OS_USERNAME
Password --os-password OS_PASSWORD Password ``--os-password`` OS_PASSWORD
Project --os-project-name OS_PROJECT_NAME Project ``--os-project-name`` OS_PROJECT_NAME
Tenant --os-tenant-name (depr.) OS_TENANT_NAME Tenant ``--os-tenant-name`` (depr.) OS_TENANT_NAME
Domain --os-domain-name OS_DOMAIN_NAME Domain ``--os-domain-name`` OS_DOMAIN_NAME
======== ======================== ==================== ======== ============================ ====================
Optional Arguments Optional Arguments
------------------ ------------------
**-h**, **--help** (Optional) Shows help message with the description of * ``-h, --help`` (Optional) Shows help message with the description of
utility and its arguments, and exits. utility and its arguments, and exits.
**-c /etc/tempest.conf**, **--config-file /etc/tempest.conf** (Optional) Path * ``-c, --config-file /etc/tempest.conf`` (Optional) Path
to tempest config file. If not specified, it searches for tempest.conf in these to tempest config file. If not specified, it searches for tempest.conf in
locations: these locations:
- ./etc/ - ./etc/
- /etc/tempest - /etc/tempest
- ~/.tempest/ - ~/.tempest/
- ~/ - ~/
- /etc/ - /etc/
**--os-username <auth-user-name>** (Optional) Name used for authentication with * ``--os-username <auth-user-name>`` (Optional) Name used for authentication
the OpenStack Identity service. Defaults to env[OS_USERNAME]. Note: User should with the OpenStack Identity service. Defaults to env[OS_USERNAME]. Note: User
have permissions to create new user accounts and tenants. should have permissions to create new user accounts and tenants.
**--os-password <auth-password>** (Optional) Password used for authentication * ``--os-password <auth-password>`` (Optional) Password used for authentication
with the OpenStack Identity service. Defaults to env[OS_PASSWORD]. with the OpenStack Identity service. Defaults to env[OS_PASSWORD].
**--os-project-name <auth-project-name>** (Optional) Project to request * ``--os-project-name <auth-project-name>`` (Optional) Project to request
authorization on. Defaults to env[OS_PROJECT_NAME]. authorization on. Defaults to env[OS_PROJECT_NAME].
**--os-tenant-name <auth-tenant-name>** (Optional, deprecated) Tenant to * ``--os-tenant-name <auth-tenant-name>`` (Optional, deprecated) Tenant to
request authorization on. Defaults to env[OS_TENANT_NAME]. request authorization on. Defaults to env[OS_TENANT_NAME].
**--os-domain-name <auth-domain-name>** (Optional) Domain the user and project * ``--os-domain-name <auth-domain-name>`` (Optional) Domain the user and
belong to. Defaults to env[OS_DOMAIN_NAME]. project belong to. Defaults to env[OS_DOMAIN_NAME].
**--tag TAG** (Optional) Resources tag. Each created resource (user, project) * ``--tag TAG`` (Optional) Resources tag. Each created resource (user, project)
will have the prefix with the given TAG in its name. Using tag is recommended will have the prefix with the given TAG in its name. Using tag is recommended
for the further using, cleaning resources. for the further using, cleaning resources.
**-r CONCURRENCY**, **--concurrency CONCURRENCY** (Optional) Concurrency count * ``-r, --concurrency CONCURRENCY`` (Optional) Concurrency count
(default: 1). The number of accounts required can be estimated as (default: 1). The number of accounts required can be estimated as
CONCURRENCY x 2. Each user provided in *accounts.yaml* file will be in CONCURRENCY x 2. Each user provided in *accounts.yaml* file will be in
a different tenant. This is required to provide isolation between test for a different tenant. This is required to provide isolation between test for
running in parallel. running in parallel.
**--with-admin** (Optional) Creates admin for each concurrent group * ``--with-admin`` (Optional) Creates admin for each concurrent group
(default: False). (default: False).
**-i VERSION**, **--identity-version VERSION** (Optional) Provisions accounts * ``-i, --identity-version VERSION`` (Optional) Provisions accounts
using the specified version of the identity API. (default: '3'). using the specified version of the identity API. (default: '3').
To see help on specific argument, please do: ``tempest account-generator To see help on specific argument, please do: ``tempest account-generator
[OPTIONS] <accounts_file.yaml> -h``. [OPTIONS] <accounts_file.yaml> -h``.

49
tempest/cmd/cleanup.py
View File

@ -28,45 +28,48 @@ specified in ``tempest.conf`` is never deleted.
Example Run Example Run
----------- -----------
**WARNING: If step 1 is skipped in the example below, the cleanup procedure .. warning::
may delete resources that existed in the cloud before the test run. This
may cause an unwanted destruction of cloud resources, so use caution with
this command.**
``$ tempest cleanup --init-saved-state`` If step 1 is skipped in the example below, the cleanup procedure
may delete resources that existed in the cloud before the test run. This
may cause an unwanted destruction of cloud resources, so use caution with
this command.
``$ # Actual running of Tempest tests`` Examples::
``$ tempest cleanup`` $ tempest cleanup --init-saved-state
$ # Actual running of Tempest tests
$ tempest cleanup
Runtime Arguments Runtime Arguments
----------------- -----------------
**--init-saved-state**: Initializes the saved state of the OpenStack deployment * ``--init-saved-state``: Initializes the saved state of the OpenStack
and will output a ``saved_state.json`` file containing resources from your deployment and will output a ``saved_state.json`` file containing resources
deployment that will be preserved from the cleanup command. This should be from your deployment that will be preserved from the cleanup command. This
done prior to running Tempest tests. should be done prior to running Tempest tests.
**--delete-tempest-conf-objects**: If option is present, then the command will * ``--delete-tempest-conf-objects``: If option is present, then the command
delete the admin project in addition to the resources associated with them on will delete the admin project in addition to the resources associated with
clean up. If option is not present, the command will delete the resources them on clean up. If option is not present, the command will delete the
associated with the Tempest and alternate Tempest users and projects but will resources associated with the Tempest and alternate Tempest users and
not delete the projects themselves. projects but will not delete the projects themselves.
**--dry-run**: Creates a report (``./dry_run.json``) of the projects that will * ``--dry-run``: Creates a report (``./dry_run.json``) of the projects that
be cleaned up (in the ``_projects_to_clean`` dictionary [1]_) and the global will be cleaned up (in the ``_projects_to_clean`` dictionary [1]_) and the
objects that will be removed (domains, flavors, images, roles, projects, global objects that will be removed (domains, flavors, images, roles,
and users). Once the cleanup command is executed (e.g. run without projects, and users). Once the cleanup command is executed (e.g. run without
parameters), running it again with **--dry-run** should yield an empty report. parameters), running it again with ``--dry-run`` should yield an empty
report.
**--help**: Print the help text for the command and parameters. * ``--help``: Print the help text for the command and parameters.
.. [1] The ``_projects_to_clean`` dictionary in ``dry_run.json`` lists the .. [1] The ``_projects_to_clean`` dictionary in ``dry_run.json`` lists the
projects that ``tempest cleanup`` will loop through to delete child projects that ``tempest cleanup`` will loop through to delete child
objects, but the command will, by default, not delete the projects objects, but the command will, by default, not delete the projects
themselves. This may differ from the ``projects`` list as you can clean themselves. This may differ from the ``projects`` list as you can clean
the Tempest and alternate Tempest users and projects but they will not be the Tempest and alternate Tempest users and projects but they will not be
deleted unless the **--delete-tempest-conf-objects** flag is used to deleted unless the ``--delete-tempest-conf-objects`` flag is used to
force their deletion. force their deletion.
""" """

24
tempest/cmd/run.py
View File

@ -19,11 +19,11 @@ Test Selection
============== ==============
Tempest run has several options: Tempest run has several options:
* **--regex/-r**: This is a selection regex like what testr uses. It will run * ``--regex, -r``: This is a selection regex like what testr uses. It will run
any tests that match on re.match() with the regex any tests that match on re.match() with the regex
* **--smoke/-s**: Run all the tests tagged as smoke * ``--smoke, -s``: Run all the tests tagged as smoke
There are also the **--blacklist-file** and **--whitelist-file** options that There are also the ``--blacklist-file`` and ``--whitelist-file`` options that
let you pass a filepath to tempest run with the file format being a line let you pass a filepath to tempest run with the file format being a line
separated regex, with '#' used to signify the start of a comment on a line. separated regex, with '#' used to signify the start of a comment on a line.
For example:: For example::
@ -44,21 +44,21 @@ something like::
When combined with a whitelist file all the regexes from the file and the CLI When combined with a whitelist file all the regexes from the file and the CLI
regexes will be ORed. regexes will be ORed.
You can also use the **--list-tests** option in conjunction with selection You can also use the ``--list-tests`` option in conjunction with selection
arguments to list which tests will be run. arguments to list which tests will be run.
You can also use the **--load-list** option that lets you pass a filepath to You can also use the ``--load-list`` option that lets you pass a filepath to
tempest run with the file format being in a non-regex format, similar to the tempest run with the file format being in a non-regex format, similar to the
tests generated by the **--list-tests** option. You can specify target tests tests generated by the ``--list-tests`` option. You can specify target tests
by removing unnecessary tests from a list file which is generated from by removing unnecessary tests from a list file which is generated from
**--list-tests** option. ``--list-tests`` option.
Test Execution Test Execution
============== ==============
There are several options to control how the tests are executed. By default There are several options to control how the tests are executed. By default
tempest will run in parallel with a worker for each CPU present on the machine. tempest will run in parallel with a worker for each CPU present on the machine.
If you want to adjust the number of workers use the **--concurrency** option If you want to adjust the number of workers use the ``--concurrency`` option
and if you want to run tests serially use **--serial/-t** and if you want to run tests serially use ``--serial/-t``
Running with Workspaces Running with Workspaces
----------------------- -----------------------
@ -82,7 +82,7 @@ Test Output
=========== ===========
By default tempest run's output to STDOUT will be generated using the By default tempest run's output to STDOUT will be generated using the
subunit-trace output filter. But, if you would prefer a subunit v2 stream be subunit-trace output filter. But, if you would prefer a subunit v2 stream be
output to STDOUT use the **--subunit** flag output to STDOUT use the ``--subunit`` flag
Combining Runs Combining Runs
============== ==============
@ -90,7 +90,7 @@ Combining Runs
There are certain situations in which you want to split a single run of tempest There are certain situations in which you want to split a single run of tempest
across 2 executions of tempest run. (for example to run part of the tests across 2 executions of tempest run. (for example to run part of the tests
serially and others in parallel) To accomplish this but still treat the results serially and others in parallel) To accomplish this but still treat the results
as a single run you can leverage the **--combine** option which will append as a single run you can leverage the ``--combine`` option which will append
the current run's results with the previous runs. the current run's results with the previous runs.
""" """

19
tempest/cmd/subunit_describe_calls.py
View File

@ -21,17 +21,14 @@ API calls are made inside of a test and in what order they are called.
Runtime Arguments Runtime Arguments
----------------- -----------------
**--subunit, -s**: (Optional) The path to the subunit file being parsed, * ``--subunit, -s``: (Optional) The path to the subunit file being parsed,
defaults to stdin defaults to stdin
* ``--non-subunit-name, -n``: (Optional) The file_name that the logs are being
**--non-subunit-name, -n**: (Optional) The file_name that the logs are being stored in
stored in * ``--output-file, -o``: (Optional) The path where the JSON output will be
written to. This contains more information than is present in stdout.
**--output-file, -o**: (Optional) The path where the JSON output will be * ``--ports, -p``: (Optional) The path to a JSON file describing the ports
written to. This contains more information than is present in stdout. being used by different services
**--ports, -p**: (Optional) The path to a JSON file describing the ports being
used by different services
Usage Usage
----- -----

16
tempest/cmd/workspace.py
View File

@ -26,28 +26,28 @@ Outputs the name and path of all known tempest workspaces
register register
-------- --------
Registers a new tempest workspace via a given --name and --path Registers a new tempest workspace via a given ``--name`` and ``--path``
rename rename
------ ------
Renames a tempest workspace from --old-name to --new-name Renames a tempest workspace from ``--old-name`` to ``--new-name``
move move
---- ----
Changes the path of a given tempest workspace --name to --path Changes the path of a given tempest workspace ``--name`` to ``--path``
remove remove
------ ------
Deletes the entry for a given tempest workspace --name Deletes the entry for a given tempest workspace ``--name``
--rmdir Deletes the given tempest workspace directory ``--rmdir`` Deletes the given tempest workspace directory
General Options General Options
=============== ===============
**--workspace_path**: Allows the user to specify a different location for the * ``--workspace_path``: Allows the user to specify a different location for the
workspace.yaml file containing the workspace definitions workspace.yaml file containing the workspace definitions instead of
instead of ~/.tempest/workspace.yaml ``~/.tempest/workspace.yaml``
""" """
import os import os

1
tempest/scenario/README.rst
View File

@ -15,6 +15,7 @@ multiple OpenStack services to exercise the touch points between them.
Any scenario test should have a real-life use case. An example would be: Any scenario test should have a real-life use case. An example would be:
- "As operator I want to start with a blank environment": - "As operator I want to start with a blank environment":
1. upload a glance image 1. upload a glance image
2. deploy a vm from it 2. deploy a vm from it
3. ssh to the guest 3. ssh to the guest