Fix indentation in docs
This commit fixes indentation in tempest docs. These indentations are not necessary and it causes a weird html outputs. Change-Id: I9c8714558a3327b7ad0b0ab0d3fdc7e770c3c75b
This commit is contained in:
parent
10d9e73349
commit
b78b923e5a
@ -9,14 +9,14 @@ Tempest Specific Commandments
|
|||||||
------------------------------
|
------------------------------
|
||||||
|
|
||||||
- [T102] Cannot import OpenStack python clients in tempest/api &
|
- [T102] Cannot import OpenStack python clients in tempest/api &
|
||||||
tempest/scenario tests
|
tempest/scenario tests
|
||||||
- [T104] Scenario tests require a services decorator
|
- [T104] Scenario tests require a services decorator
|
||||||
- [T105] Tests cannot use setUpClass/tearDownClass
|
- [T105] Tests cannot use setUpClass/tearDownClass
|
||||||
- [T106] vim configuration should not be kept in source files.
|
- [T106] vim configuration should not be kept in source files.
|
||||||
- [T107] Check that a service tag isn't in the module path
|
- [T107] Check that a service tag isn't in the module path
|
||||||
- [T108] Check no hyphen at the end of rand_name() argument
|
- [T108] Check no hyphen at the end of rand_name() argument
|
||||||
- [T109] Cannot use testtools.skip decorator; instead use
|
- [T109] Cannot use testtools.skip decorator; instead use
|
||||||
decorators.skip_because from tempest.lib
|
decorators.skip_because from tempest.lib
|
||||||
- [T110] Check that service client names of GET should be consistent
|
- [T110] Check that service client names of GET should be consistent
|
||||||
- [T111] Check that service client names of DELETE should be consistent
|
- [T111] Check that service client names of DELETE should be consistent
|
||||||
- [T112] Check that tempest.lib should not import local tempest code
|
- [T112] Check that tempest.lib should not import local tempest code
|
||||||
|
@ -121,8 +121,8 @@ fix it. When it will happen, we will deal with it on a case-by-case basis.
|
|||||||
|
|
||||||
When to approve
|
When to approve
|
||||||
---------------
|
---------------
|
||||||
* Every patch needs two +2s before being approved.
|
* Every patch needs two +2s before being approved.
|
||||||
* Its ok to hold off on an approval until a subject matter expert reviews it
|
* Its ok to hold off on an approval until a subject matter expert reviews it
|
||||||
* If a patch has already been approved but requires a trivial rebase to merge,
|
* If a patch has already been approved but requires a trivial rebase to merge,
|
||||||
you do not have to wait for a second +2, since the patch has already had
|
you do not have to wait for a second +2, since the patch has already had
|
||||||
two +2s.
|
two +2s.
|
||||||
|
@ -17,10 +17,10 @@ Test Credentials
|
|||||||
Tempest allows for configuring a set of admin credentials in the ``auth``
|
Tempest allows for configuring a set of admin credentials in the ``auth``
|
||||||
section, via the following parameters:
|
section, via the following parameters:
|
||||||
|
|
||||||
#. ``admin_username``
|
#. ``admin_username``
|
||||||
#. ``admin_password``
|
#. ``admin_password``
|
||||||
#. ``admin_project_name``
|
#. ``admin_project_name``
|
||||||
#. ``admin_domain_name``
|
#. ``admin_domain_name``
|
||||||
|
|
||||||
Admin credentials are not mandatory to run Tempest, but when provided they
|
Admin credentials are not mandatory to run Tempest, but when provided they
|
||||||
can be used to:
|
can be used to:
|
||||||
@ -47,9 +47,9 @@ In order for Tempest to be able to talk to your OpenStack deployment you need
|
|||||||
to provide it with information about how it communicates with keystone.
|
to provide it with information about how it communicates with keystone.
|
||||||
This involves configuring the following options in the ``identity`` section:
|
This involves configuring the following options in the ``identity`` section:
|
||||||
|
|
||||||
- ``auth_version``
|
- ``auth_version``
|
||||||
- ``uri``
|
- ``uri``
|
||||||
- ``uri_v3``
|
- ``uri_v3``
|
||||||
|
|
||||||
The ``auth_version`` option is used to tell Tempest whether it should be using
|
The ``auth_version`` option is used to tell Tempest whether it should be using
|
||||||
Keystone's v2 or v3 api for communicating with Keystone. The two uri options are
|
Keystone's v2 or v3 api for communicating with Keystone. The two uri options are
|
||||||
@ -74,12 +74,12 @@ three sets of username, password, and project names for a primary user,
|
|||||||
an admin user, and an alternate user. To enable and use dynamic credentials you
|
an admin user, and an alternate user. To enable and use dynamic credentials you
|
||||||
only need to configure two things:
|
only need to configure two things:
|
||||||
|
|
||||||
#. A set of admin credentials with permissions to create users and
|
#. A set of admin credentials with permissions to create users and
|
||||||
projects. This is specified in the ``auth`` section with the
|
projects. This is specified in the ``auth`` section with the
|
||||||
``admin_username``, ``admin_project_name``, ``admin_domain_name`` and
|
``admin_username``, ``admin_project_name``, ``admin_domain_name`` and
|
||||||
``admin_password`` options
|
``admin_password`` options
|
||||||
#. To enable dynamic credentials in the ``auth`` section with the
|
#. To enable dynamic credentials in the ``auth`` section with the
|
||||||
``use_dynamic_credentials`` option.
|
``use_dynamic_credentials`` option.
|
||||||
|
|
||||||
This is also currently the default credential provider enabled by Tempest, due
|
This is also currently the default credential provider enabled by Tempest, due
|
||||||
to its common use and ease of configuration.
|
to its common use and ease of configuration.
|
||||||
@ -115,21 +115,21 @@ like with dynamic credentials.
|
|||||||
|
|
||||||
To enable and use locking test accounts you need do a few things:
|
To enable and use locking test accounts you need do a few things:
|
||||||
|
|
||||||
#. Create an accounts.yaml file which contains the set of pre-existing
|
#. Create an accounts.yaml file which contains the set of pre-existing
|
||||||
credentials to use for testing. To make sure you don't have a credentials
|
credentials to use for testing. To make sure you don't have a credentials
|
||||||
starvation issue when running in parallel make sure you have at least two
|
starvation issue when running in parallel make sure you have at least two
|
||||||
times the number of worker processes you are using to execute Tempest
|
times the number of worker processes you are using to execute Tempest
|
||||||
available in the file. (If running serially the worker count is 1.)
|
available in the file. (If running serially the worker count is 1.)
|
||||||
|
|
||||||
You can check the accounts.yaml.sample file packaged in Tempest for the yaml
|
You can check the accounts.yaml.sample file packaged in Tempest for the yaml
|
||||||
format.
|
format.
|
||||||
#. Provide Tempest with the location of your accounts.yaml file with the
|
#. Provide Tempest with the location of your accounts.yaml file with the
|
||||||
``test_accounts_file`` option in the ``auth`` section
|
``test_accounts_file`` option in the ``auth`` section
|
||||||
|
|
||||||
*NOTE: Be sure to use a full path for the file; otherwise Tempest will
|
*NOTE: Be sure to use a full path for the file; otherwise Tempest will
|
||||||
likely not find it.*
|
likely not find it.*
|
||||||
|
|
||||||
#. Set ``use_dynamic_credentials = False`` in the ``auth`` group
|
#. Set ``use_dynamic_credentials = False`` in the ``auth`` group
|
||||||
|
|
||||||
It is worth pointing out that each set of credentials in the accounts.yaml
|
It is worth pointing out that each set of credentials in the accounts.yaml
|
||||||
should have a unique project. This is required to provide proper isolation
|
should have a unique project. This is required to provide proper isolation
|
||||||
@ -162,8 +162,8 @@ For Tempest to be able to create servers you need to specify flavors that it
|
|||||||
can use to boot the servers with. There are two options in the Tempest config
|
can use to boot the servers with. There are two options in the Tempest config
|
||||||
for doing this:
|
for doing this:
|
||||||
|
|
||||||
#. ``flavor_ref``
|
#. ``flavor_ref``
|
||||||
#. ``flavor_ref_alt``
|
#. ``flavor_ref_alt``
|
||||||
|
|
||||||
Both of these options are in the ``compute`` section of the config file and take
|
Both of these options are in the ``compute`` section of the config file and take
|
||||||
in the flavor id (not the name) from Nova. The ``flavor_ref`` option is what
|
in the flavor id (not the name) from Nova. The ``flavor_ref`` option is what
|
||||||
@ -181,8 +181,8 @@ Images
|
|||||||
Just like with flavors, Tempest needs to know which images to use for booting
|
Just like with flavors, Tempest needs to know which images to use for booting
|
||||||
servers. There are two options in the compute section just like with flavors:
|
servers. There are two options in the compute section just like with flavors:
|
||||||
|
|
||||||
#. ``image_ref``
|
#. ``image_ref``
|
||||||
#. ``image_ref_alt``
|
#. ``image_ref_alt``
|
||||||
|
|
||||||
Both options are expecting an image id (not name) from Nova. The ``image_ref``
|
Both options are expecting an image id (not name) from Nova. The ``image_ref``
|
||||||
option is what will be used for booting the majority of servers in Tempest.
|
option is what will be used for booting the majority of servers in Tempest.
|
||||||
@ -192,13 +192,13 @@ those tests will be skipped.
|
|||||||
|
|
||||||
There are also options in the ``scenario`` section for images:
|
There are also options in the ``scenario`` section for images:
|
||||||
|
|
||||||
#. ``img_file``
|
#. ``img_file``
|
||||||
#. ``img_dir``
|
#. ``img_dir``
|
||||||
#. ``aki_img_file``
|
#. ``aki_img_file``
|
||||||
#. ``ari_img_file``
|
#. ``ari_img_file``
|
||||||
#. ``ami_img_file``
|
#. ``ami_img_file``
|
||||||
#. ``img_container_format``
|
#. ``img_container_format``
|
||||||
#. ``img_disk_format``
|
#. ``img_disk_format``
|
||||||
|
|
||||||
However, unlike the other image options, these are used for a very small subset
|
However, unlike the other image options, these are used for a very small subset
|
||||||
of scenario tests which are uploading an image. These options are used to tell
|
of scenario tests which are uploading an image. These options are used to tell
|
||||||
@ -261,7 +261,7 @@ booting.
|
|||||||
|
|
||||||
To set a fixed network name simply:
|
To set a fixed network name simply:
|
||||||
|
|
||||||
#. Set the ``fixed_network_name`` option in the ``compute`` group
|
#. Set the ``fixed_network_name`` option in the ``compute`` group
|
||||||
|
|
||||||
In the case that the configured fixed network name can not be found by a user
|
In the case that the configured fixed network name can not be found by a user
|
||||||
network list call, it will be treated like one was not provided except that a
|
network list call, it will be treated like one was not provided except that a
|
||||||
@ -329,9 +329,9 @@ connecting to and remotely accessing the created servers.
|
|||||||
|
|
||||||
To enable remote access to servers, there are 3 options at a minimum that are used:
|
To enable remote access to servers, there are 3 options at a minimum that are used:
|
||||||
|
|
||||||
#. ``run_validation``
|
#. ``run_validation``
|
||||||
#. ``connect_method``
|
#. ``connect_method``
|
||||||
#. ``auth_method``
|
#. ``auth_method``
|
||||||
|
|
||||||
The ``run_validation`` is used to enable or disable ssh connectivity for
|
The ``run_validation`` is used to enable or disable ssh connectivity for
|
||||||
all tests (with the exception of scenario tests which do not have a flag for
|
all tests (with the exception of scenario tests which do not have a flag for
|
||||||
@ -370,9 +370,9 @@ API calls for each service, it needs to know how that project is defined in the
|
|||||||
service catalog. There are three options for each service section to accomplish
|
service catalog. There are three options for each service section to accomplish
|
||||||
this:
|
this:
|
||||||
|
|
||||||
#. ``catalog_type``
|
#. ``catalog_type``
|
||||||
#. ``endpoint_type``
|
#. ``endpoint_type``
|
||||||
#. ``region``
|
#. ``region``
|
||||||
|
|
||||||
Setting ``catalog_type`` and ``endpoint_type`` should normally give Tempest
|
Setting ``catalog_type`` and ``endpoint_type`` should normally give Tempest
|
||||||
enough information to determine which endpoint it should pull from the service
|
enough information to determine which endpoint it should pull from the service
|
||||||
|
@ -44,9 +44,9 @@ ensure that we don't break backwards compatibility. For patches that change
|
|||||||
existing interfaces we have to be careful to make sure we don't break any
|
existing interfaces we have to be careful to make sure we don't break any
|
||||||
external consumers. Some common red flags are:
|
external consumers. Some common red flags are:
|
||||||
|
|
||||||
* a change to an existing API requires a change outside the library directory
|
* a change to an existing API requires a change outside the library directory
|
||||||
where the interface is being consumed
|
where the interface is being consumed
|
||||||
* a unit test has to be significantly changed to make the proposed change pass
|
* a unit test has to be significantly changed to make the proposed change pass
|
||||||
|
|
||||||
Testing
|
Testing
|
||||||
'''''''
|
'''''''
|
||||||
|
@ -33,46 +33,46 @@ project guidelines.
|
|||||||
Tempest will cover only integration testing of applicable microversions with
|
Tempest will cover only integration testing of applicable microversions with
|
||||||
below exceptions:
|
below exceptions:
|
||||||
|
|
||||||
#. Test covers a feature which is important for interoperability. This covers tests requirement
|
#. Test covers a feature which is important for interoperability. This covers tests requirement
|
||||||
from Defcore.
|
from Defcore.
|
||||||
#. Test needed to fill Schema gaps.
|
#. Test needed to fill Schema gaps.
|
||||||
Tempest validates API responses with defined JSON schema. API responses can be different on
|
Tempest validates API responses with defined JSON schema. API responses can be different on
|
||||||
each microversion and the JSON schemas need to be defined separately for the microversion.
|
each microversion and the JSON schemas need to be defined separately for the microversion.
|
||||||
While implementing new integration tests for a specific microversion, there
|
While implementing new integration tests for a specific microversion, there
|
||||||
may be a gap in the JSON schemas (caused by previous microversions) implemented
|
may be a gap in the JSON schemas (caused by previous microversions) implemented
|
||||||
in Tempest.
|
in Tempest.
|
||||||
Filling that gap while implementing the new integration test cases is not efficient due to
|
Filling that gap while implementing the new integration test cases is not efficient due to
|
||||||
many reasons:
|
many reasons:
|
||||||
|
|
||||||
* Hard to review
|
* Hard to review
|
||||||
* Sync between multiple integration tests patches which try to fill the same schema gap at same
|
* Sync between multiple integration tests patches which try to fill the same schema gap at same
|
||||||
time
|
time
|
||||||
* Might delay the microversion change on project side where project team wants Tempest
|
* Might delay the microversion change on project side where project team wants Tempest
|
||||||
tests to verify the results.
|
tests to verify the results.
|
||||||
|
|
||||||
Tempest will allow to fill the schema gaps at the end of each cycle, or more
|
Tempest will allow to fill the schema gaps at the end of each cycle, or more
|
||||||
often if required.
|
often if required.
|
||||||
Schema gap can be filled with testing those with a minimal set of tests. Those
|
Schema gap can be filled with testing those with a minimal set of tests. Those
|
||||||
tests might not be integration tests and might be already covered on project
|
tests might not be integration tests and might be already covered on project
|
||||||
side also.
|
side also.
|
||||||
This exception is needed because:
|
This exception is needed because:
|
||||||
|
|
||||||
* Allow to create microversion response schema in Tempest at the same time that projects are
|
* Allow to create microversion response schema in Tempest at the same time that projects are
|
||||||
implementing their API microversions. This will make implementation easier for adding
|
implementing their API microversions. This will make implementation easier for adding
|
||||||
required tests before a new microversion change can be merged in the corresponding project
|
required tests before a new microversion change can be merged in the corresponding project
|
||||||
and hence accelerate the development of microversions.
|
and hence accelerate the development of microversions.
|
||||||
* New schema must be verified by at least one test case which exercises such schema.
|
* New schema must be verified by at least one test case which exercises such schema.
|
||||||
|
|
||||||
For example:
|
For example:
|
||||||
If any projects implemented 4 API microversion say- v2.3, v2.4, v2.5, v2.6
|
If any projects implemented 4 API microversion say- v2.3, v2.4, v2.5, v2.6
|
||||||
Assume microversion v2.3, v2.4, v2.6 change the API Response which means Tempest
|
Assume microversion v2.3, v2.4, v2.6 change the API Response which means Tempest
|
||||||
needs to add JSON schema for v2.3, v2.4, v2.6.
|
needs to add JSON schema for v2.3, v2.4, v2.6.
|
||||||
In that case if only 1 or 2 tests can verify all new schemas then we do not need
|
In that case if only 1 or 2 tests can verify all new schemas then we do not need
|
||||||
separate tests for each new schemas. In worst case, we have to add 3 separate tests.
|
separate tests for each new schemas. In worst case, we have to add 3 separate tests.
|
||||||
#. Test covers service behavior at large scale with involvement of more deep layer like hypervisor
|
#. Test covers service behavior at large scale with involvement of more deep layer like hypervisor
|
||||||
etc not just API/DB layer. This type of tests will be added case by case basis and
|
etc not just API/DB layer. This type of tests will be added case by case basis and
|
||||||
with project team consultation about why it cannot be covered on project side and worth to test
|
with project team consultation about why it cannot be covered on project side and worth to test
|
||||||
in Tempest.
|
in Tempest.
|
||||||
|
|
||||||
Project Scope For Microversion Testing
|
Project Scope For Microversion Testing
|
||||||
""""""""""""""""""""""""""""""""""""""
|
""""""""""""""""""""""""""""""""""""""
|
||||||
@ -294,72 +294,72 @@ Microversion tests implemented in Tempest
|
|||||||
|
|
||||||
* Compute
|
* Compute
|
||||||
|
|
||||||
* `2.1`_
|
* `2.1`_
|
||||||
|
|
||||||
.. _2.1: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id1
|
.. _2.1: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id1
|
||||||
|
|
||||||
* `2.2`_
|
* `2.2`_
|
||||||
|
|
||||||
.. _2.2: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id2
|
.. _2.2: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id2
|
||||||
|
|
||||||
* `2.10`_
|
* `2.10`_
|
||||||
|
|
||||||
.. _2.10: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id9
|
.. _2.10: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id9
|
||||||
|
|
||||||
* `2.20`_
|
* `2.20`_
|
||||||
|
|
||||||
.. _2.20: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id18
|
.. _2.20: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id18
|
||||||
|
|
||||||
* `2.25`_
|
* `2.25`_
|
||||||
|
|
||||||
.. _2.25: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#maximum-in-mitaka
|
.. _2.25: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#maximum-in-mitaka
|
||||||
|
|
||||||
* `2.32`_
|
* `2.32`_
|
||||||
|
|
||||||
.. _2.32: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id29
|
.. _2.32: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id29
|
||||||
|
|
||||||
* `2.37`_
|
* `2.37`_
|
||||||
|
|
||||||
.. _2.37: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id34
|
.. _2.37: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id34
|
||||||
|
|
||||||
* `2.42`_
|
* `2.42`_
|
||||||
|
|
||||||
.. _2.42: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#maximum-in-ocata
|
.. _2.42: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#maximum-in-ocata
|
||||||
|
|
||||||
* `2.47`_
|
* `2.47`_
|
||||||
|
|
||||||
.. _2.47: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id42
|
.. _2.47: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id42
|
||||||
|
|
||||||
* `2.48`_
|
* `2.48`_
|
||||||
|
|
||||||
.. _2.48: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id43
|
.. _2.48: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id43
|
||||||
|
|
||||||
* Volume
|
* Volume
|
||||||
|
|
||||||
* `3.3`_
|
* `3.3`_
|
||||||
|
|
||||||
.. _3.3: https://docs.openstack.org/cinder/latest/contributor/api_microversion_history.html#id3
|
.. _3.3: https://docs.openstack.org/cinder/latest/contributor/api_microversion_history.html#id3
|
||||||
|
|
||||||
* `3.9`_
|
* `3.9`_
|
||||||
|
|
||||||
.. _3.9: https://docs.openstack.org/cinder/latest/contributor/api_microversion_history.html#id9
|
.. _3.9: https://docs.openstack.org/cinder/latest/contributor/api_microversion_history.html#id9
|
||||||
|
|
||||||
* `3.11`_
|
* `3.11`_
|
||||||
|
|
||||||
.. _3.11: https://docs.openstack.org/cinder/latest/contributor/api_microversion_history.html#id11
|
.. _3.11: https://docs.openstack.org/cinder/latest/contributor/api_microversion_history.html#id11
|
||||||
|
|
||||||
* `3.12`_
|
* `3.12`_
|
||||||
|
|
||||||
.. _3.12: https://docs.openstack.org/cinder/latest/contributor/api_microversion_history.html#id12
|
.. _3.12: https://docs.openstack.org/cinder/latest/contributor/api_microversion_history.html#id12
|
||||||
|
|
||||||
* `3.14`_
|
* `3.14`_
|
||||||
|
|
||||||
.. _3.14: https://docs.openstack.org/cinder/latest/contributor/api_microversion_history.html#id14
|
.. _3.14: https://docs.openstack.org/cinder/latest/contributor/api_microversion_history.html#id14
|
||||||
|
|
||||||
* `3.19`_
|
* `3.19`_
|
||||||
|
|
||||||
.. _3.19: https://docs.openstack.org/cinder/latest/contributor/api_microversion_history.html#id18
|
.. _3.19: https://docs.openstack.org/cinder/latest/contributor/api_microversion_history.html#id18
|
||||||
|
|
||||||
* `3.20`_
|
* `3.20`_
|
||||||
|
|
||||||
.. _3.20: https://docs.openstack.org/cinder/latest/contributor/api_microversion_history.html#id19
|
.. _3.20: https://docs.openstack.org/cinder/latest/contributor/api_microversion_history.html#id19
|
||||||
|
@ -29,19 +29,19 @@ Proposing a test removal
|
|||||||
|
|
||||||
In the proposal etherpad we'll be looking for answers to 3 questions
|
In the proposal etherpad we'll be looking for answers to 3 questions
|
||||||
|
|
||||||
#. The tests proposed for removal must have equiv. coverage in a different
|
#. The tests proposed for removal must have equiv. coverage in a different
|
||||||
project's test suite (whether this is another gating test project, or an in
|
project's test suite (whether this is another gating test project, or an in
|
||||||
tree functional test suite). For API tests preferably the other project will
|
tree functional test suite). For API tests preferably the other project will
|
||||||
have a similar source of friction in place to prevent breaking api changes
|
have a similar source of friction in place to prevent breaking api changes
|
||||||
so that we don't regress and let breaking api changes slip through the
|
so that we don't regress and let breaking api changes slip through the
|
||||||
gate.
|
gate.
|
||||||
#. The test proposed for removal has a failure rate < 0.50% in the gate over
|
#. The test proposed for removal has a failure rate < 0.50% in the gate over
|
||||||
the past release (the value and interval will likely be adjusted in the
|
the past release (the value and interval will likely be adjusted in the
|
||||||
future)
|
future)
|
||||||
|
|
||||||
.. _`prong #3`:
|
.. _`prong #3`:
|
||||||
#. There must not be an external user/consumer of tempest
|
#. There must not be an external user/consumer of tempest
|
||||||
that depends on the test proposed for removal
|
that depends on the test proposed for removal
|
||||||
|
|
||||||
The answers to 1 and 2 are easy to verify. For 1 just provide a link to the new
|
The answers to 1 and 2 are easy to verify. For 1 just provide a link to the new
|
||||||
test location. If you are linking to the tempest removal patch please also put
|
test location. If you are linking to the tempest removal patch please also put
|
||||||
@ -68,23 +68,23 @@ setUpClass or tearDownClass failures)
|
|||||||
|
|
||||||
You can access the infra mysql subunit2sql db w/ read-only permissions with:
|
You can access the infra mysql subunit2sql db w/ read-only permissions with:
|
||||||
|
|
||||||
* hostname: logstash.openstack.org
|
* hostname: logstash.openstack.org
|
||||||
* username: query
|
* username: query
|
||||||
* password: query
|
* password: query
|
||||||
* db_name: subunit2sql
|
* db_name: subunit2sql
|
||||||
|
|
||||||
For example if you were trying to remove the test with the id:
|
For example if you were trying to remove the test with the id:
|
||||||
tempest.api.compute.admin.test_flavors_negative.FlavorsAdminNegativeTestJSON.test_get_flavor_details_for_deleted_flavor
|
tempest.api.compute.admin.test_flavors_negative.FlavorsAdminNegativeTestJSON.test_get_flavor_details_for_deleted_flavor
|
||||||
you would run the following:
|
you would run the following:
|
||||||
|
|
||||||
#. run: "mysql -u query -p -h logstash.openstack.org subunit2sql" to connect
|
#. run: "mysql -u query -p -h logstash.openstack.org subunit2sql" to connect
|
||||||
to the subunit2sql db
|
to the subunit2sql db
|
||||||
#. run the query: MySQL [subunit2sql]> select * from tests where test_id like
|
#. run the query: MySQL [subunit2sql]> select * from tests where test_id like
|
||||||
"tempest.api.compute.admin.test_flavors_negative.FlavorsAdminNegativeTestJSON%";
|
"tempest.api.compute.admin.test_flavors_negative.FlavorsAdminNegativeTestJSON%";
|
||||||
which will return a table of all the tests in the class (but it will also
|
which will return a table of all the tests in the class (but it will also
|
||||||
catch failures in setUpClass and tearDownClass)
|
catch failures in setUpClass and tearDownClass)
|
||||||
#. paste the output table with numbers and the mysql command you ran to
|
#. paste the output table with numbers and the mysql command you ran to
|
||||||
generate it into the etherpad.
|
generate it into the etherpad.
|
||||||
|
|
||||||
Eventually a cli interface will be created to make that a bit more friendly.
|
Eventually a cli interface will be created to make that a bit more friendly.
|
||||||
Also a dashboard is in the works so we don't need to manually run the command.
|
Also a dashboard is in the works so we don't need to manually run the command.
|
||||||
@ -131,23 +131,23 @@ Exceptions to this procedure
|
|||||||
For the most part all tempest test removals have to go through this procedure
|
For the most part all tempest test removals have to go through this procedure
|
||||||
there are a couple of exceptions though:
|
there are a couple of exceptions though:
|
||||||
|
|
||||||
#. The class of testing has been decided to be outside the scope of tempest.
|
#. The class of testing has been decided to be outside the scope of tempest.
|
||||||
#. A revert for a patch which added a broken test, or testing which didn't
|
#. A revert for a patch which added a broken test, or testing which didn't
|
||||||
actually run in the gate (basically any revert for something which
|
actually run in the gate (basically any revert for something which
|
||||||
shouldn't have been added)
|
shouldn't have been added)
|
||||||
#. Tests that would become out of scope as a consequence of an API change,
|
#. Tests that would become out of scope as a consequence of an API change,
|
||||||
as described in `API Compatibility`_.
|
as described in `API Compatibility`_.
|
||||||
Such tests cannot live in Tempest because of the branchless nature of
|
Such tests cannot live in Tempest because of the branchless nature of
|
||||||
Tempest. Such test must still honor `prong #3`_.
|
Tempest. Such test must still honor `prong #3`_.
|
||||||
|
|
||||||
For the first exception type the only types of testing in tree which have been
|
For the first exception type the only types of testing in tree which have been
|
||||||
declared out of scope at this point are:
|
declared out of scope at this point are:
|
||||||
|
|
||||||
* The CLI tests (which should be completely removed at this point)
|
* The CLI tests (which should be completely removed at this point)
|
||||||
* Neutron Adv. Services testing (which should be completely removed at this
|
* Neutron Adv. Services testing (which should be completely removed at this
|
||||||
point)
|
point)
|
||||||
* XML API Tests (which should be completely removed at this point)
|
* XML API Tests (which should be completely removed at this point)
|
||||||
* EC2 API/boto tests (which should be completely removed at this point)
|
* EC2 API/boto tests (which should be completely removed at this point)
|
||||||
|
|
||||||
For tests that fit into this category the only criteria for removal is that
|
For tests that fit into this category the only criteria for removal is that
|
||||||
there is equivalent testing elsewhere.
|
there is equivalent testing elsewhere.
|
||||||
@ -159,12 +159,12 @@ Starting in the liberty cycle tempest has defined a set of projects which
|
|||||||
are defined as in scope for direct testing in tempest. As of today that list
|
are defined as in scope for direct testing in tempest. As of today that list
|
||||||
is:
|
is:
|
||||||
|
|
||||||
* Keystone
|
* Keystone
|
||||||
* Nova
|
* Nova
|
||||||
* Glance
|
* Glance
|
||||||
* Cinder
|
* Cinder
|
||||||
* Neutron
|
* Neutron
|
||||||
* Swift
|
* Swift
|
||||||
|
|
||||||
anything that lives in tempest which doesn't test one of these projects can be
|
anything that lives in tempest which doesn't test one of these projects can be
|
||||||
removed assuming there is equivalent testing elsewhere. Preferably using the
|
removed assuming there is equivalent testing elsewhere. Preferably using the
|
||||||
|
@ -36,12 +36,12 @@ methods.
|
|||||||
In standard unittest the lifecycle of a TestCase can be described in the
|
In standard unittest the lifecycle of a TestCase can be described in the
|
||||||
following phases:
|
following phases:
|
||||||
|
|
||||||
#. setUpClass
|
#. setUpClass
|
||||||
#. setUp
|
#. setUp
|
||||||
#. Test Execution
|
#. Test Execution
|
||||||
#. tearDown
|
#. tearDown
|
||||||
#. doCleanups
|
#. doCleanups
|
||||||
#. tearDownClass
|
#. tearDownClass
|
||||||
|
|
||||||
setUpClass
|
setUpClass
|
||||||
----------
|
----------
|
||||||
@ -54,10 +54,10 @@ setup for interacting with the configured cloud.
|
|||||||
To accomplish this you do **not** define a setUpClass function, instead there
|
To accomplish this you do **not** define a setUpClass function, instead there
|
||||||
are a number of predefined phases to setUpClass that are used. The phases are:
|
are a number of predefined phases to setUpClass that are used. The phases are:
|
||||||
|
|
||||||
* skip_checks
|
* skip_checks
|
||||||
* setup_credentials
|
* setup_credentials
|
||||||
* setup_clients
|
* setup_clients
|
||||||
* resource_setup
|
* resource_setup
|
||||||
|
|
||||||
which is executed in that order. Cleanup of resources provisioned during
|
which is executed in that order. Cleanup of resources provisioned during
|
||||||
the resource_setup must be scheduled right after provisioning using
|
the resource_setup must be scheduled right after provisioning using
|
||||||
@ -65,7 +65,7 @@ the addClassResourceCleanp helper. The resource cleanups stacked this way
|
|||||||
are executed in reverse order during tearDownClass, before the cleanup of
|
are executed in reverse order during tearDownClass, before the cleanup of
|
||||||
test credentials takes place. An example of a TestCase which defines all
|
test credentials takes place. An example of a TestCase which defines all
|
||||||
of these would be::
|
of these would be::
|
||||||
|
|
||||||
from tempest.common import waiters
|
from tempest.common import waiters
|
||||||
from tempest import config
|
from tempest import config
|
||||||
from tempest.lib.common.utils import test_utils
|
from tempest.lib.common.utils import test_utils
|
||||||
|
@ -14,11 +14,11 @@ multiple OpenStack services to exercise the touch points between them.
|
|||||||
|
|
||||||
Any scenario test should have a real-life use case. An example would be:
|
Any scenario test should have a real-life use case. An example would be:
|
||||||
|
|
||||||
- "As operator I want to start with a blank environment":
|
- "As operator I want to start with a blank environment":
|
||||||
1. upload a glance image
|
1. upload a glance image
|
||||||
2. deploy a vm from it
|
2. deploy a vm from it
|
||||||
3. ssh to the guest
|
3. ssh to the guest
|
||||||
4. create a snapshot of the vm
|
4. create a snapshot of the vm
|
||||||
|
|
||||||
|
|
||||||
Why are these tests in Tempest?
|
Why are these tests in Tempest?
|
||||||
|
Loading…
x
Reference in New Issue
Block a user