The primary goal is to be able to run all Tempest tests
and verify OpenStack when profiling is enabled. Also this patch
allows to:
* manually verify that certain services are properly instrumented
and produce trace events when a scenario is executed;
* write automatic tests for trace coverage;
* profile certain tests from performance perspective.
A new parameter is introduced into tempest.conf:
* profiler.key - the key used to enable OSProfiler (should
match the one configured in OpenStack services)
To test the patch on DevStack:
1. Enable osprofiler with Redis collector in local.conf:
enable_plugin osprofiler https://git.openstack.org/openstack/osprofiler master
OSPROFILER_COLLECTOR=redis
2. Run all Tempest tests or select some, e.g.:
tempest run --regex tempest.api.network.test_networks.NetworksTest.test_list_networks*
Change-Id: I64f30c36adbf7fb26609142f22d3e305ac9e82b5
Several docstrings have incorrect format, what prevents to generate
a documentation using sphinx-apidoc extension.
Change-Id: I66431c7030089173d7635c1e2a72765db21f9780
identity_utils is an helper used to obtain a cred_client object
by some tempest tests; in an effort of keeping the interface
in test.py to the minimum required, moving that helper to
tempest.common.identity, so that it can still be used by a
few tempest tests but we don't have to support it as a stable
interface.
Change-Id: I6692bcf2b02d3d023a1db0dd0255b17e7a869a5e
We ensure that if setUpClass is implemented by a test class,
it must invoke the super setUpClass. Adding unit test for it.
Change-Id: I50993b6d76cc462e533b243b7bf28d1b2c2228ff
The skip_checks and setup_credentials "fixtures" are implemented
in the base class. The implementation can be extended by test
classes but it should not be overwritten.
Enforcing this by checking that the base class version has been
invoked. Added unit tests for it as well.
Change-Id: Ie5d994519f9d56de423d1a0bede7ee8d703b9390
The network_resources attribute should not be a public interface,
so renaming it to _network_resources. Document set_network_resources
better; ensure it's only invoked before super's setup_credentials is
invoked to avoid the call being ignored and generate errors that
may be difficult to debug.
Change-Id: I4eab8f2a722b47edc20e4aab0ef453bec16842f3
The provisioning of validation resources is unreadable and hard to
use. Tests have to call an helper before super's resource_setup is
invoked and they will find resources in a dictionary in a class
attribute.
Changing to a model where the test.py base class provides helpers
that takes care of common tasks:
- pulling the right parameters from configuration
- scheduling cleanup
There are two helpers available, one to be used when validation
resources are provisioned for a server created at class setup time;
the second one shall be used with servers provisioned durint tests
or test setup.
The new helper returns the provisioned resources to the test.
Other helpers are affected by this change since they cannot pull
validation resource from class anymore safely, and they have
been updated to accept validation resources as input.
Change-Id: I1106e40c6d7483f66d645f3bb560c6d74a612d0f
Add docstrings, unit tests and an helper for class resource
cleanup. Check super's resource_cleanup is invoked when
overriden to ensure the cleanup stack is processed.
Change-Id: I9c89ba4efd715634dde6b1182c2025ddf9c2f7d2
If force_tenant_isolation is set to True, admin credential is
needed, and this check should be put in skip_checks in test.py,
while should not be done by individual testcases.
Change-Id: I9506120a088cde270b9c787bab26bf6e4c4a23d3
services and extension decorators are based on CONF and needed by
Tempest only, so moving them to tempest.common.
Since a few plugnis use them today, use debtcollector to avoid
breaking all dependencies.
Change-Id: Ibd52153d00b8e60fb8c89e38d94e358ddc787251
Create and cleanup both expect a dict as input. Even though the
expected content of the dict is documented, using dedicated args
makes the interface easer to use and to test.
Change-Id: Id8dca4244008d5944dfd8ba32ce1a726b4fda973
Validation resources implicitly depends on tempest/clients.py as
it uses client aliases defined there. This is not ok since that
module does not live under tempest/lib.
Change validation resources to use clients as defined in
tempest/lib/services/clients.py instead. The ServiceClients class
defined in there is the superclass of the Manager class that was
assumed before, so it will just work with any further change.
Clients were also not used consistently with the use_neutron
flag, so making sure that only neutron clients are used if neutron
is selected and viceversa.
Change-Id: If83610b06af13aa736b03d463a94ac6ed02296bd
Validation resources is going to be a stable interface in lib, so
the first step is to remove dependencies from configuration items.
Validation resources are directly used by Tempest only, so the
change in interface should not cause any issue with plugins.
Change-Id: I9ad52d9985139ce373f927c838fcaa8812ea9d17
As part of the work of making test.py stable for plugins, the
is_scheduler_filter_enabled helper has to be moved, since it's
compute specific, legacy and not useful outside of Tempest.
Change-Id: I6efda7372d1e8a2028853a63f6793f680df86d64
All tempest plugin now switched to use related_bug
from tempest.lib.decorators
Last patch is proposed in mistral to use the lib interface
which is depends-on of this patch.
We can remove deprecated tempest.test.related_bug.
Depends-On: I20497d48a46195644a114dd9f6b86aca59303097
Change-Id: Ic9ea144750a216d15a372c390d3160934220d2f9
This commit migrates the preprovisioned credentials provider from
tempest/common into tempest/lib. As part of this 2 dependencies are
also migrated to lib: the InvalidTestResource exception class and the
tempest.common.fixed_network module.
Change-Id: Ib9e909aaaf81f7cb57e2382cec44598e918747b8
When pause_teardown flag in tempest.conf is set to True a pdb breakpoint
is added to tearDown and tearDownClass methods in test.py.
This allows to pause cleaning resources process, so that, used resources
can be examined. Closer examination of used resources may lead to faster
debugging.
Change-Id: I09b7721e64cda161289f915d30888ec54bbed821
This commit adds notes for network and identity service in
get_service_list(). Sometimes people (including me) are confusing that
only the two services are `True`. So, this commit is trying to clarify
it.
Change-Id: I0465cb6de2b2f0cfdfee0948e2c3ebc0133357ae
It could be confusing to new comers that the variables
cls.os and cls.os_primary or cls.alt_manager and cls.os_alt
actually are aliases. Besides we are not consistent in their usage.
This patch normalizes the usage around os_admin, os_primary and
os_alt. We keep the old aliases for compatibility but we should not
use them anymore.
This fix a long standing TODO item.
Change-Id: I5f7164f7a7ec5d4380ca22885000caa0183a0bf7
The config value identity.auth_version defines the identity API
version used for all common API tests, such as identity API calls
performed by non-identity tests, as well as provisioning of
dynamic accounts, with the exception of identity tests.
Since the v2 API is deprecated, we don't need to run v2 admin
tests in the gate anymore. However v2 user APIs are most likely
still widely used by OpenStack users, and we should keep those
test runnings.
Right now identity tests control which version of the identity
API is used to allocate / provision test accounts by setting
an identity_version class attribute.
This behaviour forces us to run an identity v2 admin endpoint
only to be able to provision accounts for identity v2 tests.
This commit changes the current behaviour to always provision
accounts with the identity API version specified in
CONF.identity.auth_version. When v3 is configured (which is
the case in all gate jobs), v3 credentials will be provisioned
and then used for v2 non-admin tests.
Depends-on: I801e6740258ddea2a1b628a209970e0307d39d12
Change-Id: I2601433585740030ab61433f0e8ca1a919275d9f
We are going to make test.py stable for plugins, so it should
only depend on tempest.lib and config.
InvalidServiceTag is in the exceptions module.
It cannot be moved in tempest.lib.exceptions since it has no meaning
in there, so moving it to test.py directly.
Change-Id: I969fb45a44ce08c176d64dfe1c58d75215eacdf5
This commit makes to use sequence directly instead of using
len(SEQUENCE). The original code works correctly, and it's really
straight forward. However, PEP8 recommends like below[1]. And it
makes code more simple, too.
```
For sequences, (strings, lists, tuples), use the fact that empty
sequences are false.
Yes: if not seq:
if seq:
No: if len(seq):
if not len(seq):
```
[1] https://www.python.org/dev/peps/pep-0008/#programming-recommendations
Change-Id: I8d41e16d82b1b3860a98e5217cb7a541fc83b907
I think it's a good idea to move all utility decorators into
tempest/lib/decorators.py. This patch does that for the `attr`
decorator.
Change-Id: Iaafbb112b6eee458089cc49918359a8a8d0485e2
I think it's a good idea to move all utility decorators into
tempest/lib/decorators.py. This patch does that for the `related_bug`
decorator.
Change-Id: I846d575e41f4dddfd5642b7750e988f75a717e7d
Rather than repeating the list of services twice, use the
one defined in get_service_list. Note that this removes
baremetal from the list of accepted services.
Change-Id: Ib5a9b33a54ef1064f5dc8f6206ddd1c7d218fc8e
Don't use 'list' as a parameter name since this assert can be used
for any sequence or collection. Add docstring for both helpers.
Change-Id: I747507c3a5d0cf6f84915c8dd9222f4c082cbc04
When assertEmpty fails, it will print:
raise mismatch_error
MismatchError: 0 != 5
this is to improve the error info as:
raise mismatch_error
MismatchError: 0 != 1: list is not empty: [u'1b95a8ea7e124a76bfadf6ce1906aa27']
and when assertNotEmpty fails, it will print:
raise self.failureException(msg)
AssertionError: 0 not greater than 0
this is to improve the error info as:
raise self.failureException(msg)
AssertionError: list is empty.
Change-Id: Icdf69b596fa135387721c49df164a68799ab9a70
The call_until_true of test module is marked as deprecated and
we have a plan to remove it in Ocata cycle.
This patch removes it after switching to use test_utils' one on
all OpenStack projects.
Change-Id: Ide11a7434a4714e5d2211af1803333535f557370
Most modules are switched to using decorators.idempotent_id and
this patch covers the remaining and removes the old decorator check.
NOTE: The old decorator test.idempotent_id is still necessary to be
used on the other projects.
Change-Id: Ice7f593fbc109863eba11402c2f3af75fc6ab9a1
Closes-Bug: #1616913
Ilya Shakhat