The primary goal is to be able to run all Tempest tests
and verify OpenStack when profiling is enabled. Also this patch
* manually verify that certain services are properly instrumented
and produce trace events when a scenario is executed;
* write automatic tests for trace coverage;
* profile certain tests from performance perspective.
A new parameter is introduced into tempest.conf:
* profiler.key - the key used to enable OSProfiler (should
match the one configured in OpenStack services)
To test the patch on DevStack:
1. Enable osprofiler with Redis collector in local.conf:
enable_plugin osprofiler https://git.openstack.org/openstack/osprofiler master
2. Run all Tempest tests or select some, e.g.:
tempest run --regex tempest.api.network.test_networks.NetworksTest.test_list_networks*
identity_utils is an helper used to obtain a cred_client object
by some tempest tests; in an effort of keeping the interface
in test.py to the minimum required, moving that helper to
tempest.common.identity, so that it can still be used by a
few tempest tests but we don't have to support it as a stable
The skip_checks and setup_credentials "fixtures" are implemented
in the base class. The implementation can be extended by test
classes but it should not be overwritten.
Enforcing this by checking that the base class version has been
invoked. Added unit tests for it as well.
The network_resources attribute should not be a public interface,
so renaming it to _network_resources. Document set_network_resources
better; ensure it's only invoked before super's setup_credentials is
invoked to avoid the call being ignored and generate errors that
may be difficult to debug.
The provisioning of validation resources is unreadable and hard to
use. Tests have to call an helper before super's resource_setup is
invoked and they will find resources in a dictionary in a class
Changing to a model where the test.py base class provides helpers
that takes care of common tasks:
- pulling the right parameters from configuration
- scheduling cleanup
There are two helpers available, one to be used when validation
resources are provisioned for a server created at class setup time;
the second one shall be used with servers provisioned durint tests
or test setup.
The new helper returns the provisioned resources to the test.
Other helpers are affected by this change since they cannot pull
validation resource from class anymore safely, and they have
been updated to accept validation resources as input.
Add docstrings, unit tests and an helper for class resource
cleanup. Check super's resource_cleanup is invoked when
overriden to ensure the cleanup stack is processed.
If force_tenant_isolation is set to True, admin credential is
needed, and this check should be put in skip_checks in test.py,
while should not be done by individual testcases.
services and extension decorators are based on CONF and needed by
Tempest only, so moving them to tempest.common.
Since a few plugnis use them today, use debtcollector to avoid
breaking all dependencies.
Create and cleanup both expect a dict as input. Even though the
expected content of the dict is documented, using dedicated args
makes the interface easer to use and to test.
Validation resources implicitly depends on tempest/clients.py as
it uses client aliases defined there. This is not ok since that
module does not live under tempest/lib.
Change validation resources to use clients as defined in
tempest/lib/services/clients.py instead. The ServiceClients class
defined in there is the superclass of the Manager class that was
assumed before, so it will just work with any further change.
Clients were also not used consistently with the use_neutron
flag, so making sure that only neutron clients are used if neutron
is selected and viceversa.
Validation resources is going to be a stable interface in lib, so
the first step is to remove dependencies from configuration items.
Validation resources are directly used by Tempest only, so the
change in interface should not cause any issue with plugins.
As part of the work of making test.py stable for plugins, the
is_scheduler_filter_enabled helper has to be moved, since it's
compute specific, legacy and not useful outside of Tempest.
All tempest plugin now switched to use related_bug
Last patch is proposed in mistral to use the lib interface
which is depends-on of this patch.
We can remove deprecated tempest.test.related_bug.
This commit migrates the preprovisioned credentials provider from
tempest/common into tempest/lib. As part of this 2 dependencies are
also migrated to lib: the InvalidTestResource exception class and the
When pause_teardown flag in tempest.conf is set to True a pdb breakpoint
is added to tearDown and tearDownClass methods in test.py.
This allows to pause cleaning resources process, so that, used resources
can be examined. Closer examination of used resources may lead to faster
This commit adds notes for network and identity service in
get_service_list(). Sometimes people (including me) are confusing that
only the two services are `True`. So, this commit is trying to clarify
It could be confusing to new comers that the variables
cls.os and cls.os_primary or cls.alt_manager and cls.os_alt
actually are aliases. Besides we are not consistent in their usage.
This patch normalizes the usage around os_admin, os_primary and
os_alt. We keep the old aliases for compatibility but we should not
use them anymore.
This fix a long standing TODO item.
The config value identity.auth_version defines the identity API
version used for all common API tests, such as identity API calls
performed by non-identity tests, as well as provisioning of
dynamic accounts, with the exception of identity tests.
Since the v2 API is deprecated, we don't need to run v2 admin
tests in the gate anymore. However v2 user APIs are most likely
still widely used by OpenStack users, and we should keep those
Right now identity tests control which version of the identity
API is used to allocate / provision test accounts by setting
an identity_version class attribute.
This behaviour forces us to run an identity v2 admin endpoint
only to be able to provision accounts for identity v2 tests.
This commit changes the current behaviour to always provision
accounts with the identity API version specified in
CONF.identity.auth_version. When v3 is configured (which is
the case in all gate jobs), v3 credentials will be provisioned
and then used for v2 non-admin tests.
We are going to make test.py stable for plugins, so it should
only depend on tempest.lib and config.
InvalidServiceTag is in the exceptions module.
It cannot be moved in tempest.lib.exceptions since it has no meaning
in there, so moving it to test.py directly.
This commit makes to use sequence directly instead of using
len(SEQUENCE). The original code works correctly, and it's really
straight forward. However, PEP8 recommends like below. And it
makes code more simple, too.
For sequences, (strings, lists, tuples), use the fact that empty
sequences are false.
Yes: if not seq:
No: if len(seq):
if not len(seq):
Rather than repeating the list of services twice, use the
one defined in get_service_list. Note that this removes
baremetal from the list of accepted services.
When assertEmpty fails, it will print:
MismatchError: 0 != 5
this is to improve the error info as:
MismatchError: 0 != 1: list is not empty: [u'1b95a8ea7e124a76bfadf6ce1906aa27']
and when assertNotEmpty fails, it will print:
AssertionError: 0 not greater than 0
this is to improve the error info as:
AssertionError: list is empty.
The call_until_true of test module is marked as deprecated and
we have a plan to remove it in Ocata cycle.
This patch removes it after switching to use test_utils' one on
all OpenStack projects.
Most modules are switched to using decorators.idempotent_id and
this patch covers the remaining and removes the old decorator check.
NOTE: The old decorator test.idempotent_id is still necessary to be
used on the other projects.