Commit Graph

273 Commits (0ea2c01d757375d54aa9a9de34c2c9995bb48d13)

Author SHA1 Message Date
Ilya Shakhat 1291bb4736 Add profiler support into Tempest
The primary goal is to be able to run all Tempest tests
and verify OpenStack when profiling is enabled. Also this patch
allows to:
 * manually verify that certain services are properly instrumented
   and produce trace events when a scenario is executed;
 * write automatic tests for trace coverage;
 * profile certain tests from performance perspective.

A new parameter is introduced into tempest.conf:
 * profiler.key - the key used to enable OSProfiler (should
     match the one configured in OpenStack services)

To test the patch on DevStack:
 1. Enable osprofiler with Redis collector in local.conf:

   enable_plugin osprofiler master

 2. Run all Tempest tests or select some, e.g.:

   tempest run --regex*

Change-Id: I64f30c36adbf7fb26609142f22d3e305ac9e82b5
4 years ago
Sergey Vilgelm eac094a8cf
docs: Fix incorrect docstrings in reStructuredText format
Several docstrings have incorrect format, what prevents to generate
a documentation using sphinx-apidoc extension.

Change-Id: I66431c7030089173d7635c1e2a72765db21f9780
5 years ago
zhufl de67637682 Add missing ws seperator between words
This is to add missing ws seperator between words, usually
in log messages.

Change-Id: I228573a755717bb9725267fad61706ce2ef3da87
5 years ago
Federico Ressi 2d6bcaa7e5 Fix code to pass pep8 Tox environment.
Command tox -e pep8 is failing making CI to fail: this fixes it.

Change-Id: I5b1acfc4ba2e0eb16f50bfd1cdaa8edc361923d1
5 years ago
zhufl 33289a2347 Default to using neutron networks_client in get_tenant_network
Proxy apis to neutron like '/os-networks' are deprecated in
microverion 2.36, and calling to them will get 404 error.
So this is to use neutron networks_client as default in get_tenant_network.

Change-Id: I0496d2d0c3c9fe29d61124275c7f5cca78729bf2
5 years ago
Andrea Frittoli e1ed695996 Move identity_utils to common.identity
identity_utils is an helper used to obtain a cred_client object
by some tempest tests; in an effort of keeping the interface
in to the minimum required, moving that helper to
tempest.common.identity, so that it can still be used by a
few tempest tests but we don't have to support it as a stable

Change-Id: I6692bcf2b02d3d023a1db0dd0255b17e7a869a5e
6 years ago
Andrea Frittoli ba712ac26c Add unit tests for not overriding setUpClass
We ensure that if setUpClass is implemented by a test class,
it must invoke the super setUpClass. Adding unit test for it.

Change-Id: I50993b6d76cc462e533b243b7bf28d1b2c2228ff
6 years ago
Andrea Frittoli (andreaf) 08e42d4325 Add unit tests for test class fixtures
Add unit tests to enures class fixtures behave as expected in
normal and error conditions.

Change-Id: I79a9460387dc68e5d514a2a0660e9daed0443cce
6 years ago
Andrea Frittoli 9958367148 Prepare setup_clients and resource_setup
Prepare setup_clients and resource_setup as stable interfaces:
add docstrings.

Change-Id: I41de277bf087200136362779f6d74e5949e0f726
6 years ago
Andrea Frittoli a6c885a8f8 Prepare setup_credentials as stable
Add docs and unit tests.

Change-Id: If77885219736d47ace430f44ed1bebf60ac0e9fc
6 years ago
Andrea Frittoli a5440c8f90 Prepare skip_checks as stable interface
Fix docstring and add unit tests.

Change-Id: I97c468936465e4677591a77b98aa8296e528ed3b
6 years ago
Andrea Frittoli 421dc3c744 Make skip_checks and setup_creds safe
The skip_checks and setup_credentials "fixtures" are implemented
in the base class. The implementation can be extended by test
classes but it should not be overwritten.

Enforcing this by checking that the base class version has been
invoked. Added unit tests for it as well.

Change-Id: Ie5d994519f9d56de423d1a0bede7ee8d703b9390
6 years ago
Andrea Frittoli bcbf1af1e3 Prepare network_resources as a stable interface
The network_resources attribute should not be a public interface,
so renaming it to _network_resources. Document set_network_resources
better; ensure it's only invoked before super's setup_credentials is
invoked to avoid the call being ignored and generate errors that
may be difficult to debug.

Change-Id: I4eab8f2a722b47edc20e4aab0ef453bec16842f3
6 years ago
Andrea Frittoli 0477accd32 Make validation_resources a stable interface
Move the module to tempest.lib, add unit tests, release notes and
update imports.

Change-Id: I76b3b8e9a83bd83a6b5742b6d2ceb84642f653ad
6 years ago
Andrea Frittoli 9f416dd25a Stop implicit validation_resources provisioning
The provisioning of validation resources is unreadable and hard to
use. Tests have to call an helper before super's resource_setup is
invoked and they will find resources in a dictionary in a class

Changing to a model where the base class provides helpers
that takes care of common tasks:
- pulling the right parameters from configuration
- scheduling cleanup

There are two helpers available, one to be used when validation
resources are provisioned for a server created at class setup time;
the second one shall be used with servers provisioned durint tests
or test setup.

The new helper returns the provisioned resources to the test.
Other helpers are affected by this change since they cannot pull
validation resource from class anymore safely, and they have
been updated to accept validation resources as input.

Change-Id: I1106e40c6d7483f66d645f3bb560c6d74a612d0f
6 years ago
Andrea Frittoli 3be574898c Make resource_cleanup stable
Add docstrings, unit tests and an helper for class resource
cleanup. Check super's resource_cleanup is invoked when
overriden to ensure the cleanup stack is processed.

Change-Id: I9c89ba4efd715634dde6b1182c2025ddf9c2f7d2
6 years ago
zhufl 75498bae68 check is_admin_available for force_tenant_isolation in
If force_tenant_isolation is set to True, admin credential is
needed, and this check should be put in skip_checks in,
while should not be done by individual testcases.

Change-Id: I9506120a088cde270b9c787bab26bf6e4c4a23d3
6 years ago
Andrea Frittoli cd36841ca2 Move test decorators to common
services and extension decorators are based on CONF and needed by
Tempest only, so moving them to tempest.common.
Since a few plugnis use them today, use debtcollector to avoid
breaking all dependencies.

Change-Id: Ibd52153d00b8e60fb8c89e38d94e358ddc787251
6 years ago
Jenkins 29330ba31e Merge "Stop passing around dicts in validation resources" 6 years ago
Jenkins 3ea343ecc4 Merge "Fix client usage in validation resources" 6 years ago
Jenkins cd65d422b0 Merge "Drop validation resources dependency from CONF" 6 years ago
Andrea Frittoli 8871fca323 Stop passing around dicts in validation resources
Create and cleanup both expect a dict as input. Even though the
expected content of the dict is documented, using dedicated args
makes the interface easer to use and to test.

Change-Id: Id8dca4244008d5944dfd8ba32ce1a726b4fda973
6 years ago
Andrea Frittoli 463a8a64ef Fix client usage in validation resources
Validation resources implicitly depends on tempest/ as
it uses client aliases defined there. This is not ok since that
module does not live under tempest/lib.

Change validation resources to use clients as defined in
tempest/lib/services/ instead. The ServiceClients class
defined in there is the superclass of the Manager class that was
assumed before, so it will just work with any further change.

Clients were also not used consistently with the use_neutron
flag, so making sure that only neutron clients are used if neutron
is selected and viceversa.

Change-Id: If83610b06af13aa736b03d463a94ac6ed02296bd
6 years ago
Andrea Frittoli 1fa7a60e26 Drop validation resources dependency from CONF
Validation resources is going to be a stable interface in lib, so
the first step is to remove dependencies from configuration items.

Validation resources are directly used by Tempest only, so the
change in interface should not cause any issue with plugins.

Change-Id: I9ad52d9985139ce373f927c838fcaa8812ea9d17
6 years ago
Jenkins 9619b9818f Merge "Remove deprecated test.related_bug decorator" 6 years ago
Andrea Frittoli 88eb677da9 Move schedule filter check to compute module
As part of the work of making stable for plugins, the
is_scheduler_filter_enabled helper has to be moved, since it's
compute specific, legacy and not useful outside of Tempest.

Change-Id: I6efda7372d1e8a2028853a63f6793f680df86d64
6 years ago
ghanshyam 64485cc674 Remove deprecated test.related_bug decorator
All tempest plugin now switched to use related_bug
from tempest.lib.decorators
Last patch is proposed in mistral to use the lib interface
which is depends-on of this patch.

We can remove deprecated tempest.test.related_bug.
Depends-On: I20497d48a46195644a114dd9f6b86aca59303097

Change-Id: Ic9ea144750a216d15a372c390d3160934220d2f9
6 years ago
Matthew Treinish b19c55df4b
Migrate the preprov creds module to tempest lib
This commit migrates the preprovisioned credentials provider from
tempest/common into tempest/lib. As part of this 2 dependencies are
also migrated to lib: the InvalidTestResource exception class and the
tempest.common.fixed_network module.

Change-Id: Ib9e909aaaf81f7cb57e2382cec44598e918747b8
6 years ago
Jenkins 138c0fb79f Merge "Add notes for network and identity in get_service_list" 6 years ago
Martin Kopec ae155b7095 Pause resource cleanup
When pause_teardown flag in tempest.conf is set to True a pdb breakpoint
is added to tearDown and tearDownClass methods in
This allows to pause cleaning resources process, so that, used resources
can be examined. Closer examination of used resources may lead to faster

Change-Id: I09b7721e64cda161289f915d30888ec54bbed821
6 years ago
Masayuki Igawa 9012ba31a6
Add notes for network and identity in get_service_list
This commit adds notes for network and identity service in
get_service_list(). Sometimes people (including me) are confusing that
only the two services are `True`. So, this commit is trying to clarify

Change-Id: I0465cb6de2b2f0cfdfee0948e2c3ebc0133357ae
6 years ago
Andrea Frittoli cf999a83f0 cls.os is deprecated use cls.os_primary
Fixing a couple of cases left over from the patch where the
deprecation was introduced.

Change-Id: Ic1875708e0f6e8e9ff04b124d2443a7dfd08e972
6 years ago
Jakub Libosvar 7835ca1771 Fix deprecation warnings
Those warnings produce wrong removal version message. Queens is the one
after Pike.

Change-Id: I2245722899521cbecc17d1b9ca05d221cce9d5c5
6 years ago
Jordan Pittier 8160d31e58 stop using aliases for creds manager
It could be confusing to new comers that the variables
cls.os and cls.os_primary or cls.alt_manager and cls.os_alt
actually are aliases. Besides we are not consistent in their usage.

This patch normalizes the usage around os_admin, os_primary and
os_alt. We keep the old aliases for compatibility but we should not
use them anymore.

This fix a long standing TODO item.

Change-Id: I5f7164f7a7ec5d4380ca22885000caa0183a0bf7
6 years ago
Andrea Frittoli 9e01dbbe40 Always provision accounts with auth_version
The config value identity.auth_version defines the identity API
version used for all common API tests, such as identity API calls
performed by non-identity tests, as well as provisioning of
dynamic accounts, with the exception of identity tests.

Since the v2 API is deprecated, we don't need to run v2 admin
tests in the gate anymore. However v2 user APIs are most likely
still widely used by OpenStack users, and we should keep those
test runnings.

Right now identity tests control which version of the identity
API is used to allocate / provision test accounts by setting
an identity_version class attribute.

This behaviour forces us to run an identity v2 admin endpoint
only to be able to provision accounts for identity v2 tests.

This commit changes the current behaviour to always provision
accounts with the identity API version specified in
CONF.identity.auth_version. When v3 is configured (which is
the case in all gate jobs), v3 credentials will be provisioned
and then used for v2 non-admin tests.

Depends-on: I801e6740258ddea2a1b628a209970e0307d39d12
Change-Id: I2601433585740030ab61433f0e8ca1a919275d9f
6 years ago
Andrea Frittoli 07acf26015 Move InvalidServiceTag
We are going to make stable for plugins, so it should
only depend on tempest.lib and config.

InvalidServiceTag is in the exceptions module.
It cannot be moved in tempest.lib.exceptions since it has no meaning
in there, so moving it to directly.

Change-Id: I969fb45a44ce08c176d64dfe1c58d75215eacdf5
6 years ago
Jenkins 9902bd3eaa Merge "Use sequence directly instead of using len()" 6 years ago
Masayuki Igawa 0c0f0143e1
Use sequence directly instead of using len()
This commit makes to use sequence directly instead of using
len(SEQUENCE). The original code works correctly, and it's really
straight forward. However, PEP8 recommends like below[1]. And it
makes code more simple, too.

For sequences, (strings, lists, tuples), use the fact that empty
sequences are false.

Yes: if not seq:
     if seq:

No: if len(seq):
    if not len(seq):


Change-Id: I8d41e16d82b1b3860a98e5217cb7a541fc83b907
6 years ago
Jordan Pittier 3b46d27c90 Move the `attr` decorator from to tempest/lib
I think it's a good idea to move all utility decorators into
tempest/lib/ This patch does that for the `attr`

Change-Id: Iaafbb112b6eee458089cc49918359a8a8d0485e2
6 years ago
Jenkins 9d52e99ef1 Merge "Move the `related_bug` decorator from to tempest/lib" 6 years ago
Jordan Pittier c5665a6cc7 Move the `related_bug` decorator from to tempest/lib
I think it's a good idea to move all utility decorators into
tempest/lib/ This patch does that for the `related_bug`

Change-Id: I846d575e41f4dddfd5642b7750e988f75a717e7d
6 years ago
Andrea Frittoli 6f05307629 Cleanup services decorator
Rather than repeating the list of services twice, use the
one defined in get_service_list. Note that this removes
baremetal from the list of accepted services.

Change-Id: Ib5a9b33a54ef1064f5dc8f6206ddd1c7d218fc8e
6 years ago
Andrea Frittoli 71c71e9444 Beautify assertEmpty and assertNotEmpty
Don't use 'list' as a parameter name since this assert can be used
for any sequence or collection. Add docstring for both helpers.

Change-Id: I747507c3a5d0cf6f84915c8dd9222f4c082cbc04
6 years ago
zhufl 92ade4b8b1 Improve error info in assertEmpty and assertNotEmpty
When assertEmpty fails, it will print:
        raise mismatch_error
    MismatchError: 0 != 5

this is to improve the error info as:
        raise mismatch_error
    MismatchError: 0 != 1: list is not empty: [u'1b95a8ea7e124a76bfadf6ce1906aa27']

and when assertNotEmpty fails, it will print:
        raise self.failureException(msg)
    AssertionError: 0 not greater than 0

this is to improve the error info as:
        raise self.failureException(msg)
    AssertionError: list is empty.

Change-Id: Icdf69b596fa135387721c49df164a68799ab9a70
6 years ago
Jenkins f2088f3b7b Merge "Remove call_until_true from test module" 6 years ago
Ken'ichi Ohmichi f2ba23f783 Remove call_until_true from test module
The call_until_true of test module is marked as deprecated and
we have a plan to remove it in Ocata cycle.
This patch removes it after switching to use test_utils' one on
all OpenStack projects.

Change-Id: Ide11a7434a4714e5d2211af1803333535f557370
6 years ago
Ken'ichi Ohmichi 44f01273cf Finish switching to decorators.idempotent_id
Most modules are switched to using decorators.idempotent_id and
this patch covers the remaining and removes the old decorator check.

NOTE: The old decorator test.idempotent_id is still necessary to be
      used on the other projects.

Change-Id: Ice7f593fbc109863eba11402c2f3af75fc6ab9a1
Closes-Bug: #1616913
6 years ago
Jordan Pittier e4be907e34 Remove an unused variable in the BaseTestCase class
The variable has become useless with the migration of the clients
to tempest/lib.

Change-Id: I98b655026a9cf2f9ba6d94d98c20fa46cc5dddbd
7 years ago
Jordan Pittier 525ec718f6 Hacking: enable H904
It's a really cool rule that will avoid suboptimal usage of the
logging library.

Change-Id: I6414624f6b51333d477dd88290158aaeb51b851f
7 years ago
Jenkins d3685ca17e Merge "Removing baremetal tests from tempest tree" 7 years ago