194 Commits

Author SHA1 Message Date
Jenkins
a8a6f0334b Merge "Add script to use tempest-lib files in tempest" 2015-12-04 17:42:54 +00:00
Matthew Treinish
c1802bc956
Use the idempotent id and tooling from tempest-lib
The idempotent id decorator and the check-uuid script were migrated to
tempest-lib as part of tempest-lib 0.10.0. However, tempest was never
updated to use the library version of idempotent id or check-uuid.
This commit updates tempest to remove it's local copies of the
idempotent_id decorator and the check-uuid tool and use them from
tempest-lib instead.

Also, instead of updating the import of every single test module in
tempest an alias is just added to tempest.test to let all the old
imports work seamlessly. Updating the imports and removing the alias
may or may not be worth it in the future, but that will be a follow
on patch or discussion.

Change-Id: Iae3f491ada8f7b55c1fd817e033d968c2102d11e
2015-12-03 19:20:44 -05:00
ghanshyam
e73744bbd3 Add script to use tempest-lib files in tempest
As many files have been migrated to tempest-lib and
we need to remove those from tempest and make use of lib.

That is easy work but doing it manually takes times.
Automating that process same way it was done for migrating files to lib.

This commit adds new script use_tempest_lib for using the files
from tempest-lib.

In short this script does below steps-

- Get the latest tempest commit-id of file.
- Clone the tempest-lib and fetch the migration commit information
  - Get the tempest commit-id of file which was migrated to lib.
- Check Diff of both (latest commit-id and migrated commit-id).
- If both are same then delete those files from tempest and edit import of
  those file using tempest-lib file.

Sample - Id28c615534235c345e31b895a77a53ceac4af1a4

Change-Id: I8177f47113b6ab9b23af8ff86b87bb946ad549ad
2015-12-03 15:43:29 +09:00
Matthew Treinish
3898474abd
Switch run_tempest.sh to use subunit-trace
This commit switches the run_tempest.sh script to use subunit-trace
instead of the colorizer output filter. Since the colorizer isn't used
anymore it is removed.

The colorizer output has several issues. Firstly the color is actually
meaningless. It's based on arbitrary run times for the tests but don't
actually indicate anything useful. Additionally, it lacks a lot of the
debugging information that subunit-trace provides. Things like skip
messages, the worker breakdown, etc. are missing with the colorizor
output. The last aspect is that switching to use subunit-trace will
provide a consistent output format for running tempest whether you
use tox or run_tempest.sh.

Change-Id: I26e9c45f8e674bdb889b6bc65ad30c4abf6e7655
2015-11-23 13:09:51 -05:00
Ken'ichi Ohmichi
7616d19360 Fix H404/405 violations for tools
There is a lot of H404/405 violations in Tempest now, and that leads
difficult to migrate the code to tempest-lib or the other projects'
repos. This patch fixes these violations for tools for enabling
H404/405 rules on Tempest.

Change-Id: I4e1d78532d8f96edd5fd28a7bd2d62181fe9144b
2015-11-18 07:04:12 +00:00
David Paterson
0bf52d497b Fix init command
Init command was not working properly as it could not locate the
configuration sample file.
Steps taken:
- Move config-generator.tempest.conf to etc so it will be installed
with pbr.
- Update all files with new path to config-generator-tempest.conf as
needed.
- Refactor init command so if it detects we are not in a virtual
environment, try to find the global config directory /etc/tempest.
If that fails fallback to [sys.prefix]/etc/tempest.

Closes-Bug: #1491058
Closes-Bug: #1490670
Change-Id: I960bc711ff78ac2b0441ef63dff8ec4fb268fd3a
2015-09-23 09:42:07 -04:00
Matthew Treinish
c8a39b488b
Stop gating on sample generation check
This removes the check on in tree sample generation to ensure the file
matches the config options. However, to counter the loss of the in-tree
the tempest init command will now automatically generate a new sample
config in the local etc dir when run. This will include all of the oslo
options that were previously included in the in-tree sample file.
Additionally, the docs now contain the complete text of the generated
sample output. These also get published on docs.openstack.org on each
commit to tempest. Between this 2 alternatives there shouldn't be a need
to keep the file in tree anymore, especially given the weekly headaches
with new oslo releases changing the config option help breaking the job.

Change-Id: Ie3bc368329d240ca9604cc2f9b2cac544b3ee715
Depends-On: I4064a041a965ed2419b68efc8dc31fce37b24cfd
2015-08-04 11:00:11 -04:00
ghanshyam
f5360ff14f Sync venv scripts from oslo-incubator
Currently install_venv.py and with_venv.sh create virtual
environment in .venv directory and there is no way to specify
the virtual environment path explicitly.

This has been fixed in oslo-incubator.
This patch sync those virtual environment scripts from oslo-incubator.
Also make run_test.sh and run_tempest.sh scripts to use 'VENV' env variable.

Change-Id: I55311cff4dea1f6c5adaa2a6d51eeee2f6cc71ea
Closes-Bug: 1370086
2015-07-10 11:53:58 +09:00
Joe Gordon
4f10e45db7 Clarify how to resolve a uuid collision
Previously 'tox -epep8' just says run 'tox -v -euuidgen' but without
manually removing the duplicate uuid it fails to fix things and says run
'tox -v -euuidgen' again.

Change-Id: I1948c2d038d89af8dcfe69d01dcc49d024ae3210
2015-03-27 13:15:44 -04:00
Marc Koderer
07a5a6f58d Make check_uuid executable
Check_uuid is a script and not a module. This makes it executable
from CLI.

Change-Id: I1c5b6a79e412471f27bc9779dd821a52ec83c645
2015-03-17 09:14:55 +01:00
Marc Koderer
fb199c0516 Add additional checks for attributes
Some decorators (like dtt) do have a func attribute
but no "attr" and "value" attribute defined.
This leads to errors during pep8 gate check (see [1]).

[1]: https://review.openstack.org/#/c/153234/4

Closes-Bug: #1431267
Change-Id: Id16bce40e30ecb6bc5bff9b55650acc538ece3a9
2015-03-16 12:04:11 +01:00
Doug Hellmann
583ce2c043 Update all Oslo module use
Use graduated libraries where available and update the incubated modules
that remain.

Update imports for libraries that were being used already to avoid the
'oslo' namespace package.

Update config file for creating example configuration files and provide
a new sample file based on the options defined in the libraries.

Note: because of the number of libraries involved there is no real way
to break this change up into smaller pieces. Fortunately, most of the
changes are just to import statements.

Change-Id: Ia0de957b681cb924a57af98d99a9389ee234ed5b
2015-03-11 21:07:56 +00:00
Chris Hoge
7579c1a40e Add UUIDs to all tempest tests and gate check
Adds uuid4 as a decorator of the form:
@test.idempotent_id('12345678-1234-1234-1234-123456789abc')
to every test in the Tempest tree. Includes a gate check to
ensure the existence and uniqueness of the ids.

Modify check tool to ignore Tempest unit tests.

Change-Id: I19e3c7dd555a3ea09d585fb9091c357a300e6559
Co-Authored-By: Sergey Slipushenko <sslypushenko@mirantis.com>
Implements: bp test-uuid
2015-02-26 16:05:16 -08:00
Chris Hoge
296558cf41 Add UUIDs to tests with tools and checks
This patch adds a new tool to check for existence and uniqueness
of UUIDs across Tempest or any other test repository based on
Tempest. The tool also includes an option to automatically tag
a test repository with UUIDs if they don't exist. The tool
will be used in the gate to ensure UUID existence.

Change-Id: I25aa83c7836f5a607af2aaa4bf862fa72766f799
Co-Authored-By: Sergey Slipushenko <sslypushenko@mirantis.com>
Partially-Implements: bp test-uuid
2015-02-26 13:59:05 -08:00
Matthew Treinish
16b6826b94 Fix slowest test output after test run
This commit fixes the output from pretty_tox.sh so that the testr
slowest output isn't gobbled up by the pipe into subunit-trace.

Change-Id: I11e8fe80c35a1d3a90a4d7c8bf2b36c4114b5cc6
2014-12-18 15:14:39 +00:00
Jenkins
31a91ee725 Merge "Fixed a comment on dirty logs list" 2014-11-11 16:52:04 +00:00
Ihar Hrachyshka
226de0b8f8 Fixed a comment on dirty logs list
We've recently changed default behaviour to forbid any dirty logs for
services outside the list instead of explicitly enabling log checks for
specific services. The comment was not updated to reflect the change
though.

Change-Id: I5d5ba64d156a9b5318cfc2740b2a69d75de3c127
2014-10-31 13:56:36 +01:00
Matthew Treinish
cdb7f549a9 Switch to using subunit-trace from tempest-lib
This commit removes the tempest copy of subunit-trace and uses the
entry-point from tempest-lib instead.

Partially implements bp tempest-library

Change-Id: Ic1aeddc2d0b217c41b6b9914ac3f269b80c4ec8e
2014-10-23 15:07:49 +00:00
Jon Grimm
270bd7f8ce Move tempest to oslo-config-generator
Also fixes tempest reported bug with old generator in oslo-incubator,
when identical options are registered in different groups.

Change-Id: I274f48503c1b61ca09cabc8d8c50245ba1059943
Closes-bug: 1282887
2014-10-16 16:06:43 +00:00
Jenkins
cc6ca0d184 Merge "Make output from check_logs less verbose" 2014-09-20 07:32:35 +00:00
Jenkins
c6dac5315e Merge "Fail tempest if 0 tests are run" 2014-09-16 18:20:49 +00:00
Jenkins
24d946e188 Merge "Remove test start output in pretty tracer" 2014-09-16 07:24:45 +00:00
Matthew Treinish
178cc4a605 Fail tempest if 0 tests are run
This commit adds a check to subunit-trace to fail if the test run
doesn't actually run anything. Despite technically passing without
running anything, if you're passing a regex to testr that runs 0 tests
that is probably an error, so treat it as such. This also uncovered a
bug in the test_wrappers unit test where the test was using a bad test
regex but still passsing. Since we start enforcing it here the unit
test is also updated to use the proper regex.

Change-Id: I109a8cd6eb731e5b8daf5cc2d6a76498ddd48c6d
2014-09-15 20:13:44 +00:00
Clark Boylan
a5c669d7f0 Properly detect grenade in check_logs script
The check logs script was not properly detecting grenade and would fail
grenade jobs if the logs were not clean. We don't want to do this
because the upgrade process is not log clean yet. Correct this by
setting grenade flag to true only if the DEVSTACK_GATE_GRENADE variable
has content.

Change-Id: Id8a339d78d981376b9af3a80aae8e4e18a68a85f
2014-09-03 21:48:49 -07:00
Matthew Treinish
1179205dbc Make output from check_logs less verbose
This commit changes the output of check_logs.py to be a bit more
concise and clear. Instead of printing out every instance of errors
found in the log files it will only print out which individual files
had errors. If a file is marked as not allowing ERRORS or TRACES it
also prints that along with the filename. This should hopefully avoid
confusion around the operation of this script running in the gate.

Change-Id: Ib2a9a9736470cd3b445fb51a95a1a6c6696bb2cc
2014-09-03 21:42:59 -04:00
Robert Collins
b5e4a9872b Remove test start output in pretty tracer
Sean does not want and its currently broken.

Change-Id: I7c3cf17bfd8d273823774cd162ae011161c36d3d
2014-08-18 13:20:07 +12:00
Matthew Treinish
96e9e88dc0 Enable H407,H305,H307,E122 ignore E123
After bumping the hacking version to the 0.9.x series ignores were
added for several rules. This commit fixes the violations for a subset
of these rules and re-enables the checks.

E123 is ignored because this rule is ignored in the default pep8 by
default, due to a lack of consensus around it.

Change-Id: I8f14cd2ca6afc38d3fe8ee758272071111022896
2014-08-12 17:39:16 -04:00
Robert Mizielski
e1d8899945 Correct misspelled words
Correct some misspelled words:
responCe => responSe,
unkown => unkNown
infomation => infoRmation.

Change-Id: If0f8c2b71381be087694654969d7ea1483da7b17
2014-07-15 14:54:58 +00:00
Matthew Treinish
afeb91a5da Use config generator rc instead of wrapper script
This commit removes the generate_sample wrapper script and adds an
rc file for the config generator to specify the alternate path.
This makes console messages from the generator tooling point
consistent with what is actually being used. Before because a wrapper
was used error messages would recommend running the incorrect tool.

Change-Id: I0ac792454777af6e3f1d718836031f9a14850229
2014-06-19 16:54:48 +00:00
Matthew Treinish
4251685fac Update config generator from oslo-incubator
This commit pulls in the latests version of the config generator from
oslo.

Change-Id: I3a2b06dd9ca3285a3d93298ea5727b705f6109e4
2014-06-19 16:45:48 +00:00
Matthew Treinish
53eef72128 Add total runtime to summary output
This commit adds the total runtime in secs to the summary view from
subunit-trace.py

Change-Id: If56f2a048b3d1ab3d51425d5b3560e1f365a6c3a
2014-06-12 23:06:30 -04:00
Matthew Treinish
1d5f32b48e Change how tempest debug logs are displayed
This commit cleans up how tempest failure logs are displayed. We'll
no longer dump the failure information to the console as a test fails.
Instead all the failure logs will be printed after the tests are run.

Change-Id: I7ecdc349d913b43f4fb0505d5c17c66f811774b4
2014-06-10 15:19:49 +00:00
Jenkins
ad681f9655 Merge "Remove tempest_auto_config script" 2014-05-10 13:24:32 +00:00
Matthew Treinish
2adef1a87f Remove tempest_auto_config script
This script isn't being used by anyone to my knowledge nor has it
really been maintained since it was first added. We will be working
on a plan for automating tempest configuration soon and this
doesn't really fit the initial set requirements of requirements. It
also does not have any testing, so I expect that it has bit rotted
and probably doesn't work anymore.

Change-Id: Id896d9877312dfb6b12b54ea1c598a1940cdd9ce
2014-05-09 15:18:39 -04:00
Sean Dague
04867448e1 create test and worker summary report
a summary report that includes counts of the test results so far
as well as a worker report to let us understand how our workers
are balanced at the moment, as well as if we are missing worker
0 in our data (which seems to happen quite often).

Change-Id: Ide0b63b86e87f8a18c2f0e45264692ac3797c791
2014-05-08 10:25:38 -04:00
Jenkins
a5ddec8309 Merge "Removing unnecessary pass instructions" 2014-05-08 13:15:06 +00:00
Sean Dague
50af5d5ecc Add in a concurrency aware subunit filter
A live filter for subunit stream that will let us display the
concurrency of the streams as we go.

* This includes the worker id as {#} at the beginning of the lines
* Dumps out stdout/stderr if they are found inline (makes for easier
  debug
* Dumps out pythonlogging on failures if found inline
* Prints skip reasons

Based on I1b529546e005f47aba56b451e1c0d8b0da09fca3, but because
that started as Robert's I couldn't un Abandon it.

Change-Id: Icc99b652e4e8ae85b73bb905a3b704447a63195f
Co-Authored-By: Robert Collins <rbtcollins@hp.com>
2014-05-06 10:59:00 -04:00
Yaroslav Lobankov
4693783f7e Removing unnecessary pass instructions
There are some files in Tempest where code has unnecessary
pass instructions. This CR fixes that.

Change-Id: Ie32399d731edad8a4d0b9cdec8aa7e9444eda254
2014-05-06 14:14:44 +04:00
Matthew Treinish
f8b816af07 Add cmd entry point for verify_tempest_config
This commit moves the verify_tempest_config script from the tools dir
to tempest.cmd and adds a pbr entry point for the script. This means
that the script will in effect be a packaged binary for tempest.

Partially implements bp config-verification

Change-Id: I75b9743a8117d55c6c445db05eb39404b5708624
2014-04-30 17:36:12 +00:00
Matthew Treinish
f09717137a Add support for updating the config file
This commit adds a new flag to the verify_tempest_config.py script,
'-u' that will update the config file from values found during the
verification process.

Partially implements bp config-verification

Change-Id: I20b8bca7d3368b56da0a0496ed7aa3c1a6eacc9e
2014-04-30 13:35:29 -04:00
Matthew Treinish
2e439637f2 Add cinder api version detection to verify_tempest_config
This commit adds a check for the cinder api versions that are
supported and checks that with what is set in the config file.

Partially implements bp config-verification

Change-Id: Ifce002a6f42c1565992299965bcc490f20aa5887
2014-04-11 20:13:06 +00:00
Matthew Treinish
42d50f61e1 Fix service list in verify_tempest_config
This commit updates the service list for verify_tempest_config. The
list was not updated to include new options for marconi and trove. It
also wasn't updated for the sahara rename.

Partially implements bp config-verification

Change-Id: I00500e104dda34098cf445f4bbd522ea843fcabf
2014-04-11 20:13:06 +00:00
Matthew Treinish
1722f0b17c Fix url parsing for api version check
This commit fixes a couple of oversights in the common api version
check function. The previous version of the method would never work
because the url wasn't passed to urlparse, and the resulting endpoint
url would be missing a ':' even if it were.

Change-Id: I4b6bb47f0601063aa8164ded386ca71adc6ccc80
2014-04-11 20:13:06 +00:00
Sean Dague
ef2865af43 remove n-sch from the watch list
we've seen no n-sch errors in passing gate runs in the last week,
I consider this sufficient proof that we're clean on n-sch for now.

Part of bp:clean-logs

Change-Id: Ife0e0602767e155fc645e07546bed36c2c060a20
2014-04-10 07:26:09 -04:00
Matthew Treinish
864fe07b13 Add keystone api version detection to verify_tempest_config
This commit adds a check for the keystone api versions that are
supported and checks that with the config file. In the process of
adding the additional check the version verification code is reworked
to use a common get versions method.

Partially implements bp config-verification

Change-Id: Ibb15c53e2b2a1e2a2ad95351692d06389f2ebb06
2014-04-09 17:27:36 +00:00
Sean Dague
e2cda41ad5 ceilometer-collector now has errors
need to add this to the dirty list to make forward progress.

In order to make the results more readable we also need a line
break before the log file stanza.

Change-Id: I863a8a30384edf27a4ba761c707b5cf673c1f86e
2014-03-26 15:39:05 -04:00
Jenkins
fd5aa5e1cd Merge "Update Oslo config sample generator" 2014-03-26 08:29:33 +00:00
Jenkins
590cde1f79 Merge "Add service/endpoint discover to verify_tempest_config" 2014-03-21 23:02:12 +00:00
Jenkins
02b4c04697 Merge "change dirty logs to work off a whitelist" 2014-03-19 17:37:47 +00:00
Jenkins
2258bddb36 Merge "Add swift discoverable_api support to verify_tempest_config" 2014-03-19 09:02:01 +00:00