140 Commits

Author SHA1 Message Date
Matthew Treinish
bdef1c77ab
Move oslo config generator config inside package
This commit moves the oslo config generator config file from living as
a data file, which makes it very difficult to rely on, to just living
in the package. By including it in the package we have a guaranteed
relative path we can rely on for building utilities on it, like the
tempest init command.

Change-Id: I5b9deae733f42fe3ed39418103304f71b43dccb3
Partially-implements: bp tempest-run-cmd
2016-06-22 14:39:42 -04:00
Matthew Treinish
0e25a90793
Use a single venv for all tempest jobs
This commit simplifies the venv management via tox for tempest jobs.
All tox targets that actually run tempest will now share a single tox
venv .tox/tempest. This also adds a new target tempest-venv which is
the same as the pre-existing venv target, but uses the common venv.
We can't simply use the common venv with the venv target because
external tooling, like the docs jobs, rely on the .tox/venv path
existing when using the venv target.

Depends-On: I34a69020eee07156e64026781a3c0bffdb5ab415
Change-Id: Iab2e6e04b6c5795a4d0c8214564106525b942308
2016-05-04 13:10:49 -04:00
Matthew Treinish
3a9d3e70d5 Add ostestr tox jobs
This commit adds tox jobs to run tempest with ostestr instead of the
pretty_tox*.sh wrappers. ostestr provides essentially the same
functionality except it exposes slightly more rich selection logic
and additional debugging options. However, since the cli isn't
backwards compatible with pretty_tox we can't simply remove the old
tox jobs.

Change-Id: I347438b1d561af7e05fdce895e2e3e51d2b38b0c
2016-04-21 13:14:50 +09:00
Masayuki Igawa
d81d3ceed7 Align multiple lines in tox
This commit aligns multiple lines in tox for visual consistency.
I fixed them to 4 white spaces because the tox example[1] shows 4 white
spaces indentation.

[1] http://tox.readthedocs.org/en/latest/example/basic.html

Change-Id: I4f3b9c5c4982a0e96bb120ce8546dd48b0fa8a4b
2016-04-20 15:11:35 +09:00
Jenkins
0aeea56f18 Merge "remove largeops test" 2016-03-01 06:11:22 +00:00
Sean Dague
1976da83a5 remove largeops test
This isn't really a scenario test, it's a performance test, however
without any of the infrastructure to do performance testing /
comparisons. It's time has come to say goodbye.

Depends-On: I788cd70dc4e6a9c3f8e6805d5bdcdf7f549fce35

Which removes it's use from the gate.

Change-Id: I93b2fb33e97381f7c1e0cb1ef09ebc5c42c16ecc
2016-02-24 20:24:58 -05:00
Andrea Frittoli (andreaf)
897ecfb602 Add reno to tempest
Tempest is going to be on pypi and expose a stable interface in
the tempest.lib namespace, so adding reno to it.

Partially-implements: bp tempest-lib-reintegration
Change-Id: Ice4c27f7bc6d0154cfced9789f3c82236106a9e3
2016-02-24 11:31:32 -05:00
Ghanshyam
3a59e1ef92 Raise tox min version to avoid maximum recursion issue
There was regression in tox 2.2.0 for env-substitution
with setenv.

https://bitbucket.org/hpk42/tox/issues/99/environment-variable-usage-schema-env-xxx

It is fixed in 2.3.1
https://bitbucket.org/hpk42/tox/issues/285/tox-220-breaks-some-toxini-config-files

So no issue with 2.3.1, but let's raise the min version in tox.ini
to avoid maximum recursion issue.

Change-Id: Ie808e46a90a681b21c30998d2fc867981c2c05fd
Closes-Bug: #1534006
2016-01-14 17:15:29 +09:00
Sergey Reshetnyak
e05246d035 Add support override OS_TEST_TIMEOUT for all environments
Change-Id: Ic615c3958c30aa14f46784923d4cc12d634ab099
2015-12-28 18:06:33 +00:00
Jenkins
02ef4c2956 Merge "Put py34 first in the env order of tox" 2015-12-11 15:01:56 +00:00
Matthew Treinish
c1802bc956
Use the idempotent id and tooling from tempest-lib
The idempotent id decorator and the check-uuid script were migrated to
tempest-lib as part of tempest-lib 0.10.0. However, tempest was never
updated to use the library version of idempotent id or check-uuid.
This commit updates tempest to remove it's local copies of the
idempotent_id decorator and the check-uuid tool and use them from
tempest-lib instead.

Also, instead of updating the import of every single test module in
tempest an alias is just added to tempest.test to let all the old
imports work seamlessly. Updating the imports and removing the alias
may or may not be worth it in the future, but that will be a follow
on patch or discussion.

Change-Id: Iae3f491ada8f7b55c1fd817e033d968c2102d11e
2015-12-03 19:20:44 -05:00
lvdongbing
768c807657 Put py34 first in the env order of tox
To solve the problem of "db type could not be determined" on py34
we have to run first the py34 env to, then, run py27. This patch
puts py34 first in the tox.ini list of envs to avoid this problem
to happen.
Closes-bug: #1489059
Change-Id: Ia11ecf7bd147e35b7c8ad3db0eaad17f893e78ba
2015-12-01 22:46:38 -05:00
Masayuki Igawa
835220498e Fix typo in tox.ini
This commit fixes typos in tox.ini.

Change-Id: I990b5e3d713de5a19300f4ae7ead66d75a423429
2015-12-01 16:51:34 +09:00
Ken'ichi Ohmichi
2e2ee19179 Fix H404/405 violations for remaining
There is a lot of H404/405 violations in Tempest now, and that leads
difficult to migrate the code to tempest-lib or the other projects'
repos. This patch fixes these violations for remaining.

Change-Id: I3abac40ebb33836980c119d59bc97f035b213c0a
2015-11-19 11:22:29 +00:00
David Paterson
0bf52d497b Fix init command
Init command was not working properly as it could not locate the
configuration sample file.
Steps taken:
- Move config-generator.tempest.conf to etc so it will be installed
with pbr.
- Update all files with new path to config-generator-tempest.conf as
needed.
- Refactor init command so if it detects we are not in a virtual
environment, try to find the global config directory /etc/tempest.
If that fails fallback to [sys.prefix]/etc/tempest.

Closes-Bug: #1491058
Closes-Bug: #1490670
Change-Id: I960bc711ff78ac2b0441ef63dff8ec4fb268fd3a
2015-09-23 09:42:07 -04:00
hayderimran7
cd95f122ee Update tox.ini for sphinx build instructions.
Earlier we specified, that we must generate sample config
file before generating docs. Now after commit 13ed69fff
got merged, we must update the comments or any info related
to generating config file.

Change-Id: Ic0686322f07045700ff390eea8faa05eebe3df7c
2015-08-21 11:08:52 -07:00
Matthew Treinish
13ed69fffc
Bake in oslo sample config generation to sphinx build
The expectation from users is that to build docs locally all you need
to do is run build_sphinx on the repo and the docs will be fully
created. When we added the sample config to the tempest docs we ignored
this and just added the sample config generation to the docs tox job.
This commit fixes this by baking in the sample config generation to
the sphinx conf.py file which should ensure we have the sample config
generated every time that we call build_sphinx.

Change-Id: Ifd570707847f52592f5bc9bc743b8d3c98512639
2015-08-11 10:13:03 -04:00
Matthew Treinish
c8a39b488b
Stop gating on sample generation check
This removes the check on in tree sample generation to ensure the file
matches the config options. However, to counter the loss of the in-tree
the tempest init command will now automatically generate a new sample
config in the local etc dir when run. This will include all of the oslo
options that were previously included in the in-tree sample file.
Additionally, the docs now contain the complete text of the generated
sample output. These also get published on docs.openstack.org on each
commit to tempest. Between this 2 alternatives there shouldn't be a need
to keep the file in tree anymore, especially given the weekly headaches
with new oslo releases changing the config option help breaking the job.

Change-Id: Ie3bc368329d240ca9604cc2f9b2cac544b3ee715
Depends-On: I4064a041a965ed2419b68efc8dc31fce37b24cfd
2015-08-04 11:00:11 -04:00
Mathew Odden
b389faa22f Include sample configuration in dev docs
Since including the sample configuration in tree is
becoming difficult to maintain, lets try moving it to
the developer docs. We still have the ability to view it
easily for users, and it can be auto-generated on doc build,
so it should also always be up to date.

Change-Id: I7ec2d76448feef5ecb9af7c238bbd12619fa31d8
2015-08-04 10:59:38 -04:00
Marc Koderer
c2f9b3a3fe Activate site-packages for tempest plugins
If site-packages is deactivated tempest plugins aren't visible.
Create a new tox entry point instead of activating it globally.

Change-Id: If6cd204d4c1e64cf7bcf79bd5d32d3c10a1b2559
Closes-bug: #1478559
2015-07-27 15:19:08 +02:00
yan.haifeng
658544f6e6 Fix failed to access pip server when run behind proxy
When we run tempest on the machine behind proxy,the pip command failed to
connect to pypi.python.org to download packages.

Fix it by add another option "passenv" to [testenv] section.

Change-Id: Iedc40a2266be59a2298d5909486a1439f2757b8c
Closes-Bug: #1463978
2015-06-18 09:13:01 +08:00
Jenkins
7ebf503a9b Merge "Add passenv option to tox.ini" 2015-06-15 22:08:24 +00:00
Jenkins
cb169eec2d Merge "Remove heat-slow tox entry" 2015-05-29 15:39:46 +00:00
Attila Fazekas
83c6d08483 Remove heat-slow tox entry
The heat-slow tox entry tries to run 0 tests at the
moment.

Change-Id: I582c697d59d121bdd43d9c41eeba1b67c64a4861
2015-05-20 07:25:15 +02:00
Matthew Treinish
43073a2bf8 Add passenv option to tox.ini
Tox 2.0 introduced env variable separation, meaning that env vars set
in the env calling tox will not automatically be in the test env. This
breaks some assumptions we have in tempest. This commit adds all the
env vars which are used by testr or tempest directly for running tests
in the hope of making things work on tox 2.0 and older versions.

Change-Id: Icfe4bb5df0e48cbc9fcdbc602656672295d1887e
2015-05-14 09:05:35 -04:00
Matthew Treinish
3460aaaf3e
Add py34 to readme, pkg metadata, and envlist
This commit adds references explaining the current state of python 3.4
and tempest to the readme, pkg metadata, and adds an etry to the tox
envlist for running the unit tests on python 3.4.

Change-Id: I9c1e3cdf53dbae723a9fee67ad96932709d2305f
2015-05-13 10:17:58 -04:00
Matthew Treinish
464d287f5b
Remove CLI testing once and for all
The cli tests have been marked for removal for ~6 months and the
framework was the first thing included in tempest lib. There has
been more than enough time for all the projects to pick this up
in the client repos. So let's remove it all!

As part of this a couple of missing entries for tempest's
requirements.txt were found. These dependencies were being installed
by the clients so the fact they were missing was never noticed prior
to this. This commit also adds these missing entries back into the
requirements file.

Change-Id: I4f8638f1c048bbdb598dd181f4af272ef9923806
2015-05-07 14:17:44 -04:00
Sean Dague
3c634d1419 not all scenario tests should be in smoke
The decision to include all of scenario by default makes for an
extremely heavy test of tests in smoke, we should instead only include
specific scenario tests that expose some basic sanity
checking. Advanced scenario tests should be dropped from this list.

This change means we no longer have to exclude 'slow' tests, as we'll
just tag the right tests with smoke.

Change-Id: I34aa8aa9274e02bb68f62863d64c47ba9916f2e4
2015-04-29 06:05:50 -04:00
Jenkins
72c76893a1 Merge "Fix and re-enable H105" 2015-03-04 07:44:28 +00:00
Jenkins
d32c845a34 Merge "Move to hacking 0.10" 2015-03-04 07:43:34 +00:00
Chris Hoge
7579c1a40e Add UUIDs to all tempest tests and gate check
Adds uuid4 as a decorator of the form:
@test.idempotent_id('12345678-1234-1234-1234-123456789abc')
to every test in the Tempest tree. Includes a gate check to
ensure the existence and uniqueness of the ids.

Modify check tool to ignore Tempest unit tests.

Change-Id: I19e3c7dd555a3ea09d585fb9091c357a300e6559
Co-Authored-By: Sergey Slipushenko <sslypushenko@mirantis.com>
Implements: bp test-uuid
2015-02-26 16:05:16 -08:00
Chris Hoge
296558cf41 Add UUIDs to tests with tools and checks
This patch adds a new tool to check for existence and uniqueness
of UUIDs across Tempest or any other test repository based on
Tempest. The tool also includes an option to automatically tag
a test repository with UUIDs if they don't exist. The tool
will be used in the gate to ensure UUID existence.

Change-Id: I25aa83c7836f5a607af2aaa4bf862fa72766f799
Co-Authored-By: Sergey Slipushenko <sslypushenko@mirantis.com>
Partially-Implements: bp test-uuid
2015-02-26 13:59:05 -08:00
Joe Gordon
a62baa099d Fix and re-enable H105
Enable Hacking Hacking H105 (Don't use author tags, as we have git for
that)

Change-Id: I7c5379da34d21717feab19684ffc5ead6d50399b
2015-02-25 12:51:46 -08:00
Joe Gordon
28788b4ec0 Move to hacking 0.10
Release notes:
http://lists.openstack.org/pipermail/openstack-dev/2015-January/054165.html

Fix minor changes for new/improved rules.

Remove references to deleted rules. Skip new rule H105, enable in
subsequent patch.

Change-Id: Ibe07679034f575c403aced46c0221b96d299ea83
2015-02-25 12:42:37 -08:00
Joe Gordon
9f1612e7af Turn off sitepackages
Fully decouple the versions needed for tempest from the underlying
system. This will help make it possible to pin library versions on
stable branches to the library version used when the branch was
cut, while allowing master global requirements to move beyond that
version.

Note: We previously enabled sitepackages everywhere to avoid
recompiling indirect dependencies that were pulled in via the
pythonclients. To address this we were working on removing the CLI tests
from tempest.

Change-Id: I3b355fb592dbbd38cb6805ec413e02644088b088
2015-01-28 13:35:56 -08:00
Jeremy Stanley
0ae5d52a31 Preupgrade setuptools in tox tempestenv
* tox.ini: If setuptools is declared as a dependency, even
transitively, it can trigger a race in some versions of pip when
called with -U wherein pip removes setuptools in preparation for
upgrading it, and then proceeds to install something else using
setuptools. Work around this behavior by upgrading setuptools within
the virtualenv in advance.

Change-Id: I6d9bf61c1ce41e46ff6f621bfa88d511f6c16120
Related-Bug: #1405579
2014-12-29 20:37:14 +00:00
Sean Dague
c7b4d88c5a rm pyc files before test runs
It's extremely easy for incorrect local results to happen if a file
has been moved or deleted in an upstream pull. Add the rm pyc cantrip
to ensure local testing matches what a clean upstream is going to be
like as much as possible.

Change-Id: Iecf18d7c2310918ec90c14c64cd3083e46459980
2014-11-25 08:18:23 -05:00
David Kranz
03fbe5cae1 Fix tox stress invocation to not quote arguments
Change-Id: I6fc1e64c3e52a02126235d5cc42831ee2d25f997
2014-11-11 15:29:00 -05:00
David Kranz
f9695d21b3 Increase the 'all' timeout because slow tests are included
The default for tempest in .testr.conf is 500s. This was too short for
the heat slow tests so the time for that was increased to 1200. This
change is to a periodic job so if we need to it can be increased more
in the future.

Change-Id: If27ddef9865c54b997130c707382d2014d82e0a6
2014-10-30 14:53:05 -04:00
Jon Grimm
edcb24c858 Remove PYTHONHASHSEED=0 from tox pep8 job
Tempest has moved to new oslo-config-generator and this bug
workaround is no longer needed.

Change-Id: I50ffdcbc8a8f4830ba9c274af4c908e96467dad5
Closes-Bug: 1365136
2014-10-16 18:09:40 +00:00
Jon Grimm
270bd7f8ce Move tempest to oslo-config-generator
Also fixes tempest reported bug with old generator in oslo-incubator,
when identical options are registered in different groups.

Change-Id: I274f48503c1b61ca09cabc8d8c50245ba1059943
Closes-bug: 1282887
2014-10-16 16:06:43 +00:00
David Kranz
ea202d789e Don't use a specific arg list for stress in tox.ini
Callers should be able to specify the arguments.

Change-Id: I1d60af5e5300ff91ce3777ac46df655442098013
2014-10-10 12:13:05 -04:00
Matthew Treinish
5b7626a123 Clean up tox.ini and make tempest and unit test distinct
This commit creates a separate base env for tempest tests and
tempest's unit tests. However, to keep the same interface the tempest
env is setup by overloading the testenv using substitution from a
common base section.

As part of this internal reorganization in the tox.ini several other
things are cleaned up or removed. Usless or duplicate job definitions
are removed. The unit tests are switched to pretty_tox. Previously
they were run without any output which made it difficult to figure out
exactly what was running. Additionally, this removes the unnecessary
regex filter on the unit test job. The unit tests use a different
discovery path so there is no need to also filter it on the same path
this will allow the use of more specific regexes with tox to filter
tests.

Change-Id: I996d81dec2739bbe907f5fe87a335994c01697b3
2014-09-29 14:26:51 -04:00
Jenkins
5b45439078 Merge "Work toward Python 3.4 support and testing" 2014-09-29 16:41:59 +00:00
Jenkins
0c8dc861af Merge "Add bash to tox whitelist_externals" 2014-09-29 16:38:50 +00:00
Joe Gordon
fdd0663a55 Add bash to tox whitelist_externals
Stop triggering triggering a “not installed in virtualenv” every time
bash is called. Since we use this in all tempest based jobs, we have
these unnecessary warnings all over the place.

Change-Id: I8af53c0241dbcd656c274944338384be0f2f250d
2014-09-24 17:35:02 -07:00
Adam Gandelman
a10f768ac1 Remove PYTHONHASHSEED=0 from tox.ini
We've ironed out remaining issues that crop up when running
with a random seed. This updates tox.ini to revert to the default
behavior of running with PYTHONHASHSEED=random.  The pep8 check
continues to run with PYTHONHASHSEED=0 to workaround oslo issues
around config value ordering in the check_uptodate.sh test.

Change-Id: Ifa0dba8c3970fc676d374f33c129ef0f9d526c94
2014-09-15 12:09:52 -07:00
Jeremy Stanley
dd2e07acb9 Work toward Python 3.4 support and testing
Change-Id: If36e1a5e51cf63e499acb374d4aeace608c13512
2014-09-03 19:40:14 +00:00
Matthew Treinish
3a48181b1d Fix tox coverage job
To run the tempest unit tests with coverage enabled the coverage
module needs to be installed. This requirement is properly listed in
test-requirements.txt, however the coverage job definition wasn't
installing coverage. This commit fixes this oversight by adding the
test-requirements file as a dependency for the job.

Change-Id: Ic443c6c4a3166e6b198cbe2af0a77ff2f0a38e83
2014-08-26 19:08:43 -04:00
Adam Gandelman
359738a2be Ensure PYTHONHASHSEED=0 for the 'all' tox env
This removes the setenv override and sets the required
PYTHONHASHSEED=0 for the 'all' tox target, which devstack-vm-gate.sh
ends up calling for the current Ironic job.

Change-Id: Icc6a1cae1e49140c4e4648f6c75ad415f92479ce
Closes-bug: #1359929
2014-08-21 14:46:23 -07:00