64 Commits

Author SHA1 Message Date
Hai Shi
6f52fc599a [DOC] Fix details of tox
We need put a space between a param's name and a param's value in tox
and it would be more clear.
eg: from `tox -epy35` to `tox -e py35`.

Change-Id: I86750f42669bb02dcfac2b62e294d557bf44bd4c
2017-04-07 00:47:00 +00:00
Masayuki Igawa
5a3ad34fe5
Change API-WG guideline link to specs.o.o
This commit changes API-WG guideline link to spec.openstack.org instead
of github.com. We should use openstack.org links as possible.

Change-Id: Id0265bd6136a0c6f4adedf5c3aef34d34bbc933e
2017-03-22 16:27:53 +09:00
Ken'ichi Ohmichi
8a08211b4b Drop allowing old test.idempotent_id
test.idempotent_id should not be allowed anymore on the check
to block containing such wrong usage into tempest anymore.

Change-Id: Ie45b5d4a28b645ad5145ac37decf70f2f08d4229
2017-03-06 16:08:28 -08:00
Jordan Pittier
2e7ae7c6e5 Remove the Stress framework
It's not used anymore. There was general consensus in Feb 2016
to deprecate it (see [1]) and remove it in Newton.

[1]  [qa] deprecating Tempest stress framework

Change-Id: Ib229985ea2a1fee495c9492c9ce1781e6bac1dc6
2016-10-19 14:31:13 +02:00
Ken'ichi Ohmichi
8db407576b Clarify the guideline of negative tests
We have discussed negative tests at OpenStack Summit Austin before
as https://etherpad.openstack.org/p/newton-qa-negative-testing

The conclusion is
* Keep the existing negative tests
  Tempest should not cover all negative test cases as integration tests.
  Such tests should be covered with function tests in each projects.
  Function tests are implemented in Nova, [Ceilometer -> Gabbi], Barbican
  36 negative tests are used in Refstack as required tests and they are valuable

So we need some guideline about negative tests for our future, and this
patch adds it.

Change-Id: I2ac1a4cf91c9357caca2d490e6bc15699b7f3dad
2016-10-04 15:52:22 -07:00
Chris Hoge
2b47841c4d Add guidance on negative tests in HACKING.rst
Adds guidance on negative tests based on work items from the Newton
design summit in Austin[1].

[1] https://etherpad.openstack.org/p/newton-qa-negative-testing

Change-Id: Ic44b0e17540a5e984006a4a74c8318ad7e74db31
2016-06-23 16:05:01 -07:00
Jenkins
55fd59831c Merge "Add hacking rule to enfore no config in tempest.lib" 2016-06-02 04:57:15 +00:00
Matthew Treinish
59d9eaabdd
Add hacking rule to enfore no config in tempest.lib
This commit adds a hacking rule to enforce that we never add a config
dependency on tempest/lib. Right now we're completely dependent on
reviewers catching this, it is a strong rule so we should ensure we
can't ever land a change that does this.

Change-Id: I1ab1ba52573c6706a50abcd021759c93dd19aa44
2016-06-01 15:05:59 -04:00
Masayuki Igawa
e63cf0fedf Fix docs errors and warnings
This commit fixes docs errors and warnings. However, 'WARNING: nonlocal
image URI found' is still remaining because it needs an ugly workaround
that is replacing it to raw html.

Change-Id: I4524c1ff4126c87979301d0fb8ac7c9eb6f2f708
2016-05-25 10:25:21 +09:00
Franklin Naval
e8896de357 Docs: Fix Hacking guide bulleting
* fix list formatting of fixtures section

Change-Id: I9b2c11c59c34cae0086e2af507aa85313b8480bc
Closes-Bug: #1582350
2016-05-16 15:21:00 -05:00
Jenkins
43d742dd6d Merge "Remove negative test framework documentation" 2016-04-30 19:12:43 +00:00
Andrea Frittoli (andreaf)
1370bafda0 Remove few refercences left to tempest-lib
A few places in docs / comments still refer to tempest-lib.
Cleaning those up.

Change-Id: I6a15a6e7e58a49a39b3955c60e16d0c196ec154c
2016-04-29 20:30:34 +00:00
Luz Cazares
e28c18f656 Remove negative test framework documentation
We have a conclusion to remove the description of negative test
framework during Austin summit, details at:
https://etherpad.openstack.org/p/newton-qa-negative-testing

Change-Id: Ia3d36b8d1c8e9721b88dbc35bc350af822b8dc6f
2016-04-29 13:25:26 -07:00
Ken'ichi Ohmichi
d079c8925d Add pep8 check to use data_utils.rand_uuid()
There is a common method for generating random uuid value.
So it is nice to avoid pointing it out on human review to reduce
reviewing workload.
In addition, this patch removes str() around the method calls
because str() is called in the method.

Change-Id: Ic8caea352d27c479fdb765a699d277b59f1c5bb4
2016-04-19 14:23:00 -07:00
Sean Dague
ed6e586b9f deprecate use of tenant in configs
OpenStack has settled (a very long time ago) on project to represent
the container which has users. We should make tempest configs always
use project versions of these names. This deprecates the old tenant
versions of these names, and makes project version the prefered
option.

Change-Id: Ied68c0326cf891a4e6eb3db51dc5f43241e4b8d2
2016-04-06 22:26:18 +01:00
Ken'ichi Ohmichi
0dc9747519 Add pep8 check for tempest.lib import
tempest.lib should not import local tempest code to avoid circular dependency,
so this patch adds pep8 check to block such kind of code.

Change-Id: I392d28b3195040a800d96171ef275c6e73f9fef4
2016-03-29 11:14:05 -07:00
Ken'ichi Ohmichi
4f525f7cca Add T111 history to HACKING.rst
On the commit I36592a91001361d816e302745fe8417aee5e7565 , I forgot
to add T111 history to HACKING.rst. So this patch fixes it.

Change-Id: Ib4fd6e6d6f20cbe6c8b2914319aca721fb223850
2016-03-25 15:21:10 -07:00
Matthew Treinish
c1802bc956
Use the idempotent id and tooling from tempest-lib
The idempotent id decorator and the check-uuid script were migrated to
tempest-lib as part of tempest-lib 0.10.0. However, tempest was never
updated to use the library version of idempotent id or check-uuid.
This commit updates tempest to remove it's local copies of the
idempotent_id decorator and the check-uuid tool and use them from
tempest-lib instead.

Also, instead of updating the import of every single test module in
tempest an alias is just added to tempest.test to let all the old
imports work seamlessly. Updating the imports and removing the alias
may or may not be worth it in the future, but that will be a follow
on patch or discussion.

Change-Id: Iae3f491ada8f7b55c1fd817e033d968c2102d11e
2015-12-03 19:20:44 -05:00
Ken'ichi Ohmichi
c0d96bec35 Add hacking rule for "GET /resources"
This patch is a prototype for "GET /resources" hacking rule.
black_list_T110.txt file contains the service client files which
are against this rule. So we need to fix them with removing them
from this file.

Partially implements blueprint consistent-service-method-names

Change-Id: I150fe2ef21d4d4d246a46d9baf2fb14cc7d79ee5
2015-11-16 05:32:01 +00:00
Andrea Frittoli (andreaf)
17209bb083 Rename isolated creds to dynamic creds
Renaming the isolated_creds module to dynamic_creds module, and rename
the IsolatedCreds class to DynamicCredentialProvider in preparation
to migration to tempest-lib.

Partially implements bp tempest-library

Change-Id: I78a4884e980ef7d0103639cb3792a54c69fb7761
2015-10-06 15:02:01 +01:00
Jenkins
9c0bb46692 Merge "Add hacking check for testtools.skip" 2015-09-07 15:30:10 +00:00
John Warren
3059a09e24 Add hacking check for testtools.skip
Adding hacking check for testtools.skip decorator as described
in the bug referenced below.

Closes-Bug: #1490590
Change-Id: I9c456bb8b57a54fbcd1cd5249eec2292916d4ff9
2015-09-02 15:22:10 -04:00
Matthew Treinish
f45ba2e604
Cleanup tempest docs a bit
This commit fixes a few things in the tempest docs. First it fixes all
of the sphinx warnings and enables fail on warn to ensure we're using
valid sphinx everywhere. It also adds a link from the configuration
guide to the sample config file.

Change-Id: I3ad645a6bbfa46b4498e1732410743e46d6eb0cb
2015-09-02 11:06:44 -04:00
Naomichi Wakui
dbe9aab53f Fix typo in tempest coding guide
"funcionality" was mis-spelled as "functionality " in
tempest/HACKING.html

Closes-Bug #1488729

Change-Id: I3c981852e8b9fe776cf4c111e98c9c0e5586dc3f
2015-08-26 03:36:02 +00:00
Xicheng Chang
6fb98ec084 Fix typo in Tempest Coding Guide.
"doctstrings" was mis-spelled as "docstings" in
tempest/HACKING.html

Change-Id: I0ae2b84af718cc032cd031612a898895a2b3c21d
Closes-bug: #1484356
2015-08-13 15:30:16 -07:00
Chris Hoge
0e000ed1dd Add documentation to HACKING.rst describing idempotent_id
All Tempest tests require an idempotent_id decorator to uniquely
identify test functionlity. Previously this requirement was not
documented, leading to some confusion regarding its use and
implemtation. This patch includes documentation in the HACKING.rst
file that describes the purpose, format, and tool for working
with idempotent_id metadata.

Change-Id: I569bb68adc498f428664ed0b49d5943a353124b9
2015-07-28 14:23:54 -05:00
Ken'ichi Ohmichi
80369a96a2 Add a rule for blocking "-" from rand_name call
data_utils.rand_name() appends randam charactors with a hypen like:

 def rand_name(name=''):
     randbits = str(random.randint(1, 0x7fffffff))
     if name:
         return name + '-' + randbits

So it is not necessary to specify a hypen in caller side.
This patch adds a hacking rule for blocking "-" at the end of argument
of rand_name() calls.

Change-Id: I9b9f25dbe5a3ef5ac5900113bcc46e0d911becc9
2015-06-18 00:41:22 +00:00
Matthew Treinish
a970d65486 Cleanup the tempest readme
This commit makes several needed cleanups and improvements to the
tempest readme to make things clearer and elaborate on some sections.
Also as the readme is intended to be a tempest users guide, this moves
sections which are more developer focused to the hacking guide.

Change-Id: I4180ce18268443873fe7d8d5e7d0aaebc2cd1e08
2015-03-12 10:03:11 -04:00
Ken'ichi Ohmichi
7581bcdd22 Add the description of hacking rule T107
HACKING.rst doesn't contain T107 description, so this patch adds it.

Change-Id: I2e9a0c190016c3007ab37b502013bff222a9fc3d
2015-02-16 04:09:58 +00:00
Jenkins
07a3f45625 Merge "Framework for staged setup" 2014-12-11 01:52:04 +00:00
Andrea Frittoli
a5ddd556c6 Framework for staged setup
Split up the resource_setup in a number of methods to avoid
calls to tearDownClass when not needed, and enforce good
practices in the structure of the setUpClass.

Partially-implements bp:resource-cleanup

Change-Id: I3eae85f8e42f8ed58c87e6336792faf66c8f13aa
2014-12-10 21:42:29 +00:00
Marc Koderer
b3875b0f13 Adapt negative testing documentation
- Removes the json file documentation
 - Removes XML part

Change-Id: I6a7fdcb12c84189f9adde6f83fee8b303c28a4ee
2014-11-28 14:59:01 +01:00
ghanshyam
50f1947484 Fix doc for usage of python clients in scenario tests
Scenario tests has been migrated from official python clients to
Tempest clients.
Documents for the same needs to be fixed.

This patch fix the README & HACKING file for above changes.

This patch also extends hacking rule of not import python clients
for scenario tests too.

Change-Id: Ieb19a2c0b09f00fb3d4f6c7c73541275a4cf24ae
2014-11-26 17:04:37 +09:00
Jenkins
149b4a43c2 Merge "Update docs for regenerating sample conf file" 2014-11-18 08:41:18 +00:00
Mithil Arun
be067ec235 Fixes grammatical errors in HACKING.rst
There are a few lines in HACKING.rst that were incorrect
grammatically. This fix changes those.

Change-Id: I8173dae3cf3a3e78a504b4edd4a7c1ff5b7fb9a4
Closes-Bug: #1389603
2014-11-13 18:35:58 +05:30
David Kranz
fb0f51ff9f Update docs for regenerating sample conf file
Change-Id: I0999b75f43248b7e3e826845aaabff239aeab013
2014-11-11 14:07:20 -05:00
Dougal Matthews
4bebca0a28 Correct docstring indentation
Change-Id: Ifb4ad284f2a36eb6ab42ae2f344f030215186d69
2014-10-29 08:51:56 +00:00
Masayuki Igawa
93424e5021 Remove OfficialClient dependency from HACKING.rst
This commit migrates OfficialClientTest to ScenarioTest about
TestVolumeBootPattern in HACKING.rst. We should not have such an example
in the HACKING.rst because we don't use OfficialClientTest class in the
scenario tests anymore.

Partially-implements: bp:tempest-client-scenarios

Change-Id: Id95d011f3d080effc7bfee6232578451f7ba72d9
2014-10-06 13:59:47 +09:00
Matthew Treinish
550788819a Add a test documentation section to the docs
This commit adds a new section the HACKING and REVIEWING docs. It
explains the guidelines around inline documentation when
contributing new tests.

Change-Id: Ib56723d0c748baf41439975102d1363f81085f50
2014-09-23 10:04:40 -04:00
Ghanshyam
2a180b8398 Fix "mutable" object as default value
This patch fix the coding issue where "mutable" objects like list, dict
are being used as default value of param.

Current functions with mutable object as default value can create issues
if they will be called in more random order.
For example - first call with non default value and then with default
value. In this case original default will be overridden by the first
function call (as mutable object keep their state among function calls).

This commit also add the hacking check for the same and its
corresponding test.

Closes-Bug: #1330322

Change-Id: I251b316ef6d37f4b95c5e8d73a20a39005c22870
2014-09-01 11:13:22 +09:00
Marc Koderer
313cbd554d Update documentation for negative testing
Since generate_scenario() call is not needed any longer update the
documentation to the current state.

Change-Id: I34e8b974d5c4851a27f53a20778a181985ff6b5b
2014-03-26 10:08:42 +01:00
Jenkins
47e3ac11fd Merge "Adapt documentation for negative testing" 2014-02-25 08:45:49 +00:00
Masayuki Igawa
fcacf96204 Introduce T106 rule for vi modelines
We don't need to have the vi modelines in each source file anymore.
We've already fixed them several times.
 https://review.openstack.org/#/c/66507/
 https://review.openstack.org/#/c/68552/
 https://review.openstack.org/#/c/69318/
 https://review.openstack.org/#/c/70133/
However, newly some files still have it in its header. So we should
check it automatically with our HACKING rule.
This commit introduces T106 rule for that.

Note: This code is copied from Nova's hacking rule.
   Change-Id: I347307a5145b2760c69085b6ca850d6a9137ffc6

Change-Id: I5c94ef041a39c2377ba6321ace8934f324287bcf
Closes-Bug: #1229324
2014-02-20 11:42:35 +09:00
Marc Koderer
a5afb4fc3e Adapt documentation for negative testing
With the introduction of the new negative testing framework the workflow
for new test has changed.

Partially-implements: bp negative-tests
Change-Id: I522c2be227a120a6a714349fa29b31e60b5e0bf4
2014-02-11 19:02:47 +01:00
Bruce R. Montague
44a6a194ff Fix three accidentally formatted paragraphs
Three paragraphs are misformated in the output due to
a leading space in a line. The leading space causes a
line break and the indent of the line.

Change-Id: I5c0d03a47a2ca305b2b077068df26fadccd72212
2013-12-17 09:06:04 -08:00
Matthew Treinish
ecf212c49f Update tempest hacking regarding unit tests
This commit just updates some of the hacking documentation around unit
tests. It also adds a new hacking rule to ensure that setUpClass isn't
used for the unit tests.

Partially implements bp unit-tests

Change-Id: Ie8b1c9f1312a467265d53bc28ee905fa1b5fbb53
2013-12-06 20:00:13 +00:00
Matthew Treinish
6eb0585f70 Improve the UX around sample config generation
This commit just adds some documentation around using the sample config
generator. It also adds a wrapper script that sets the env variable for
running the oslo generator script with tempest.

Change-Id: I860b17defeeb85174e68b1e8b046b0e43c30683b
2013-11-26 15:47:52 +00:00
Marc Koderer
31fe483851 Add additional documentation for stress tests
In order to encourage people to add more and more stress tests we
need more documentation about it. This documents the stresstest
decorator and gives a global overview.

Change-Id: I65294fb278faba7a0d324782bf50e2fdedc5c1de
2013-11-11 08:16:03 +01:00
Matthew Treinish
8b79bb395c Add section for negative tests to HACKING.rst
This commit adds a new section to HACKING.rst to outline the
guidelines for adding negative tests to tempest.

Change-Id: Ieb3fc3670bb9229c6791e5c8c0a7ac4fbde03c4d
2013-10-10 17:11:05 -04:00
Giulio Fidente
83181a9703 introduces skip_because decorator
this change introduces a skip_because decorator which accepts two args
a bug and a condition; also updates the skip tracker accordingly

Change-Id: If53f2ef81d6bddbce284267216254b467046855f
2013-10-04 15:48:28 +02:00