As per discussion on ML[1], we do not have any job which runs only
slow tests.
This commit provide a separate job which run only slow tests which can
be API or scenario tests. Further this job can be used cross projects
like nova, cinder, neutron.
Why we need separate job for slow tests:
- slow tests create gate issues on job timeout which we observed\
during Rocky cycle[2]. Not to effect gate let's start them executing
on separate job.
Why to convert the tempest-scenario-all job
- tempest-scenario-all job was created to run the slow tests along with
other scenario tests. But slow tests can be API tests[3] and mixing them
with other scenario or API tests is not good idea.
- All scenario tests excluding slow tests are run as part of tempest-full
job so let's not run scenario tests twice on gate.
[1] http://lists.openstack.org/pipermail/openstack-dev/2018-July/132527.html
[2] https://bugs.launchpad.net/tempest/+bug/1783405
[3] https://ethercalc.openstack.org/dorupfz6s9qt
Change-Id: I5b1d3041c0140bea11705ad287f1472e74aacb5c
We stopped running all scenarios in parallel,
which leads to longer job times and less concurrent test run.
The full parallel run had too high failure rate,
this job is supposed to prove it is possible to return back to
the old way. If the issue(s) are still exists it will
produce usable artifacts for further debugging.
Change-Id: I776ebb7761d90d2490cf49f9df9b322c2b34da59
This commit fixes an oversight in the tox job definitions for unit
tests. Because for better or worse we still use oslotest as the base
test class for tempest's unit tests the stdout, stderr, and logging
capture fixtures are only enabled by an env var. We were never setting
that, so these things weren't captured. This commit fixes the oversight
to enable the capture of these once again.
Change-Id: I81e4077dfda9e9a8a8001f52955ab0164f39d31d
If GENERATE_TEMPEST_PLUGIN_LIST=False, the plugin-registry page is not
generated, and then sphinx fails with:
doc/source/index.rst:69:toctree contains reference to nonexisting
document u'plugin-registry'
Setting the variable to False is a legit use case, used in isolated
build environments like CBS used for RDO packages and should be
supported.
Alternatively, we could make inclusion of the page conditional to the
environment variable value, for example using 'only' directive for
sphinx and passing an appropriate 'tag' as a flag indicating the page is
generated; setup.py interface doesn't seem to provide an explicit way to
pass tags but SPHINXOPTS variable would be handy. Sadly, table of
contents can't be controlled with the directive:
https://github.com/sphinx-doc/sphinx/issues/1420
Another alternative would be to generate a stub page with no useful
info just to upease sphinx. I figured it's easier to tweak sphinx
configuration though.
The fix still leaves warnings-as-errors enabled for tox environments
with assumption that if tox is used, the build environment is not as
constrained.
Change-Id: Id174d1679c7b141b0327d8c23d6b14f7db83fae8
Commit 173b11cb50e115fe12baa097a05db36f24f2e181 moved the reno
dependency from test-requirements to doc/requirements which weren't
used by the venv tox target, which means you can't create release
notes the easy way "tox -e venv -- reno new <slug>". This fixes it
by adding the doc/requirements into the venv target.
Change-Id: Ic3724435176eab6f7fcc53cf5379f7eee44baa71
We do not need tox_install.sh, pip can handle constraints itself
and install the project correctly. Thus update tox.ini and remove
the now obsolete tools/tox_install.sh file.
This follows https://review.openstack.org/#/c/508061 to remove
tools/tox_install.sh.
Change-Id: Ifcc108dc4f91fcdf1d9d4c21ed8ad2983c7ff2c1
Now that we're starting to unify everything on a single test runner we
should only be using either tempest run or stestr (which tempest run
will use internally in the near future). This commit changes the
coverage job to use stestr and removes the ostestr job as part of that
effort.
Change-Id: I5972346b8dbef4a57c733f85ceeeeca425bf263e
This commit switches to use stestr command to run unit tests directly.
Recently, the latest ostestr command uses stestr instead of testr.
However, we should use stestr directly because it's simple and straight
forward. There is no reason to use ostestr for unit tests anymore.
We still have the other ostestr/testr usage such as tempest run command
and documentations. So, that will be changed with following patches.
Change-Id: Iff4abef50178bdc83b868eed4a906e22d790762b
The unit test logs are mixed up in the output of subunit-trace.
Adding OS_LOG_CAPTURE=1 by default to avoid that.
Change-Id: Ie322795c8638963b276c4bd54b845331c3567370
1. exclude "build" in flake8 section.
2. to ensure the old files are not left over when running these
document builds.
Change-Id: Ia5590b7816b36351559c692eebfecb3c8e6fd1a3
This commit adds a flag to generate a tempest plugin list. In a local
environment, we don't need to generate a tempest plugin list, always
when we just need to see the other documents. Besides, generating the
tempest plugin list takes a long time. When GENERATE_TEMPEST_PLUGIN_LIST
environment variable is 'true'(default), the plugin list is created.
When the other value such as 'false' is specified, it isn't.
Change-Id: I6fd3cb3045ba8bbfb39c597413e13faa3cd24ae3
It will help to catch tempest plugin config and dependency issues
with tempest.
sh ./tools/tools/tempest-plugin-sanity.sh or
tox -e plugin-sanity-check will invoke the same.
Change-Id: If42561bfc1d6dcffbb6537634d7769ef90fab21d
This commit removes the setenv parameter of testenv:cover section in
tox.ini. Recently, ''tox -e cover doesn't'' work, actually.
Change-Id: I82ac3dace47c187160dc683e0f740b0589773c0b
Now gate job for Python 3.5 exists and not 3.4,
we should remove those references to the 3.4 that is untested
Change-Id: I2bfedb9a7cc48413dee35d519c52cc2fd3bd3f13
The scenario environment forces serial execution, and it selects
all scenario tests, including those tagged as slow.
Change-Id: I3ad362da88f770cd03f32534683888ea59cfe5c6
The scenario tests end up generating a large load on target
environments. In most cases this means the whole system is running at
an extremely unrealisticly high load, and many of our scenario
failures are probably traceable back to this.
This classifies many more scenario tests as slow so that during normal
full runs they are not run, they are still accessible for running with
the tox -e all flag.
Selection based on https://ethercalc.openstack.org/nu56u2wrfb2b
Depends-on: Ia303773cabffc25d910cd1c0950d6739592bade6
Change-Id: I35cf4a77ff61d04ed8be58acff3edf47b1ae36f7
Pin tox environments to upper-constraints to avoid conflicts with
library releases.
Change-Id: I6e564f3035e2fdc5461f57f15168f498c99fc164
Closes-Bug:#1628597
This commit introduces flake8-import-order to check import orders
automatically. We shouldn't do it manually.
Depends-On: Ia2140f5566ce169b076ffa5c9ebfbdbfc41e5bed
Change-Id: Iff75ebec42b295870edd1c04001adfb17460a682
[H106] Don't put vim configuration in source files.
[H203] Use assertIs(Not)None to check for None.
Change-Id: I1837e4d2228740290bb58c92e6e557800843deec
It's not used anymore. There was general consensus in Feb 2016
to deprecate it (see [1]) and remove it in Newton.
[1] [qa] deprecating Tempest stress framework
Change-Id: Ib229985ea2a1fee495c9492c9ce1781e6bac1dc6
Now that we've deprecated all our bash wrappers for running tests
we shouldn't be using them for running unit tests either. This commit
updates the unit test tox job definitions so that we no longer are
relying on deprecated code in the gate and we can remove the wrappers
when the time is right.
Change-Id: I127786537c0792456d26b13fc3a2044afc4e93a1
That new tox target checks that we don't have missing requirements in
requirements.txt AND that we don't have superfluous (unused) requirements
in requirements.txt.
That patch is copy/pasted from I014375a5ecfebc723541c2f2db6c2e0fba636aa2
which tried to do the same for Nova (unmerged yet for an unknown reason).
This change also includes the fixes to requirements.txt needed
to pass the new tests.
Change-Id: Iff37af95269c38e4d435a53a21e5d60e2c5a3fa8
Oslo.utils provides same method get_ipv6_addr_by_EUI64,
so deprecate it in Newton and remove it in Ocata.
Co-Authored-By: guo yunxian <yunxian.guo@easystack.cn>
Change-Id: Ia72f866cda69b4f98e88936df32c76e455619d07
This commit switches the tox commands that run tempest to use tempest
run instead of pretty_tox.sh. Tempest run is the primary method of
running tempest moving forward so we should dogfood it where we can.
This also officially starts the deprecation timer (which was long
overdue as these were unofficially deprecated before) on the
run_tempest.sh and pretty_tox scripts as we move to having a unified
method of running things.
Partially implements bp tempest-run-cmd
Depends-On: I8786b9709584378aaf48393115dda94397e22544
Depends-On: I0529c2922242d8ed9c92ef718328e09f3ea9e05b
Depends-On: If0064b9e8358332972ef4a1eee8f150e66f8c50f
Depends-On: Ibaffa3a8568ea058d964463df4b76196c4d2bc7a
Change-Id: I3684fce66a799579fa68af119652cafef25a9f03
Since the change Ib5e24e19bcba9808a9f49fe7f328668df77fe4f9 , the
directory thirdparty has been removed. So we don't need to specify
it on tox.ini anymore.
Change-Id: I7eddcb975ebf20a78920acd6f48ba8260f6cb75a
Now that there is a passing gate job, we can claim support for
Python 3.5 in the classifier. This patch also adds the convenience
py35 venv.
Change-Id: Iaec0c9e88cb38ab876b5bc957f53989615cab246
This commit moves the oslo config generator config file from living as
a data file, which makes it very difficult to rely on, to just living
in the package. By including it in the package we have a guaranteed
relative path we can rely on for building utilities on it, like the
tempest init command.
Change-Id: I5b9deae733f42fe3ed39418103304f71b43dccb3
Partially-implements: bp tempest-run-cmd
This commit simplifies the venv management via tox for tempest jobs.
All tox targets that actually run tempest will now share a single tox
venv .tox/tempest. This also adds a new target tempest-venv which is
the same as the pre-existing venv target, but uses the common venv.
We can't simply use the common venv with the venv target because
external tooling, like the docs jobs, rely on the .tox/venv path
existing when using the venv target.
Depends-On: I34a69020eee07156e64026781a3c0bffdb5ab415
Change-Id: Iab2e6e04b6c5795a4d0c8214564106525b942308
This commit adds tox jobs to run tempest with ostestr instead of the
pretty_tox*.sh wrappers. ostestr provides essentially the same
functionality except it exposes slightly more rich selection logic
and additional debugging options. However, since the cli isn't
backwards compatible with pretty_tox we can't simply remove the old
tox jobs.
Change-Id: I347438b1d561af7e05fdce895e2e3e51d2b38b0c
This commit aligns multiple lines in tox for visual consistency.
I fixed them to 4 white spaces because the tox example[1] shows 4 white
spaces indentation.
[1] http://tox.readthedocs.org/en/latest/example/basic.html
Change-Id: I4f3b9c5c4982a0e96bb120ce8546dd48b0fa8a4b
This isn't really a scenario test, it's a performance test, however
without any of the infrastructure to do performance testing /
comparisons. It's time has come to say goodbye.
Depends-On: I788cd70dc4e6a9c3f8e6805d5bdcdf7f549fce35
Which removes it's use from the gate.
Change-Id: I93b2fb33e97381f7c1e0cb1ef09ebc5c42c16ecc
