Commit Graph

116 Commits (cc2db2aeb5f91730a6fa9c6253e683e856a4878f)

Author SHA1 Message Date
James E. Blair 6c8855ede4 Add noxfile and switch to nox
Tox v4 behaves significantly differently than v3, and some of the
more complex things we do with tox would need an overhaul to
continue to use it.  Meanwhile, nox is much simpler and more
flexible, so let's try using it.

This adds a noxfile which should be equivalent to our tox.ini file.
We still need to update the docs build (which involves changes to
base jobs) before we can completely remove tox.

Change-Id: Ibebb0988d2702d310e46c437e58917db3f091382
2022-12-20 08:57:53 -08:00
Jeremy Stanley 90d7dcb1ed Update unit test container setup and instructions
Update the TESTING.rst file to reflect that just installing and
starting zookeeperd is no longer sufficient now that we require TLS
and auth for the connection, as well as running database servers.
Suggest the container-based setup script instead. Also improve that
script to allow it to be invoked as a normal user, with root command
escalation tool choice (e.g. "sudo") supplied through a ROOTCMD
environment variable, so that things created inside the git worktree
like the CA don't end up root-owned.

Related, the tox-docker plugin previously suggested in the document
is no longer a viable option for the same reasons, so clean up the
plumbing for it in tox.ini as well.

Change-Id: Iac32799425a5bd4b1bdbf56f34a2310241ac4499
2022-08-05 21:00:02 +00:00
Ian Wienand f592c31263 nodepool-zuul-functional: remove
This job has been broken for a long time, and the paths it tests are
better covered by the quickstart tests which bring up nodepool with

It's a bit of an odd job because nodepool sets itself up, but then the
test calls the tox "nodepool" environment in Zuul here.  So remove the
job inclusion, but also the tox/unit-tests being run by the job (it's
already failing and non-voting on nodepool, so this won't affect

Change-Id: I8483b1d66a6a58d4bd2f2fce82b023d8f0446ae7
2022-01-28 08:18:02 +11:00
djager 1419356ed8 Allow to specify MySQL port for test environment
In case for whatever reason there is already another
MySQL server running on the standard port developers
may want to run the MySQL test-server on a different
port. This can be done by specifying an environment
variable "ZUUL_MYSQL_PORT":

export ZUUL_MYSQL_PORT=3307
tox -e py3

Change-Id: Id32bac1a87619803d973b066762d525bd9a4f913
2022-01-19 09:28:14 +01:00
Simon Westphahl edb4985305 Allow running tests with multiple schedulers
The number of schedulers can be defined via the ZUUL_SCHEDULER_COUNT
environment variable (also forwarded by tox).

Disclaimer: running the tests with multiple schedulers will still fail
as not all test support multiple schedulers yet.

Change-Id: Ie11467dcf0fc36a5e167adcbd29ac4f256e0b3c5
2021-11-24 13:30:28 -08:00
Clark Boylan 7413e5000a Use all but one CPU when unittesting
Previously we removed the CPU limit from stestr to speed up testing.
However, based on log data we have occasionaly failures now due to
exceeding system load limits in the Zuul executor governors.

Based on logs we appear to be just over that limit and a small pullback
may be sufficient to make things reliable. Previously we were dividing
the total cpu count by 2, but now lets try reducing it by 1 instead. On
OpenDev's CI systems this should have us using 7 test processes instead
of 4 when we were dividing by 2.

Change-Id: I1ded37b799780c52dbab6a84c1fde9fae099a08e
2021-10-26 10:07:27 -07:00
James E. Blair df338db03b Uncap concurrency in tests
This is 2 changes:

* Run dstat in unit test jobs so that we can get an idea of
  system performance/utilization.

* Remove the stestr concurrency cap.  With 8 cores, we have
  enough headroom to run the test dependencies (sql/zk) while
  the tests are running too.  Use all the CPU that's available.

Change-Id: I9f250865f7043fdbb1fa8a01f1bc9508490accc1
2021-10-25 08:53:37 -07:00
Clark Boylan f52c54ab91 Prepare for SQLAlchemy 2.0
SQLAlchemy 2.0 will introduce a number of changes. Thankfully Current
sqlalchemy has deprecation warnings and can be run with future flags set
to True to enforce 2.0 behavior. We use these tools to prepare Zuul for
the SQLAlchemy 2.0 release.

In tox.ini configure the environment to always emit DeprecationWarnings
for modules that touch sqlalchemy.

Update sqlconnection to use the new location for declarative_base and
set future=True on our Engine and Session objects.

Finally update the database migration tests to use transaction based
connections for executing raw sql statements. Also we switch to the
exec_driver_sql method for that. SQLAlchemy 2.0 will not do implicit
autocommiting and doesn't support executing strings directly. has tons
of info on these changes. Reviews should probably pay attention to the
transaction behavior changes as well as any alembic code that might also
need updating.

Change-Id: I4e7a56d24d0f52b6d5b00a8c12fed52d6fae92ef
2021-08-12 12:53:40 -07:00
James E. Blair f718b16844 Stop running mypy in linters
We aren't generally using type annotations, and mypy itself is
somewhat flawed, occasionally producing false positives and rarely
catching errors.  Stop running it.

Change-Id: I6f24457f7d99ca11ec9228e505e6edec558baf9e
2021-08-09 08:49:29 -07:00
James E. Blair 1736ea8ca2 Retry yarn package installs
We're seeing occasional npm install failures in jobs; retry the
"yarn install" command if it fails.

Also, pass the YARN_REGISTRY env var in the tox remote job.

Change-Id: Ic80dccf0869ce805368f678eafcfdf48f8bf9bea
2021-05-03 11:26:52 -07:00
James E. Blair 74a9c9de9b Use ZooKeeper TLS in tests
This mirrors the configuration in Nodepool for using TLS-enabled
ZooKeeper in tests.  We use the ensure-zookeeper role in order
to get a newer ZooKeeper than is supplied in bionic.

Change-Id: I14413fccbc9a6a7a75b6233d667e2a1d2856d894
2021-03-08 06:49:57 -08:00
Sorin Sbarnea c911e26fa6 Decouple mypy from tox
This allows mypy to produce the same results regardless how is called.
For example IDEs will produce the same results as tox -e linters
instead of displaying violations which the core team decided to

Change-Id: I20eeafc0549f78e3d4f9636793cfe031389b6bbe
2020-12-09 17:04:45 +00:00
Tobias Henkel 6ab497ad3d
Bump mypy to 0.790
The currently used version 0.740 throws syntax errors when used with
python 3.9.

Change-Id: If5cd0da2d5a77126e9ac4c4d6db7a6552f1bd067
2020-12-09 16:28:27 +01:00
Matthieu Huin 8b29c09f40 Add zuul-client testing
Define unit testing for the zuul-client CLI. The actual
testing job will be defined in a subsequent patch to
the zuul-client project.

These tests are stored in the zuul repository rather than in
zuul-client's because we rely on zuul's test framework to bootstrap
zuul and zuul-web, which is not available through zuul's python
package. This is similar to what is done for nodepool's functional
testing with zuul.

Change-Id: I156162279c5d638ef39d22fcfb043655890c2d6e
2020-09-29 12:14:55 +02:00
Tobias Henkel 9cb2e52c59
Use npm mirrors in tox jobs
Opendev runs a caching mirror for npm packages. Use it when running
yarn install during tox tests. Due to the way yarn.lock files work at
the moment we need to patch the lockfile on the fly to inject the
mirror urls [1]. This can be handlel inside the where we patch
the lockfile, run yarn and reset it afterwards.


Change-Id: Id1f2daac8327f3110e6beeb2d6391baccfd3ce34
2020-07-28 22:21:17 +02:00
Zuul c5c0308347 Merge "Add DOCKER_* to passenv" 2020-04-29 16:09:49 +00:00
Sorin Sbarnea 1ce6cb860f Add DOCKER_* to passenv
We missed including DOCKER_* to tox passenv and that prevented
use of docker for users that are using other hosts. Most notable
option is DOCKER_HOST but any docker specific options should be
passed, so the docker commands keep their behaviors.

This also sorts passenv list, to ease maintenance.

Change-Id: Iac436da1202066541fc98eb94a2a228e6cf8b337
2020-04-28 10:55:13 +01:00
Clark Boylan 92dba202ed Bump mypy for py3.8 support
mypy depends on typed-ast which relatively recently added python 3.8
support. Unfortunately the mypy we were using requires an older
typed-ast and fails under python 3.8. Fix this by bumping mypy up to a
level where minimal code changes are necessary.

The code changes we made ignore type conflicts due to redefinition on
conditional imports. Comments include link back to a bug describing this
issue and workaround in mypy.

Change-Id: Iec69a27b16a1e09eb6bfbcf8d68deb1ae68d42a3
2020-04-27 17:43:50 -07:00
Antoine Musso 9583905244 tox: do not install bindep for linters
The binary dpeendencies are not needed to run flake8, mypy or

Change-Id: I373b503f90b05183a3ad39b1707259068c5d4cc5
2020-04-02 10:58:30 +02:00
Antoine Musso 2625a6f3f7 tox: rename pep8 to linters
pep8 is legacy, we now use flake8 and the virtual environment also runs
mypy and openapi-spec-validator.

Rename the virtualenv from 'pep8' to 'linters'.

Use tox-linters instead of tox-pep8.

tox-pep8 skips Ia140b331ae1502fcf1f685faa6dace117d77d35d

Done for tox-linters by Idc72c7d1a88fff8e738c896e2482b5cb1d844600

Depends-On: Idc72c7d1a88fff8e738c896e2482b5cb1d844600
Change-Id: I1fbd8335a2353f750f49b6a763eefc6f399a7143
2020-04-02 10:57:55 +02:00
Jeremy Stanley 7e53f87d06 Declare support for Python3.8
Replace our tox-py37 job with tox-py38, extend the list of trove
classifiers for Python versions in package metadata, and replace the
"py35" in the tox.ini envlist with just "py3" so that folks running
`tox` unqualified on their systems will use whatever python3
interpreter they have on hand (odds are it's in our supported range
these days). Also uncap python-daemon so we use a version compatible
with Python >=3.8.

Change-Id: I4b382bb89847d3ae2b20292d923c3c2211dfb254
2020-03-11 20:22:21 +00:00
Tobias Henkel 4402668367 Don't set OS_LOG_DEFAULTS if unset
We currently set the OS_LOG_DEFAULTS in tox to an empty string if it's
unset. This essentially overwrites the default levels defined in our
base test class to debug level for everything. This makes the tox logs
hard to read because especially kazoo and gear are very chatty in the
debug level. Further the excessive logging often exceeds the lengh
limitation of stestr when the test fails.

To mitigate this remove OS_LOG_DEFAULTS from setenv and leave it in
passenv so we take the zuul test case default when it's not defined
from outside.

Change-Id: I8f634f1ca5cc028018f181a07e204b55788759fa
2020-02-17 21:41:00 +00:00
Zuul d06e71a55d Merge "tox: reduce deps used for pep8 env" 2020-01-22 09:47:47 +00:00
Antoine Musso 73110b4909 tox: reduce deps used for pep8 env
Linters are only used by the 'pep8' virtual environment, we do not need
them in other test environment.  Also 'pep8' only requires the three
linters it used and the other modules are not needed.

Move the pep8 dependencies to the pep8 testenv.

Change-Id: I3a8436cbff9e2386cf0e0f5b0d1742283e647b56
2020-01-21 16:05:18 +01:00
Antoine Musso 875f1e1ad7 tox: pass --slowest to stestr
Since version 2.0.0, `stestr run` supports passing `--slowest` to show
the slowest tests.

Change-Id: I015781402962e5518f650057f32ba8c7f168aea9
2020-01-21 10:44:10 +01:00
Zuul 0c353ad9ac Merge "Don't install zuul in the bindep tox testenv" 2019-10-14 20:45:09 +00:00
Jeremy Stanley f10566af34
Don't install zuul in the bindep tox testenv
Running bindep doesn't rely on the zuul source or any of its
dependencies installed, so set usedevelop to False ensuring only
bindep and its dependencies are installed (which is much quicker and
less likely to raise unrelated dependency errors in the process).

Change-Id: I886857e22ed953efdaab5b6247d1e15039806f48
2019-10-14 20:42:08 +02:00
Zuul 26e6d99e7c Merge "web: add OpenAPI documentation" 2019-07-16 11:52:26 +00:00
Zuul 56fe484aea Merge "Increase test timeout to 6 minutes" 2019-06-12 21:49:14 +00:00
Tristan Cacqueray 85616c4c09 web: add OpenAPI documentation
This change adds a swagger description of the REST API. The description is
rendered in the sphinx user documentations and in the web interface.

Change-Id: I753524f40a09874dab5952f14ab17025525bbab9
2019-06-12 22:35:13 +02:00
Jeremy Stanley 94ef36eb7f Increase test timeout to 6 minutes
We're regularly seeing the test_plugins unit test exceed our 260s
hard timeout fixture on slow job nodes, so increase the base timeout
from 4 minutes to 6 to accommodate this. Also bump the wait timeout
in test_plugins from 3 to 4 minutes.

Change-Id: I46a0f84b43c3e7c15919f3681d50e32aa7d3b89f
2019-06-12 15:39:41 +00:00
Tobias Henkel e912cd15a9
Fix utility for multi-ansible
Since the multi-ansible work the script is broken and
needs to be fixed to use the managed ansible.

Change-Id: I271f45d9f096a05949900ba1dd2bf8321c459815
2019-05-25 09:07:32 +02:00
Zuul a0c2993411 Merge "tox: Integrate tox-docker" 2019-05-17 05:42:08 +00:00
Tobias Henkel 5d828097ff
Restore posargs when running stestr
The posargs got missing which removed the ability to customize test
runs during local execution.

Change-Id: Ic8687f817c83367c5da07a3d4444e96912569852
2019-05-03 21:40:00 +02:00
Zuul 5d910885d7 Merge "Halve stestr concurrency" 2019-05-02 15:11:14 +00:00
Zuul dde6a7470a Merge "Fix tox.ini cover target install command" 2019-04-27 06:44:57 +00:00
James E. Blair c9f4a862ce Halve stestr concurrency
To see if reducing the concurrency of tests makes them more reliable,
set them to half the number of cpus (by default, stestr uses the
number of cpus).

Change-Id: I7977c853b759cf980a6717a70876857ef7e9f77e
2019-04-25 14:55:56 -07:00
Monty Taylor b0252eef9d Update references for opendev
Change-Id: I3c1ac5478efed4dee1d525deb036d457287fa136
2019-04-24 12:59:17 +00:00
Clark Boylan 929108725a Fix tox.ini cover target install command
The cover target wasn't working because the various ansible versions
weren't getting installed. We can fix that by removing the custom pip
install command for the cover target and instead rely on the default
install command which handles this for us.

Change-Id: Ieb7adce16356f0f080ceac580fbddab0d2dbc537
2019-04-15 09:44:25 -07:00
Stephen Finucane 390f071563 tox: Integrate tox-docker
This allows us to start a zookeeper instance via docker as part of the
standard tox process, avoiding issues like those described in [1]. It's
exposed as a separate target, to ensure users can use local
installations if necessary. This requires bumping the minimum version of
tox to 3.2 so we can make use of the 'requires' config option [2].


Change-Id: Ie848d409533b2a53523cd930299acf21ac10e13c
Signed-off-by: Stephen Finucane <>
2019-04-03 15:05:42 +01:00
Stephen Finucane 1cbaa2969b tox: Fix indentation
Yes, this is purely stylistic and does screw up git log, but it's a heck
of a lot easier to work with in this way.

Change-Id: Ie20ad2fc62be83181826cbdad64ba951d43534e8
Signed-off-by: Stephen Finucane <>
2019-04-01 14:35:15 +01:00
Stephen Finucane db5cc667ee tox: Change default 'install_command'
Means we don't need to set this for the default factors (py3x), meaning
we get new versions for free going forward.

Change-Id: Idba181e96adab16181a7f776093c9dfe33e071c6
Signed-off-by: Stephen Finucane <>
2019-04-01 14:32:23 +01:00
Stephen Finucane 9c96f36833 tox: Remove 'commands_pre'
These were added in I5ce1385245c76818777aa34230786a9dbaf723e5 but are
not necessary since I680777e5406368d35366602ac37e029db91c8a5c, the
latter reverting most of the former. Removed them.

Change-Id: Ic820ac288cc8b97b417ba9fa25328c30c99562a3
Signed-off-by: Stephen Finucane <>
2019-04-01 14:30:00 +01:00
Tobias Henkel 947aae0d99
Increase global test timeout
The OS_TEST_TIMEOUT setting is used to create a TestTimeout fixture
which aborts tests. This is currently set to 150s which is too low
often for the test_plugin and test_playbook tests.

Change-Id: Ic638dfdad513f3a61e4026c3667b1ca18d464fc9
2019-03-23 11:01:42 +01:00
Tobias Henkel e173757908
Add py37 environment to tox
On some newer distros like Fedora we need to be able to run tox with
py37. For the py3* environments we need to use the pip wrapper so
declare this as well.

Change-Id: I666098238383ebc70982772dbeb449ab37b3b37c
2019-03-19 10:09:36 +01:00
Tobias Henkel fa0715eeac
Move ansible installation from pre_command to install phase in tox
We currently install ansible during the pre_command phase in tox. This
breaks the following use case:

  tox -e py36 --notest
  ttrun <test case>

This is broken now because ttrun doesn't run pre_commands. Instead we
already have a pip wrapper that does the nodeenv installation. We can
use this to also call zuul-manage-ansible on zuul installation.

Change-Id: I680777e5406368d35366602ac37e029db91c8a5c
2019-03-19 10:09:36 +01:00
Tobias Henkel fe0dfd6ede
Run tox remote concurrent
Adding more ansible versions lead to longer runtimes so we need to
either run those tests concurrently or split them per ansible version
and run one job per ansible version.

Change-Id: I41574d8f5c0f7f5fa2c5272044d6b387def5bfa7
2019-03-15 09:09:17 +01:00
Tobias Henkel cd9827e664
Manage ansible installations within zuul
As a first step towards supporting multiple ansible versions we need
tooling to manage ansible installations. This moves the installation
of ansible from the requirements.txt into zuul. This is called as a
setup hook to install the ansible versions into
<prefix>/lib/zuul/ansible. Further this tooling abstracts knowledge
that the executor must know in order to actually run the correct
version of ansible.

The actual usage of multiple ansible versions will be done in

For better maintainability the ansible plugins live in
zuul/ansible/base where plugins can be kept in different versions if
necessary. For each supported ansible version there is a specific
folder that symlinks the according plugins.

Change-Id: I5ce1385245c76818777aa34230786a9dbaf723e5
2019-03-15 09:09:16 +01:00
Monty Taylor a5736ba07f
Add ignore_basepython_conflict option to tox.ini
Without this, basepython overrides the python in all envs, even the
builtin ones. That means that py35 and py36 would both be testing with
whatever the default python3 is on the system, not 3.5 and 3.6

Change-Id: I1af5bd99934638244ea29375fc4f170e5616f01b
2018-11-06 08:17:49 -06:00
Zuul 5f7f7eaf40 Merge "Use doc/requirements.txt" 2018-10-29 07:11:37 +00:00