We ran into issues getting ansible-lint to run on Noble without updating
everything to current ish versions due to Python 3.12 compatibility
issues. Updating everything in this way created a bunch of new lint
errors we need to fix so we worked around the problem previously by
pinning back to Jammy.
Now we move the job back to Noble (the default Nodeset) and update the
tooling and fix the linter errors. This should allow us to move forward
in a productive manner on modern platforms.
The linter errors we fix include:
* Naming every play
* Ensuring every play name starts with a capital letter
* Using fully qualified collection names for action modules
(archive, git_config, and synchronize)
* Quoting octal file modes
Change-Id: I96560c5ce2a5af39d39b3fc339862932a856bd13
We often see failures on requirements patches generates after a release
was tagged that are due to the newly created version not being found yet
in the index. These require a recheck to resolve. Add a pause of 10
minutes before creating the requirements review in an attempt to reduce
the frequency of these failures.
Also fix some minor issues:
- The extraction of the SHORTNAME for the repo from remote.origin.url
git config is broken, because zuul overrides that url to be /dev/null.
Use the working directory name instead.
- Current python minor versions have two digits, amend the regex we are
using to match these.
- Drop a workaround that was needed when Python 3.8 was new and shiny.
- Switch the nodeset this job is running on from ubuntu-focal to
ubuntu-noble.
Change-Id: I6751ea91499bbc81c71dce6da7fbe982bbc03efe
There is no real reason we should be using some of the
terms we do, they're outdated, and we're behind other
open-source projects in this respect. Let's switch to
using more inclusive terms in all possible places.
Updated playbooks and related code accordingly.
Change-Id: Ia471193921660aa5f2152ab63eaf570bee3ebcd0
Depends-on: https://review.opendev.org/c/openstack/requirements/+/917786
Between September 4 and 5, our wheel cache publication began failing
for Debian distributions. The ansible_distribution_version fact
began to differentiate between Debian minor versions (so e.g.
bullseye went from "11" to "11.7"), and this does not match our
cache scheme in AFS so resulted in attempts to write to a
nonexistent directory. This corresponds to when
I9527fc847e18a80414139ebd6f19a9dbb9a5778e merged to switch the
tenant's default Ansible version to 8, so is very likely a behavior
change in Ansible itself.
Switch the logic in our wheel-mirror playbook to use
ansible_distribution_major_version like we do for Ubuntu, and rename
the tasks to be a little more generic.
Change-Id: Ie95e99bce1ed01aa9c284871b517e6c28c278d16
The publish jobs build and copy the wheels to the AFS r/w partitions,
but they are not released to the mirrors until a final "vos release"
of their volume.
Previously this happened in a final job that would not trigger if any
of the wheel builds failed. This has meant thing like out-of-sync
centos mirrors or arm64 node failures have stopped all wheel
publishing previously. There's really no need to block other
platforms publishing if one of these jobs fails.
This converts the jobs to have a release job syncrhonized by a
semaphore. We don't want many "vos release" processes running all at
once, as it has been a source of failure with openafs before. Since
releasing requires the admin key, we keep it in a separate job from
the building.
Change-Id: I10c307c2d46c0e5b86732943208c3167da28a694
This updates the script to automatically update puppet constraints so
that Puppetfile_unit is updated as well as Puppetfile. We introduced
this separate file[1] a while ago but have not been updated it
properly. Updating it periodically helps us catch up with any update
in the dependent modules more timely.
[1] 047ea125ddefa8151d6592d4772560a1b23a13dc
Depends-on: https://review.opendev.org/875301
Change-Id: I4d89a2985781f14d9ff7f76f412f0b8ccc4c6ddb
A new job openstack-fips is created. This job is expected to be
a base job for most OpenStack jobs, so that FIPS testing can be
easily enabled throughout OpenStack CI.
A base job is required here because, for Ubuntu nodes, we need
to enable an Ubuntu Advantage subscription, as Ubuntu considers
FIPS to be a UA feature. The subscription key is stored here in
project-config.
Depends-On: I47a31f680172b47584510adb672b68498a85bd32
Change-Id: I8a88d6a9bcf5725986b00b063e03686d3225b48e
In order to be able to safely re-enqueue tags which previously
failed release jobs after successfully uploading at least some
artifacts to PyPI, instruct twine to treat "file already exists"
responses as benign and ignore them, proceeding to upload any others
which aren't yet there.
Depends-On: https://review.opendev.org/864004
Change-Id: Iab38df6386ce5219a52787fda5e64a8faab23a06
Since the PTI switched the supported python versions for the current
cycle from 3.8+3.9 to 3.9+3.10, we want to update the constraints
generation accordingly. Since we don't have a stable platform where we
can run both py3.9 and py3.10 in parallel, generate constraints only for
py3.10 for now.
Change-Id: I4a4d9cb6e292bb693ae424be0928501fed81113f
A later version of ansible-lint picks this up; fix lack of whitespace
around filters. No operational change.
Change-Id: If042a0f92a74ea06a312aad68de21f9e44f16582
A later version of ansible-lint enforces names on blocks. This is
generally a good rule; fix a few missing blocks here.
Change-Id: Ia87a0c21ec0ed1662e37cbc9e17a0df344b54e57
These were found by a later version of ansible-lint. This should have
no effect, but just fixes some inconsistent whitespace issues.
Change-Id: I7bcde4942c97cfe743e8aba74833aeb5844c8290
This doesn't need a name: when included like this. I guess Ansible
accepts it, but it doesn't document this working anywhere.
Change-Id: Ibe6fa9296041af54f39a3ef69957d0e8a0df0259
Previously we had this error:
No key/value pairs provided, at least one is required for this action to succeed
Which appears to be due to the default(omit) condition when setting
ensure_pip_virtualenv_command in focal proposal jobs. The issue is
default(omit) doesn't do what you think it does when defining a
variable. It is really only useful for defining module parameters that
may be omitted.
Instead we need to call ensure-tox two different ways depending on
whether or not we want to override the defaults for
ensure_pip_virtualenv_command. One method for Bionic and another for
Focal keyed off of whether or not _venv_command is defined.
Change-Id: I0cbca64f4a31c8b4eacb5e1c50f2e9fb289ce18e
This fixes a typo in I683a61b1bae1a809caf724aee87a21af2e18fb0c -- it
should be "ansible_distribution_release"
Change-Id: I074f089371054856a8b96bd6e650496a0bd53685
I think we have quite a tangle of fixes on fixes in the translation
jobs, and I hope this refactor makes things a bit better.
The fundamental problem seems to be that python3.6 can't work with
master requirements.txt. But we can not move some jobs off bionic
(python 3.6) because of java version contsraints.
The first thing that happened was these jobs got "ensure-python" with
"python_version: 3.8" set to bring in python3.8 on Bionic. This role
was not bringing in python3.8-venv, so we had no way to create 3.8
virtual environments. Thus ensure-pip ended up getting modified to
drag this package in (reverted with notes in
372f3af706c796331578ef81c2680e8c4c03c3b6). This has a fairly easy
solution; let's just make ensure-python bring in the venv packages for
the udpated python -- see dependency
Ie3c03fea82bcec80a897f0905c15f35405a50396.
The git-review install here uses 'ensure_pip_virtualenv_command' to
install git-review in a virtualenv. git-review still supports bionic,
and does not need to be installed in any speical way. However, by
overriding the 'ensure_pip_virtualenv_command' for the entire job,
this unrelated install now grew a dependency on python3.8 and created
more confusion over why it was failing.
I believe what we actually want to do on Bionic is just run tox under
python3.8. So this pulls out all the job-level defintions and
encapsulates them in a single block scoped to run on bionic. It will
pull in python3.8 packages, and then install tox under python3.8 with
an override of the virtulenv command just for that step. The jobs
just maintain their node definition and note on why it is required.
The long-term future of these jobs is obviously a question; but I
believe this keeps them running on bionic in as sane a way as possible
for now.
The second part of this is the python3.9 interpreter required on
Focal. Setting "python_version: 3.9" in the job definition is a bit
confusing, because the job actually requires both Python 3.8 AND 3.9.
This variable is passed to ensure-python, and the naming is a bit
unfortunate because it is so broad (but we can't change that without
zuul-jobs updates). Here I've pulled out the Python 3.9 install into
a separate focal-gated section in the pre.yaml that explains what is
going on (note that Python 3.8 is just the base python on focal, so
needs no extra installation). I've also updated the description on
the job node with this info. There's no need to install the tox
environment any differently on focal; it just uses the base system
python (3.8) which is fine.
Depends-On: https://review.opendev.org/c/zuul/zuul-jobs/+/850957
Change-Id: I547e5eb7faabcd9d3983ae85f3291b1a740bc77c
Currently both python3.8 and python3.9 venv package
is being installed, but since python3.9-venv is not
available in bionic, python3.8-venv installation also
fails as mentioned in [1].
This patch moves ensure_pip_from_upstream_interpreters
var from proposal pre playbook to job definition and
override it in translation jobs to attempt installation
only for python3.8-venv.
Also switch back to "python3.8 -m venv" virtualenv_cmd
as with this patch python3.8-venv will get installed.
[1] https://review.opendev.org/c/zuul/zuul-jobs/+/846201
[2] https://review.opendev.org/c/openstack/project-config/+/846390
Change-Id: I9d9a423e3147d3f04c1951a796636e5520202a6d
Install git-review task relies on ensure_pip_virtualenv_command
variable but with [1] this is set to python3.8 in translation
update jobs but it get's installed later by ensure-python role,
let's run it earlier to fix the issue.
[1] https://review.opendev.org/c/openstack/project-config/+/845907
Change-Id: Iecbdcb50063cb461b3a09ddfa205d071c7d77b75
The previous attempts at fixing the job were not successful yet, it
seems the variable in the pre playbook isn't overridden by a job
variable. Since the other uses of this job are running with ubuntu-focal
anyway, set the list of available python interpreters in the playbook
directly and remove the modifications from the requirements specific job
again. Also pin the nodeset for this job to guard against changes in the
base job, which will likely require another change to the interpreter
list.
[0] I9735f34639777c928fbe06febc16dd24667b1405
[1] I52cc316ce9756c4411c18c8339d44c457cb59897
Change-Id: Iddd61e1be766f2bb6da4bf62766829247fa143ce
3.10 will come later when there is an image that supports all three
Change-Id: I663de65a971337fce65a51f0e5a7f959b8da2c03
Signed-off-by: Matthew Thode <mthode@mthode.org>
The configure-mirrors role in zuul-jobs expects CentOS wheels will
be in URLs like distro-ver-arch and so doesn't include "-stream"
like our wheel publishing jobs are adding. This was a necessary
distinction in order to differentiate between CentOS Linux 8 and
CentOS Stream 8 wheels, but since configure-mirrors didn't know to
tell Stream nodes to look in the other location, this was actually
broken from the moment it was introduced and we simply never noticed
until we removed the old centos-8 volume (at which point
centos-8-stream nodes no longer found any wheels).
As we no longer have centos-8 nodes, rather than try to change how
configure-mirrors works (since it may be successfully used in other
sites already and would need advance warning or complex overrides in
base jobs), simply rearrange our mirrors to mount the volumes in the
paths it expects. Once this merges I'll adjust AFS accordingly.
This partially reverts 3d776f6734ff0c5197471c7a89aa84511a0eff88.
Change-Id: I07ee7ccdd10e68f6f38ec362ce7e5d333d5fc2f7
The addition of ACLs to zuul/main.yaml broke the naive parsing done
by the maintain-github-mirror playbook, as it assumed all list items
were tenant definitions. Add a conditional check to allow it to skip
past any which aren't.
Change-Id: I31ab916a17fd116f00e94ea7bfa4d7243c0e1c2c
This was missed with Ia44c384072c0482cfd11c642013fd51004f85c8b when we
removed the Xenial ARM64 jobs.
Also cleanup the grafana; additionally remove CentOS 7 ARM64
references which are not present.
Change-Id: If5fb2a1fe2cce058f1d516d5f5f7180a6cab06cf
The irc-meetings are now published to AFS
(I6f3a9970d907f2ae1420e1523cdf8a7c50647235) and served statically from
there.
Depends-On: https://review.opendev.org/c/opendev/system-config/+/794089
Change-Id: Ib852eac630f5aedcb4cd1d240fee63405d50188e
Since Debian Buster can not be used with nova 23.0.0 because of the
min required libvirt version, we should make Bullseye available for CI
to ensure that OpenStack Wallaby release will run on it smoothly.
Depends-On: https://review.opendev.org/c/openstack/diskimage-builder/+/783790
Change-Id: I9c1bb7aaa02ba60ee52e2d7b990e2e6e1212317f
OpenStack's Python release jobs don't run tox, and don't install the
projects being packaged, not even in the branch tarball variants
these days. Remove the bindep, ensure-tox and collect-tox-logs roles
from pti-python-tarball and python-branch-tarball playbooks so that
the jobs using them can be more portable between distro versions
with less work. This should also make the jobs considerably faster.
Change-Id: Icc30a79f770ed78674354f6226d28907bab7eb1d
Following the deletion of the foundation-board-repos.yaml file,
the script failed as it was expecting it.
Remove that file from the scanned list, and make the script more
resilient in case other extra files go missing in the future (like
user-committee.yaml which should soon be cleaned up).
Change-Id: I83cff14f19a829b5e56771442ffb3a0341e82d69
maintain-github-mirror makes direct use of the requests module. That
module is currently installed as a dependency of PyGithub, but that's
brittle, as PyGithub may well opt for a different library in the
future.
Let's proactively fix that before it bites us.
Change-Id: I8fc288e6c3978e20ddf48f905968931f89834e53
Job definition referenced opendev/governance and opendev/project_config
instead of openstack/governance and openstack/project-config, causing
the job to fail.
Also replace variable names so that they make more sense.
Change-Id: I7cd026a1d0bc1a8ad2d0c6b7bf3a5a0d37ecd423
The script needs the 'github' module, but this is provided by the
'PyGithub' library on Pypi, not the 'github' library.
Change-Id: I7c6d4b30a11d38e060fb4e6b2cdf29301f6f03e3
Opendev no longer automatically creates repositories on the
GitHub mirror, nor does it update descriptions or closes open PRs.
Add a playbook and a job for periodically maintaining the GitHub
mirror for the 'openstack' organization:
- updating descriptions based on Gerrit project descriptions
- creating on GitHub newly-added openstack repositories
- archiving from GitHub recently-retired openstack repositories
- closing any open PR with a healpful message
This job makes use of a GitHub API token (from the openstack-mirroring
user) and is defined to run periodically on project-config.
Change-Id: Ic02f436eb655dcbe84824b304ea2933e16312e67
If00afb50c37b45aefbb45070da4efef3e43e62b2 updated the requirements
update job to use tox as part of the update checking script. We have
ensure-tox as one of the roles in the pre playbook for this job, but it
is not set to be made available globally, which results in the script
failing with being unable to find tox.
This updates the role inclusion to set ensure_global_symlinks: true so
tox will be available in the script.
Change-Id: I5d3ed4ce62cd8df604802fedf522d644d5664698
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
This script creates a virtualenv to run a requirements script to update
constraints. virtualenv is not globally available anymore. Rather than
trying to call it directly, this now uses tox to set up the virtual
environment.
Change-Id: If00afb50c37b45aefbb45070da4efef3e43e62b2
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
The publish-openstack-python-branch-tarball job run by many projects
uses tox (in python-branch-tarball/post.yaml), so needs it installed.
Change-Id: Iba219ad14523ce900d9562ef56c78cc6c15aa01d