The docker image that we build the zuul executor from is a Debian
image, but it does not follow the same python3 policies as Debian
itself. While we would not necessarily expect all roles to work
on the executor, it is reasonable to want to use the ensure-pip
role (which logically should be a no-op on the executor) for the
side effect of finding and returning the appropriate pip command.
Currently, the role fails on the executor because it mistakenly
concludes that it must install python3-venv to get a working
venv module. By increasing the precision of the check for what
is missing (the actual error is a missing "ensurepip" python module
(oh irony!), we can avoid attempting an installation of
python3-venv on python docker images (including the Zuul executor
images).
This also adds the ensure-pip-localhost job
This tests that the ensure-pip role works on the Zuul executor.
The executor is a debian host with a working python environment,
so it should be a no-op (and no packages should need to be installed).
Change-Id: Id7f13f2f73d45e680f79c00a83751b185212a63d
This sorts the supported test platforms so that we can more easily
modify them in a future change.
Change-Id: I3a3a0347b89ab90d6ee74bc3070f7ad3604d59df
This does a full upload of the sandbox project to the test.pypi.org
service.
It uses the inline token with the caveats noted in the comment. Since
you can't upload the same version twice, always running this in the
check job will mean it will work the first time, but then fail (unless
something has been committed to the sandbox to update the version
number). As noted, the idea is that we develop the change and then do
one run that does a full test, and reviewers can check that before
committing. There's no real way to use a secret here because we
actually want this to run at check time, not in any post pipeline
where it would already be in production. We don't want to do
something like require committing something to sandbox every time you
run this in the check queue, etc. Given how much this is actually
updated (most of it hasn't been touched since 2019) I think this is a
reasonable compromise.
Change-Id: Ida4ad07c82a6301107c938565656988aba3bf250
Very basic testing to include the ensure-twine and upload-pypi role.
This really just does a syntax and sanity check.
Change-Id: Ie40ec32b3b2ffc4646301c6e145c527fe6a3bae2
First up we need python2.7 specific msrest version as azure-storage-blob
depends on it and it doesn't properly specify python versions for its
releases.
Next we cannot use zuul/zuul to test ensure-pip can install packages
because zuul/zuul requires python3.8 or newer now. Some platforms we run
ensure-pip on still only have python3.6. We address this by using
zuul-client instead.
Change-Id: I3d91fd3f85e3d7845ab2d7615a1ebfb614064573
CentOS 8 EOL'd at the end of 2021. OpenDev is beginning the process of
removing the image and starting with some job cleanups like the job
cleanups in this change. Users can use CentOS 8 Stream or some other
RHEL alike.
Change-Id: I9dab417b28a0503e1409e70953ed275f2c6f8c30
OpenDev doesn't currently have an image built for tumblweed. Rather than
try and fix this image up and get it running again OpenDev has proposed
it simply be removed. In preparation for that and to avoid running jobs
for a system that doesn't currently have an image we remove the jobs
here.
It was hoped that tumbleweed would be a good forward looking platform
but it seems to need a lot more care than we are able to give it.
Additionally Zuul is probably the major consumer of the platform and it
doesn't see much consumption by other projects.
Change-Id: Iad351b6da1f08183bc4c49c049c303a9491af844
Ubuntu Xenial reached end of standard support almost 6 months ago.
Its default python3 is 3.5, which new versions of many libraries are
dropping support for as that hit EOL earlier this month. We are now
faced with a situation where we cannot test the fetch-sphinx-tarball
role on Xenial because latest SetupTools breaks old funcparserlib
(indirectly needed for our blockdiag and seqdiag elements in
zuul-jobs' docker-image.rst), and the only funcparserlib release
which is installable has dropped Python 3.5 support.
Cease running Ubuntu Xenial platform and Python 3.5 tests for
changes to zuul-jobs now. This is a soft removal, since we're not
actually removing Xenial platform checks from playbooks or roles
yet, nor removing the tox-py35 job definition.
Change-Id: I46b9b887427133147481e92a1e7c523e6592fd2d
Fedora 32 is now EOL, we should test against the newly released
version of Fedora which is 34.
The podman job is made non-voting while we investigate issues running
as non-root.
Depends-On: https://review.opendev.org/c/openstack/project-config/+/795604
Change-Id: I962a123e1fbf68f064a197700d0fd0da239fca72
Debian Buster is the current stable, and Bullseye is the impending
stable (likely within the month). The parent of this change aims to
fix security mirror configuration for Bullseye, but to do that we
need to add testing for it. While we're at it, drop the Stretch
(oldoldstable) jobs to keep the job count from exploding too much.
Change-Id: I278246eff347dd4bdb3471a8db95d30fcba0f17a
The zuul-jobs-test-base-roles-gentoo-17-0-systemd job isn't passing
at the moment because Gentoo now needs Python 3.8 to install
pip/wheel, but the images are still using 3.6. A change has been
merged to correct this, but OpenDev has been unable to build new
Gentoo images since May 2020, so that will need to be solved before
the job will work again. In the meantime, stop running it so our
changes aren't indefinitely blocked. We'll need to apply extra
scrutiny to (or simply hold) any Gentoo-related changes for now
until we're testing that again.
Change-Id: I276bc596c50e7185fae828afa79c73ca9d955ec5
The merge-output-to-logs role is not doing anything in this test. As
described inline, this merges files in
zuul.executor.work_root/<docs,artifacts> into the
zuul.executor.log_root directory so they are available in change
results.
Since this job doesn't publish anything there, this role is unused.
merge-output-to-logs currently can't run because it tries to run shell
scripts on the executor. Thus we can remove this unused role and
restore the job.
Change-Id: I1afc905aa8d9c420bed316e99760ad7ad1d838ce
This reverts commit 51a8ed8e95.
This has a typo ("exector"). The fix is obvious, but the bigger
issue is that it was not caught in testing, even though the main
purpose of the change was to re-enable tests. We should understand
why it wasn't caught in testing and resolve that before fixing and
unreverting.
Change-Id: I3ed407546fecc52d4a039f7959c0521511e6a00b
This reverts commit 69a238df46.
The role is re-written with executor-safe methods.
Depends-On: https://review.opendev.org/753222
Change-Id: I0b52eff66bfdca776e0e5c426bf1fc57deb3fc49
Recent security fixes made to zuul made impossible to run this
test. Disabling this test until we can find a way to test it as
this is not an easy change.
This is needed because there were at least two other changes that
failed to pass because triggered this broken job. This was broken
for more than a month.
Main challenge here is that Ansible does not have any `mv` module, so
we still need to use shell. Replacing mv with copy could risk producing
out of disk space failures sooner or later.
Change-Id: If09e0430033fd2f1d6fc5e35e5ec0b10dc7c2dfe
Needed-By: https://review.opendev.org/#/c/748480/
Needed-By: https://review.opendev.org/#/c/748682/
To avoid getting zuul-jobs getting rotten without known, we need
to run them weekly.
This should solve issue where some less common job get broken and
we find out much later, when we have to make other unrelated changes.
Change-Id: I3a95526fdb83b8aef12ee2ee09a089990e75cb59
Depends-On: https://review.opendev.org/#/c/748706
We have Fedora 32 nodes now, retire the Fedora 31 testing and replace
with 32.
Docker isn't supported on Fedora 32. There's a problem with cgroups
v2 and you can read all about it at [1]. Drop the install test.
Haskell upstream has dropped F32 from the COPR repo as it is now
packaged; a new package with the correct dependencies has been pushed
but isn't available yet (see the inline github issue), so workaround
the missing package.
[1] https://github.com/docker/for-linux/issues/955
Change-Id: I025613bee8c0a8513afa90331431d2c18b616464
Add focal testing for automatically generated jobs.
Seems only one minor quoting change for ovs is required.
Change-Id: I5de5d5bd420092729de251d6bfcfe80b8af79f1a
This partially reverts commit
3f961ce202.
This alternative installs wheel with the ensure-pip role instead of in
a separate role. wheel is very closely linked with pip install
operations so this isn't a large overreach of the role.
I suggest this for several reasons; firstly the python-wheel role
doesn't try to install packages, so we end up with mixed system pip
and upstream versions of wheel most of the time. This is the type of
thing that has proven problematic in the past. It also installs via
pip --user; something we've already had problems with tox when for
various reasons roles want to run this as non-zuul user. Using
ensure-pip we keep the packaged versions together.
[1] did try to install wheel with root, but during runtime which
didn't work due to sudo being revoked. This should work for the
existing build-python-release job, because it already includes
ensure-pip in pre-run via playbooks/python/pre.yaml
I believe our conclusion on the ensure-* roles was that requiring
root/become: for installation is OK, but we should have a no-op path
if the tools are found. This is consistent with that approach
(i.e. if you want wheel and can't do sudo, you should pre-install it
on your image using whatever you build that with).
This adds a check to the existing "is pip installed" check to also
check if wheel packages are available. If not we trigger the install
path.
This revealed some issues with RedHat.yaml -- we can always install
Python 3 (packages available for CentOS 7) so remove that check, and
if Ansible is running under Python 2; ensure we install the
dependencies too (not only if it is forced).
Update the documentation to describe that it will enable support for
bdist_wheel, and add a basic sanity test that wheels are produced by
pip. The existing build-python-release job is kept; although it is
modified to use the playbooks/python/pre.yaml playbook as the build
job does.
Change-Id: I2ab11bb45b6b2a49d54db39195228ab40141185c
[1] https://review.opendev.org/#/c/736001/5/roles/build-python-release/tasks/main.yaml
With the merge of I76befd9e2a054798e8e36500b723870f7c7ae328 the main
images have dropped pip-and-virtualenv. Remove the plain variant
testing.
Change-Id: If442e15a0264821207725c1cc271b3ff646b0720
Since tox_envlist has a default value it cannot be undefined
so the fail task will never run. Instead handle the case when
tox_envlist is an empty string by getting the default configured
envlist from tox. Also handle the casewhen tox_envlist is 'ALL'.
This also updates tox_install_sibling_packages to correctly
handle multiple testenvs and uses configuration supplied by
'tox --showconfig -e <envlist>' instead of guessing where the
envdir and logdir are located.
We also cannot run tox inside python because it gets complicated
to know which tox_executable we should call during the python test
cases so run these commands in ansible and pass the output to
tox_install_sibling_packages.
Since role params have higher precedence than set_fact we set an
internal _tox_envlist fact that is a comma separated list of testenvs
that should be run.
Change-Id: I9e5a1b041f653cbcff7b8ed62e4a95a0a040fdd7
This change enables using fetch-sphinx-tarball role along with
the fetch-output role. By default the role still synchronizes
artifacts back to the executor.
Change-Id: I7452f34bccdca49e256053f9630f77953b27f99c
This change enables using fetch-sphinx-output role along with
the fetch-output role. By default the role still synchronizes
artifacts back to the executor.
Change-Id: I81c8a2faf566be694dbf1c59cbbd157dc4df5146
With the dependent change pip-and-virtualenv is dropped from the
opensuse-15 image, thus we don't need the -plain testing any more.
Change-Id: Iccf70042833093ebeca92c171dfe05dbf14a55aa
Depends-On: https://review.opendev.org/725749
This adds more of the plain nodes (without pip-and-virtualenv) to our
testing environment to facilitate the eventual switch in opendev
infra.
Change-Id: I091ab667c5f071ba5844ea567464b1c29052638d
This is part of the efforts to remove pip-and-virtualenv from our base
images [1].
There are some users who specifically require the virtualenv command
(perhaps, like dib, they have some code that uses the activate_this.py
mechanisms it provides wich venv does not).
This installs the virtualenv package for the currently running
distribution.
One of the main maintenance issues of pip-and-virtualenv is that tried
to ensure that "virtualenv" created a Python 2 environment always by
default. Now that we have Python 3 only distributions like current
Fedora, this is not something we can continue to do (even if we wanted
to, which we don't).
What owns virtualenv and what it produces in our heterogeneous
environment is messy, and I think the best we can do is document it as
done here.
[1] https://docs.opendev.org/opendev/infra-specs/latest/specs/cleanup-test-node-python.html
Change-Id: I97d8bfb970ed2b5aaa02a0813899014c94994066
We discovered that role fails on some platform because of the missing
bzip2 package. We can avoid this by using gzip which is much more
likely to exist on machines (and exists in the container images for
most operating systems).
Change-Id: Ia33a0f77343df809cd3c591a4a988e8e3712c133
Co-Authored-By: Mohammed Naser <mnaser@vexxhost.com>
Otherwise this role could fail when tox is not installed in PATH.
Also updates the default behaviour to only collect the environments
configured with 'envlist' in tox.ini and collects all environments if
the envlist is set to 'ALL'.
Change-Id: I60bcc25cb853d2db3ee767a719b9ac54e421532e
We have enough infrastructure in place now to have the
pip-and-virtualenv free node type voting, while we progress towards
making this the default.
Change-Id: Ie2677364d8696d5c4f71eefe2054da1b39e8bb35
Update the Fedora test platform to 31. Since we have dropped the
pip-and-virtualenv element with this image, we do not need the
work-around for excluded packages any more.
Story: #2007386
Task: #39310
Change-Id: I713a3cc790039dee3d28d83198f5afff5bebc491
This role is intended to ensure that the `pip:` module, and jobs that
may wish to use `pip` from the shell, have the necessary requirements.
It is intended as a partial replacement for the pip-and-virtualenv
element in diskimage-builder, which currently pre-installs pip on our
infra CI images during image build.
We wish to remain broady compatible with this element, but not
replicate some of the more problematic areas of its implementation.
By default, this installs the system packages for pip and setuptools
(the latter being a requirement of the Ansible pip module, which
imports it directly, despite pip itself not requiring it).
In this role, we ensure the libraries for the currently running
ansible_python_interpreter version are installed.
There is provision to provide a flag to install the packages directly
from upstream via get-pip.py, although this is not recommended.
Story: #2007386
Task: #39309
Change-Id: Iac2d518a66caf1b801273225f75a0a748412903c
This is a temporary node type that does not install
"pip-and-virtualenv" so does not have pip or virtualenv pre-installed.
Add this for non-voting testing as we migrate to this node type.
Change-Id: Ia9006873470d76d29cc9229f11650981662b2de9
Story: #2007386
Task: #39306
It seems some versions of ruamel do a better job at preserving
comments than others. Create a tox env to regenerate the lists with a
version we know works.
Change-Id: I13503890fd3c18487281e3394b003e9f36cd24f2
This change enables using fetch-subunit-output role along with
the fetch-output role. By default the role still synchronizes
artifacts back to the executor.
Change-Id: I50a50856b96a9112c1b4be82ca04b612fd29d939