2048 Commits

Author SHA1 Message Date
James E. Blair
8074d20b8f Fix ensure-docker for Ansible 6
The systemd_service module is known as "systemd" in Ansible 6.  Use
the backwards compat format until Ansible 6 is removed from Zuul.

Change-Id: Ifff1cdfdf4d03426f289355673ee1472f0d37dd6
2024-02-16 06:55:05 -08:00
James E. Blair
509880073e Remove command.warn usage
This is no longer present in Ansible 9.

Removing these upsets ansible-lint, so those errors are ignored.

The base roles job has bitrotted on centos-7 and bionic due to
a bad voluptuous release used in an stestr test.  That is fixed in
this change as well.

Change-Id: I67886d5ad82ab590979f82bd102d6f974b9d4421
2024-02-14 15:17:52 -08:00
Clark Boylan
9519fafd10 Override DOCKER_MIN_API_VERSION for skopeo when installing docker
Skopeo hardcoded the docker api version for image manipulation to
version 1.22 of the api until very recently. Docker 25 sets 1.24 as a
minimum version by default introducing an incompatibility between the
tools. It isn't straightforward to install an updated skopeo everywhere
we need it (due to golang requirements). As a workaround we override
the min version to 1.22 when installing the docker daemon. This should
work until Docker 26 is released and removed the override option.

Note we also pin microk8s from latest/stable (which is currently
1.29/stable) to 1.28/stable to workaround
https://github.com/canonical/microk8s/issues/4361. This is necessary to
get the CI jobs for docker/registry/k8s testing working in order to land
this fixup.

Change-Id: I377ac84d532749eba578c4b32eb2ed6a5ce7a0c9
2024-02-14 10:42:47 -08:00
Zuul
df5756f765 Merge "Add zuul-tenant-conf-check role/job" 2024-02-13 16:09:34 +00:00
Zuul
d2998dfbcd Merge "Introduce LogJuicer roles" 2024-02-13 15:41:51 +00:00
James E. Blair
73bdf1f2df Add zuul-tenant-conf-check role/job
This performs static validation of Zuul tenant config files.

Change-Id: I5d439d6cfb963e55d07b2a0058de76f030fe47b3
2024-02-01 15:56:29 -08:00
Tristan Cacqueray
26db5b3b24 Introduce LogJuicer roles
This change adds new roles to run logjuicer in zuul jobs:
  https://github.com/logjuicer/logjuicer

Change-Id: I02824a18285a16c8f0be6bb96b5404aa0d601c16
2024-01-08 16:09:17 +00:00
Felix Edel
7761396303 mirror-workspace-git-repos: Retry on failure in git update task
We occasionally see the this task fail for the first element in the
zuul.projects list with a MODULE FAILURE and a return code of -13
(SIGPIPE) [1]. So far we couldn't identify the root cause, so try to
mitigate this issue by retrying on failure. This solution is similar to
the one used for the "Synchronize repos" task[2].

There is a bug report in Ansible that fits

Since it's only the first element in the loop that is failing while
subsequent elements are successful, we currently have two assumptions:

  1. As the task before is using a `delegate_to: localhost' [3],
     there might be a problem with Ansible when switching the connection
     from localhost to the remote host (node).
  2. Since the task before is using the same SSH connection [4] that is
     used by Ansible to push the git repository, there might be some
     "leftovers" on the connection that make the next task fail.
  3. There is also a bug report in Ansible [5] which might be causing
     that error.

[1]:
    {
        "ansible_loop_var": "zj_project",
        "changed": false,
        "failed": true,
        "module_stderr": "",
        "module_stdout": "",
        "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
        "rc": -13,
        "zj_project": {...}
    }

[2]: 3b3495e255/roles/mirror-workspace-git-repos/tasks/main.yaml (L32)
[3]: 3b3495e255/roles/mirror-workspace-git-repos/tasks/main.yaml (L25)
[4]: 3b3495e255/roles/mirror-workspace-git-repos/tasks/main.yaml (L16)
[5]: https://github.com/ansible/ansible/issues/81777

Change-Id: I0c4cb87bb076b9b40c9c446dbe5db437daff5897
2023-12-08 06:37:55 -08:00
Roman Kuznecov
6710f84a90 tox: Do not concat stdout and stderr in getting siblings
Several packages in calling "python setup.py --name ..." may return
warning message, e.g.:

  ...
  _DeprecatedInstaller: setuptools.installer and fetch_build_eggs are deprecated.
  !!

          ********************************************************************************
          Requirements should be satisfied by a PEP 517 installer.
          If you are using pip, you can try `pip install --use-pep517`.
          ********************************************************************************

  !!
  dist.fetch_build_eggs(dist.setup_requires)
  WARNING setuptools_scm.pyproject_reading toml section missing 'pyproject.toml does not contain a tool.setuptools_scm section'
  <PACKAGE_NAME>

and then this huge log places into the package_name variable. But
script expects that package_name will contain only package name. Because of this situation could not find siblings packages and
install them.

Change-Id: I5bf9a19233c48d1260b5ab17d749bfc58a8ef2fa
2023-11-24 06:32:19 +00:00
Zuul
3b3495e255 Merge "Deprecate mirror-workspace-git-repos" 2023-09-26 23:24:36 +00:00
Lukas Kranz
ce2bea51d4 Deprecate mirror-workspace-git-repos
This change is preparation for https://review.opendev.org/c/zuul/zuul-jobs/+/887917

In the beginning, there was only prepare-workspace[0] which rsynced repos.

Then we added mirror-workspace-git[1] to make it more efficient by using git operation, but it required some openstack-specific code in project-config to work.

Then we added prepare-workspace-git[2] which completed the git-based sync solution by locating everything requried in zuul-jobs.  It used mirror-workspace-git by reference and added this TODO:

  # TODO(tobiash): we might want to deprecate the role mirror-workspace-git-repos
  # and move it here.

This change completes that TODO by moving the mirror-workspace-git-repos code into prepare-workspace-git and places the repo in a sensible and maintainable state with two simple and good options:

 * prepare-workspace (rsync)
 * prepare-workspace-git (git)

In the unlikely event anyone is still using mirror-workspace-git-repos standalone (OpenStack/OpenDev is not, and that solution was haphazard as described above) they would be well served by a notification that there is a better alternative which is what most of the community actually uses now.

[0] cfffd4431b8efc2f4df1999ecb89384a29c59238
[1] 348598e96aac742954fa326a15a4ef8fd7f71b8b
[2] 7cee7156bcde8bc396ac4b6581bf2cae02eea0e9

Change-Id: Ib80e0447d49363182fd0d4c4d4e269841bc3aa95
2023-09-25 14:51:26 -07:00
Zuul
6ad6ee7a65 Merge "roles/ensure-python: Fix 'python_use_stow' option" 2023-09-05 14:52:59 +00:00
Zuul
5aca055b8b Merge "Drop Helm v2 support to fix v3 issue" 2023-09-04 14:07:26 +00:00
James E. Blair
56653e0470 Support ensure-kubernetes on bookworm
Change-Id: I88cd78e73611f97ad0cfb1eebe86626b79fd1699
2023-08-14 11:15:36 -07:00
James E. Blair
df906ec2f7 Fix zookeeper install on debian bookworm
This is a workaround for https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1030129
which prevents installation of the JDK on bookworm.

This also updates the test platforms for this role.

Change-Id: I1efe59a4e7884fffe15bc623f717f891189f48ca
2023-08-11 14:59:50 -07:00
Moritz Haase
d8af2889f8 roles/ensure-python: Fix 'python_use_stow' option
Fix a syntax error in stow.yaml that broke the 'python_use_stow' option. While
at it, make sure that a proper error message is shown if no stow environments
are installed on the system.

Change-Id: Ifa696daa367997fb705ba22527137f4f051ca558
2023-07-11 06:17:29 +00:00
Zuul
d490441a61 Merge "Use zuul_workspace_root for prepare/mirror workspace roles" 2023-06-29 16:24:26 +00:00
Zuul
1d9fcd8017 Merge "Use zuul_workspace_root for prepare/mirror workspace test roles" 2023-06-28 18:35:59 +00:00
Benedikt Loeffler
7f400a0d32 Use zuul_workspace_root for prepare/mirror workspace roles
Use zuul_workspace_root as the destination folder for preparing and
mirroring the workspace, which points to ansible_user_dir by default.
(As is currently the case).
The override of this variable can be useful when using a static node
with multiple slots to use a separate workspace for each slot.

Change-Id: If50abc7ccc2dfb6f4ea9f9d5c6a5091ff6967aef
2023-06-27 14:22:08 +02:00
Benedikt Loeffler
258add7bb4 Use zuul_workspace_root for prepare/mirror workspace test roles
Use zuul_workspace_root as the destination folder for preparing and
mirroring the workspace, which points to ansible_user_dir by default.
(As is currently the case).
The override of this variable can be useful when using a static node
with multiple slots to use a separate workspace for each slot.

Change-Id: I2014b9c876907cfc17a0ec7fa2b6d4e67cb2331c
2023-06-27 14:21:38 +02:00
James E. Blair
97ae13d7a2 Pin podman on jammy
The jammy-updates/universe repo recently added
podman 3.4.4+ds1-1ubuntu1.22.04.1 which produces the error:

  Error validating CNI config file
  /home/zuul/.config/cni/net.d/zuul-tutorial_zuul.conflist:
  [plugin bridge does not support config version \"1.0.0\"
  plugin portmap does not support config version \"1.0.0\"
  plugin firewall does not support config version \"1.0.0\"
  plugin tuning does not support config version \"1.0.0\"]

Pin to 3.4.4+ds1-1ubuntu1 until this is fixed.

Change-Id: I944c6cbb8129abfc8e3383e0f08920520345f5be
2023-06-20 18:49:14 -07:00
Radosław Piliszek
0f662a23d0 Drop Helm v2 support to fix v3 issue
Helm v2 is long gone and there seems to be no interest in
supporting it any longer. [1]

Dropping support for v2 allows for an easy fix to the
command used so that it names the release properly
as it was supposed to instead of trying to set the
namespace name (the -n switch changed purpose [1]).

[1] https://lists.zuul-ci.org/archives/list/zuul-discuss@lists.zuul-ci.org/thread/WUWBM5F3PXXDLKK6JNSP4UR4VTWDNPZ4/

Story: 2010783
Task: 48201
Change-Id: I00676bcc7e8b4f3ec014f2292c6c0ee4c17acd96
2023-06-20 17:06:36 +02:00
Zuul
29f27ebaaa Merge "Use release CentOS SIGS repo to install openvswitch in C9S" 2023-06-13 14:44:44 +00:00
Zuul
efdedc66a3 Merge "Force cgroupfs cgroup manager with podman on ubuntu" 2023-06-01 01:15:00 +00:00
Zuul
90fcbf8ee5 Merge "use-docker-mirror: Also run task if docker_mirror is defined" 2023-05-26 17:50:35 +00:00
Alfredo Moralejo
b778eaa960 Use release CentOS SIGS repo to install openvswitch in C9S
After moving to C9S we started using testing repository for ovs and we
never switched to release one after NFV SIG started releasing packages
to the official mirrors.

Change-Id: I0f199c9fafe9d5ac3a47b9e55a2d33c4d6735604
2023-05-22 10:06:03 +02:00
Clark Boylan
2b37cf7a9e Force cgroupfs cgroup manager with podman on ubuntu
Per https://github.com/containers/podman-compose/issues/209 it seems
this is a known thing with non root containers, podman, and Ubuntu. Just
apply the necessary woraround in a config file so that we don't have to
be aware of it every time we try to use podman in this way. It also
makes using podman-compose simpler.

Change-Id: Ic024e242d99b7f0d4fbd4cf2c3489022b364672d
2023-05-18 14:25:52 -07:00
James E. Blair
45b70f4a56 Add podman dns plugin to ensure-podman
This is available in this package in Ubuntu 22.04 and later.  I
am unaware of how it might be installed in any other distro.

Change-Id: Ibde5b8af4907eb6c1a7dd2e445a560850d85552f
2023-05-18 09:59:38 -07:00
Clark Boylan
5994ce4049 Gracefully handle use of intermediate registry in container upload role
For symmetry and ease of transition between the docker specific
jobs/roles and generic container jobs/roles it is advantageous to have
the container upload role skip pushing artifacts to the final registry
location if we are relying on the intermediate registry instead.

Update the container upload role to skip pushing to the actual registry
if the promote var is set to intermediate registry. This allows us to
avoid reshuffling all of our jobs as we migrate between the two
implementations.

Change-Id: I3cae9e03517cb0a5ce8e9369bf43fd052cac97ff
2023-05-09 16:18:42 -07:00
James E. Blair
839de7f899 Add labels to build-container-image
The "docker build" path is missing the "--label" arguments which are
present in all other paths (docker buildx in both build-container-image
and build-docker-image roles).  This means that images are being built
without expected metadata labels.

This change adds the extra arguments to match.

Change-Id: I607ae89cace87fe87b6dcfb95a1bae4fca516d3c
2023-05-01 10:13:32 -07:00
Clark Boylan
3db2bfe8d1 Skip quay repo creation if necessary info is missing
We try to be smart about when we attempt to create repos in quay. One
thing that was missing was checking if necessary info for the API
request is present. Skip attempting creating if not present.

Change-Id: I61e603c4d07280079e560215685bf09ebc0c4263
2023-04-28 15:08:04 -07:00
Clark Boylan
58f408cfac Update ensure-quay-repo to run opportunistically
This updates the new ensure-quay-repo to run opportunistically if the
registry_type image flag is set to quay and the registry credentials
matching the container image has an api token defined. This will allow
us to include this role in base jobs and it will do what we need it to
do without impacting docker based images or quay managed images that
don't need automatic creation.

Change-Id: Ia419578bf0a27293757c5f723873e9930ee2c489
2023-04-26 08:45:50 -07:00
Zuul
0e6df8d38f Merge "Add ensure-quay-repo role" 2023-04-24 23:08:00 +00:00
Clark Boylan
4617d0835a Use full image url in container buildx path
The container roles assume a full image url path and do not work with
shorted names like the docker roles do. The buildx path used the shorted
path when it should use the full path. Fix this.

Change-Id: Icdeee0ebb1c0d280968f425f8560cc5208ebd18d
2023-04-21 16:33:02 -07:00
Clark Boylan
18b32703ed Add ensure-quay-repo role
This adds a new role that can be used to ensure a quay repo exists
before publishing to it. This is particularly useful for creating public
repos in quay as simply pushing to a repo with quay will create a
private repo by default.

Change-Id: I979f1b9b64f901bb8d54b8991bb9142b18b6330f
2023-04-21 15:09:42 -07:00
Zuul
b7f983c621 Merge "Move containerfile setting in container build" 2023-04-21 21:34:25 +00:00
James E. Blair
2c6bc6d7a3 Move containerfile setting in container build
When setting the containerfile variable, we need to be inside the
zk_images loop in order to set the correct default for each image.

Change-Id: I216ffd19dd797752b2e5ca2332e651b8e6ac8a3c
2023-04-21 10:19:40 -07:00
Zuul
0354a8c52d Merge "container role docs : clarify requirements" 2023-04-13 18:05:57 +00:00
James E. Blair
f381cc328b Update promote-container-image to copy from intermediate registry
Change-Id: Ia24bbd101e01ab371ceacfed006b5ff806418a97
2023-04-12 11:36:26 -07:00
Zuul
bdb6f558fe Merge "remove-registry-tag: no_log assert" 2023-04-04 22:04:10 +00:00
Zuul
a9725e8cfd Merge "remove-registry-tag: update docker age match" 2023-04-04 22:04:09 +00:00
Zuul
e3804cb3c4 Merge "promote-container-image: use generic tag removal role" 2023-04-04 22:04:07 +00:00
Zuul
d8caef726f Merge "remove-registry-tag: role to delete tags from registry" 2023-04-04 22:04:04 +00:00
Zuul
9e82c7a24c Merge "promote-container-image: add promote_container_image_method" 2023-04-04 22:04:02 +00:00
Zuul
35d57db9a4 Merge "build-container-image: expand docs" 2023-04-04 22:02:28 +00:00
Zuul
29de51b1f0 Merge "promote-image-container: do not delete tags" 2023-04-04 22:02:26 +00:00
Ian Wienand
7bbe8f5b0d
remove-registry-tag: no_log assert
Just to be sure, add no_log here.  Since the actual authenticated call
is no_log too; if this fails you at least know you didn't get the
credentials layout right which narrows down the problems the actual
call might be facing.

Change-Id: Ic7174c10f4e693f6a2c0554dc66ea22fd28d678f
2023-04-05 07:38:54 +10:00
Ian Wienand
507dfdad6b
remove-registry-tag: update docker age match
Update this match to be a little shorter and use datetimes more
directly.

Change-Id: I2012091a16b5b5ddb522a5e5ac6976c4f4c580af
2023-04-04 09:56:26 +10:00
Ian Wienand
9df7c8eb70
promote-container-image: use generic tag removal role
This uses the generic tag removal role added with
I7f2d9d00024e34451e2d20b2c2f8171ecd151943 to cleanup the promote tag
and any leaked tags.

Change-Id: I3f1b82d63874ee886048b9ccabe616a60dc09434
2023-04-04 09:56:18 +10:00
Ian Wienand
fec27296c8
remove-registry-tag: role to delete tags from registry
This is a role to abstract removal of tags from registries, which is
an operation that practically has to be done via the registry API.

This implements removing tags from the quay and docker API's.

For the common case of working with a repository like
"quay.io/org/project" there is minimal configuration.  However, if you
run a private repository, this is flexible with a few extra variables
to tell the role to use the quay API but your own URL.

By default it clears out old tags from the Zuul promote pipeline.
However if you set registry_tag_remove_tag it will only remove that
one tag.

This is inspired by the current work done in promote-docker-image
role.

Change-Id: I7f2d9d00024e34451e2d20b2c2f8171ecd151943
2023-04-04 09:53:18 +10:00