2591 Commits

Author SHA1 Message Date
Clark Boylan
3db2bfe8d1 Skip quay repo creation if necessary info is missing
We try to be smart about when we attempt to create repos in quay. One
thing that was missing was checking if necessary info for the API
request is present. Skip attempting creating if not present.

Change-Id: I61e603c4d07280079e560215685bf09ebc0c4263
2023-04-28 15:08:04 -07:00
Clark Boylan
58f408cfac Update ensure-quay-repo to run opportunistically
This updates the new ensure-quay-repo to run opportunistically if the
registry_type image flag is set to quay and the registry credentials
matching the container image has an api token defined. This will allow
us to include this role in base jobs and it will do what we need it to
do without impacting docker based images or quay managed images that
don't need automatic creation.

Change-Id: Ia419578bf0a27293757c5f723873e9930ee2c489
2023-04-26 08:45:50 -07:00
Zuul
0e6df8d38f Merge "Add ensure-quay-repo role" 2023-04-24 23:08:00 +00:00
Clark Boylan
4617d0835a Use full image url in container buildx path
The container roles assume a full image url path and do not work with
shorted names like the docker roles do. The buildx path used the shorted
path when it should use the full path. Fix this.

Change-Id: Icdeee0ebb1c0d280968f425f8560cc5208ebd18d
2023-04-21 16:33:02 -07:00
Clark Boylan
18b32703ed Add ensure-quay-repo role
This adds a new role that can be used to ensure a quay repo exists
before publishing to it. This is particularly useful for creating public
repos in quay as simply pushing to a repo with quay will create a
private repo by default.

Change-Id: I979f1b9b64f901bb8d54b8991bb9142b18b6330f
2023-04-21 15:09:42 -07:00
Zuul
b7f983c621 Merge "Move containerfile setting in container build" 2023-04-21 21:34:25 +00:00
Clark Boylan
f910bbac74 Pin virtualenv in tox environments
Virtualenv 20.22.0 dropped support for python<=3.6 (including 2.7). We
still want to run tests under 2.7. Work around this by capping off
virtualenv in tox's requires list.

Change-Id: I4868cd5a8a958f04ac782e0963c52a118b2f5ebf
2023-04-21 20:43:45 +00:00
James E. Blair
2c6bc6d7a3 Move containerfile setting in container build
When setting the containerfile variable, we need to be inside the
zk_images loop in order to set the correct default for each image.

Change-Id: I216ffd19dd797752b2e5ca2332e651b8e6ac8a3c
2023-04-21 10:19:40 -07:00
Zuul
0354a8c52d Merge "container role docs : clarify requirements" 2023-04-13 18:05:57 +00:00
Zuul
73e145a493 Merge "containers : update test variable" 2023-04-13 18:02:29 +00:00
Zuul
1733210f17 Merge "Update promote-container-image to copy from intermediate registry" 2023-04-13 15:13:40 +00:00
Zuul
7f4efe3ebe Merge "test-registry: split docker and container paths" 2023-04-12 23:46:41 +00:00
James E. Blair
f381cc328b Update promote-container-image to copy from intermediate registry
Change-Id: Ia24bbd101e01ab371ceacfed006b5ff806418a97
2023-04-12 11:36:26 -07:00
Zuul
bdb6f558fe Merge "remove-registry-tag: no_log assert" 2023-04-04 22:04:10 +00:00
Zuul
a9725e8cfd Merge "remove-registry-tag: update docker age match" 2023-04-04 22:04:09 +00:00
Zuul
e3804cb3c4 Merge "promote-container-image: use generic tag removal role" 2023-04-04 22:04:07 +00:00
Zuul
d8caef726f Merge "remove-registry-tag: role to delete tags from registry" 2023-04-04 22:04:04 +00:00
Zuul
9e82c7a24c Merge "promote-container-image: add promote_container_image_method" 2023-04-04 22:04:02 +00:00
Zuul
35d57db9a4 Merge "build-container-image: expand docs" 2023-04-04 22:02:28 +00:00
Zuul
29de51b1f0 Merge "promote-image-container: do not delete tags" 2023-04-04 22:02:26 +00:00
Ian Wienand
7bbe8f5b0d
remove-registry-tag: no_log assert
Just to be sure, add no_log here.  Since the actual authenticated call
is no_log too; if this fails you at least know you didn't get the
credentials layout right which narrows down the problems the actual
call might be facing.

Change-Id: Ic7174c10f4e693f6a2c0554dc66ea22fd28d678f
2023-04-05 07:38:54 +10:00
Ian Wienand
507dfdad6b
remove-registry-tag: update docker age match
Update this match to be a little shorter and use datetimes more
directly.

Change-Id: I2012091a16b5b5ddb522a5e5ac6976c4f4c580af
2023-04-04 09:56:26 +10:00
Ian Wienand
9df7c8eb70
promote-container-image: use generic tag removal role
This uses the generic tag removal role added with
I7f2d9d00024e34451e2d20b2c2f8171ecd151943 to cleanup the promote tag
and any leaked tags.

Change-Id: I3f1b82d63874ee886048b9ccabe616a60dc09434
2023-04-04 09:56:18 +10:00
Ian Wienand
fec27296c8
remove-registry-tag: role to delete tags from registry
This is a role to abstract removal of tags from registries, which is
an operation that practically has to be done via the registry API.

This implements removing tags from the quay and docker API's.

For the common case of working with a repository like
"quay.io/org/project" there is minimal configuration.  However, if you
run a private repository, this is flexible with a few extra variables
to tell the role to use the quay API but your own URL.

By default it clears out old tags from the Zuul promote pipeline.
However if you set registry_tag_remove_tag it will only remove that
one tag.

This is inspired by the current work done in promote-docker-image
role.

Change-Id: I7f2d9d00024e34451e2d20b2c2f8171ecd151943
2023-04-04 09:53:18 +10:00
Ian Wienand
0a64d51c3d
promote-container-image: add promote_container_image_method
After recent conversations, we've come to the conclusion it will be
good to have two models of promotion

 - using tags, where gate directly uploads to the final repository and
   promote retags the image.

 - from an intermediate-registry, where upload stores the built image
   in an i-r and the promote step uploads to the final registry.

To facilitate this, we add a "promote_container_image_method" flag to
the promote roles.

The documentation is expanded to explain how all this is intended to
work together.

These roles haven't been publicised yet, but this should be a no-op as
it defaults to tags, which is the current operation.

c.f. Ia24bbd101e01ab371ceacfed006b5ff806418a97

Change-Id: I1c25f60f835b1cab983bcdd169eeffc0e250a56c
2023-04-04 09:50:17 +10:00
Ian Wienand
d7e5559e58
build-container-image: expand docs
This goes into a bit more detail on the advantages/disadvantages of
each method.

Change-Id: Ie90a52f1c0e205e9f8552156aded871b6fd30214
2023-03-30 10:38:23 +11:00
Ian Wienand
51e437c2f1
promote-image-container: do not delete tags
As noted inline, currently promote-image-container uses skopeo to
delete tags which is not the semantics we want.  This results in the
whole image being removed.

For safety we remove this call; we have two solutions in follow-ons
(deleting tags directly from the registry with a generic tag, and
promoting images from the intermediate registry).

Change-Id: I4b257f593275413da9a50a0cc64e13638e7f94cb
2023-03-30 08:30:33 +11:00
Zuul
803458668d Merge "build-container-image: directly push with buildx" 2023-03-28 21:50:23 +00:00
James E. Blair
6fd25dc10d Add --insecure-policy to skopeo promote command
We don't seem to have a policy file available in zuul-bwrap, but
we also don't really need it for this operation, so just tell skopeo
to skip it.

Also, fix the registry logout command.

Change-Id: I18ad59ffa8e8e80a720e798ab2bb81ea5d19619a
2023-03-23 18:56:02 -07:00
Ian Wienand
0ad671bfbf
test-registry: split docker and container paths
This ended up calling into push-to-intermediate-registry with both
docker_images *and* container_images variable set.

This hid from testing that push-to-intermeidate-registry was not
working with only the container_images variable set.

Split these calls up so we don't have both variables defined.

Change-Id: If84b039852f2afc4df66c98e64fcce6f30f51246
2023-03-24 12:19:47 +11:00
James E. Blair
e07ed8b5d9 Move container-image-promote login block
This should be inside the zk_image loop.  Also, try to log out
as well.

Change-Id: Id569b8a46c373e1c0b7b48d97ad45c893298fedf
2023-03-23 18:19:23 -07:00
Zuul
6daa9a5572 Merge "push-to-intermediate-registry: look for container_images variable" 2023-03-24 01:15:58 +00:00
Ian Wienand
0939c2a6be
build-container-image: directly push with buildx
The multi-stage build/push was added with
I8036a9b4d4c515c20a05994741540b999e7cbcae (2020-05) which noted

  When building multi-arch it's done in parallel which can result
  in the same layer being pushed at the same time, which is bad for
  the registries.

I believe this was probably actually a bug in zuul-registry, addressed
with Ibdf1ca554756af61247d705b2ea3cf85c39c2b83 (2022-02)

  The way the registry was previously written if two concurrent
  uploads of the same blob were happening one would fail to grab the
  lock and then return early. The uploading client would then
  immediately HEAD the blob and if it did so quickly enough would get
  a short size or 404.

So this simplifies a couple of steps by returning the push to the
build step.

Change-Id: I41b8a495eb3097b358d7634b63b7eb9de6161059
2023-03-24 11:07:06 +11:00
Zuul
b7cf56103e Merge "Handle credential repository not being defined in container roles" 2023-03-23 23:38:04 +00:00
Zuul
6e63339611 Merge "Fix container-image pre playbook container_command default" 2023-03-23 23:37:04 +00:00
Ian Wienand
d1e74606bc
push-to-intermediate-registry: look for container_images variable
When used with the container-image roles, the variable is
container_images, not docker_images.

Change-Id: I074f9523aea7d71879f722b88ff7c3e7e45fb90f
2023-03-24 10:35:35 +11:00
James E. Blair
a3a38ee2a2 Handle credential repository not being defined in container roles
This attribute is optional.

Change-Id: I0875b25d07f5fde0f6c3a3b62c0dac93770ffc49
2023-03-23 15:33:36 -07:00
Zuul
b9799f0c7f Merge "buildx: remove experimental flags" 2023-03-23 22:23:29 +00:00
Zuul
98aa36824e Merge "Add docker buildx multiarch support to container roleset" 2023-03-23 22:23:27 +00:00
James E. Blair
c14523148a Fix container-image pre playbook container_command default
This variable may be undefined, so use the default documented by
the roles.

Change-Id: I842a596402f30f1c9a1c2369a6dde23c346fd5f5
2023-03-23 12:56:45 -07:00
Zuul
54b41e3bdf Merge "Add support for passing env vars to the container build env" 2023-03-23 19:54:59 +00:00
Ian Wienand
7631526431 buildx: remove experimental flags
Tetsing this, I can't determine exactly when this might have switched.
It's not listed in.  Let's see...

[1] https://github.com/docker/cli/blob/master/experimental/README.md

Change-Id: If6c2ae98ea42c505f46563ca84a870081c4cdbe9
2023-03-23 11:14:16 -07:00
Clark Boylan
2d1c713b75 Add docker buildx multiarch support to container roleset
This adds support for multiarch container image builds when using docker
as the container command to the container roleset.

Change-Id: I48bf2e34c258e54baf013d3c04c6d4baaacde04b
2023-03-23 11:14:14 -07:00
James E. Blair
466aa92635 Add container build jobs
These jobs use the container build roles.

Change-Id: I13d1987980bc3d0b1c717878a4bc47edc6dcfe1c
2023-03-23 09:47:49 -07:00
Clark Boylan
13e44fa520 Add support for passing env vars to the container build env
This is useful if you are using docker and wish to enable buildkit for
example.

Change-Id: I69ae90d945fa3f4eabc10460da936b23c8212858
2023-03-22 14:30:35 -07:00
James E. Blair
40f9c38d0c Fix file matchers in docker/container jobs
This corrects and simplifies the file matchers for these jobs at
the expense of possibly running some extra jobs in some cases.

The alternative is a lot of duplicated lists that will be hard
to maintain.

Change-Id: I97406cd0506a438e4b14bbbee3d1b61026f40cd7
2023-03-22 14:30:35 -07:00
Ian Wienand
9baf07a104
container role docs : clarify requirements
skopeo is used from the executor in the promote process, but isn't
required to be installed on nodes with ensure-skopeo.  Clarify this
and the runtime requirements.

Change-Id: Ifaf4b788117986037b124972441b1884744374e5
2023-03-22 11:58:41 +11:00
Ian Wienand
75c0f2ebaa
containers : update test variable
Use the term "multiarch" consistently in here to make it a bit easier
to follow.

Change-Id: Ic80b39797c5885ec1d184f1ab5d03d858b83417f
2023-03-22 11:58:38 +11:00
Zuul
e4ac244bd8 Merge "Refactor docker/container image jobs" 2023-03-22 00:32:51 +00:00
Zuul
7ca2ae289c Merge "Refactor docker/container image variables" 2023-03-22 00:32:49 +00:00