40 Commits

Author SHA1 Message Date
Yaguang Tang
74215c43e7 Install ca-certificates in the buildx image
Buildx image building (for multiarch support) requires us to set up CA
certificates that are trusted by the image for access to the
registr(y|ies). The buildx image switched to alpine at some point which
doesn't include the update-ca-certificates command we were depending on
for this by default. Install that package so that we can
update-ca-certificates and trust the registr(y|ies).

Note that there may be a better way via buildkitd.toml configuration
files that allow us to configure certs upfront and have the image set
them up for us. This is documented in a todo as a future action we can
investigate.

Change-Id: Ied0b8e81ed6a7a15ea2def26b85b933223cb42e6
2025-02-11 13:46:12 -08:00
Mohammed Naser
bdae8c9433 Add other nodes to buildx builder
If you need to run native arm64 builds, you can take advantage
of this change which will rely on the remote builders in order
to build things natively giving a significant speed up in
container build time.

Change-Id: I962bb2357a2c458d5e72b334b4fe36b55b034864
2024-09-30 18:58:21 -04:00
Mohammed Naser
655e88b7ba Only update qemu-static container settings on x86_64
The image only support x86_64 so if you're running
this role against an arm64 node, this breaks.

Change-Id: Ied74ef9e4337346910c817ea5f3540af9ccfdf0b
2024-09-30 13:46:14 -04:00
Clark Boylan
5994ce4049 Gracefully handle use of intermediate registry in container upload role
For symmetry and ease of transition between the docker specific
jobs/roles and generic container jobs/roles it is advantageous to have
the container upload role skip pushing artifacts to the final registry
location if we are relying on the intermediate registry instead.

Update the container upload role to skip pushing to the actual registry
if the promote var is set to intermediate registry. This allows us to
avoid reshuffling all of our jobs as we migrate between the two
implementations.

Change-Id: I3cae9e03517cb0a5ce8e9369bf43fd052cac97ff
2023-05-09 16:18:42 -07:00
James E. Blair
839de7f899 Add labels to build-container-image
The "docker build" path is missing the "--label" arguments which are
present in all other paths (docker buildx in both build-container-image
and build-docker-image roles).  This means that images are being built
without expected metadata labels.

This change adds the extra arguments to match.

Change-Id: I607ae89cace87fe87b6dcfb95a1bae4fca516d3c
2023-05-01 10:13:32 -07:00
Clark Boylan
4617d0835a Use full image url in container buildx path
The container roles assume a full image url path and do not work with
shorted names like the docker roles do. The buildx path used the shorted
path when it should use the full path. Fix this.

Change-Id: Icdeee0ebb1c0d280968f425f8560cc5208ebd18d
2023-04-21 16:33:02 -07:00
Zuul
b7f983c621 Merge "Move containerfile setting in container build" 2023-04-21 21:34:25 +00:00
James E. Blair
2c6bc6d7a3 Move containerfile setting in container build
When setting the containerfile variable, we need to be inside the
zk_images loop in order to set the correct default for each image.

Change-Id: I216ffd19dd797752b2e5ca2332e651b8e6ac8a3c
2023-04-21 10:19:40 -07:00
Zuul
0354a8c52d Merge "container role docs : clarify requirements" 2023-04-13 18:05:57 +00:00
Zuul
e3804cb3c4 Merge "promote-container-image: use generic tag removal role" 2023-04-04 22:04:07 +00:00
Zuul
9e82c7a24c Merge "promote-container-image: add promote_container_image_method" 2023-04-04 22:04:02 +00:00
Zuul
35d57db9a4 Merge "build-container-image: expand docs" 2023-04-04 22:02:28 +00:00
Ian Wienand
9df7c8eb70
promote-container-image: use generic tag removal role
This uses the generic tag removal role added with
I7f2d9d00024e34451e2d20b2c2f8171ecd151943 to cleanup the promote tag
and any leaked tags.

Change-Id: I3f1b82d63874ee886048b9ccabe616a60dc09434
2023-04-04 09:56:18 +10:00
Ian Wienand
0a64d51c3d
promote-container-image: add promote_container_image_method
After recent conversations, we've come to the conclusion it will be
good to have two models of promotion

 - using tags, where gate directly uploads to the final repository and
   promote retags the image.

 - from an intermediate-registry, where upload stores the built image
   in an i-r and the promote step uploads to the final registry.

To facilitate this, we add a "promote_container_image_method" flag to
the promote roles.

The documentation is expanded to explain how all this is intended to
work together.

These roles haven't been publicised yet, but this should be a no-op as
it defaults to tags, which is the current operation.

c.f. Ia24bbd101e01ab371ceacfed006b5ff806418a97

Change-Id: I1c25f60f835b1cab983bcdd169eeffc0e250a56c
2023-04-04 09:50:17 +10:00
Ian Wienand
d7e5559e58
build-container-image: expand docs
This goes into a bit more detail on the advantages/disadvantages of
each method.

Change-Id: Ie90a52f1c0e205e9f8552156aded871b6fd30214
2023-03-30 10:38:23 +11:00
Ian Wienand
0939c2a6be
build-container-image: directly push with buildx
The multi-stage build/push was added with
I8036a9b4d4c515c20a05994741540b999e7cbcae (2020-05) which noted

  When building multi-arch it's done in parallel which can result
  in the same layer being pushed at the same time, which is bad for
  the registries.

I believe this was probably actually a bug in zuul-registry, addressed
with Ibdf1ca554756af61247d705b2ea3cf85c39c2b83 (2022-02)

  The way the registry was previously written if two concurrent
  uploads of the same blob were happening one would fail to grab the
  lock and then return early. The uploading client would then
  immediately HEAD the blob and if it did so quickly enough would get
  a short size or 404.

So this simplifies a couple of steps by returning the push to the
build step.

Change-Id: I41b8a495eb3097b358d7634b63b7eb9de6161059
2023-03-24 11:07:06 +11:00
Ian Wienand
7631526431 buildx: remove experimental flags
Tetsing this, I can't determine exactly when this might have switched.
It's not listed in.  Let's see...

[1] https://github.com/docker/cli/blob/master/experimental/README.md

Change-Id: If6c2ae98ea42c505f46563ca84a870081c4cdbe9
2023-03-23 11:14:16 -07:00
Clark Boylan
2d1c713b75 Add docker buildx multiarch support to container roleset
This adds support for multiarch container image builds when using docker
as the container command to the container roleset.

Change-Id: I48bf2e34c258e54baf013d3c04c6d4baaacde04b
2023-03-23 11:14:14 -07:00
Clark Boylan
13e44fa520 Add support for passing env vars to the container build env
This is useful if you are using docker and wish to enable buildkit for
example.

Change-Id: I69ae90d945fa3f4eabc10460da936b23c8212858
2023-03-22 14:30:35 -07:00
Ian Wienand
9baf07a104
container role docs : clarify requirements
skopeo is used from the executor in the promote process, but isn't
required to be installed on nodes with ensure-skopeo.  Clarify this
and the runtime requirements.

Change-Id: Ifaf4b788117986037b124972441b1884744374e5
2023-03-22 11:58:41 +11:00
okozachenko
0c3b87f20e Add promote-container-image role
This role uses skopeo to perform image operations.

Also update the container roles docs to add missing documentation
for the already existing upload-container-image role.  Clarify
some ambiguity about the registry and repository attributes of
the container images data structure.

Change-Id: Ib66c85daf0edacf0dd797ab34b0d629f99c7111b
Co-Authored-By: James E. Blair <jim@acmegating.com>
2023-03-21 10:17:49 -07:00
Ian Wienand
9a8b06fbc5 container-roles-jobs: Update tests to jammy nodes
These build tests are all running on Bionic for no particular reason
other than it was recent at the time; update them to Jammy.

Because Jammy appears to have a too old systemd to properly support
podman and non root continers [0] our image builds don't work by
default. We need to override the cgroup manager to cgroupfs which is
less than ideal but should be functional.

[0] https://github.com/containers/podman/issues/7650

Change-Id: I18c56aea776e9ef102fe9020db592fb04df372e2
2023-02-03 10:53:05 -08:00
Ian Wienand
f76cfbab11 linters: fix spaces between filters
This fixes a number of places where we do not have spaces between
filters.  I think that this is a reasonable rule for readability (I
also think it probably was enforced, but maybe later versions got
better at detecting it?).

These are detected by a later version of Ansible lint; this change
should have no operational change to any roles but prepares us to
update in a follow-on change.

Change-Id: I07e1a109b87adce86f483d14d7e02fcecb8313d5
2022-07-27 17:13:39 +10:00
Gonéri Le Bouder
ad56b0b4eb
build-container-image: improve the reliabilty
build-container-image may ends up pushing large files. Network is not
always as reliable as we want. This commit ensures we retry several
times in case of failure.

Change-Id: Ieaf92a8c3531e24ab5e41783b540de1df806c02f
2021-10-08 11:38:11 -04:00
Clark Boylan
4c40b92950 Prevent leaks of buildset registry credentials
Because buildset registries may be used by jobs that finish before other
jobs are finished using the buildset registry we must be careful not to
expose the registry credentials in the jobs that finish sooner.
Otherwise logs for the earlier job runs could potentially be used to
poison the registry for later jobs.

This is likely currently incomplete. Other Zuulians should look over it
carefully to ensure we're covering all the bases here.

The cases I've identified so far are:

* Setting facts that include passwords
* Reading and writing to files that include passwords (as content may be
  logged)
* Calling modules with passwords passed as arguments (the module
  invocation is logged)

I've also set no_log on zuul_return that passes up credentials because
while the logging for zuul_return is minimal today, I don't want to
count on it remaining that way.

We also use the yet to be merged secret_data attribute on zuul_return to
ensure that zuul_return itself does not expose anything unwanted.

Finally it would be great if others could check over the use of
buildset_registry variables to make sure there aren't any that got
missed. One thing I'm not sure of is whether or not when conditionals
get logged and if we need to be careful about their use too.

Temporarily remove some buildset-regitry jobs which are in a catch-22.

Change-Id: I2dea683e27f00b99a7766bf830981bf91b925265
2021-06-24 09:56:19 -07:00
Paul Belanger
e9226bbaab Fix typo with container_images siblings logic
Change-Id: I4c54ccd47677fde30fc0621005f0ef9d51df1f66
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2020-11-25 14:30:48 -05:00
Paul Belanger
777230be59 Create upload-container-image role
This new role will be used to replace our upload-docker-image role in
the future.

Change-Id: I0e2b0cca6575255520aa6d4d48a12128ab5f46cc
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2020-11-13 15:04:27 -05:00
Monty Taylor
f449ae20fb Set up siblings dirs for each build in the loop
The buildx patch unfortunately changed the logic associated with
siblings to set up siblings in a loop one time, rather than to
do a loop of "set up siblings, build, cleanup siblings". This causes
builds to fail when they're using siblings with an error about
siblings dir not having been cleaned up.

Change-Id: I3c45bfa77ec9f2609689e04044c18f066adc9741
2020-05-05 17:52:50 -05:00
Albin Vass
bee0c6ae2f ansible-lint: use matchplay instead of matchtask
For some reason matchtask doesn't match includes, matchplay does so use
that instead.

Change-Id: I040f7f3394503e92d06c05e8ff671a43b14baebc
2020-05-05 20:42:38 +02:00
Albin Vass
a7b4e5e906 Fix bare 'item' in build-container-image
Move the setting of the containerfile into the loop (thus in build.yaml)
to follow the semantics from roles/build-container-image/common.rst for
container_filename and container_images.container_filename.

Co-Authored-By: Andreas Jaeger <aj@suse.com>
Change-Id: Id39f13cc666e9f99557447073cc0d362991f3506
2020-05-05 20:32:57 +02:00
Zuul
9d7551e4ea Merge "Revert "Revert "Do not set buildset_fact if it's not present in results.json""" 2020-05-05 14:35:24 +00:00
Albin Vass
d0e2016592 Add loop var policy to ansible-lint
This adds a custom ansible-lint rule at .rules/ZuulJobsNamespaceLoopVar.py
that enforces the loop var policy described at:
https://zuul-ci.org/docs/zuul-jobs/policy.html#ansible-loops-in-roles

It also updates existing roles to follow the policy.

Change-Id: I92b2ff56a1c2702542fc07b316f1809087a4c92f
2020-04-29 17:20:59 +02:00
Albin Vass
a5f5091897 Revert "Revert "Do not set buildset_fact if it's not present in results.json""
This reverts commit b5e87e62f1ff03dd80067aa1ec297007770301ce.

Change-Id: I839c0fb9811ce68e2949402f12aab1387d1a812e
2020-04-28 21:26:16 +02:00
Albin Vass
b5e87e62f1 Revert "Do not set buildset_fact if it's not present in results.json"
This reverts commit bc0ca442a11fee5c610a71b50d8d0073d5293d2b.

Change-Id: If0885877a1c976ed430d6e3a7a462e06042874fb
2020-04-28 18:35:10 +00:00
Albin Vass
bc0ca442a1 Do not set buildset_fact if it's not present in results.json
Change-Id: I9a8790e6f3628924f796b4f3eff052b5e36eff03
2020-04-27 19:53:47 +02:00
Fabien Boucher
9d0a8f0812 Remove hardcoded use of /home/zuul
Change-Id: Ib19ddc2221f9f387542bc6c0c758b47e94e42081
2020-04-24 11:33:03 +02:00
vass
2cb9f249a8 Rename install-podman to ensure-podman for consistency
The old role will be kept and include ensure-podman for backwards-compatability.

Change-Id: I21cfdfef4a88f66f51d139abed04f70b3228a22d
2020-04-04 15:41:29 -04:00
vass
c80b593fb6 Rename install-docker to ensure-docker for consistency
The old role will be kept and include ensure-docker for backwards compatability.

Change-Id: Icb76a954b04631c405b09eb54343fd852a511b83
2020-04-04 15:41:29 -04:00
Ian Wienand
29d55cbf17 build-container-image: support sibling copy
Add support for "siblings:" to the build-container-image role.  This
is a port of change I079d823e7194e15b1b496aea0f53f70f6b563f02 from the
build-docker-image role.

Note that we found a couple of issues with this role, fixed with
I9f3b0a1f71d20cf7511f224648dd2fa51a039015.  However,
build-container-image has merged before this fix, so thess fixes now
follow this change.  To avoid squashing this into something confusing
this update incorporates those fixes.  However testing is left to the
follow-on change because the extant build-docker-image role would fail
the corrected testing.  They should be merged roughly together.

One minor thing is the removal of an additional duplicated "tags:"
entry from the variable documentation.

Change-Id: I71e2f2ab29446bd113644ce8c75378b8d3ef8ba6
2019-12-09 11:13:42 +11:00
James E. Blair
b583530f2b Add build-container-image role
Testing is added in a followup change.

Change-Id: Ia1ebf0075e9f6976310dff618cb06da928f55a7b
2019-12-03 14:22:59 -08:00