We have to be careful about avoiding outer loop loop_var conflicts in
ansible. Because the zuul-jobs roles are meant to be reconsumed
elsewhere we should not use 'item' loopvars and instead set them to
something a bit more unique.
We use a zj_ prefix to try and be unique to this repo and document this
convention.
Change-Id: I20b9327a914890e9eafcb2b36dc8c23fb472bc8f
The "Start the buildset registry" task, which uses the
docker_container Ansible module, was failing with:
"No module named 'requests'"
This is because the earlier package install task was installing
the packages into the wrong python version because the
ansible_python_interpreter value was 'auto' and thus never
included the 'python3' value.
Change-Id: I887fdf8d000c8b916fdab281c531b7c98bdd5ae2
If we want to run a second registry on a different port, we'll need
a different name for the docker container.
Change-Id: I887d9015c0d21e6d7f95379e6fa7fb4211e58d3d
This will allow us to run the role twice on the same host. This
will be useful when we test changes to this role.
Change-Id: I97baeb3172298648bcfef26c5be635ad4be036f0
The proxy functionality is no longer needed so it is removed.
Change-Id: I29ff75d331b433ea4ad3b66ed723eee14a90b404
Depends-On: https://review.opendev.org/689829
With the arrival of ansible-lint 4, Jinja2 variable expansions must
include spaces before and after the variable name inside the
brackets.
Adjust the new violations accordingly and remove the rule
206 exclusion.
Change-Id: Ib3ff7b0233a5d5cf99772f9c2adc81861cf34ffa
Complete a lingering TODO in run-buildset-registry to use a less
guessable password for the intermediate registry service. Timestamps
(even with microsecond precision and even running through a hash
algorithm) are inherently guessable. Someone watching a console
stream of the job could probably narrow down the time that task ran
to at least second precision, which then requires at most a million
guesses to be able to pollute or otherwise compromise the content
jobs are relying on. With reasonable network access and a typical
personal computer this is a rather small work factor to overcome.
Luckily, the Ansible community maintains an in-tree lookup plugin[*]
for strong password generation. Its default is 20 mixed-case
letters, numbers and punctuation, so the same length as the
truncated hash we're replacing.
[*] https://docs.ansible.com/ansible/latest/plugins/lookup/password.html
Depends-On: https://review.opendev.org/662870
Change-Id: I66e60f767328cc3af540ec4b755121da989b5e56
Skopeo has problems with ipv6 address literals just like docker as they
use the same underlying checks for url validity. However, we think we
can fix that by using a port forward from the executor to the buildset
registry so that skopeo can connect via ipv4
Go back to aliases the registries on test nodes via /etc/hosts.
Change-Id: I5f9316ffe84de06cb2fb2b65a7e1c31d9f8b0e35
Co-Authored-By: James E. Blair <jeblair@redhat.com>
The approach of having the proxy serve the local data as well as
the remote wasn't working -- it seems that the proxy would always
check upstream and prefer that data even if it had been pushed
locally.
To correct this, separate the data stores of the two registries,
and add both of them to the registry_mirror setting for the
docker daemon. Now we will pull from our buildset registry first,
and fall back on the proxy to talk to upstream if an image is not
found locally.
The proxy is still required in order to mask out the username and
password which dockerd will otherwise use when talking to upstream.
Change-Id: Iab11954a4b5431d3b1a4d4753f519b6b71f64094
This configuration is intended to mask the creds supplied by the
local docker configuration when the registry is used as a
pull-through proxy to upstream. But we need to set the creds
to the empty string, rather than null, which ends up as the string
"None" in the docker config.
Change-Id: Ic36a5c764dec8df4fa460f6735ff5668a1dd3b50
The docker registry daemon can either act as a private registry,
or as a pull-through proxy, but not both. Yet we need to be able
to serve private (speculative buildset) images as well as plain
upstream images. Our registry is used as a mirror and requires
authentication, therefore docker's normal behavior of falling back
on docker.io won't work because it will attempt to use our
credentials.
However, the registry daemon stores all of its state in the
filesystem, therefore we can run two instances of the registry
service, both pointing at the same data store. The first acts
as a pull-through proxy and will serve whatever files are already
in the local storage, or will fetch them from docker.io. The second
can be used to upload images into the local storage.
To make a long story short, whenever we push into the buildset
registry, we will use the second endpoint. Whenever the docker
daemon pulls from the buildset registry, it will use the first.
Change-Id: I296029068b5ef28ee56543741fe8c8deeefb5dfa
Docker has a curious behavior with respect to the registry-mirror
setting: it will only use the credentials for dockerhub itself.
Therefore, configure the buildset registry to be a pull-through
cache for dockerhub, but with no credentials, so that we can set
buildset registry credentials under the dockerhub url without
their actually being used for dockerhub.
Change-Id: I5daf93868d408d71cf2766fbe1bb17f2cfa8a3c7