14 Commits

Author SHA1 Message Date
Clark Boylan
4ed66807a0 Use unique loop vars to avoid conflicts
We have to be careful about avoiding outer loop loop_var conflicts in
ansible. Because the zuul-jobs roles are meant to be reconsumed
elsewhere we should not use 'item' loopvars and instead set them to
something a bit more unique.

We use a zj_ prefix to try and be unique to this repo and document this
convention.

Change-Id: I20b9327a914890e9eafcb2b36dc8c23fb472bc8f
2020-02-04 12:23:36 -08:00
James E. Blair
d48667a3c5 use-buildset-registry: Add podman support
Change-Id: I41718073962c8e7eb3d8810276e550fb84bd6e99
2019-12-05 11:44:15 -08:00
David Shrewsbury
ce67cef122 run-buildset-registry: Fix python version check
The "Start the buildset registry" task, which uses the
docker_container Ansible module, was failing with:

    "No module named 'requests'"

This is because the earlier package install task was installing
the packages into the wrong python version because the
ansible_python_interpreter value was 'auto' and thus never
included the 'python3' value.

Change-Id: I887fdf8d000c8b916fdab281c531b7c98bdd5ae2
2019-10-31 16:17:27 -04:00
James E. Blair
dba2c101fd Adjust buildset registry container name
If we want to run a second registry on a different port, we'll need
a different name for the docker container.

Change-Id: I887d9015c0d21e6d7f95379e6fa7fb4211e58d3d
2019-10-28 15:42:29 -07:00
James E. Blair
a321a8270d Make the buildset registry port configurable
This will allow us to run the role twice on the same host.  This
will be useful when we test changes to this role.

Change-Id: I97baeb3172298648bcfef26c5be635ad4be036f0
2019-10-28 15:42:29 -07:00
James E. Blair
8766890012 Use zuul-registry as buildset registry
The proxy functionality is no longer needed so it is removed.

Change-Id: I29ff75d331b433ea4ad3b66ed723eee14a90b404
Depends-On: https://review.opendev.org/689829
2019-10-28 15:42:26 -07:00
Jeremy Stanley
2711d10186 Be consistent about spaces before and after vars
With the arrival of ansible-lint 4, Jinja2 variable expansions must
include spaces before and after the variable name inside the
brackets.

Adjust the new violations accordingly and remove the rule
206 exclusion.

Change-Id: Ib3ff7b0233a5d5cf99772f9c2adc81861cf34ffa
2019-08-07 20:17:41 +01:00
Jeremy Stanley
3f01dc11f7 Use password lookup for run-buildset-registry role
Complete a lingering TODO in run-buildset-registry to use a less
guessable password for the intermediate registry service. Timestamps
(even with microsecond precision and even running through a hash
algorithm) are inherently guessable. Someone watching a console
stream of the job could probably narrow down the time that task ran
to at least second precision, which then requires at most a million
guesses to be able to pollute or otherwise compromise the content
jobs are relying on. With reasonable network access and a typical
personal computer this is a rather small work factor to overcome.

Luckily, the Ansible community maintains an in-tree lookup plugin[*]
for strong password generation. Its default is 20 mixed-case
letters, numbers and punctuation, so the same length as the
truncated hash we're replacing.

[*] https://docs.ansible.com/ansible/latest/plugins/lookup/password.html

Depends-On: https://review.opendev.org/662870
Change-Id: I66e60f767328cc3af540ec4b755121da989b5e56
2019-06-04 18:43:09 +00:00
Clark Boylan
23826e05a6 Only alias docker registries on test nodes
Skopeo has problems with ipv6 address literals just like docker as they
use the same underlying checks for url validity. However, we think we
can fix that by using a port forward from the executor to the buildset
registry so that skopeo can connect via ipv4

Go back to aliases the registries on test nodes via /etc/hosts.

Change-Id: I5f9316ffe84de06cb2fb2b65a7e1c31d9f8b0e35
Co-Authored-By: James E. Blair <jeblair@redhat.com>
2019-04-22 10:37:33 -07:00
James E. Blair
9c0d25f349 Fix buildset registry
The approach of having the proxy serve the local data as well as
the remote wasn't working -- it seems that the proxy would always
check upstream and prefer that data even if it had been pushed
locally.

To correct this, separate the data stores of the two registries,
and add both of them to the registry_mirror setting for the
docker daemon.  Now we will pull from our buildset registry first,
and fall back on the proxy to talk to upstream if an image is not
found locally.

The proxy is still required in order to mask out the username and
password which dockerd will otherwise use when talking to upstream.

Change-Id: Iab11954a4b5431d3b1a4d4753f519b6b71f64094
2019-03-01 15:52:01 -08:00
James E. Blair
8998838d7a run-buildset-container: fix username/password for proxy registry
This configuration is intended to mask the creds supplied by the
local docker configuration when the registry is used as a
pull-through proxy to upstream.  But we need to set the creds
to the empty string, rather than null, which ends up as the string
"None" in the docker config.

Change-Id: Ic36a5c764dec8df4fa460f6735ff5668a1dd3b50
2019-02-22 13:22:19 -08:00
James E. Blair
e7a0f0da8b run-buildset-registry: run a dual registry
The docker registry daemon can either act as a private registry,
or as a pull-through proxy, but not both.  Yet we need to be able
to serve private (speculative buildset) images as well as plain
upstream images.  Our registry is used as a mirror and requires
authentication, therefore docker's normal behavior of falling back
on docker.io won't work because it will attempt to use our
credentials.

However, the registry daemon stores all of its state in the
filesystem, therefore we can run two instances of the registry
service, both pointing at the same data store.  The first acts
as a pull-through proxy and will serve whatever files are already
in the local storage, or will fetch them from docker.io.  The second
can be used to upload images into the local storage.

To make a long story short, whenever we push into the buildset
registry, we will use the second endpoint.  Whenever the docker
daemon pulls from the buildset registry, it will use the first.

Change-Id: I296029068b5ef28ee56543741fe8c8deeefb5dfa
2019-02-21 13:49:49 -08:00
James E. Blair
c8c439e0d8 use-buildset-registry: configure as a pull-through proxy
Docker has a curious behavior with respect to the registry-mirror
setting: it will only use the credentials for dockerhub itself.
Therefore, configure the buildset registry to be a pull-through
cache for dockerhub, but with no credentials, so that we can set
buildset registry credentials under the dockerhub url without
their actually being used for dockerhub.

Change-Id: I5daf93868d408d71cf2766fbe1bb17f2cfa8a3c7
2019-02-20 17:16:38 -08:00
James E. Blair
2292ce9aed Add a role to run a buildset registry
Part of a system to interact with an intermediate registry.

Change-Id: I2f4662cc587f9379e9ba3b7b705c85793a41864e
2019-02-01 13:25:11 -08:00