Commit Graph

123 Commits

Author SHA1 Message Date
Monty Taylor
e51e289425 Add restTokenPrivateKey
We don't have this on review-dev, so it was missed.

Change-Id: I459266ac6766c204891152c161e80f3cdfc9a295
2020-03-20 14:22:04 +11:00
Monty Taylor
2e5b95a7f3 Add flag to allow skipping docker-compose up
For our rollout, we need to be able to run this without actually
running the up.

Also, split out startup tasks so that we can run them from a
dedicated start playbook by themselves.

Change-Id: I08d994e496fbd8d5adbfa1ce344b0ae52f46535c
2020-03-19 15:51:33 -05:00
Ian Wienand
b1bfee423b nodepool-builder: Add webserver
This adds the webserver that serves the logs and generated images.

Change-Id: I230f5291e0bd928af2e00966d76c3f385b749cb6
2020-03-11 09:16:31 +11:00
Zuul
b2b0cc1c83 Merge "Add install zookeeper role; use for nodepool-builder testing" 2020-03-10 00:12:14 +00:00
Jeremy Stanley
43ed9fc297 Moving FortNebula to OpenEdge
Sister change for Ia5caff34d3fafaffc459e7572a4eef6bd94422ea and
removing earlier references to the mirror server in preparation for
building and adding the new one.

Change-Id: I7d506be85326835d5e77a0c9c461f2d457b1dfd3
2020-03-06 20:43:56 +00:00
Ian Wienand
e7f1062d51 Add install zookeeper role; use for nodepool-builder testing
This adds a simple role to install Zookeeper.

Add an option to nodepool-base to use this role to install Zookeeper.

Use this in the nodepool-builder gate testing where we are just
validating that the nodepool-builder container starts and is ready to
accept connections.  It needs a zookeeper to talk to, even though it
is not going to do anything.

Change-Id: I4ae89a51e454be4ee53ad4e04407162aaa8d9f9a
2020-03-06 14:02:52 +11:00
Zuul
fd50bb63aa Merge "letsencrypt: Register email with accounts" 2020-03-05 22:23:45 +00:00
Clark Boylan
6f9c151e79 Collect docker logs as root
When testing our system-conf configuration we don't actually add zuul to
the docker group. This means the zuul user cannot access the docker
socket. This then breaks docker container log collection. Address this
by becoming root when collecting logs.

Change-Id: Ic0232f7ef458cdd07fb0853f97f2dc22ce137c71
2020-03-05 16:14:49 +11:00
Ian Wienand
3aaf87ee6d letsencrypt: Register email with accounts
Currently we don't set a contact email with our accounts.  This is an
optional feature, but would be helpful for things like [1] where we
would be notified of certificates affected by bugs, etc.

Setup the email address in the acme.sh config which will apply with
any new accounts created.  To update all the existing hosts, we see if
the account email is added/modified in the config *and* if we have
existing account details; if so we need a manual update call.

For anyone who might be poking here, we also add a note on sharing an
account based on some broadly agreed upon discussion in IRC.

[1] https://community.letsencrypt.org/t/revoking-certain-certificates-on-march-4/114864

Change-Id: Ib4dc3e179010419a1b18f355d13b62c6cc4bc7e8
2020-03-05 12:25:56 +11:00
Zuul
7fe8a64cdc Merge "Build gerrit images with bazelisk" 2020-02-24 18:20:48 +00:00
Monty Taylor
a8e1d1496d Build gerrit images with bazelisk
We need to use bazelisk to build gerrit so that we can properly
track bazel versions in the job. Use the roles developed for
gerrit-review to do that, then simplify the dockerfile to have
it simply copy the war into the target image.

Also add polymer-bridges.

Depends-On: https://review.opendev.org/709256
Change-Id: I7c13df51d3b8c117bcc9aab9caad59687471d622
2020-02-21 17:32:01 -06:00
Zuul
5f80e934c4 Merge "Use LE certs for Apache" 2020-02-14 19:08:51 +00:00
Monty Taylor
bbe8086726 Use LE certs for Apache
We're getting LE certs for the hosts now, use them in the apache
config. Also add the redirects.

Change-Id: I67d33b4c542182a2474ac0d2416357541b1c3a47
2020-02-13 10:31:59 -06:00
Zuul
e1f75c92b2 Merge "Add Apache to Ansible for Gerrit" 2020-02-05 18:33:04 +00:00
Monty Taylor
4de5f79599 Add Apache to Ansible for Gerrit
When we run gerrit, we also need to run Apache.

Change-Id: Ia2f1494808bd29d83e041e224cb2eb5fc406a93b
2020-02-03 07:57:36 -06:00
Clark Boylan
9166780b54 Add airship CI cloud
This is a new cloud provided via citycloud that will add resources
capable of running Airship jobs. The goal is to use this as a stepping
stone to having Airship jobs run on our generic CI resources. This cloud
will provide both generic and larger resources to support this.

Change-Id: I63fd9023bc11f1382424c8906dc306cee5b3f58d
2020-01-31 14:46:02 -08:00
Ian Wienand
c3c96d3797 Add Linaro US cloud
Add the credentials for the newly provisioned us.linaro.cloud cloud

Change-Id: I0b81a8eeabec4e0b00258dc4e499c1d449b21681
2020-01-22 06:44:01 +11:00
Ian Wienand
8296bf450c Remove unused linaro credentials
As a follow-on to Ie37abb4fd3eb3342b66ade52ab65024c420d7264 remove the
linaro credentials that were related to the (now removed) linaro-cn1
cloud.

Change-Id: Ia1e8dd3732164708c2e9fd82509e350829c438ba
2020-01-21 14:13:31 +11:00
Zuul
8e75bf4416 Merge "Switch to collect-container-logs" 2020-01-16 23:13:19 +00:00
Zuul
fd1f2596b9 Merge "Update registry testing to use LE" 2020-01-15 18:56:57 +00:00
Zuul
a48be16f3e Merge "afs-client: move reduced cache to group variable" 2020-01-15 17:05:13 +00:00
Clark Boylan
6f62b38dbc Update registry testing to use LE
This was missed when converting the registry server over to LE in
production. We need to test it this way too.

Change-Id: Ic2a05ebeae6991b69c000d5269165a45a0c72d38
2020-01-14 13:50:13 -08:00
Mohammed Naser
e0c627d8a6 Switch to collect-container-logs
This change switches the post bits to use a new centralized
role to collect all container logs.

Depends-On: https://review.opendev.org/701867
Change-Id: I9e982b37518c22e6d5358f7604ebc7f56b0626e3
2020-01-09 19:47:22 -05:00
Monty Taylor
4449612d20 Plumb through secure.config contents
While we're in there - fix a misspelling.

Remove auth.restTokenPrivateKey from config file. It hasn't been
used since 2.6: https://gerrit-review.googlesource.com/c/gerrit/+/70770

Change-Id: I94405cf870d57780b86f30c2bddb573ff15c05bc
2020-01-07 17:36:10 -05:00
Monty Taylor
e42862af73 Plumb through storyboard hiera data
NOTE: We should update storyboard-dev to be driven by
letsencrypt first, otherwise we need to plumb in the
self-signed cert, which gets weird with needing to
import it for java which in this case is in the container
image, meaning we either need to bind-mount java certs in
or build it in to the image.

Change-Id: Ida9dd15ca8262925c54579660fe9c16e2b573907
2019-12-17 08:13:34 -05:00
Monty Taylor
1d37be64b4 Add service playbook and test run for prod gerrit
We need to test this against production variables too.

Change-Id: I7813787506e3b70ef0960ce85dccca4eb9ec7a3f
2019-12-17 08:13:34 -05:00
Ian Wienand
f49fc87f95 afs-client: move reduced cache to group variable
For gate testing we need the smaller AFS cache size applied to
everything that might install openafs, not just the mirror nodes.
Move the definition to the afs-client group.

Change-Id: Id27efd2f12f5ac3f351f65fa1ae513624a53df90
2019-12-16 15:34:12 +11:00
Zuul
29019411eb Merge "Run a gerrit container on review-dev01" 2019-12-15 19:00:21 +00:00
Clark Boylan
5392f8a27c Manage opendev.org cert with LE
This is the first step in managing the opendev.org cert with LE. We
modify gitea01.opendev.org only to request the cert so that if this
breaks the other 7 giteas can continue to serve opendev.org. When we are
happy with the results we can merge the followup change to update the
other 7 giteas.

Depends-On: https://review.opendev.org/694182
Change-Id: I9587b8c2896975aa0148cc3d9b37f325a0be8970
2019-11-18 12:07:10 -08:00
Zuul
8c75f8137d Merge "Remove arm64ci (3/2)" 2019-11-13 20:27:07 +00:00
James E. Blair
4f9720e76e Run a gerrit container on review-dev01
This runs gerrit in a container on review-dev01 using podman.

Remove an unused web_server.py file that we found from copying it
from puppet to ansible.

Change-Id: I399d3cf8471bc8063022b0db0ff81718b2ee2941
2019-10-29 08:29:17 +09:00
Monty Taylor
36aa77937a Add jobs to build gerrit master branch
We'll use this to test the checks plugin.

We have to add jgit as a repo because it's a submodule now.

Change-Id: Ic7e9ad0265e136a9ac6b1147998f6eb5ee398180
2019-10-20 06:35:56 +09:00
Zuul
cd402000a4 Merge "Several updates because the world is a dark place" 2019-10-19 00:58:47 +00:00
Monty Taylor
9ab25e89a9 Several updates because the world is a dark place
A few things have changed and we need to fix them in one go.

Use mirror for installing docker for buildset-registry

While, we need to make this more systemic, that's hanging off of the
mirror rework. For now, since we know all of these jobs are debian
based, just set the mirror location.

Replace use of zuul cloner with git clones

You can never be a prophet in your own hometown. This is now broken
because of the git cache rework, so just replace it.

Update libjemalloc library

python:slim is based on buster now, which has libjemalloc2 not
libjemalloc1.

Remove gerrit repo remote for submodules

A recent change to the base jobs to use prepare-workspace-git
broke the gerrit image builds by actually having the origin
remote by /dev/null as intended. This breaks submodules because
for a few of them where we don't have matching stable branches
the submodule relative path behavior is actually exactly what
we want.

Since we don't care about the remote otherwise, remove the
origin remote before doing the submodule update --init so that
the submodule will clone the refs from the zuul prepared repo.

Change-Id: Ieb5b6bc8711fe971ed3445c7c267306ac4616464
2019-10-19 07:51:29 +09:00
James E. Blair
dee6a8b330 Add token secret to intermediate registry
An upcoming change will add JWT authentication to the registry;
prepare for that by establishing a server-side secret for use
in signing the tokens.

Change-Id: Ibaa15dd0c4b0d797f01a1886186fdc021dc990fa
2019-10-08 14:16:43 -07:00
James E. Blair
b5d37bfaa2 Remove arm64ci (3/2)
This cloud no longer exists.

Change-Id: Iec9d98c7bcf3c4cd3f9853bb059e3ee2efc31e87
Depends-On: https://review.opendev.org/686761
2019-10-04 09:20:33 -07:00
Zuul
f8808d6919 Merge "Remove bazel version hack" 2019-09-19 17:22:45 +00:00
Monty Taylor
f0a3f0cb37 Remove bazel version hack
The upstream patch has landed, so we don't need this anymore.

Change-Id: I08a6705f189b2a24b737ab4f52bb7f449879fdf1
2019-09-19 14:18:41 +02:00
Zuul
823298b365 Merge "Fix files matcher and bazel for gerrit base image" 2019-09-17 00:41:59 +00:00
Monty Taylor
072fcca06f Fix files matcher and bazel for gerrit base image
Use latest bazel

It seems 0.27 is now too old. This is what happens when I go on vacation
apparently.

Add in a hack to override the bazelversion. We'll remove this once
https://gerrit-review.googlesource.com/c/gerrit/+/237495 lands and
has been merged up.

Change-Id: Ib7a6d33ce8bf8498fd5cd09b25087dc09acb8df4
2019-09-16 21:20:18 +02:00
Ian Wienand
912dff49e7 Set zuul_work_dir for tox testing
Setting this to system-config allows us to run the base tests as 3rd
party ci for projects like testinfra.

Change-Id: I2d15df154dcdc7c5da6c3326fbecec2146201164
2019-09-09 09:44:43 +10:00
Monty Taylor
56ceaf1c40 Remove the extra bazel options
We had some extra bazel options that don't seem to be necessary
anymore now that we are using upstream bazel options appropriately.

Retry the build a couple of times if it goes south, inside of the
build image. This should allow re-use of the cache the second time,
and if there is a temporary error, it should pick up and move
forward.

Change-Id: I5f304acb21fd3a4d40701fc0414ae0c424c838e5
2019-08-26 11:26:19 +02:00
Ian Wienand
0751b3d481 Convert nested bridge.o.o ARA report to static HTML
With the switch to swift logging, we need to convert the nested ARA
report to static HTML

Change-Id: I9e177915099598d5d48a31c15bd6db49e4d1c7e8
2019-08-19 10:28:57 +10:00
Ian Wienand
814e4be128 Ansible roles for backup
This introduces two new roles for managing the backup-server and hosts
that we wish to back up.

Firstly the "backup" role runs on hosts we wish to backup.  This
generates and configures a separate ssh key for running bup and
installs the appropriate cron job to run the backup daily.

The "backup-server" job runs on the backup server (or, indeed
servers).  It creates users for each backup host, accepts the remote
keys mentioned above and initalises bup.  It is then ready to receive
backups from the remote hosts.

This eliminates a fairly long-standing requirement for manual setup of
the backup server users and keys; this section is removed from the
documentation.

testinfra coverage is added.

Change-Id: I9bf74df351e056791ed817180436617048224d2c
2019-08-05 16:59:57 +10:00
Monty Taylor
2a46202b9f Build gerrit images for 2.16 and 3.0 as well
Our goal is upgrading to 3.0. To do that we need to upgrade to 2.15, then
to 2.16, then to 3.0. Build all of the images so that we can do that.

2.16 and 3.0 also use bazel, so just use one copy of the Dockerfile for
all three and let zuul check out the repos to the right versions.

Depends-On: https://review.opendev.org/673147
Depends-On: https://review.opendev.org/672320
Change-Id: I35bd278e0c70c871fa44d005c60a987d1d8e3cdc
2019-07-27 11:34:42 -04:00
Jeremy Stanley
5587c299ea Re-add gitea01 replacement to inventory
Add new IP addresses to inventory for the rebuild, but don't
reactivate it in the haproxy pools yet.

Note this switches the gitea testing to use a host called gitea99 so
that it doesn't conflict with our changes of the production hosts.

Change-Id: I9779e16cca423bcf514dd3a8d9f14e91d43f1ca3
2019-07-23 16:17:41 -07:00
Ian Wienand
814b42f616 Set openafs cache sizes for mirror/mirror-update
Set the openafs cache values to the same as the puppet set values for
openafs-client role users.

Change-Id: I5a58673cad8df2a1e8dddb592c322e751d7f2ac5
2019-07-19 12:04:26 -07:00
Ian Wienand
82c6dec4fa Disable cloud launcher cron job during CI
This takes a similar approach to the extant ansible_cron_install_cron
variable to disable the cron job for the cloud launcher when running
under CI.

If you happen to have your CI jobs when the cron job decides to fire,
you end up with a harmless but confusing failed run of the cloud
launcher (that has tried to contact real clouds) in the ARA results.

Use the "disbaled" flag to ensure the cron job doesn't run.  Using
"disabled" means we can still check that the job was installed via
testinfra however.

Convert ansible_cron_install_cron to a similar method using disable,
document the variable in the README and add a test for the run_all.sh
script in crontab too.

Change-Id: If4911a5fa4116130c39b5a9717d610867ada7eb1
2019-07-16 15:01:55 +10:00
Zuul
482abf3bf0 Merge "mirror-update: export mirroring logs" 2019-07-15 22:47:34 +00:00
James E. Blair
ee3b273876 Exclude ansible_python_interpreter from write-inventory
Zuul now includes an ansible_python_interpreter hostvar in every
host in its inventory.  It defaults to python2.  The write-inventory
role, which takes the Zuul inventory and makes an inventory for
the fake bridge server in the gate passes that through.  Because it's
in /etc/ansible/inventory.yaml, it overrides any settings which may
arrive via group vars, but this is the way we set the interpreter
for all the hosts on bridge (we do not do so in the actual inventory
file).

To correct this, tell write-inventory to strip the
ansible_python_interpreter variable when it writes out the new
inventory.  This restores the behavior to match what happens on
the real bridge host.  One instance of setting the interpreter
for the fake "trusty" host used in base platform tests is moved to
a hostvars file to match the rest of the real hosts.

Change-Id: I60f0acb64e7b90ed8af266f21f2114fd598f4a3c
2019-07-10 10:10:02 -07:00