Commit Graph

122 Commits

Author SHA1 Message Date
Monty Taylor
cc619fe589 Add review-dev01.opendev.org
Add a new review-dev server on the opendev domain with LE support
enabled.

Depends-On: https://review.opendev.org/705661
Change-Id: Ie32124cd617e9986602301f230e83bb138524fdf
2020-02-05 09:58:25 -06:00
Zuul
e038eccbc3 Merge "openafs-client: add centos8" 2020-01-23 00:59:05 +00:00
Ian Wienand
3f68936a0c openafs-client: add centos8
Add CentOS 8 support for the openafs client build

Change-Id: I8290cf1eed9ee8e4af44ac209502553944c52103
Depends-On: https://review.opendev.org/702348
2020-01-22 23:14:43 +00:00
Andreas Jaeger
9b2657527e Remove trusty testing
We do not have trusty nodes anymore, remove testing for it.

Change-Id: I1e464354b4c9e6d77a59c914d6d89f1837350b6e
2020-01-17 10:52:44 +01:00
Zuul
744a272cb2 Merge "Update python-base image upload job depends" 2020-01-17 00:11:13 +00:00
Zuul
3927db22c4 Merge "Add job dependencies to haproxy-statsd" 2020-01-17 00:11:11 +00:00
Monty Taylor
9e57b3b3b7 Update python-base image upload job depends
We've got a copy-pasta error here with upload jobs depending on
build jobs.

Change-Id: I0d3e0c359bbabc46c1d177d0369658d0b6d673f9
2020-01-17 06:50:40 +08:00
Monty Taylor
99c3fbff78 Add job dependencies to haproxy-statsd
haproxy-statsd uses opendevorg/python-base already. Add that to its
job dependencies and make sure it triggers on updates to python-base.

Update the FROM line to be fully qualified.

Change-Id: I9c8e8094f5570bf44076915610cd1be6d95ed326
2020-01-17 06:50:19 +08:00
Ian Wienand
f5b5ee9336 Add roles for a basic static server
Basic implementation of the opendev static server, described in

 https://docs.opendev.org/opendev/infra-specs/latest/specs/retire-static.html

Change-Id: Ie1b92f06b71aa6069fe831b26ba1cc272ce4562c
Story: #2006598
Task:  #37757
2020-01-16 14:10:08 +11:00
Clark Boylan
6f62b38dbc Update registry testing to use LE
This was missed when converting the registry server over to LE in
production. We need to test it this way too.

Change-Id: Ic2a05ebeae6991b69c000d5269165a45a0c72d38
2020-01-14 13:50:13 -08:00
Zuul
e68e154956 Merge "Clean up review comments" 2020-01-09 22:42:22 +00:00
Zuul
564a9d2a2a Merge "Base gitea-init on opendevorg/python-base" 2020-01-06 15:41:11 +00:00
Zuul
e6f4b1fa22 Merge "Add service playbook and test run for prod gerrit" 2019-12-18 19:46:40 +00:00
Monty Taylor
a3d91d4df3 Clean up review comments
Make image and volume list in compose file templated.

Rename the gerrit-podman directory to not be based on tool.

Make sure we run the job on changes to the playbooks.

Update the job name - it's not just review-dev anymore.

Change-Id: I0341fa95caff656a2176cc2026ec0ac8903fb24e
2019-12-17 08:13:34 -05:00
Monty Taylor
1d37be64b4 Add service playbook and test run for prod gerrit
We need to test this against production variables too.

Change-Id: I7813787506e3b70ef0960ce85dccca4eb9ec7a3f
2019-12-17 08:13:34 -05:00
Ian Wienand
4441f469ad Add arm64 mirror test
Add to the new arm64 check queue

Change-Id: I7b2f700f9326580b5424656e8fa1fd9731c8f14c
2019-12-16 10:42:41 +11:00
Zuul
29019411eb Merge "Run a gerrit container on review-dev01" 2019-12-15 19:00:21 +00:00
Ian Wienand
ce15e3bed9 mirror jobs: copy acme.sh output
We have seen failures issuing keys, but can't see the output of the
letsencrypt wrapper without capturing this logfile.  Add it.

Also, when we updated the mirror to "mirror01.openafs." (because we
have WIP for non-openafs kafs mirrors too) we didn't update the
host-vars match for the apache logs either.  Fix this.

Change-Id: I810a02d309f473e8c4aa0ce1612088aba7868c33
2019-11-27 10:46:34 +11:00
Clark Boylan
f7a305afbf Manage opendev.org with LE on all giteas
This catches up gitea02-07 with 01 managing ssl certs with LE.

Change-Id: I06228edca2204c5c57ebc5cb60b9d1308a393058
2019-11-18 12:47:08 -08:00
Clark Boylan
5392f8a27c Manage opendev.org cert with LE
This is the first step in managing the opendev.org cert with LE. We
modify gitea01.opendev.org only to request the cert so that if this
breaks the other 7 giteas can continue to serve opendev.org. When we are
happy with the results we can merge the followup change to update the
other 7 giteas.

Depends-On: https://review.opendev.org/694182
Change-Id: I9587b8c2896975aa0148cc3d9b37f325a0be8970
2019-11-18 12:07:10 -08:00
James E. Blair
4f9720e76e Run a gerrit container on review-dev01
This runs gerrit in a container on review-dev01 using podman.

Remove an unused web_server.py file that we found from copying it
from puppet to ansible.

Change-Id: I399d3cf8471bc8063022b0db0ff81718b2ee2941
2019-10-29 08:29:17 +09:00
James E. Blair
a441dddaa4 Set zuul_work_dir in gerrit master job
Set the zuul_work_dir to this project so that this job may be used
in other projects (eg gerrit itself or plugins).

Change-Id: I8662ff2a26bcff342f922c28d22225b73859929c
2019-10-24 10:40:56 -07:00
Monty Taylor
ccaf54c866 Base gitea-init on opendevorg/python-base
So that we can keep one python base image for our python things,
base jinja-init and gitea-init on python-base. Also, tie jinja-init
to python-base in the dependency graph and gitea-init to jinja-init.
This way if python-base updates, we'll rebuild our python images.

Update FROM lines to use full paths to images.

Change-Id: I554bf07fa8e458e443729cf4b8f40d7ceeaafa04
2019-10-23 17:14:02 +09:00
James E. Blair
1ce80389f1 Add system-config as a required project to gerrit-master
The build process for this job includes using content from the
system-config repo, therefore we need to make sure it is present
in case this job is used in other repos (eg, gerrit repos).

Change-Id: Id15c87b4dc1330406ab52741e1c2e2ecb62583ff
2019-10-22 13:25:35 -07:00
James E. Blair
6bd0d0258d Refactor gerrit master job
Make a "base" version of the gerrit master job with no file matcher
so that we can use it in other repositories (eg, gerrit, zuul).
Inherit from it with the original name to add the file matcher back.

Change-Id: I4e428b44dd82f8dba08b219cbf8407969c6436b1
2019-10-22 09:06:24 -07:00
Monty Taylor
61b5a11bfb Always rebuild base and builder together
When we build either, it could pick up base image changes, such as
moving from stretch to buster. Make sure any time we build one we
build the other so that they stay in sync.

Change-Id: Ia28ad4f64114c88cc02289c9318a323ceb4f143d
2019-10-22 02:59:11 +09:00
Monty Taylor
92fe8eae71 Add checks plugin to gerrit master build
Change-Id: Ifb7f4b84fb07c99d3c28b78ec55a1835821202f6
2019-10-20 06:36:02 +09:00
Monty Taylor
36aa77937a Add jobs to build gerrit master branch
We'll use this to test the checks plugin.

We have to add jgit as a repo because it's a submodule now.

Change-Id: Ic7e9ad0265e136a9ac6b1147998f6eb5ee398180
2019-10-20 06:35:56 +09:00
Monty Taylor
9ab25e89a9 Several updates because the world is a dark place
A few things have changed and we need to fix them in one go.

Use mirror for installing docker for buildset-registry

While, we need to make this more systemic, that's hanging off of the
mirror rework. For now, since we know all of these jobs are debian
based, just set the mirror location.

Replace use of zuul cloner with git clones

You can never be a prophet in your own hometown. This is now broken
because of the git cache rework, so just replace it.

Update libjemalloc library

python:slim is based on buster now, which has libjemalloc2 not
libjemalloc1.

Remove gerrit repo remote for submodules

A recent change to the base jobs to use prepare-workspace-git
broke the gerrit image builds by actually having the origin
remote by /dev/null as intended. This breaks submodules because
for a few of them where we don't have matching stable branches
the submodule relative path behavior is actually exactly what
we want.

Since we don't care about the remote otherwise, remove the
origin remote before doing the submodule update --init so that
the submodule will clone the refs from the zuul prepared repo.

Change-Id: Ieb5b6bc8711fe971ed3445c7c267306ac4616464
2019-10-19 07:51:29 +09:00
Monty Taylor
f9e7fcab51 Remove dependencies from promote
These don't really make a ton of sense now do they?

Change-Id: Ic5f515dd50872387e3447eb70af25070257f61cd
2019-10-05 09:20:38 +02:00
Monty Taylor
d9fb5b3faf Install jeepyb into the gerrit images
We need jeepyb installed because the content of the gerrit hook scripts
we install is done via jeepyb commands. Use python-builder so that we
can just install the jeepyb wheel.

Should we maybe transition these hooks into being zuul jobs?

Depends-On: https://review.opendev.org/683146/
Change-Id: I8899885b05d1e9f48b3f354ca22b360b54d455a3
2019-09-19 15:17:14 +02:00
Monty Taylor
072fcca06f Fix files matcher and bazel for gerrit base image
Use latest bazel

It seems 0.27 is now too old. This is what happens when I go on vacation
apparently.

Add in a hack to override the bazelversion. We'll remove this once
https://gerrit-review.googlesource.com/c/gerrit/+/237495 lands and
has been merged up.

Change-Id: Ib7a6d33ce8bf8498fd5cd09b25087dc09acb8df4
2019-09-16 21:20:18 +02:00
Monty Taylor
9bb1c73139 Split out bazel builder and gerrit base image
There is a bunch of duplication which needs to be redone almost never.
Split those into their own images so we can run them once and reuse them.

Change-Id: I923d4bff96dae75eb52a1c271fa52d5ae79933a0
2019-08-26 11:26:23 +02:00
Ian Wienand
f3def9b84a Run ansible jobs on bridge.yaml changes
We almost merged I7ed75d253857f86b68f67023af6897af4e1b4f50 which would
have broken production Ansible runs due to a issue with the upgraded
Ansible and listener syntaxes.  CI was picking this up, but the jobs
weren't running on this change (in this case, it was noticed in a
follow-on job that triggered the letsencrypt jobs to run).

Add this file to all ansible tests so that if we bump versions of
ansible/openstacksdk/ara etc, we run all the tests in the gate.

Change-Id: I738c4e7721bd126e8e109c5ea1f38eba9e07b22b
2019-08-22 11:41:56 +10:00
Ian Wienand
814e4be128 Ansible roles for backup
This introduces two new roles for managing the backup-server and hosts
that we wish to back up.

Firstly the "backup" role runs on hosts we wish to backup.  This
generates and configures a separate ssh key for running bup and
installs the appropriate cron job to run the backup daily.

The "backup-server" job runs on the backup server (or, indeed
servers).  It creates users for each backup host, accepts the remote
keys mentioned above and initalises bup.  It is then ready to receive
backups from the remote hosts.

This eliminates a fairly long-standing requirement for manual setup of
the backup server users and keys; this section is removed from the
documentation.

testinfra coverage is added.

Change-Id: I9bf74df351e056791ed817180436617048224d2c
2019-08-05 16:59:57 +10:00
Monty Taylor
2a46202b9f Build gerrit images for 2.16 and 3.0 as well
Our goal is upgrading to 3.0. To do that we need to upgrade to 2.15, then
to 2.16, then to 3.0. Build all of the images so that we can do that.

2.16 and 3.0 also use bazel, so just use one copy of the Dockerfile for
all three and let zuul check out the repos to the right versions.

Depends-On: https://review.opendev.org/673147
Depends-On: https://review.opendev.org/672320
Change-Id: I35bd278e0c70c871fa44d005c60a987d1d8e3cdc
2019-07-27 11:34:42 -04:00
Zuul
4b092eaed7 Merge "Build docker images of gerrit" 2019-07-25 21:58:06 +00:00
Monty Taylor
943f66e3e6 Build docker images of gerrit
To provide a stepwise upgrade path from 2.13 running directly to
2.15 in a container, make a container image containing the war we're
using currently. This should let us make a change to how we run the
war without changing the war at all, and then update the war.

Instead of trying to make a clean build for gerrit 2.13 inside of a
builder image, just have it wget the already built wars and jars we
have.

There are pieces of this that duplicate what's being done in puppet,
but in this context it's not immediately clear these are important to
do. However, it's also not clear they're a bad idea.

The gerrit 2.15 build needs a newer bazel. Looking at the CI scripts
that are used by gerrithub, we find that they use bazel 0.26.1
and nodesource v10. Use the bazel image published by google to get
a bazel builder image.

Set gerrit uid/git to 3000 in both images to match the existing
directory ownership so that bindmounting doesn't face permissions
problems.

Change-Id: I3533f01c0859ed50640dcfd98023994c5867c056
2019-07-24 04:40:28 -04:00
Jeremy Stanley
5587c299ea Re-add gitea01 replacement to inventory
Add new IP addresses to inventory for the rebuild, but don't
reactivate it in the haproxy pools yet.

Note this switches the gitea testing to use a host called gitea99 so
that it doesn't conflict with our changes of the production hosts.

Change-Id: I9779e16cca423bcf514dd3a8d9f14e91d43f1ca3
2019-07-23 16:17:41 -07:00
Ian Wienand
814b42f616 Set openafs cache sizes for mirror/mirror-update
Set the openafs cache values to the same as the puppet set values for
openafs-client role users.

Change-Id: I5a58673cad8df2a1e8dddb592c322e751d7f2ac5
2019-07-19 12:04:26 -07:00
Monty Taylor
5c6b3411b7 Run actual full project creation in gitea test
Add the full remote_puppet_git playbook that we actually use in
production so that we can test the whole kit and caboodle. For
now don't add a review.o.o server to the mix, because we aren't
testing anything about it.

Change-Id: If1112a363e96148c06f8edf1e3adeaa45fc7271c
2019-07-11 13:39:22 -07:00
James E. Blair
ee3b273876 Exclude ansible_python_interpreter from write-inventory
Zuul now includes an ansible_python_interpreter hostvar in every
host in its inventory.  It defaults to python2.  The write-inventory
role, which takes the Zuul inventory and makes an inventory for
the fake bridge server in the gate passes that through.  Because it's
in /etc/ansible/inventory.yaml, it overrides any settings which may
arrive via group vars, but this is the way we set the interpreter
for all the hosts on bridge (we do not do so in the actual inventory
file).

To correct this, tell write-inventory to strip the
ansible_python_interpreter variable when it writes out the new
inventory.  This restores the behavior to match what happens on
the real bridge host.  One instance of setting the interpreter
for the fake "trusty" host used in base platform tests is moved to
a hostvars file to match the rest of the real hosts.

Change-Id: I60f0acb64e7b90ed8af266f21f2114fd598f4a3c
2019-07-10 10:10:02 -07:00
James E. Blair
6d66d7ca34 Remove .zuul.yaml file matchers
The change at https://review.opendev.org/669752 will cause the
self-testing behavior we wanted from this, but will apply more
narrowly, so that jobs are run when their own configuration
changes.  Since this is no longer needed, remove it.

Change-Id: I50a863cab3bd7a3535fd0185d4ec9d1307b1b7d6
2019-07-08 21:48:30 +00:00
Ian Wienand
b85282c046 Move rsync mirror updates to new opendev.org mirror-update host
This move was prompted by wishing to expose the mirror update logs for
the rsync updates so that debugging problems does not require a root
user (note: not actually done in this change; will be a follow-on).

Rather than start hacking at puppet, the rsync mirror scripts make a
nice delination point for starting an Ansible-first/Bionic update.

Most magic is included in the scripts, so there is not much more to do
than copy them.  The host uses the existing kerberos and openafs roles
and copies the key material into place (to be added before merge).

Note the scripts are removed from the extant puppet so we don't have
two updates happening simultaneously.  This will also require a manual
clean to remove the cron jobs as a once-off when merging.

The other part of mirror-update is the reprepro based scripts for the
various debuntu repositories.  They are left as future work for now.

Testing is added to ensure dependencies and scripts are all in place.

Change-Id: I525ac18b55f0e11b0a541b51fa97ee5d6512bf70
2019-07-02 16:42:33 +10:00
Ian Wienand
d33105535a Separate openafs CI mirror
This is an intermediate step to having both kafs and openafs testing
in the gate; this just makes it clear which host is which.

Change-Id: I8cd006227ed47ad5f2c5eec664083477dd7ba397
2019-06-17 15:56:09 +10:00
Ian Wienand
6256d26f47 Role integration-tests : use a group match for openafs
This adds a group match for the openafs hosts.  This is so a further
role can run kafs separately.

Change-Id: I5ade7a4c34c89f79012fbcd85efcefddb9c0e810
2019-06-17 15:55:05 +10:00
Ian Wienand
b6c3c2eb68 Pin ARA on devel job to stable branch
As noted in the linked thread, we need to stay on the stable branch
until we update various bits for the 1.0 version of ARA.  This should
fix the -devel job.

Change-Id: I3b5931cc9b8d55feb66971daed1ef28621da4b59
2019-06-11 18:06:45 +10:00
James E. Blair
3199e3b225 Enable SPF checking on lists
This requires an external program and only works on Debian hosts.

Newer versions of exim (4.91) have SPF functionality built-in, but
they are not yet available to us.

Change-Id: Idfe6bfa5a404b61c8761aa1bfa2212e4b4e32be9
2019-06-07 10:34:33 -07:00
Andreas Jaeger
15a5806bce Follow opendev renames
The sandbox repos moved from openstack-dev to opendev, the
zone-opendev.org and zone-zuul-ci.org as well.

Follow the rename in this repo.

Depends-On: https://review.opendev.org/657277
Change-Id: I31097568e8791cc49c623fc751bcc575268ad148
2019-05-30 16:00:30 +02:00
James E. Blair
5faf89f566 Add haproxy-statsd to haproxy server
Build a container image with the haproxy-statsd script, and run that
along with the haproxy container.

Change-Id: I18be70d339df613bf9a72e115e80a6da876111e0
2019-05-24 15:40:28 -07:00