91 Commits

Author SHA1 Message Date
Monty Taylor
072fcca06f Fix files matcher and bazel for gerrit base image
Use latest bazel

It seems 0.27 is now too old. This is what happens when I go on vacation
apparently.

Add in a hack to override the bazelversion. We'll remove this once
https://gerrit-review.googlesource.com/c/gerrit/+/237495 lands and
has been merged up.

Change-Id: Ib7a6d33ce8bf8498fd5cd09b25087dc09acb8df4
2019-09-16 21:20:18 +02:00
Monty Taylor
9bb1c73139 Split out bazel builder and gerrit base image
There is a bunch of duplication which needs to be redone almost never.
Split those into their own images so we can run them once and reuse them.

Change-Id: I923d4bff96dae75eb52a1c271fa52d5ae79933a0
2019-08-26 11:26:23 +02:00
Ian Wienand
f3def9b84a Run ansible jobs on bridge.yaml changes
We almost merged I7ed75d253857f86b68f67023af6897af4e1b4f50 which would
have broken production Ansible runs due to a issue with the upgraded
Ansible and listener syntaxes.  CI was picking this up, but the jobs
weren't running on this change (in this case, it was noticed in a
follow-on job that triggered the letsencrypt jobs to run).

Add this file to all ansible tests so that if we bump versions of
ansible/openstacksdk/ara etc, we run all the tests in the gate.

Change-Id: I738c4e7721bd126e8e109c5ea1f38eba9e07b22b
2019-08-22 11:41:56 +10:00
Ian Wienand
814e4be128 Ansible roles for backup
This introduces two new roles for managing the backup-server and hosts
that we wish to back up.

Firstly the "backup" role runs on hosts we wish to backup.  This
generates and configures a separate ssh key for running bup and
installs the appropriate cron job to run the backup daily.

The "backup-server" job runs on the backup server (or, indeed
servers).  It creates users for each backup host, accepts the remote
keys mentioned above and initalises bup.  It is then ready to receive
backups from the remote hosts.

This eliminates a fairly long-standing requirement for manual setup of
the backup server users and keys; this section is removed from the
documentation.

testinfra coverage is added.

Change-Id: I9bf74df351e056791ed817180436617048224d2c
2019-08-05 16:59:57 +10:00
Monty Taylor
2a46202b9f Build gerrit images for 2.16 and 3.0 as well
Our goal is upgrading to 3.0. To do that we need to upgrade to 2.15, then
to 2.16, then to 3.0. Build all of the images so that we can do that.

2.16 and 3.0 also use bazel, so just use one copy of the Dockerfile for
all three and let zuul check out the repos to the right versions.

Depends-On: https://review.opendev.org/673147
Depends-On: https://review.opendev.org/672320
Change-Id: I35bd278e0c70c871fa44d005c60a987d1d8e3cdc
2019-07-27 11:34:42 -04:00
Zuul
4b092eaed7 Merge "Build docker images of gerrit" 2019-07-25 21:58:06 +00:00
Monty Taylor
943f66e3e6 Build docker images of gerrit
To provide a stepwise upgrade path from 2.13 running directly to
2.15 in a container, make a container image containing the war we're
using currently. This should let us make a change to how we run the
war without changing the war at all, and then update the war.

Instead of trying to make a clean build for gerrit 2.13 inside of a
builder image, just have it wget the already built wars and jars we
have.

There are pieces of this that duplicate what's being done in puppet,
but in this context it's not immediately clear these are important to
do. However, it's also not clear they're a bad idea.

The gerrit 2.15 build needs a newer bazel. Looking at the CI scripts
that are used by gerrithub, we find that they use bazel 0.26.1
and nodesource v10. Use the bazel image published by google to get
a bazel builder image.

Set gerrit uid/git to 3000 in both images to match the existing
directory ownership so that bindmounting doesn't face permissions
problems.

Change-Id: I3533f01c0859ed50640dcfd98023994c5867c056
2019-07-24 04:40:28 -04:00
Jeremy Stanley
5587c299ea Re-add gitea01 replacement to inventory
Add new IP addresses to inventory for the rebuild, but don't
reactivate it in the haproxy pools yet.

Note this switches the gitea testing to use a host called gitea99 so
that it doesn't conflict with our changes of the production hosts.

Change-Id: I9779e16cca423bcf514dd3a8d9f14e91d43f1ca3
2019-07-23 16:17:41 -07:00
Ian Wienand
814b42f616 Set openafs cache sizes for mirror/mirror-update
Set the openafs cache values to the same as the puppet set values for
openafs-client role users.

Change-Id: I5a58673cad8df2a1e8dddb592c322e751d7f2ac5
2019-07-19 12:04:26 -07:00
Monty Taylor
5c6b3411b7 Run actual full project creation in gitea test
Add the full remote_puppet_git playbook that we actually use in
production so that we can test the whole kit and caboodle. For
now don't add a review.o.o server to the mix, because we aren't
testing anything about it.

Change-Id: If1112a363e96148c06f8edf1e3adeaa45fc7271c
2019-07-11 13:39:22 -07:00
James E. Blair
ee3b273876 Exclude ansible_python_interpreter from write-inventory
Zuul now includes an ansible_python_interpreter hostvar in every
host in its inventory.  It defaults to python2.  The write-inventory
role, which takes the Zuul inventory and makes an inventory for
the fake bridge server in the gate passes that through.  Because it's
in /etc/ansible/inventory.yaml, it overrides any settings which may
arrive via group vars, but this is the way we set the interpreter
for all the hosts on bridge (we do not do so in the actual inventory
file).

To correct this, tell write-inventory to strip the
ansible_python_interpreter variable when it writes out the new
inventory.  This restores the behavior to match what happens on
the real bridge host.  One instance of setting the interpreter
for the fake "trusty" host used in base platform tests is moved to
a hostvars file to match the rest of the real hosts.

Change-Id: I60f0acb64e7b90ed8af266f21f2114fd598f4a3c
2019-07-10 10:10:02 -07:00
James E. Blair
6d66d7ca34 Remove .zuul.yaml file matchers
The change at https://review.opendev.org/669752 will cause the
self-testing behavior we wanted from this, but will apply more
narrowly, so that jobs are run when their own configuration
changes.  Since this is no longer needed, remove it.

Change-Id: I50a863cab3bd7a3535fd0185d4ec9d1307b1b7d6
2019-07-08 21:48:30 +00:00
Ian Wienand
b85282c046 Move rsync mirror updates to new opendev.org mirror-update host
This move was prompted by wishing to expose the mirror update logs for
the rsync updates so that debugging problems does not require a root
user (note: not actually done in this change; will be a follow-on).

Rather than start hacking at puppet, the rsync mirror scripts make a
nice delination point for starting an Ansible-first/Bionic update.

Most magic is included in the scripts, so there is not much more to do
than copy them.  The host uses the existing kerberos and openafs roles
and copies the key material into place (to be added before merge).

Note the scripts are removed from the extant puppet so we don't have
two updates happening simultaneously.  This will also require a manual
clean to remove the cron jobs as a once-off when merging.

The other part of mirror-update is the reprepro based scripts for the
various debuntu repositories.  They are left as future work for now.

Testing is added to ensure dependencies and scripts are all in place.

Change-Id: I525ac18b55f0e11b0a541b51fa97ee5d6512bf70
2019-07-02 16:42:33 +10:00
Ian Wienand
d33105535a Separate openafs CI mirror
This is an intermediate step to having both kafs and openafs testing
in the gate; this just makes it clear which host is which.

Change-Id: I8cd006227ed47ad5f2c5eec664083477dd7ba397
2019-06-17 15:56:09 +10:00
Ian Wienand
6256d26f47 Role integration-tests : use a group match for openafs
This adds a group match for the openafs hosts.  This is so a further
role can run kafs separately.

Change-Id: I5ade7a4c34c89f79012fbcd85efcefddb9c0e810
2019-06-17 15:55:05 +10:00
Ian Wienand
b6c3c2eb68 Pin ARA on devel job to stable branch
As noted in the linked thread, we need to stay on the stable branch
until we update various bits for the 1.0 version of ARA.  This should
fix the -devel job.

Change-Id: I3b5931cc9b8d55feb66971daed1ef28621da4b59
2019-06-11 18:06:45 +10:00
James E. Blair
3199e3b225 Enable SPF checking on lists
This requires an external program and only works on Debian hosts.

Newer versions of exim (4.91) have SPF functionality built-in, but
they are not yet available to us.

Change-Id: Idfe6bfa5a404b61c8761aa1bfa2212e4b4e32be9
2019-06-07 10:34:33 -07:00
Andreas Jaeger
15a5806bce Follow opendev renames
The sandbox repos moved from openstack-dev to opendev, the
zone-opendev.org and zone-zuul-ci.org as well.

Follow the rename in this repo.

Depends-On: https://review.opendev.org/657277
Change-Id: I31097568e8791cc49c623fc751bcc575268ad148
2019-05-30 16:00:30 +02:00
James E. Blair
5faf89f566 Add haproxy-statsd to haproxy server
Build a container image with the haproxy-statsd script, and run that
along with the haproxy container.

Change-Id: I18be70d339df613bf9a72e115e80a6da876111e0
2019-05-24 15:40:28 -07:00
James E. Blair
a92ac59e15 Fix new mirror system errors
Fix the reported stat name for the mirror playbook.

Run the mirror job in gate.

Set follow=false so that we're telling Ansible to set the perms
on the link rather than the target (which is the default).

Change-Id: Id594cf3f7ab1dacae423cd2b7e158a701d086af6
2019-05-24 09:42:38 -07:00
Zuul
54c72ab7b9 Merge "Create opendev mirrors" 2019-05-21 23:01:28 +00:00
Ian Wienand
670107045a Create opendev mirrors
This impelements mirrors to live in the opendev.org namespace.  The
implementation is Ansible native for deployment on a Bionic node.

The hostname prefix remains the same (mirrorXX.region.provider.) but
the groups.yaml splits the opendev.org mirrors into a separate group.
The matches in the puppet group are also updated so to not run puppet
on the hosts.

The kerberos and openafs client parts do not need any updating and
works on the Bionic host.

The hosts are setup to provision certificates for themselves from
letsencrypt.  Note we've added a new handler for mirror nodes to use
that restarts apache on certificate issue/renewal.

The new "mirror" role is a port of the existing puppet mirror.pp.  It
installs apache, sets up some modules, makes some symlinks, sets up a
cleanup cron job and installs the apache vhost configuration.

The vhost configuration is also ported from the extant puppet.  It is
simplified somewhat; but the biggest change is that we have extracted
the main port 80 configuration into a macro which is applied to both
port 80 and 443; i.e. the host will have SSL support.  The other ports
are left alone for now, but can be updated in due course.

Thus we should be able to CNAME the existing mirrors to new nodes, and
any existing http access can continue.  We can update our mirror setup
scripts to point to https resources as appropriate.

Change-Id: Iec576d631dd5b02f6b9fb445ee600be060f9cf1e
2019-05-21 11:08:25 +10:00
Zuul
60f47bf05e Merge "Add testinfra master to -devel job" 2019-05-20 22:43:56 +00:00
Zuul
2c5847dad9 Merge "Split the base playbook into services" 2019-05-20 10:04:40 +00:00
James E. Blair
8ad300927e Split the base playbook into services
This is a first step toward making smaller playbooks which can be
run by Zuul in CD.

Zuul should be able to handle missing projects now, so remove it
from the puppet_git playbook and into puppet.

Make the base playbook be merely the base roles.

Make service playbooks for each service.

Remove the run-docker job because it's covered by service jobs.

Stop testing that puppet is installed in testinfra. It's accidentally
working due to the selection of non-puppeted hosts only being on
bionic nodes and not installing puppet on bionic. Instead, we can now
rely on actually *running* puppet when it's important, such as in the
eavesdrop job. Also remove the installation of puppet on the nodes in
the base job, since it's only useful to test that a synthetic test
of installing puppet on nodes we don't use works.

Don't run remote_puppet_git on gitea for now - it's too slow. A
followup patch will rework gitea project creation to not take hours.

Change-Id: Ibb78341c2c6be28005cea73542e829d8f7cfab08
2019-05-19 07:31:00 -05:00
Ian Wienand
ee4448b162 Remove puppet 3 beaker jobs
These can be removed as we don't wish to gate on puppet 3 any more.

Change-Id: I027af025ef1bdae6cd321471d2ac383711d76dea
2019-05-17 13:20:54 +10:00
Ian Wienand
829d7ef672 Remove legacy-puppet-syntax-3
We no longer need to gate against puppet 3 syntax

Change-Id: I2518eb1d85d887a98425f395b816e9c92a53282a
Needed-By: https://review.opendev.org/659696
2019-05-17 12:54:23 +10:00
Ian Wienand
d5b321b074 Handle moved puppet repos
As per [1], it seems puppet has "cleaned up" most of the packages we
are using to install.

Install the puppet-agent packages directly as puppet's archive location
is not a valid repo. With puppet 4 at least these packages should bundle
everything we need including ruby.

[1] https://groups.google.com/forum/#!msg/puppet-users/cCsGWKunBe4/OdG0T7LeDAAJ

Depends-On: https://review.opendev.org/659384
Depends-On: https://review.opendev.org/659395
Change-Id: Ie9e2b79b42f397bddd960ccdc303b536155ce123
2019-05-15 16:03:07 -07:00
Clark Boylan
6e61cbff2e Stop ansipuppeting the old cgit farm
We have replaced the cgit farm with a gitea farm. Stop managing the cgit
farm. This removes testing for centos7 as these were our only centos7
nodes.

Depends-On: https://review.opendev.org/654549
Change-Id: Ia48ff10cb88d51f609e8b28de176c72f7a9ee24f
2019-04-22 15:50:08 +00:00
OpenDev Sysadmins
1ee61397a3 OpenDev Migration Patch
This commit was bulk generated and pushed by the OpenDev sysadmins
as a part of the Git hosting and code review systems migration
detailed in these mailing list posts:

http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003603.html
http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004920.html

Attempts have been made to correct repository namespaces and
hostnames based on simple pattern matching, but it's possible some
were updated incorrectly or missed entirely. Please reach out to us
via the contact information listed at https://opendev.org/ with any
questions you may have.
2019-04-19 19:26:05 +00:00
Ian Wienand
afd907c16d letsencrypt support
This change contains the roles and testing for deploying certificates
on hosts using letsencrypt with domain authentication.

From a top level, the process is implemented in the roles as follows:

1) letsencrypt-acme-sh-install

   This role installs the acme.sh tool on hosts in the letsencrypt
   group, along with a small custom driver script to help parse output
   that is used by later roles.

2) letsencrypt-request-certs

   This role runs on each host, and reads a host variable describing
   the certificates required.  It uses the acme.sh tool (via the
   driver) to request the certificates from letsencrypt.  It populates
   a global Ansible variable with the authentication TXT records
   required.

   If the certificate exists on the host and is not within the renewal
   period, it should do nothing.

3) letsencrypt-install-txt-record

   This role runs on the adns server.  It installs the TXT records
   generated in step 2 to the acme.opendev.org domain and then
   refreshes the server.  Hosts wanting certificates will have
   pre-provisioned CNAME records for _acme-challenge.host.opendev.org
   pointing to acme.opendev.org.

4) letsencrypt-create-certs

   This role runs on each host, reading the same variable as in step
   2.  However this time the acme.sh tool is run to authenticate and
   create the certificates, which should now work correctly via the
   TXT records from step 3.  After this, the host will have the
   full certificate material.

Testing is added via testinfra.  For testing purposes requests are
made to the staging letsencrypt servers and a self-signed certificate
is provisioned in step 4 (as the authentication is not available
during CI).  We test that the DNS TXT records are created locally on
the CI adns server, however.

Related-Spec: https://review.openstack.org/587283

Change-Id: I1f66da614751a29cc565b37cdc9ff34d70fdfd3f
2019-04-02 15:31:41 +11:00
Ian Wienand
66ceb321a6 master-nameserver: Add unmanaged domains; add acme.opendev.org
This adds the concept of an unmanaged domain; for unmanaged domains we
will write out the zone file only if it doesn't already exist.

acme.opendev.org is added as an unmanaged domain.  It will be managed
by other ansible roles which add TXT records for ACME authentication.
The initial template comes from the dependent change, and this ensures
the bind configuration is always valid.

For flexibility and testing purposes, we allow passing an extra
refspec and version to the git checkout.  This is one way to pull in
changes for speculative CI runs (I looked into having the hosts under
test checkout from Zuul; but by the time we're 3-ansible call's deep
on the DNS hosts-under-test it's a real pain.  For the amount of times
we update this, it's easier to just allow a speculative change that
can take a gerrit URL; for an example see [1])

[1] https://review.openstack.org/#/c/641155/10/playbooks/group_vars/dns.yaml

Testing is enhanced to check for zone files and correct configuration
stanzas.

Depends-On: https://review.openstack.org/641154
Depends-On: https://review.openstack.org/641168
Change-Id: I9ef5cfc850c3458c63aff46cfaa0d49a5d194e87
2019-03-27 14:22:59 +11:00
Monty Taylor
9adc5ce8fe Split python-base into its own Dockerfile
There's no real need to tie these together into a multi-stage
Dockerfile as they don't really share anything. Split them.

Change-Id: Ifd7ccadcd8048eeb57797d60356aec2f9f0d2c80
Depends-On: https://review.openstack.org/641805
2019-03-08 15:49:49 -08:00
James E. Blair
a5f3c1cffe Add a soft dependency between gitea build and run
When we run the run-gitea job, make sure that it runs *after*
the gitea image build job, if the build job ran at all.  If
it didn't, we don't need to worry about it.

This has to happen in the project pipeline config because the
job dependency is different in check and gate.

Change-Id: I7cb069a6cd40a4ae8d5cbcdf23e7686b301493a1
2019-03-08 15:47:50 -08:00
James E. Blair
7a94bd060e Use the opendev docker build jobs
These now only run a registry if there isn't already one running
in the buildset, so we can use them here in that configuration.
They also take care of the provides/requires for us, so we can
remove those.

Change-Id: I3b6d825df8d658adcae51ed20a5335c612b7d9f2
Depends-On: https://review.openstack.org/642151
2019-03-08 15:47:49 -08:00
Monty Taylor
683b73b4fb Send readme parameter when creating projects
The web page from which this call is taken has a readme selector
and a default value of "Default" that we're not sending in this
request. Send it to avoid gitea not being able to find the empty
readme.

Also, add gitea-git-repos to the files section of system-config-run-gitea
so that we actually test it.

Change-Id: Ieec94aadb63fa097f10a3f325dd105b30e610dd9
2019-03-07 22:17:53 +00:00
Zuul
c5611561ea Merge "Test gitea project creation playbook" 2019-03-06 21:17:35 +00:00
James E. Blair
9ff29b108d Test gitea project creation playbook
Add an option to run a playbook (in the fake bridge context) after
running the base playbook.  Use this to run a new playbook which
exercises gitea project creation after bootstrapping the gitea
service.

Disable ansible-lint 304 because it erroneously thinks shell and
command are the same thing.

Change-Id: I0394b614771bc62b9fe23d811defd7767b3d10db
2019-03-06 18:42:39 +00:00
James E. Blair
d242dabdc6 Use a buildset registry in the container jobs
Now that nodepool and zuul support provider affinity for paused
jobs, we should be able to have a buildset registry run for
every change and configure the docker image based jobs to use it.

Change-Id: Iddf12475c8dfce2db2d79538ace58a49211eab97
2019-03-05 16:33:48 -08:00
Ian Wienand
318e66938f Add testinfra master to -devel job
Use testinfra master for the -devel job testing.  This should be
installed as a sibling project into the tox venv automatically.

Change-Id: Id916ea2c7377291c12da0797d8c731528793ddc6
Depends-On: https://review.openstack.org/640229
2019-03-01 11:30:00 +11:00
James E. Blair
287eecd9d2 Run zuul-preview
Change-Id: Ib72e2bd29d1061822e0c16c201445115a5e5c58f
2019-02-25 13:14:51 -08:00
James E. Blair
4b031f9f24 Run an haproxy load balancer for gitea
This runs an haproxy which is strikingly similar to the one we
currently run for git.openstack.org, but it is run in a docker
container.

Change-Id: I647ae8c02eb2cd4f3db2b203d61a181f7eb632d2
2019-02-22 12:54:04 -08:00
James E. Blair
67cda2c7df Deploy gitea with docker-compose
This deploys a shared-nothing gitea server using docker-compose.
It includes a mariadb server.

Change-Id: I58aff016c7108c69dfc5f2ebd46667c4117ba5da
2019-02-18 08:46:40 -08:00
James E. Blair
175a337e01 Handle registry role running under py3
Also, correct the host_vars filename.  Again.
Also, make sure we run the test on changes to the host_vars filename.

Change-Id: I95fb61531bae677f5c68f4e56ed718da6c507eb9
2019-02-08 09:13:06 -08:00
James E. Blair
12709a1c8b Run a docker registry for CI
Change-Id: If9669bb3286e25bb16ab09373e823b914b645f26
2019-02-01 10:12:51 -08:00
James E. Blair
8062f4c1ec Grab container logs at the end of run-base
So that we automatically get container logs for future jobs
which use containers.

Change-Id: I329c67eefb8c6a2ff9a8ce8ef69cc844cef6012a
2019-02-01 10:12:39 -08:00
James E. Blair
22ad414a86 Use stage-output role in system-config-run
This simplifies log collection.

Change-Id: I754637115f8c7469efbc1856e88bbcb6fb83b4ce
Depends-On: https://review.openstack.org/634293
2019-01-31 11:03:25 -08:00
James E. Blair
d145e86f71 Switch to zuul-jobs docker build jobs
There are upstream jobs in zuul-jobs with the docker build playbooks,
so use them.  The system-config jobs are kept so that we don't have
to duplicate the secret stanza.

Change-Id: Iceee55a3d0e8b243549fa988f134b1ea9bb6dac5
2019-01-23 13:44:04 -08:00
Zuul
545921ba7b Merge "Add python-builder docker image" 2019-01-22 18:19:04 +00:00
James E. Blair
208c9dbde7 Update dockerhub credential
We've made separate dockerhub users so we don't have to share
credentials across trust boundaries, nor do we need to define the
jobs which use them centrally.

There is a new opendevzuul user which has access to opendevorg,
use that.

Change-Id: Ia22188255f6e5b0f2ba0cefe20694bdec38c1444
2019-01-21 14:55:32 -08:00