15161 Commits

Author SHA1 Message Date
Jeremy Stanley
df99568f10 Replace wiki-dev02 with wiki-dev03
In order to confirm configuration management is working cleanly for
wiki-dev.openstack.org deployments, a new wiki-dev03 has been built
and the old wiki-dev02 deleted. These are not production hosts so
this change can be merged at any time. DNS has also been updated for
them accordingly.

Change-Id: I61ae138b10d51caef2cdd26ca8adaf9d59728ac8
2019-08-08 17:02:05 +00:00
Ian Wienand
78dc3e6ffd Add review-dev as a new backup client
Opt in review-dev to be a client for the new backup server

Change-Id: Ie24855a0df9f8d8d83588ae2f7221415a6535fd5
2019-08-08 13:55:33 +10:00
Zuul
23f310626b Merge "Add vexxhost backup server" 2019-08-07 00:38:48 +00:00
Clark Boylan
05e0ffdebc Collect gitea sshd logs
Currently we don't have any logs from our gitea sshd processes because
sshd logs to syslog by default and /dev/log isn't in our containers. You
can ask sshd nicely to log to stderr instead with the -e flag which
docker will pick up and store for us.

Update the sshd command to include -e then use testinfra to check we
collect logs and they are accssible from docker.

Change-Id: Ib7d6d405554c3c30be410bc08c6fee7d4363b096
2019-08-06 13:42:25 -07:00
Ian Wienand
734aaee327 Add vexxhost backup server
This is a new backup server for use with the roles in
I9bf74df351e056791ed817180436617048224d2c

Restrict the puppet group to only the openstack.org servers as this
new server doesn't need puppet.

Depends-On: https://review.opendev.org/674549
Change-Id: Ia8e2e01f579ed9475830c159bf266b63bed52c36
2019-08-05 19:00:29 +10:00
Zuul
788d91df1f Merge "Ansible roles for backup" 2019-08-05 08:48:41 +00:00
Ian Wienand
814e4be128 Ansible roles for backup
This introduces two new roles for managing the backup-server and hosts
that we wish to back up.

Firstly the "backup" role runs on hosts we wish to backup.  This
generates and configures a separate ssh key for running bup and
installs the appropriate cron job to run the backup daily.

The "backup-server" job runs on the backup server (or, indeed
servers).  It creates users for each backup host, accepts the remote
keys mentioned above and initalises bup.  It is then ready to receive
backups from the remote hosts.

This eliminates a fairly long-standing requirement for manual setup of
the backup server users and keys; this section is removed from the
documentation.

testinfra coverage is added.

Change-Id: I9bf74df351e056791ed817180436617048224d2c
2019-08-05 16:59:57 +10:00
Zuul
123305f29c Merge "Re-add the Debian 8/jessie key to reprepro" 2019-08-05 01:29:20 +00:00
Clark Boylan
3ff4bed27c Trim fedora mirror
The fedora mirror is our largest mirror (850GB about twice as big as the
next mirror). Much of this size is due to the fedora atomic images we
mirror.

On further investigation I notice that we are mirroring ppc images (for
which we do not have cpus to run them on), image for fedora 25 and 36
which are quite EOL'd, and our exclusion of the raw.xz and vagrant
images is failing.

Update the rsync excludes to ensure we don't mirror any of these images
we don't need.

Change-Id: I86856cb4e51b0e687aac45a1f014f87c5141318f
2019-08-02 14:35:21 -07:00
Clark Boylan
f686ec39f5 Switch fedora mirroring to pubmirror2.math.uh.edu
pubmirror1.math.uh.edu is currently offline and listed as an altonly.
pubmirror2 seems to work fine so switch to it.

Change-Id: I2562f8686146d17d4fad3997b9be22361fa05fca
2019-08-02 14:27:00 -07:00
Jeremy Stanley
5a096f3705 Re-add the Debian 8/jessie key to reprepro
Because of a limitation in GnuPG we need to have the Jessie archive
signing key in the list of VerifyRelease key IDs for the Debian
reprepro mirror. Also some suites (currently buster-backports) are
signed by a subkey of an archive signing key, so add the "+" suffix
to all these key IDs indicating subkey signatures are also allowed.
As always, Debian signing keys are published and available here:
https://ftp-master.debian.org/keys.html

Change-Id: Iedce38318718a18ace7b2c638755a7d7d4dcd69b
2019-08-02 19:04:25 +00:00
Ian Wienand
35f1321e14 AFS server restart and audit logging : helper script
This script helps restart the AFS servers, which is useful when
updating parameters.  It can also enable audit logging.

It can also stop and start the servers, although it's unlikely we'd
want all the servers offline at the same time so stopping has a
warning included.

Documentation is updated to refer to the helper script

Change-Id: Idcb3e43a3f6e614cdb787d4334e692a98bffdd15
2019-08-02 16:37:00 +10:00
Sorin Sbarnea
3792315db5 Recognize DISK_FULL failure messages (review_dev)
When a job is killed by zuul due to failure like DISK_FULL, a different
message ends up in as a comment.

<li>job-name
finger://ze09.openstack.org/8b6d...6f : DISK_FULL in 2h 59m 50s</li>

This adds another pattern that recognize these messages as failures,
regardess the case (DISK_FULL in this case).

Change-Id: Ib17f05a043430362b02a2826d69572f6b2dbd64a
Needed-By: https://review.opendev.org/#/c/631509/
2019-08-01 11:47:09 +01:00
Zuul
b0ea150b89 Merge "Correct emergency file reference in launch script" 2019-07-31 23:22:38 +00:00
James E. Blair
96aec261da Add logs.opendev.org vhost
This is a near-copy of the vhost template from puppet-openstackci.

Change-Id: I191e41b501629e2cdd82381d66daa3b850e0be81
2019-07-31 14:32:00 -07:00
James E. Blair
48cafd19f8 Add LE cert for logs.opendev.org to static
This can be used in an apache vhost later, but should be fine to
merge now.

Depends-On: https://review.opendev.org/673902
Change-Id: Ic2cb7585433351ec1bdabd88915fa1ca07da44e7
2019-07-31 13:00:50 -07:00
Jens Harbott
7df5981e12 Be explicit about fortnebula networks with nodepool
We don't want nodepool to use floating IPs in the fn cloud as it is an
ipv6 only cloud. We explicitly tell it there is no fip source and that
the tenant network routes ipv6 externally. This config is based on the
limestone configuration which is a similar cloud network wise.

Change-Id: I4a27a22a5beb9c5fc9d3e16cd2ca5b41aecbb46f
2019-07-31 08:36:23 -07:00
Zuul
3e03b7481d Merge "Add tool to analyze check and gate success rates" 2019-07-31 00:28:06 +00:00
Zuul
710821f7a2 Merge "Replace the fn mirror again" 2019-07-30 23:32:50 +00:00
Clark Boylan
4b4eb02f32 Replace the fn mirror again
Networking got weird on the previous host so we rebuilt this one going
back to networking we expect to work (FIPs and all that). This updates
the inventory so that we configure the host properly.

Change-Id: I0dcdbc9efdd330d66b57da0b01d23dd3d747f79b
2019-07-30 15:15:01 -07:00
Clark Boylan
b1de301261 Use public_v4 addr when ignoring ipv6
In our launch node script we have the option to ignore ipv6 to deal with
clouds like ovh that report an ipv6 address but don't actually provide
that data to the instance so it cannot configure ipv6. When we ignore
ipv6 we should not try to use the ipv6 address at all.

Use the public_v4 address in this case when writing out an ansible
inventory to run the base.yaml playbook when launching the node.
Otherwise we could use ipv6 which doesn't work.

Change-Id: I2ce5cc0db9852d3426828cf88965819f88b3ebd5
2019-07-30 15:00:53 -07:00
Zuul
cb2976a976 Merge "Add archive signing key for Debian 10/buster" 2019-07-30 19:41:59 +00:00
Zuul
107943e60d Merge "Build gerrit images for 2.16 and 3.0 as well" 2019-07-30 18:29:50 +00:00
Jeremy Stanley
a22df8264f Add archive signing key for Debian 10/buster
The buster-backports suite on Debian mirrors is not signed by the
old jessie signing key we have set to verify in reprepro, but also
we're not mirroring Debian 8/jessie any longer anyway. Replace that
list with the 9/stretch and 10/buster signing keys and switch to
longer key IDs which match the names used for them in the Puppet
manifest. Also add Puppetry and keyfile for the buster keys so that
they will be installed accordingly. The official list of keys can be
found here: https://ftp-master.debian.org/keys.html

Change-Id: Ia193f040b2b707329948955eb091a186eabf8096
2019-07-30 17:41:07 +00:00
Jeremy Stanley
6631b899c5 Put gitea07 and gitea08 back into service
Add the gitea07.opendev.org and gitea08.opendev.org servers into the
haproxy pools now that they've been seeded with current data. Remove
the create repos task disable list entries for them as well.

Change-Id: I69390e6a32b01cc1713839f326fa930c376282af
2019-07-29 23:35:36 +00:00
Jeremy Stanley
2ed6775780 Add gitea07 and gitea08 replacements to inventory
Add new IP addresses to inventory for the rebuilds, but don't
reactivate them in the haproxy pools yet (they're already excluded
from the repository creation task).

Change-Id: I1e3fc1ba56015eeab2c6256b3f90188ecabf23cc
2019-07-29 19:20:26 +00:00
Jeremy Stanley
56a0b08aa5 Swap gitea05 into service and bring down 07 and 08
Add the gitea05.opendev.org server into the haproxy pools now that
it's been seeded with current data. Switch the create repos task
disable list from 05 to 07 and 08, and remove 07 and 08 from the
Ansible inventory and comment them out in the haproxy pools in
preparation for replacement.

To the casual observer it may appear gitea06 is being skipped, but
it was replaced first out of sequence due to filesystem corruption
during the PTG. The increased performance of the 75% of the nodes
which have already been replaced means we can get by doing the final
25% at the same time (so two servers at once).

Change-Id: Ia49157c16582b7ed0dbef3eb9d07bf7f1d4450b9
2019-07-29 16:56:39 +00:00
Zuul
bcb07033f5 Merge "Add gitea05 replacement to inventory" 2019-07-29 14:49:46 +00:00
Jeremy Stanley
b45c672de5 Replace fortnebula mirror
The fortnebula mirror is being rebuilt while the environment there
is under some refactoring. The old mirror isn't reachable any longer
so removing it from our inventory while adding its replacement
should be safe.

Also update the letsencrypt playbooks for the new name.

Change-Id: I789248e4216f4cf059ccc5b071c2a784f9c629e9
2019-07-29 13:08:58 +00:00
Jeremy Stanley
00b814cabb Add gitea05 replacement to inventory
Add new IP addresses to inventory for the rebuild, but don't
reactivate it in the haproxy pools yet (it's already excluded from
the repository creation task).

Change-Id: I36c188992f4787d4e7c5c952eac5fb0bbdc5a627
2019-07-28 21:41:36 +00:00
Jeremy Stanley
79c86cfe3d Swap gitea04 into service and bring down gitea05
Add the gitea04.opendev.org server into the haproxy pools now that
it's been seeded with current data. Switch the create repos task
disable list from 04 to 05, and remove 05 from the Ansible inventory
and comment it out in the haproxy pools in preparation for
replacement.

Change-Id: I4cd1fef399e527771a26efee8a39952694f3ce6b
2019-07-28 12:15:41 +00:00
Monty Taylor
2a46202b9f Build gerrit images for 2.16 and 3.0 as well
Our goal is upgrading to 3.0. To do that we need to upgrade to 2.15, then
to 2.16, then to 3.0. Build all of the images so that we can do that.

2.16 and 3.0 also use bazel, so just use one copy of the Dockerfile for
all three and let zuul check out the repos to the right versions.

Depends-On: https://review.opendev.org/673147
Depends-On: https://review.opendev.org/672320
Change-Id: I35bd278e0c70c871fa44d005c60a987d1d8e3cdc
2019-07-27 11:34:42 -04:00
Jeremy Stanley
a603b4bd38 Add gitea04 replacement to inventory
Add new IP addresses to inventory for the rebuild, but don't
reactivate it in the haproxy pools yet (it's already excluded from
the repository creation task).

Change-Id: I8b43c6f9cb41452c7f64862a2b401dc0d1b7ef3d
2019-07-27 15:28:44 +00:00
Jeremy Stanley
0256ba5219 Swap gitea03 into service and bring down gitea04
Add the gitea03.opendev.org server into the haproxy pools now that
it's been seeded with current data. Switch the create repos task
disable list from 03 to 04, and remove 04 from the Ansible inventory
and comment it out in the haproxy pools in preparation for
replacement.

Change-Id: Id5817f8265996862a7e0810b9fb9e3d78be5d066
2019-07-27 02:07:13 +00:00
Jeremy Stanley
01a97664ea Add gitea03 replacement to inventory
Add new IP addresses to inventory for the rebuild, but don't
reactivate it in the haproxy pools yet (it's already excluded from
the repository creation task).

Change-Id: Id4076e179bee82b03822f59803865eaa60118334
2019-07-26 22:09:18 +00:00
Jeremy Stanley
55f657c68d Swap gitea02 into service and bring down gitea03
Add the gitea02.opendev.org server into the haproxy pools now that
it's been seeded with current data. Switch the create repos task
disable list from 02 to 03, and remove 03 from the Ansible inventory
and comment it out in the haproxy pools in preparation for
replacement.

Change-Id: I4b51291311064c60d4bb2d90bec6e5cb90a54f3c
2019-07-26 18:00:52 +00:00
Jeremy Stanley
9c5e54a89c Add gitea02 replacement to inventory
Add new IP addresses to inventory for the rebuild, but don't
reactivate it in the haproxy pools yet. Also switch the exclusion
for 01 to 02 for the repository creation task.

Change-Id: I6c4a437316627a723e6bb6c15fdce86a5e847042
2019-07-26 15:11:08 +00:00
Jeremy Stanley
4c04ad5436 Correct emergency file reference in launch script
The launch script is referring to the wrong path for the emergency
inventory. Also correct the references in the sysadmin guide and
update the example for using it.

Change-Id: I80bdbd440ec451bcd6fb1a3eb552ffda32407c44
2019-07-26 14:55:32 +00:00
Zuul
4b092eaed7 Merge "Build docker images of gerrit" 2019-07-25 21:58:06 +00:00
Clark Boylan
c23ac25264 Remove gitea02 from inventory so we can replace it
The global inventory is used when launching nodes so if we want to
replace a server we have to remove it from the inventory first. This is
that step for replacing gitea02.

Note that when adding it back for the new server there are some edits to
make to the playbooks as noted in the gitea sysadmin docs.

We do also remove this instance from haproxy as well to prevent unwanted
connections while we flip things over.

Change-Id: I53a3f517d46d046cb59e3185ca19ba3df55d8466
2019-07-24 20:12:16 -07:00
Zuul
788f3b5b85 Merge "Readd gitea01 to haproxy pools" 2019-07-24 22:17:42 +00:00
Jeremy Stanley
866b52f9fb Readd gitea01 to haproxy pools
Now that the replacement gitea01 server has up to date content, add
it back to the haproxy configuration.

Change-Id: I24b4659603efa1861fed1238b8eda6c3f6c11a14
2019-07-24 21:08:00 +00:00
Zuul
1b135f7d46 Merge "Install GNU Privacy Guard on Gitea servers" 2019-07-24 16:48:25 +00:00
Jeremy Stanley
f8bf371583 Install GNU Privacy Guard on Gitea servers
The install-docker role uses the apt-key utility which expects to
have GPG installed, so include the package for it (this seems to
have been manually installed or preinstalled on the images for our
existing Gitea servers, but our new images do not include it).

Change-Id: I28d748fab35e22219a7278603ed984aaa7658ef0
2019-07-24 15:46:50 +00:00
Clark Boylan
36c14e4325 Remove centos mirror from openstack mirror update
This rsync'd mirror is now being managed by the opendev mirror update
server. Remove it from the older openstack server to avoid a conflict in
excludes around sclo repo.

Currently we have opendev adding sclo and openstack removing it.

Change-Id: I599ee7d0fab8c5e2a060aff86bce20f1f8d4f54b
2019-07-24 08:11:44 -07:00
Monty Taylor
943f66e3e6 Build docker images of gerrit
To provide a stepwise upgrade path from 2.13 running directly to
2.15 in a container, make a container image containing the war we're
using currently. This should let us make a change to how we run the
war without changing the war at all, and then update the war.

Instead of trying to make a clean build for gerrit 2.13 inside of a
builder image, just have it wget the already built wars and jars we
have.

There are pieces of this that duplicate what's being done in puppet,
but in this context it's not immediately clear these are important to
do. However, it's also not clear they're a bad idea.

The gerrit 2.15 build needs a newer bazel. Looking at the CI scripts
that are used by gerrithub, we find that they use bazel 0.26.1
and nodesource v10. Use the bazel image published by google to get
a bazel builder image.

Set gerrit uid/git to 3000 in both images to match the existing
directory ownership so that bindmounting doesn't face permissions
problems.

Change-Id: I3533f01c0859ed50640dcfd98023994c5867c056
2019-07-24 04:40:28 -04:00
Zuul
c2f3f53389 Merge "Re-add gitea01 replacement to inventory" 2019-07-24 00:09:30 +00:00
Jeremy Stanley
5587c299ea Re-add gitea01 replacement to inventory
Add new IP addresses to inventory for the rebuild, but don't
reactivate it in the haproxy pools yet.

Note this switches the gitea testing to use a host called gitea99 so
that it doesn't conflict with our changes of the production hosts.

Change-Id: I9779e16cca423bcf514dd3a8d9f14e91d43f1ca3
2019-07-23 16:17:41 -07:00
Zuul
60d4a1bdd3 Merge "Reload haproxy configuration when config changes" 2019-07-23 18:55:05 +00:00
Monty Taylor
64da74a7a5 Serialize the gitea role
The gitea role will restart gitea if images have updated. We'd like
to not stop them all at the same time. Do serial: 1 so that we update
one backend at a time.

Change-Id: I5ce7f6d8d25a1cf7ddbe901ec6b91860ceaf5bd1
2019-07-23 13:25:29 -04:00