233 Commits

Author SHA1 Message Date
Zuul
a934f91f17 Merge "run_all.sh : add backup playbook" 2019-08-20 20:18:39 +00:00
Zuul
1c29fffa1a Merge "Replace wiki-dev02 with wiki-dev03" 2019-08-09 14:42:37 +00:00
Ian Wienand
376915e17a run_all.sh : add backup playbook
The backup roles have been debugged and are ready to run.

A note is added about having the backup server in a default disabled
state.  This was discussed at an infra meeting where consensus was to
keep it disabled [1].

[1] http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-06-11-19.01.log.html#l-184

Change-Id: I2a3d2d08a9d1514bf6bdcf15bc5bc95689f3020f
2019-08-09 16:43:55 +10:00
Jeremy Stanley
df99568f10 Replace wiki-dev02 with wiki-dev03
In order to confirm configuration management is working cleanly for
wiki-dev.openstack.org deployments, a new wiki-dev03 has been built
and the old wiki-dev02 deleted. These are not production hosts so
this change can be merged at any time. DNS has also been updated for
them accordingly.

Change-Id: I61ae138b10d51caef2cdd26ca8adaf9d59728ac8
2019-08-08 17:02:05 +00:00
Ian Wienand
78dc3e6ffd Add review-dev as a new backup client
Opt in review-dev to be a client for the new backup server

Change-Id: Ie24855a0df9f8d8d83588ae2f7221415a6535fd5
2019-08-08 13:55:33 +10:00
Ian Wienand
734aaee327 Add vexxhost backup server
This is a new backup server for use with the roles in
I9bf74df351e056791ed817180436617048224d2c

Restrict the puppet group to only the openstack.org servers as this
new server doesn't need puppet.

Depends-On: https://review.opendev.org/674549
Change-Id: Ia8e2e01f579ed9475830c159bf266b63bed52c36
2019-08-05 19:00:29 +10:00
James E. Blair
48cafd19f8 Add LE cert for logs.opendev.org to static
This can be used in an apache vhost later, but should be fine to
merge now.

Depends-On: https://review.opendev.org/673902
Change-Id: Ic2cb7585433351ec1bdabd88915fa1ca07da44e7
2019-07-31 13:00:50 -07:00
Clark Boylan
4b4eb02f32 Replace the fn mirror again
Networking got weird on the previous host so we rebuilt this one going
back to networking we expect to work (FIPs and all that). This updates
the inventory so that we configure the host properly.

Change-Id: I0dcdbc9efdd330d66b57da0b01d23dd3d747f79b
2019-07-30 15:15:01 -07:00
Jeremy Stanley
2ed6775780 Add gitea07 and gitea08 replacements to inventory
Add new IP addresses to inventory for the rebuilds, but don't
reactivate them in the haproxy pools yet (they're already excluded
from the repository creation task).

Change-Id: I1e3fc1ba56015eeab2c6256b3f90188ecabf23cc
2019-07-29 19:20:26 +00:00
Jeremy Stanley
56a0b08aa5 Swap gitea05 into service and bring down 07 and 08
Add the gitea05.opendev.org server into the haproxy pools now that
it's been seeded with current data. Switch the create repos task
disable list from 05 to 07 and 08, and remove 07 and 08 from the
Ansible inventory and comment them out in the haproxy pools in
preparation for replacement.

To the casual observer it may appear gitea06 is being skipped, but
it was replaced first out of sequence due to filesystem corruption
during the PTG. The increased performance of the 75% of the nodes
which have already been replaced means we can get by doing the final
25% at the same time (so two servers at once).

Change-Id: Ia49157c16582b7ed0dbef3eb9d07bf7f1d4450b9
2019-07-29 16:56:39 +00:00
Zuul
bcb07033f5 Merge "Add gitea05 replacement to inventory" 2019-07-29 14:49:46 +00:00
Jeremy Stanley
b45c672de5 Replace fortnebula mirror
The fortnebula mirror is being rebuilt while the environment there
is under some refactoring. The old mirror isn't reachable any longer
so removing it from our inventory while adding its replacement
should be safe.

Also update the letsencrypt playbooks for the new name.

Change-Id: I789248e4216f4cf059ccc5b071c2a784f9c629e9
2019-07-29 13:08:58 +00:00
Jeremy Stanley
00b814cabb Add gitea05 replacement to inventory
Add new IP addresses to inventory for the rebuild, but don't
reactivate it in the haproxy pools yet (it's already excluded from
the repository creation task).

Change-Id: I36c188992f4787d4e7c5c952eac5fb0bbdc5a627
2019-07-28 21:41:36 +00:00
Jeremy Stanley
79c86cfe3d Swap gitea04 into service and bring down gitea05
Add the gitea04.opendev.org server into the haproxy pools now that
it's been seeded with current data. Switch the create repos task
disable list from 04 to 05, and remove 05 from the Ansible inventory
and comment it out in the haproxy pools in preparation for
replacement.

Change-Id: I4cd1fef399e527771a26efee8a39952694f3ce6b
2019-07-28 12:15:41 +00:00
Jeremy Stanley
a603b4bd38 Add gitea04 replacement to inventory
Add new IP addresses to inventory for the rebuild, but don't
reactivate it in the haproxy pools yet (it's already excluded from
the repository creation task).

Change-Id: I8b43c6f9cb41452c7f64862a2b401dc0d1b7ef3d
2019-07-27 15:28:44 +00:00
Jeremy Stanley
0256ba5219 Swap gitea03 into service and bring down gitea04
Add the gitea03.opendev.org server into the haproxy pools now that
it's been seeded with current data. Switch the create repos task
disable list from 03 to 04, and remove 04 from the Ansible inventory
and comment it out in the haproxy pools in preparation for
replacement.

Change-Id: Id5817f8265996862a7e0810b9fb9e3d78be5d066
2019-07-27 02:07:13 +00:00
Jeremy Stanley
01a97664ea Add gitea03 replacement to inventory
Add new IP addresses to inventory for the rebuild, but don't
reactivate it in the haproxy pools yet (it's already excluded from
the repository creation task).

Change-Id: Id4076e179bee82b03822f59803865eaa60118334
2019-07-26 22:09:18 +00:00
Jeremy Stanley
55f657c68d Swap gitea02 into service and bring down gitea03
Add the gitea02.opendev.org server into the haproxy pools now that
it's been seeded with current data. Switch the create repos task
disable list from 02 to 03, and remove 03 from the Ansible inventory
and comment it out in the haproxy pools in preparation for
replacement.

Change-Id: I4b51291311064c60d4bb2d90bec6e5cb90a54f3c
2019-07-26 18:00:52 +00:00
Jeremy Stanley
9c5e54a89c Add gitea02 replacement to inventory
Add new IP addresses to inventory for the rebuild, but don't
reactivate it in the haproxy pools yet. Also switch the exclusion
for 01 to 02 for the repository creation task.

Change-Id: I6c4a437316627a723e6bb6c15fdce86a5e847042
2019-07-26 15:11:08 +00:00
Clark Boylan
c23ac25264 Remove gitea02 from inventory so we can replace it
The global inventory is used when launching nodes so if we want to
replace a server we have to remove it from the inventory first. This is
that step for replacing gitea02.

Note that when adding it back for the new server there are some edits to
make to the playbooks as noted in the gitea sysadmin docs.

We do also remove this instance from haproxy as well to prevent unwanted
connections while we flip things over.

Change-Id: I53a3f517d46d046cb59e3185ca19ba3df55d8466
2019-07-24 20:12:16 -07:00
Jeremy Stanley
5587c299ea Re-add gitea01 replacement to inventory
Add new IP addresses to inventory for the rebuild, but don't
reactivate it in the haproxy pools yet.

Note this switches the gitea testing to use a host called gitea99 so
that it doesn't conflict with our changes of the production hosts.

Change-Id: I9779e16cca423bcf514dd3a8d9f14e91d43f1ca3
2019-07-23 16:17:41 -07:00
Clark Boylan
ffcd1791bf Cleanup nodepool builder clouds.yaml
We ended up running into a problem with nodepool built control plane
images (has to do with boot from volume not allowing us to delete images
that are in use by a nova instance). We have decided to clean this up
and go back to not doing this until we can do it more properly.

Note this isn't a revert because having a group for access to control
plane clouds does seem like a good idea in general and I believe there
have been changes we'd have to resolve in the clouds.yaml files anyway.

Depends-On: https://review.opendev.org/#/c/665012/
Change-Id: I5e72928ec2dec37afa9c8567eff30eb6e9c04f1d
2019-07-22 13:55:29 -07:00
Clark Boylan
a2af942fa3 Remove gitea01 from inventory so we can replace it
The global inventory is used when launching nodes so if we want to
replace a server we have to remove it from the inventory first. This is
that step for replacing gitea01.

Note that when adding it back for the new server there are some edits to
make to the playbooks as noted in the gitea sysadmin docs.

We do also remove this instance from haproxy as well to prevent unwanted
connections while we flip things over.

Change-Id: If32405b1302353f1f262a30b7392533f86fec1e4
2019-07-22 09:20:17 -07:00
Zuul
efc8e17c6e Merge "Add fortnebula ci mirror" 2019-07-03 18:14:48 +00:00
Clark Boylan
95f02c7aa7 Add fortnebula ci mirror
Note we depends on the DNS updates so that LE cert provisioning works
on the first pass.

Depends-On: https://review.opendev.org/668929
Change-Id: I953938b77bfce67be0cb55af5cf4bd64044100f4
2019-07-03 07:53:41 -07:00
Ian Wienand
ece14bbfb0 Add mirror-update01.opendev.org server
Add the new mirror-update server as a follow-on to
I525ac18b55f0e11b0a541b51fa97ee5d6512bf70.

Also ensure that the new mirror server isn't in the puppet groups by
only matching the openstack.org one.

Also remove from the afsadmin group.  This group is only used for
keytabs stored on bridge.o.o.  I don't think that we need group for
the keytabs -- a keytab should only ever be in use on one host at a
time, so we are better off keeping the keytabs in a specific host_var
for the host they are used on, rather than being in a group and
possibly deployed on servers where they are not used.

Depends-On: https://review.opendev.org/668610
Change-Id: Icda92bb234adc00f6718c1c656e8f069ce2704c4
2019-07-02 17:34:09 +10:00
Ian Wienand
b85282c046 Move rsync mirror updates to new opendev.org mirror-update host
This move was prompted by wishing to expose the mirror update logs for
the rsync updates so that debugging problems does not require a root
user (note: not actually done in this change; will be a follow-on).

Rather than start hacking at puppet, the rsync mirror scripts make a
nice delination point for starting an Ansible-first/Bionic update.

Most magic is included in the scripts, so there is not much more to do
than copy them.  The host uses the existing kerberos and openafs roles
and copies the key material into place (to be added before merge).

Note the scripts are removed from the extant puppet so we don't have
two updates happening simultaneously.  This will also require a manual
clean to remove the cron jobs as a once-off when merging.

The other part of mirror-update is the reprepro based scripts for the
various debuntu repositories.  They are left as future work for now.

Testing is added to ensure dependencies and scripts are all in place.

Change-Id: I525ac18b55f0e11b0a541b51fa97ee5d6512bf70
2019-07-02 16:42:33 +10:00
Zuul
b3776ca3b0 Merge "Add OVH GRA1 mirror" 2019-07-01 02:15:40 +00:00
Ian Wienand
7810230408 Add OVH GRA1 mirror
This mirror will be manually configured with kafs (see
https://review.opendev.org/623974).  This should be a nice distant
geographic counterpoint to the IAD RAX server.

This will need to be manually configured with a custom kernel for now,
but fixes are making their way upstream and this host will be
converted when available.

Depends-On: https://review.opendev.org/667529
Change-Id: I6a22933029c096c781c93c33e6edf03bf59223c9
2019-06-27 10:07:44 +10:00
Clark Boylan
3734f000f9 Enroll new gitea06 into ansible inventory
We add the new host so that it will get configured as a gitea backend
server. We exclude this server from the list of gitea hosts to configure
git repos on because we want to recover its DB from one of the other
sibling nodes first. This should preserve the http redirects for us.

Once we have the db recovered we can enable replication from gerrit then
readd this host to the haproxy load balancer.

Change-Id: Ia2a98e5ded43cad044db36ca8d0da5a96277afee
2019-06-25 15:16:59 -07:00
Clark Boylan
263ab148fe Remove gitea06 from our inventory file
Note we don't fully remove it from cacti and hiera and so on because we
are replacing this server and we just want ansible to ignore the old
gitea06 for a bit while we bootstrap the new server.

Change-Id: Iaa89e77c055d8099a7d3d511723782fead43ce74
2019-06-25 14:03:44 -07:00
Ian Wienand
5981df66ea Fix syntax for new mirror hosts
Fix for I0e55d2c575427e404709e78d0c7a10a974117ac4 ... how this passed
gate testing to be determined ...

Change-Id: I834411ef2dee458ae15fb99a3c88b6d2fee4cf1e
2019-06-12 09:34:10 +10:00
Ian Wienand
66495f7a84 Add RAX IAD/ORD opendev.org mirrors
Change-Id: I0e55d2c575427e404709e78d0c7a10a974117ac4
Depends-On: https://review.opendev.org/663849
2019-06-11 09:36:16 +10:00
James E. Blair
2e5291f377 Get an LE cert for tarballs.opendev.org
Depends-On: https://review.opendev.org/663424
Change-Id: I4faa12b5d241144463ccf7ec59ef2d0b11479c35
2019-06-05 13:56:34 -07:00
Zuul
1fe34e00d4 Merge "Add control plane clouds to nodepool builder clouds.yaml" 2019-06-04 20:15:24 +00:00
Zuul
4166abf258 Merge "Remove opendev k8s cluster from inventory" 2019-05-29 15:23:09 +00:00
Monty Taylor
ff1b8a94c6 Add control plane clouds to nodepool builder clouds.yaml
In order to have nodepool build images and upload them to control
plane clouds, add them to the clouds.yaml on the nodepool-builder
hosts. Keep them out of the launcher configs by splitting the config
templates. So that we can keep our copies of things to a minimum,
create a group called "control-plane-clouds" and put bridge and nb0*
in it.

There are clouds mentions in here that we no longer use, a followup
patch will clean those up.

NOTE: Requires shifting the clouds config dict from
host_vars/bridge.openstack.org.yaml to group_vars/control-plane-clouds.yaml
in the secrets on bridge.

Needed-By: https://review.opendev.org/640044
Change-Id: Id1161bca8f23129202599dba299c288a6aa29212
2019-05-23 14:34:10 -05:00
Clark Boylan
0848e0760b Remove the ask.openstack.org inventory entry
This trusty server has been replaced by xenial ask01.openstack.org.

Change-Id: I33090c9ce45982e19d4ef85c156e76e7583a07af
2019-05-23 12:20:09 -07:00
Clark Boylan
08152aa22f Remove groups configuration
This removes the groups servers from our inventory as well as our
manifests/modules. We don't run the groups service anymore as many
groups migrated to meetup.com independent of us and the others have
transitioned there.

Change-Id: I7cb76611e6d30e7189821923f36a38dec9ea7241
2019-05-23 12:20:04 -07:00
Zuul
41c06cdf49 Merge "Bringup mirror01.dfw.rax.opendev.org" 2019-05-21 23:42:57 +00:00
Zuul
54c72ab7b9 Merge "Create opendev mirrors" 2019-05-21 23:01:28 +00:00
Zuul
05300b6268 Merge "Update ask.openstack.org to puppet 4" 2019-05-21 19:35:04 +00:00
Zuul
82e498fb59 Merge "Remove ask-staging* from disabled list" 2019-05-21 08:39:28 +00:00
Ian Wienand
73bbc6787f Bringup mirror01.dfw.rax.opendev.org
This is an initial host for testing opendev.org mirrors

Change-Id: I26b9ed1e21e2111f48bc7ecc384880c274eed213
Depends-On: https://review.opendev.org/660235
2019-05-21 11:08:30 +10:00
Ian Wienand
670107045a Create opendev mirrors
This impelements mirrors to live in the opendev.org namespace.  The
implementation is Ansible native for deployment on a Bionic node.

The hostname prefix remains the same (mirrorXX.region.provider.) but
the groups.yaml splits the opendev.org mirrors into a separate group.
The matches in the puppet group are also updated so to not run puppet
on the hosts.

The kerberos and openafs client parts do not need any updating and
works on the Bionic host.

The hosts are setup to provision certificates for themselves from
letsencrypt.  Note we've added a new handler for mirror nodes to use
that restarts apache on certificate issue/renewal.

The new "mirror" role is a port of the existing puppet mirror.pp.  It
installs apache, sets up some modules, makes some symlinks, sets up a
cleanup cron job and installs the apache vhost configuration.

The vhost configuration is also ported from the extant puppet.  It is
simplified somewhat; but the biggest change is that we have extracted
the main port 80 configuration into a macro which is applied to both
port 80 and 443; i.e. the host will have SSL support.  The other ports
are left alone for now, but can be updated in due course.

Thus we should be able to CNAME the existing mirrors to new nodes, and
any existing http access can continue.  We can update our mirror setup
scripts to point to https resources as appropriate.

Change-Id: Iec576d631dd5b02f6b9fb445ee600be060f9cf1e
2019-05-21 11:08:25 +10:00
Zuul
695a064036 Merge "Remove grafana01.openstack.org from inventory" 2019-05-20 22:23:33 +00:00
Ian Wienand
c796021bcb Add ask01.openstack.org to inventory
Change-Id: I474c0cf7bab51d2ec73a87af0a4ecbf910109c97
2019-05-20 17:56:55 +10:00
Ian Wienand
2e83c579f6 Remove ask-staging* from disabled list
These servers have been removed

Change-Id: I26ebd650866f9a71dd8b41f889878659785e4255
2019-05-20 17:25:20 +10:00
Monty Taylor
6bc8754b87 Remove opendev k8s cluster from inventory
We're not really using/maintaining this at the moment. Before we do
put it back in production, we're likely to simply rebuild it from
scratch.

Change-Id: I469f00e90903a010f2cec45031b049556eb268a2
2019-05-19 07:36:39 -05:00
Monty Taylor
7c54c2781b Remove unreachable hosts from inventory
None of these can be reached from bridge.

Change-Id: I2f4d419a7ea9993e90dba6d25681807f98ea1db5
2019-05-19 07:36:39 -05:00