The backup roles have been debugged and are ready to run.
A note is added about having the backup server in a default disabled
state. This was discussed at an infra meeting where consensus was to
keep it disabled [1].
[1] http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-06-11-19.01.log.html#l-184
Change-Id: I2a3d2d08a9d1514bf6bdcf15bc5bc95689f3020f
In order to confirm configuration management is working cleanly for
wiki-dev.openstack.org deployments, a new wiki-dev03 has been built
and the old wiki-dev02 deleted. These are not production hosts so
this change can be merged at any time. DNS has also been updated for
them accordingly.
Change-Id: I61ae138b10d51caef2cdd26ca8adaf9d59728ac8
This is a new backup server for use with the roles in
I9bf74df351e056791ed817180436617048224d2c
Restrict the puppet group to only the openstack.org servers as this
new server doesn't need puppet.
Depends-On: https://review.opendev.org/674549
Change-Id: Ia8e2e01f579ed9475830c159bf266b63bed52c36
This can be used in an apache vhost later, but should be fine to
merge now.
Depends-On: https://review.opendev.org/673902
Change-Id: Ic2cb7585433351ec1bdabd88915fa1ca07da44e7
Networking got weird on the previous host so we rebuilt this one going
back to networking we expect to work (FIPs and all that). This updates
the inventory so that we configure the host properly.
Change-Id: I0dcdbc9efdd330d66b57da0b01d23dd3d747f79b
Add new IP addresses to inventory for the rebuilds, but don't
reactivate them in the haproxy pools yet (they're already excluded
from the repository creation task).
Change-Id: I1e3fc1ba56015eeab2c6256b3f90188ecabf23cc
Add the gitea05.opendev.org server into the haproxy pools now that
it's been seeded with current data. Switch the create repos task
disable list from 05 to 07 and 08, and remove 07 and 08 from the
Ansible inventory and comment them out in the haproxy pools in
preparation for replacement.
To the casual observer it may appear gitea06 is being skipped, but
it was replaced first out of sequence due to filesystem corruption
during the PTG. The increased performance of the 75% of the nodes
which have already been replaced means we can get by doing the final
25% at the same time (so two servers at once).
Change-Id: Ia49157c16582b7ed0dbef3eb9d07bf7f1d4450b9
The fortnebula mirror is being rebuilt while the environment there
is under some refactoring. The old mirror isn't reachable any longer
so removing it from our inventory while adding its replacement
should be safe.
Also update the letsencrypt playbooks for the new name.
Change-Id: I789248e4216f4cf059ccc5b071c2a784f9c629e9
Add new IP addresses to inventory for the rebuild, but don't
reactivate it in the haproxy pools yet (it's already excluded from
the repository creation task).
Change-Id: I36c188992f4787d4e7c5c952eac5fb0bbdc5a627
Add the gitea04.opendev.org server into the haproxy pools now that
it's been seeded with current data. Switch the create repos task
disable list from 04 to 05, and remove 05 from the Ansible inventory
and comment it out in the haproxy pools in preparation for
replacement.
Change-Id: I4cd1fef399e527771a26efee8a39952694f3ce6b
Add new IP addresses to inventory for the rebuild, but don't
reactivate it in the haproxy pools yet (it's already excluded from
the repository creation task).
Change-Id: I8b43c6f9cb41452c7f64862a2b401dc0d1b7ef3d
Add the gitea03.opendev.org server into the haproxy pools now that
it's been seeded with current data. Switch the create repos task
disable list from 03 to 04, and remove 04 from the Ansible inventory
and comment it out in the haproxy pools in preparation for
replacement.
Change-Id: Id5817f8265996862a7e0810b9fb9e3d78be5d066
Add new IP addresses to inventory for the rebuild, but don't
reactivate it in the haproxy pools yet (it's already excluded from
the repository creation task).
Change-Id: Id4076e179bee82b03822f59803865eaa60118334
Add the gitea02.opendev.org server into the haproxy pools now that
it's been seeded with current data. Switch the create repos task
disable list from 02 to 03, and remove 03 from the Ansible inventory
and comment it out in the haproxy pools in preparation for
replacement.
Change-Id: I4b51291311064c60d4bb2d90bec6e5cb90a54f3c
Add new IP addresses to inventory for the rebuild, but don't
reactivate it in the haproxy pools yet. Also switch the exclusion
for 01 to 02 for the repository creation task.
Change-Id: I6c4a437316627a723e6bb6c15fdce86a5e847042
The global inventory is used when launching nodes so if we want to
replace a server we have to remove it from the inventory first. This is
that step for replacing gitea02.
Note that when adding it back for the new server there are some edits to
make to the playbooks as noted in the gitea sysadmin docs.
We do also remove this instance from haproxy as well to prevent unwanted
connections while we flip things over.
Change-Id: I53a3f517d46d046cb59e3185ca19ba3df55d8466
Add new IP addresses to inventory for the rebuild, but don't
reactivate it in the haproxy pools yet.
Note this switches the gitea testing to use a host called gitea99 so
that it doesn't conflict with our changes of the production hosts.
Change-Id: I9779e16cca423bcf514dd3a8d9f14e91d43f1ca3
We ended up running into a problem with nodepool built control plane
images (has to do with boot from volume not allowing us to delete images
that are in use by a nova instance). We have decided to clean this up
and go back to not doing this until we can do it more properly.
Note this isn't a revert because having a group for access to control
plane clouds does seem like a good idea in general and I believe there
have been changes we'd have to resolve in the clouds.yaml files anyway.
Depends-On: https://review.opendev.org/#/c/665012/
Change-Id: I5e72928ec2dec37afa9c8567eff30eb6e9c04f1d
The global inventory is used when launching nodes so if we want to
replace a server we have to remove it from the inventory first. This is
that step for replacing gitea01.
Note that when adding it back for the new server there are some edits to
make to the playbooks as noted in the gitea sysadmin docs.
We do also remove this instance from haproxy as well to prevent unwanted
connections while we flip things over.
Change-Id: If32405b1302353f1f262a30b7392533f86fec1e4
Note we depends on the DNS updates so that LE cert provisioning works
on the first pass.
Depends-On: https://review.opendev.org/668929
Change-Id: I953938b77bfce67be0cb55af5cf4bd64044100f4
Add the new mirror-update server as a follow-on to
I525ac18b55f0e11b0a541b51fa97ee5d6512bf70.
Also ensure that the new mirror server isn't in the puppet groups by
only matching the openstack.org one.
Also remove from the afsadmin group. This group is only used for
keytabs stored on bridge.o.o. I don't think that we need group for
the keytabs -- a keytab should only ever be in use on one host at a
time, so we are better off keeping the keytabs in a specific host_var
for the host they are used on, rather than being in a group and
possibly deployed on servers where they are not used.
Depends-On: https://review.opendev.org/668610
Change-Id: Icda92bb234adc00f6718c1c656e8f069ce2704c4
This move was prompted by wishing to expose the mirror update logs for
the rsync updates so that debugging problems does not require a root
user (note: not actually done in this change; will be a follow-on).
Rather than start hacking at puppet, the rsync mirror scripts make a
nice delination point for starting an Ansible-first/Bionic update.
Most magic is included in the scripts, so there is not much more to do
than copy them. The host uses the existing kerberos and openafs roles
and copies the key material into place (to be added before merge).
Note the scripts are removed from the extant puppet so we don't have
two updates happening simultaneously. This will also require a manual
clean to remove the cron jobs as a once-off when merging.
The other part of mirror-update is the reprepro based scripts for the
various debuntu repositories. They are left as future work for now.
Testing is added to ensure dependencies and scripts are all in place.
Change-Id: I525ac18b55f0e11b0a541b51fa97ee5d6512bf70
This mirror will be manually configured with kafs (see
https://review.opendev.org/623974). This should be a nice distant
geographic counterpoint to the IAD RAX server.
This will need to be manually configured with a custom kernel for now,
but fixes are making their way upstream and this host will be
converted when available.
Depends-On: https://review.opendev.org/667529
Change-Id: I6a22933029c096c781c93c33e6edf03bf59223c9
We add the new host so that it will get configured as a gitea backend
server. We exclude this server from the list of gitea hosts to configure
git repos on because we want to recover its DB from one of the other
sibling nodes first. This should preserve the http redirects for us.
Once we have the db recovered we can enable replication from gerrit then
readd this host to the haproxy load balancer.
Change-Id: Ia2a98e5ded43cad044db36ca8d0da5a96277afee
Note we don't fully remove it from cacti and hiera and so on because we
are replacing this server and we just want ansible to ignore the old
gitea06 for a bit while we bootstrap the new server.
Change-Id: Iaa89e77c055d8099a7d3d511723782fead43ce74
Fix for I0e55d2c575427e404709e78d0c7a10a974117ac4 ... how this passed
gate testing to be determined ...
Change-Id: I834411ef2dee458ae15fb99a3c88b6d2fee4cf1e
In order to have nodepool build images and upload them to control
plane clouds, add them to the clouds.yaml on the nodepool-builder
hosts. Keep them out of the launcher configs by splitting the config
templates. So that we can keep our copies of things to a minimum,
create a group called "control-plane-clouds" and put bridge and nb0*
in it.
There are clouds mentions in here that we no longer use, a followup
patch will clean those up.
NOTE: Requires shifting the clouds config dict from
host_vars/bridge.openstack.org.yaml to group_vars/control-plane-clouds.yaml
in the secrets on bridge.
Needed-By: https://review.opendev.org/640044
Change-Id: Id1161bca8f23129202599dba299c288a6aa29212
This removes the groups servers from our inventory as well as our
manifests/modules. We don't run the groups service anymore as many
groups migrated to meetup.com independent of us and the others have
transitioned there.
Change-Id: I7cb76611e6d30e7189821923f36a38dec9ea7241
This is an initial host for testing opendev.org mirrors
Change-Id: I26b9ed1e21e2111f48bc7ecc384880c274eed213
Depends-On: https://review.opendev.org/660235
This impelements mirrors to live in the opendev.org namespace. The
implementation is Ansible native for deployment on a Bionic node.
The hostname prefix remains the same (mirrorXX.region.provider.) but
the groups.yaml splits the opendev.org mirrors into a separate group.
The matches in the puppet group are also updated so to not run puppet
on the hosts.
The kerberos and openafs client parts do not need any updating and
works on the Bionic host.
The hosts are setup to provision certificates for themselves from
letsencrypt. Note we've added a new handler for mirror nodes to use
that restarts apache on certificate issue/renewal.
The new "mirror" role is a port of the existing puppet mirror.pp. It
installs apache, sets up some modules, makes some symlinks, sets up a
cleanup cron job and installs the apache vhost configuration.
The vhost configuration is also ported from the extant puppet. It is
simplified somewhat; but the biggest change is that we have extracted
the main port 80 configuration into a macro which is applied to both
port 80 and 443; i.e. the host will have SSL support. The other ports
are left alone for now, but can be updated in due course.
Thus we should be able to CNAME the existing mirrors to new nodes, and
any existing http access can continue. We can update our mirror setup
scripts to point to https resources as appropriate.
Change-Id: Iec576d631dd5b02f6b9fb445ee600be060f9cf1e
We're not really using/maintaining this at the moment. Before we do
put it back in production, we're likely to simply rebuild it from
scratch.
Change-Id: I469f00e90903a010f2cec45031b049556eb268a2