This adds roles to implement backup with borg [1].
Our current tool "bup" has no Python 3 support and is not packaged for
Ubuntu Focal. This means it is effectively end-of-life. borg fits
our model of servers backing themselves up to a central location, is
well documented and seems well supported. It also has the clarkb seal
of approval :)
As mentioned, borg works in the same manner as bup by doing an
efficient back up over ssh to a remote server. The core of these
roles are the same as the bup based ones; in terms of creating a
separate user for each host and deploying keys and ssh config.
This chooses to install borg in a virtualenv on /opt. This was chosen
for a number of reasons; firstly reading the history of borg there
have been incompatible updates (although they provide a tool to update
repository formats); it seems important that we both pin the version
we are using and keep clients and server in sync. Since we have a
hetrogenous distribution collection we don't want to rely on the
packaged tools which may differ. I don't feel like this is a great
application for a container; we actually don't want it that isolated
from the base system because it's goal is to read and copy it offsite
with as little chance of things going wrong as possible.
Borg has a lot of support for encrypting the data at rest in various
ways. However, that introduces the possibility we could lose both the
key and the backup data. Really the only thing stopping this is key
management, and if we want to go down this path we can do it as a
follow-on.
The remote end server is configured via ssh command rules to run in
append-only mode. This means a misbehaving client can't delete its
old backups. In theory we can prune backups on the server side --
something we could not do with bup. The documentation has been
updated but is vague on this part; I think we should get some hosts in
operation, see how the de-duplication is working out and then decide
how we want to mange things long term.
Testing is added; a focal and bionic host both run a full backup of
themselves to the backup server. Pretty cool, the logs are in
/var/log/borg-backup-<host>.log.
No hosts are currently in the borg groups, so this can be applied
without affecting production. I'd suggest the next steps are to bring
up a borg-based backup server and put a few hosts into this. After
running for a while, we can add all hosts, and then deprecate the
current bup-based backup server in vexxhost and replace that with a
borg-based one; giving us dual offsite backups.
[1] https://borgbackup.readthedocs.io/en/stable/
Change-Id: I2a125f2fac11d8e3a3279eb7fa7adb33a3acaa4e
This uses the Grafana container created with
Iddfafe852166fe95b3e433420e2e2a4a6380fc64 to run the
grafana.opendev.org service.
We retain the old model of an Apache reverse-proxy; it's well tested
and understood, it's much easier than trying to map all the SSL
termination/renewal/etc. into the Grafana container and we don't have
to convince ourselves the container is safe to be directly web-facing.
Otherwise this is a fairly straight forward deployment of the
container. As before, it uses the graph configuration kept in
project-config which is loaded in with grafyaml, which is included in
the container.
Once nice advantage is that it makes it quite easy to develop graphs
locally, using the container which can talk to the public graphite
instance. The documentation has been updated with a reference on how
to do this.
Change-Id: I0cc76d29b6911aecfebc71e5fdfe7cf4fcd071a4
This is a new Focal based host, which we want for it's more recent
rsync which hopefully causes less issues resyncing things to AFS
volumes.
See 4918594aa472010a8a112f5f4ed0a471a3351a91 for discussion of the
original issues; we have found that without "-t" all new data seems to
be copied continuously. Empirical testing shows later rsync doesn't
have this issue.
Depends-On: https://review.opendev.org/736859
Change-Id: Iebfffdf8aea6f123e36f264c87d6775771ce2dd8
We've got a section on using the emergency file and disabled ansible
group. Add info about the special DISABLE-ANSIBLE file there to help
make that info easier to find.
Change-Id: I2e750b9b87ca7a4f800d3ac161a195d49543a7da
refstack was moved to osf namespace, update documentation.
Refstack also uses storyboard, update the link.
Change-Id: Id47017eee7a4d1e864c2f0369c73f7047c2df6cd
Two mistypings of the string "ansible" (case insensitive) as
"ansbile" appear in our documentation. One of them is in a sample
command, which is particularly dangerous. Correct both.
Change-Id: Ib644f57060f467d4bfd70be60225e39385d38737
We've stopped using many of these, but we never got around to
removing them from lists.
Also, we should probably retire the repos.
Depends-On: https://review.opendev.org/717620
Depends-On: https://review.opendev.org/720527
Change-Id: I8e012c5bfa48d274dbd7f5484a9e75fee080cb5e
Make inventory/service for service-specific things, including the
groups.yaml group definitions, and inventory/base for hostvars
related to the base system, including the list of hosts.
Move the exisitng host_vars into inventory/service, since most of
them are likely service-specific. Move group_vars/all.yaml into
base/group_vars as almost all of it is related to base things,
with the execption of the gerrit public key.
A followup patch will move host-specific values into equivilent
files in inventory/base.
This should let us override hostvars in gate jobs. It should also
allow us to do better file matchers - and to be able to organize
our playbooks move if we want to.
Depends-On: https://review.opendev.org/731583
Change-Id: Iddf57b5be47c2e9de16b83a1bc83bee25db995cf
We are trying to use this file in our docs config but the file was
mistakently not added in that change. Add it now.
Change-Id: I8f5f9d62f96d8532477c42a7076c57aa6548c9cf
In places like crontab entries we use full paths to executables because
PATH is different under cron. Unfortunately, this meant we broke
docker-compose commands using /usr/bin/docker-compose when we installed
it under /usr/local/bin/docker-compose. In particular this impacted
database backups on gitea nodes and etherpad.
Update these paths so that everything is happy again.
Change-Id: Ib001baab419325ef1a43ac8e3364e755a6655617
We've got some old out of date docs in some places. This isn't even
a full reworking, but at least tries to remove some of the more
egregiously wrong things.
Change-Id: I9033acb9572e1ce1b3e4426564b92706a4385dcb
We use project-config for gerrit, gitea and nodepool config. That's
cool, because can clone that from zuul too and make sure that each
prod run we're doing runs with the contents of the patch in question.
Introduce a flag file that can be touched in /home/zuulcd that will
block zuul from running prod playbooks. By default, if the file is
there, zuul will wait for an hour before giving up.
Rename zuulcd to zuul
To better align prod and test, name the zuul user zuul.
Change-Id: I83c38c9c430218059579f3763e02d6b9f40c7b89
We had the clouds split from back when we used the openstack
dynamic inventory plugin. We don't use that anymore, so we don't
need these to be split. Any other usage we have directly references
a cloud.
Change-Id: I5d95bf910fb8e2cbca64f92c6ad4acd3aaeed1a3
These are OpenDev docs now so the OpenStack theming doesn't quite fit.
Switch to Alabaster + OpenDev logo which is what we did with
infra-manual.
Change-Id: Id211e8e0b4dab7282fb5ca5fce494a028a826fba
The OpenDev community is moving its discussions off the old
openstack-infra mailing list, so make sure to refer to the correct
new address(es).
Change-Id: I558b60ea0aa3421285d46be449f04198441cf285
These are useful for the times when a secret needs to be decrypted
for debugging but seem to have been deleted when we did the zuulv3
migration removal.
Change-Id: Ib1544d9032df9bd25c50eeca032f643e40f035b0
Update conf.py and index.rst for OpenDev.
Use newer openstackdocstheme and update conf.py for this.
Change-Id: I62312ca1d3fda9221660b7bb664c8ea55dac68a4
There are two different concerns here. One is configuring the gitea
and gerrit services. This is independent from the management of
projects running inside them.
Make a manage-projects playbook which currently runs gitea-git-repos
but will also get a gerrit-git-repos role in a bit. Make a
service-gitea playbook for deploying gitea itself and update
run_all to take all of that into account. This should make our
future world of turning these into zuul jobs easier.
Add several missing files to the files matchers for run-gitea
and run-review.
Also - nothing about this has anything to do with puppet.
Change-Id: I5eaf75129d76138c61013a3a7ed7c381d567bb8b
We keep track of these files now in the opendev/project-config repo,
so make sure that they are committed there.
Change-Id: Icf4b4e32ac4f209811ba8361bbb9d8458c79251a
With the move from OpenStack governance to our own OpenDev team, we
should also move to use the #opendev IRC channel in preference to
the #openstack-infra channel which will remain in use for OpenStack
specific discussions.
Update the references in our docs accordingly.
Change-Id: I448704f5d2664fd233a69a2ad12578ca24d9878a
Based on #openstack-infra talks from Feb 17th, I am proposing some clarifications regarding how logging should
be done by 3rd party CI.
These should help 3rd party integrators create a better
experience for developers, making logs more accessible.
Change-Id: I2ebc788505ba1319afc038d0aa1406da3823a911
This change effectively converts the OpenStack Infra project description
into an OpenDev project description in our documentation. Since OpenDev
is largely an evolution of the preexisting infra team much of the
content remains the same. I have added a section on governance as we'll
not be able to run off of the OpenStack governance any longer.
Note this leaves what becomes the OpenStack Infra project without a
project document. However, the remaining scope of that OpenStack project
will be small and I don't think it will need to same level of team
organization. I think we can get by with OpenStack's default governance
for its teams there. Then should we need something more explicit or
different we can write that up within openstack itself.
Depends-On: https://review.opendev.org/#/c/703134/
Change-Id: I56aab771510768211386325e6466d2f94fe298fb
