Update the backup instructions for some recent changes. Make a note
of the streaming backup method, discuss some caveats with append-only
mode and discuss the pruning scripts and when to run
(c.f. I9559bb8aeeef06b95fb9e172a2c5bfb5be5b480e,
I250d84c4a9f707e63fef6f70cfdcc1fb7807d3a7).
Change-Id: Idb04ebfa5666cd3c20bc0132683d187e705da3f1
The hound project has undergone a small re-birth and moved to
https://github.com/hound-search/hound
which has broken our deployment. We've talked about leaving
codesearch up to gitea, but it's not quite there yet. There seems to
be no point working on the puppet now.
This builds a container than runs houndd. It's an opendev specific
container; the config is pulled from project-config directly.
There's some custom scripts that drive things. Some points for
reviewers:
- update-hound-config.sh uses "create-hound-config" (which is in
jeepyb for historical reasons) to generate the config file. It
grabs the latest projects.yaml from project-config and exits with a
return code to indicate if things changed.
- when the container starts, it runs update-hound-config.sh to
populate the initial config. There is a testing environment flag
and small config so it doesn't have to clone the entire opendev for
functional testing.
- it runs under supervisord so we can restart the daemon when
projects are updated. Unlike earlier versions that didn't start
listening till indexing was done, this version now puts up a "Hound
is not ready yet" message when while it is working; so we can drop
all the magic we were doing to probe if hound is listening via
netstat and making Apache redirect to a status page.
- resync-hound.sh is run from an external cron job daily, and does
this update and restart check. Since it only reloads if changes
are made, this should be relatively rare anyway.
- There is a PR to monitor the config file
(https://github.com/hound-search/hound/pull/357) which would mean
the restart is unnecessary. This would be good in the near and we
could remove the cron job.
- playbooks/roles/codesearch is unexciting and deploys the container,
certificates and an apache proxy back to localhost:6080 where hound
is listening.
I've combined removal of the old puppet bits here as the "-codesearch"
namespace was already being used.
Change-Id: I8c773b5ea6b87e8f7dfd8db2556626f7b2500473
Add the FUSE dependencies for our hosts backed up with borg, along
with a small script to make mounting the backups easier. This is the
best way to recover something quickly in what is sure to be a
stressful situation.
Documentation and testing is updated.
Change-Id: I1f409b2df952281deedff2ff8f09e3132a2aff08
Our Gerrit admins follow this model of access management now, in
order to shield Administrators permission from external identity
provider risks.
Change-Id: I3070c28c26548d364da38d366bfa2ac8b2fb4668
We stopped serving this content and the next step is to stop managing it
internally. This depends on a change to jeepyb that makes the local git
dir management on the jeepyb side optional. Once that lands we can
update our configs to tell jeepyb to stop managing it.
We also stop doing garbage collection, mounting it into containers that
don't need it, etc.
Depends-On: https://review.opendev.org/758597
Change-Id: I2185e90edfcac71941bc29a4e11b7b2d4c7c2e13
This adds roles to implement backup with borg [1].
Our current tool "bup" has no Python 3 support and is not packaged for
Ubuntu Focal. This means it is effectively end-of-life. borg fits
our model of servers backing themselves up to a central location, is
well documented and seems well supported. It also has the clarkb seal
of approval :)
As mentioned, borg works in the same manner as bup by doing an
efficient back up over ssh to a remote server. The core of these
roles are the same as the bup based ones; in terms of creating a
separate user for each host and deploying keys and ssh config.
This chooses to install borg in a virtualenv on /opt. This was chosen
for a number of reasons; firstly reading the history of borg there
have been incompatible updates (although they provide a tool to update
repository formats); it seems important that we both pin the version
we are using and keep clients and server in sync. Since we have a
hetrogenous distribution collection we don't want to rely on the
packaged tools which may differ. I don't feel like this is a great
application for a container; we actually don't want it that isolated
from the base system because it's goal is to read and copy it offsite
with as little chance of things going wrong as possible.
Borg has a lot of support for encrypting the data at rest in various
ways. However, that introduces the possibility we could lose both the
key and the backup data. Really the only thing stopping this is key
management, and if we want to go down this path we can do it as a
follow-on.
The remote end server is configured via ssh command rules to run in
append-only mode. This means a misbehaving client can't delete its
old backups. In theory we can prune backups on the server side --
something we could not do with bup. The documentation has been
updated but is vague on this part; I think we should get some hosts in
operation, see how the de-duplication is working out and then decide
how we want to mange things long term.
Testing is added; a focal and bionic host both run a full backup of
themselves to the backup server. Pretty cool, the logs are in
/var/log/borg-backup-<host>.log.
No hosts are currently in the borg groups, so this can be applied
without affecting production. I'd suggest the next steps are to bring
up a borg-based backup server and put a few hosts into this. After
running for a while, we can add all hosts, and then deprecate the
current bup-based backup server in vexxhost and replace that with a
borg-based one; giving us dual offsite backups.
[1] https://borgbackup.readthedocs.io/en/stable/
Change-Id: I2a125f2fac11d8e3a3279eb7fa7adb33a3acaa4e
This uses the Grafana container created with
Iddfafe852166fe95b3e433420e2e2a4a6380fc64 to run the
grafana.opendev.org service.
We retain the old model of an Apache reverse-proxy; it's well tested
and understood, it's much easier than trying to map all the SSL
termination/renewal/etc. into the Grafana container and we don't have
to convince ourselves the container is safe to be directly web-facing.
Otherwise this is a fairly straight forward deployment of the
container. As before, it uses the graph configuration kept in
project-config which is loaded in with grafyaml, which is included in
the container.
Once nice advantage is that it makes it quite easy to develop graphs
locally, using the container which can talk to the public graphite
instance. The documentation has been updated with a reference on how
to do this.
Change-Id: I0cc76d29b6911aecfebc71e5fdfe7cf4fcd071a4
This is a new Focal based host, which we want for it's more recent
rsync which hopefully causes less issues resyncing things to AFS
volumes.
See 4918594aa472010a8a112f5f4ed0a471a3351a91 for discussion of the
original issues; we have found that without "-t" all new data seems to
be copied continuously. Empirical testing shows later rsync doesn't
have this issue.
Depends-On: https://review.opendev.org/736859
Change-Id: Iebfffdf8aea6f123e36f264c87d6775771ce2dd8
We've got a section on using the emergency file and disabled ansible
group. Add info about the special DISABLE-ANSIBLE file there to help
make that info easier to find.
Change-Id: I2e750b9b87ca7a4f800d3ac161a195d49543a7da
refstack was moved to osf namespace, update documentation.
Refstack also uses storyboard, update the link.
Change-Id: Id47017eee7a4d1e864c2f0369c73f7047c2df6cd
Two mistypings of the string "ansible" (case insensitive) as
"ansbile" appear in our documentation. One of them is in a sample
command, which is particularly dangerous. Correct both.
Change-Id: Ib644f57060f467d4bfd70be60225e39385d38737
We've stopped using many of these, but we never got around to
removing them from lists.
Also, we should probably retire the repos.
Depends-On: https://review.opendev.org/717620
Depends-On: https://review.opendev.org/720527
Change-Id: I8e012c5bfa48d274dbd7f5484a9e75fee080cb5e
Make inventory/service for service-specific things, including the
groups.yaml group definitions, and inventory/base for hostvars
related to the base system, including the list of hosts.
Move the exisitng host_vars into inventory/service, since most of
them are likely service-specific. Move group_vars/all.yaml into
base/group_vars as almost all of it is related to base things,
with the execption of the gerrit public key.
A followup patch will move host-specific values into equivilent
files in inventory/base.
This should let us override hostvars in gate jobs. It should also
allow us to do better file matchers - and to be able to organize
our playbooks move if we want to.
Depends-On: https://review.opendev.org/731583
Change-Id: Iddf57b5be47c2e9de16b83a1bc83bee25db995cf
We are trying to use this file in our docs config but the file was
mistakently not added in that change. Add it now.
Change-Id: I8f5f9d62f96d8532477c42a7076c57aa6548c9cf
In places like crontab entries we use full paths to executables because
PATH is different under cron. Unfortunately, this meant we broke
docker-compose commands using /usr/bin/docker-compose when we installed
it under /usr/local/bin/docker-compose. In particular this impacted
database backups on gitea nodes and etherpad.
Update these paths so that everything is happy again.
Change-Id: Ib001baab419325ef1a43ac8e3364e755a6655617
We've got some old out of date docs in some places. This isn't even
a full reworking, but at least tries to remove some of the more
egregiously wrong things.
Change-Id: I9033acb9572e1ce1b3e4426564b92706a4385dcb
We use project-config for gerrit, gitea and nodepool config. That's
cool, because can clone that from zuul too and make sure that each
prod run we're doing runs with the contents of the patch in question.
Introduce a flag file that can be touched in /home/zuulcd that will
block zuul from running prod playbooks. By default, if the file is
there, zuul will wait for an hour before giving up.
Rename zuulcd to zuul
To better align prod and test, name the zuul user zuul.
Change-Id: I83c38c9c430218059579f3763e02d6b9f40c7b89
We had the clouds split from back when we used the openstack
dynamic inventory plugin. We don't use that anymore, so we don't
need these to be split. Any other usage we have directly references
a cloud.
Change-Id: I5d95bf910fb8e2cbca64f92c6ad4acd3aaeed1a3
These are OpenDev docs now so the OpenStack theming doesn't quite fit.
Switch to Alabaster + OpenDev logo which is what we did with
infra-manual.
Change-Id: Id211e8e0b4dab7282fb5ca5fce494a028a826fba
The OpenDev community is moving its discussions off the old
openstack-infra mailing list, so make sure to refer to the correct
new address(es).
Change-Id: I558b60ea0aa3421285d46be449f04198441cf285
These are useful for the times when a secret needs to be decrypted
for debugging but seem to have been deleted when we did the zuulv3
migration removal.
Change-Id: Ib1544d9032df9bd25c50eeca032f643e40f035b0
Update conf.py and index.rst for OpenDev.
Use newer openstackdocstheme and update conf.py for this.
Change-Id: I62312ca1d3fda9221660b7bb664c8ea55dac68a4
There are two different concerns here. One is configuring the gitea
and gerrit services. This is independent from the management of
projects running inside them.
Make a manage-projects playbook which currently runs gitea-git-repos
but will also get a gerrit-git-repos role in a bit. Make a
service-gitea playbook for deploying gitea itself and update
run_all to take all of that into account. This should make our
future world of turning these into zuul jobs easier.
Add several missing files to the files matchers for run-gitea
and run-review.
Also - nothing about this has anything to do with puppet.
Change-Id: I5eaf75129d76138c61013a3a7ed7c381d567bb8b
We keep track of these files now in the opendev/project-config repo,
so make sure that they are committed there.
Change-Id: Icf4b4e32ac4f209811ba8361bbb9d8458c79251a
