Our production jobs currently only put their logging locally on the
bastion host. This means that to help maintain a production system,
you effectively need full access to the bastion host to debug any
misbehaviour.
We've long discussed publishing these Ansible runs as public logs, or
via a reporting system (ARA, etc.) but, despite our best efforts at
no_log and similar, we are not 100% sure that secret values may not
leak.
This is the infrastructure for an in-between solution, where we
publish the production run logs encrypted to specific GPG public keys.
Here we are capturing and encrypting the logs of the
system-config-run-* jobs, and providing a small download script to
automatically grab and unencrypt the log files. Obviously this is
just to exercise the encryption/log-download path for these jobs, as
the logs are public.
Once this has landed, I will propose similar for the production jobs
(because these are post-pipeline this takes a bit more fiddling and
doens't run in CI). The variables will be setup in such a way that if
someone wishes to help maintain a production system, they can add
their public-key and then add themselves to the particular
infra-prod-* job they wish to view the logs for.
It is planned that the extant operators will be in the default list;
however this is still useful over the status quo -- instead of having
to search through the log history on the bastion host when debugging a
failed run, they can simply view the logs from the failing build in
Zuul directly.
Depends-On: https://review.opendev.org/c/zuul/zuul-jobs/+/828818/
Change-Id: I5b9f9dd53eb896bb542652e8175c570877842584
ccf00b7673