We previously had two manually issued certs (one each for opendev.org
and openstack.org) but now have a single cert with all the appropriate
names in it automatically issued by LE. Use this new cert before the old
one expires.
Change-Id: I635d2bfd820fe138ee951833dd66f157b2b7c097
* migrated user storage to idp
* created users crud
* created groups crud
* migrated from eloquent to doctrine orm
* reafactoring
Change-Id: I766bbb75c0e65f504880e8c59951f63494a1e13f
Signed-off-by: smarcet <smarcet@gmail.com>
Removed all variables related to Silverstripe
Dependency
Change-Id: Ib5e6834686c4952dd8e7220a31abe71a9278e397
Signed-off-by: smarcet <smarcet@gmail.com>
This reverts commit c25e91f496.
This script parses the Apache logs and writes out a local count of the
404 data to files.openstack.org, and then exports it via
files.openstack.org.
As part of the spec [1] we're trying to remove publishing from local
volumes, in general.
Since this is not widely used, there is only one link to it, it's not
discoverable from the landing page of files.openstack.org (which just
shows the afs directory listing), it has a very long latency making it
not that useful for debugging and grepping the logs there have been no
accesses in the past 2 weeks (as far back as logs go) I propose we
remove it.
If we want to retain this, we should publish the output alongside the
docs AFS volume. That could certainly be done by distributing the
docs keytab to the host and having it write out in a similar cron job.
Another option could be to setup a keypair for remote login and keep
that as a secret in Zuul, and do the same from a periodic job
(complicated by apache logs being root only, so needs some sudo magic
or similar). Or, we could figure out an altogether better, privacy
respecting client analytics solution.
[1] https://docs.opendev.org/opendev/infra-specs/latest/specs/retire-static.html
Depends-On: https://review.opendev.org/709036
Change-Id: Iccf24a72cf82592bae8c699f9f857aa54fc74f10
This migrates the afsmon script from puppet deploying on
mirror-update.openstack.org to ansible deploying on
mirror-update.opendev.org.
There is nothing particularly special and this just a straight install
with some minor dependencies. Since we have log publishing running on
the opendev.org server, we publish the update logs alongside the
others.
Change-Id: Ifa3b4d59f8d0fc23a4492e50348bab30766d5779
This follows-on from I62ae941e70c7d58e00bc663a50d52e79dfa5a684 to
remove the old publishing job from the afs server and enable live
publishing in the cron job on mirror-update.opendev.org.
Change-Id: Ib6cc094e6f02b513c5218264657cec9064fe867c
We have been running out of disk recently with some indexes requiring
more than 400GB of space per index replica. Actual disk space
requirements are double that as we run with a replica. On top of that
the idea is that 5 of 6 elasticsearch nodes have enough space for all
our data so that we are resilient to losing a node.
Napkin math:
400 * 10 * 2 = ~8TB of disk
400 * 7 * 2 = ~5.6TB of disk
Each of the six ES nodes has 1TB of disk allocated to ES so 5.6TB should
get us just under the limit. Then for handling a node outage weekends
tend to not have as many records so our actual usage should be a little
lower.
Change-Id: Ie677bd47a9886870bc83876d2407742133299861
yui-compressor is not happy with flot's jquery.flot.js file. These files
are actually pretty small especially when compared to our input json
data. Lets just serve them as is.
Depends-On: https://review.opendev.org/704716
Change-Id: Ibfd081bb73a6c352798a7822ab781c972ace4bc3
Flot sources moved from the repo top level dir into the source dir.
Accomodate this when we minimize and copy those js files.
Change-Id: I3522271361fc43550ac1c6dc2a690c5cc5ce9c64
The review-dev service playbook should do everything now that
the puppet did. Update how we're running things.
Change-Id: I70303c48328ea6713c24bf9c6f63d4808d30b95c
We were setting the cert file contents to the paths rather than updating
the paths to point at the new LE certs. Fix this by setting the _file
vars which update the path.
This includes a partial revert of the previous change to not switch
git.zuul-ci.org over to LE as we haven't provisioned an LE cert for it
yet.
Change-Id: I41c2aa1d03afba4ebf6378e9abf8276154666df7
This switches the zuul-ci.org/zuulci.org vhost to use newly issued
letsencrypt certs. It also does the same for git.zuul-ci.org, which
is a different vhost. Since that vhost is tied into a configuration
which can't accept cert file paths (only content), adjust it to use
the newer "website" manifest pattern which can.
Change-Id: I0cd0407754466327147917390c578da336e61269
The Airship working committee is in the process of establishing a
vulnerability management process. This change adds two mailing lists,
airship-security and embargo-notice, which will provide public and
private advisories related to reported security vulnerabilities.
Change-Id: I1aa4d35cb12e4f3f45665688908af7e2cd5041a1
Signed-off-by: Drew Walters <andrew.walters@att.com>
When apache2 gets reloaded multiple times in quick succession, it may
crash and fail completely. Lately this has been seen very often on our
ask.openstack.org instance, so let us use the more intrusive, but also
hopefully more stable in the end result method of restarting instead.
Change-Id: I44e4561f8696415471f65b75d683c48636fb413f
Ceph Nautilus is released and the official mirror
is available. This adds the Ceph Nautilus mirror
so we can sync it for Stretch and Bionic.
Based on the same change that was done when Mimic
was released [1]
[1] https://review.opendev.org/#/c/571989/
Change-Id: I9424d1f4df58acde8ea70dc16283d4de89189bae
Sharing an updates file between the Debian and Ubuntu reprepro runs
causes some warnings, and is generally just unclean. They use
different release naming and repositories, so should just have
separate updates files to track them (they're already separate on
the server, they were just being copied from the same source file in
the module).
While here, remove the label and suite parameters from the Debian
reprepro distribution templates, as they're unnecessary and
potentially confusing (job nodes should never be relying on the
suite names as they change at the next release).
Also allow signatures from subkeys of the listed keys to be
sufficient to verify the debian-security mirror's release files,
like we do for the debian mirror.
Change-Id: Id0ff476864f936bbd7c4637f3dc9e2c219c6e465
The elections-committee and openstack-content mailing lists have
been defunct for years. They receive no new posts and their
moderators have stopped watching them. The stakeholders in the
OpenStack Foundation have given the okay to have them closed down so
only their archives remain for historical reference.
Change-Id: Ie8230265518f4b114a34849b8b1d48b6a3675700
Full replication is very costly and makes gerrit restarts expensive
these days. Turn off replicate_on_startup.
Depends-On: https://review.opendev.org/678486
Change-Id: I31d81821c645697e72a8702c60e2482156e01bb0
Because of a limitation in GnuPG we need to have the Jessie archive
signing key in the list of VerifyRelease key IDs for the Debian
reprepro mirror. Also some suites (currently buster-backports) are
signed by a subkey of an archive signing key, so add the "+" suffix
to all these key IDs indicating subkey signatures are also allowed.
As always, Debian signing keys are published and available here:
https://ftp-master.debian.org/keys.html
Change-Id: Iedce38318718a18ace7b2c638755a7d7d4dcd69b
When a job is killed by zuul due to failure like DISK_FULL, a different
message ends up in as a comment.
<li>job-name
finger://ze09.openstack.org/8b6d...6f : DISK_FULL in 2h 59m 50s</li>
This adds another pattern that recognize these messages as failures,
regardess the case (DISK_FULL in this case).
Change-Id: Ib17f05a043430362b02a2826d69572f6b2dbd64a
Needed-By: https://review.opendev.org/#/c/631509/
The buster-backports suite on Debian mirrors is not signed by the
old jessie signing key we have set to verify in reprepro, but also
we're not mirroring Debian 8/jessie any longer anyway. Replace that
list with the 9/stretch and 10/buster signing keys and switch to
longer key IDs which match the names used for them in the Puppet
manifest. Also add Puppetry and keyfile for the buster keys so that
they will be installed accordingly. The official list of keys can be
found here: https://ftp-master.debian.org/keys.html
Change-Id: Ia193f040b2b707329948955eb091a186eabf8096
This rsync'd mirror is now being managed by the opendev mirror update
server. Remove it from the older openstack server to avoid a conflict in
excludes around sclo repo.
Currently we have opendev adding sclo and openstack removing it.
Change-Id: I599ee7d0fab8c5e2a060aff86bce20f1f8d4f54b
Zuul has hit a scenario where a git repo update was unable to talk to
gerrit via ssh because it had reached its per user connection limit [0].
This then led to some openstack job failing [1].
The default limit (which we were using) is 64 connection per user.
Apparently this is not quite enough for a busy zuul? Increase this by
50% up to 96.
[0] http://paste.openstack.org/show/754741/
[1] http://lists.openstack.org/pipermail/release-job-failures/2019-July/001193.html
Change-Id: Ibeca2208485608f3b61aa716184165342bfcc3c9
This used to be mirrored, however there were issues when upstream
dropped the PC1 repositories a few months back. The puppet openstack
jobs are still trying to leverage this mirror but it does not exist in
some regions because it was disabled on the afs content. This change
fixes the reprepo configuration to still pull down puppet5/6 for xenial
and strech and add the symlink back to the mirrors.
Change-Id: I71ad5afe086a503d75a365543ad8869e35ef873b
This move was prompted by wishing to expose the mirror update logs for
the rsync updates so that debugging problems does not require a root
user (note: not actually done in this change; will be a follow-on).
Rather than start hacking at puppet, the rsync mirror scripts make a
nice delination point for starting an Ansible-first/Bionic update.
Most magic is included in the scripts, so there is not much more to do
than copy them. The host uses the existing kerberos and openafs roles
and copies the key material into place (to be added before merge).
Note the scripts are removed from the extant puppet so we don't have
two updates happening simultaneously. This will also require a manual
clean to remove the cron jobs as a once-off when merging.
The other part of mirror-update is the reprepro based scripts for the
various debuntu repositories. They are left as future work for now.
Testing is added to ensure dependencies and scripts are all in place.
Change-Id: I525ac18b55f0e11b0a541b51fa97ee5d6512bf70