Commit Graph

3042 Commits

Author SHA1 Message Date
smarcet
d138243bd5 OpenStackId production release 3.0.5
Change-Id: I8d2e2d96ba3543de6b47c5001f9bc2ebab115286
Signed-off-by: smarcet <smarcet@gmail.com>
2020-03-20 17:28:04 -03:00
Monty Taylor
b8f9cc40aa Pin lodgeit to the current version
So we can be careful about landing lodgeit changes.

Depends-On: https://review.opendev.org/711344
Change-Id: If1ae66fc94d5ceed458b93cb10f0bb061df85021
2020-03-04 23:54:41 +00:00
Clark Boylan
61caec5b77 Use LE cert on review.open*.org
We previously had two manually issued certs (one each for opendev.org
and openstack.org) but now have a single cert with all the appropriate
names in it automatically issued by LE. Use this new cert before the old
one expires.

Change-Id: I635d2bfd820fe138ee951833dd66f157b2b7c097
2020-02-28 08:10:24 -08:00
smarcet
eed9116625 OpenStackId v3.0.4 Deployment
Fixed inconsistent ORM mappings

Change-Id: I1806f0c22a21ebf495aa725998ba0e0e57bf3b4a
Signed-off-by: smarcet <smarcet@gmail.com>
2020-02-26 16:58:58 -03:00
smarcet
19662d7d3f OpenStackId v3.0.3 Deployment
Change-Id: I239f8311139d59260430baee0c6cb910c31caf3f
Signed-off-by: smarcet <smarcet@gmail.com>
2020-02-25 22:21:20 -03:00
Zuul
9566bd0e2a Merge "Revert "Generate list of 404s for docs.o.o"" 2020-02-25 23:39:02 +00:00
smarcet
88b26edf24 OpenStackId v3.0.1 deployment
Change-Id: I8a49dbde1bbcd935b380f003160040d634b2062c
Signed-off-by: smarcet <smarcet@gmail.com>
2020-02-25 15:35:17 -03:00
smarcet
09f35f14bf OpenStackID v3.0.0
* migrated user storage to idp
* created users crud
* created groups crud
* migrated from eloquent to doctrine orm
* reafactoring

Change-Id: I766bbb75c0e65f504880e8c59951f63494a1e13f
Signed-off-by: smarcet <smarcet@gmail.com>
2020-02-25 09:45:06 -03:00
smarcet
ce6de87ea7 Removed OpenStackID stale config variables
Removed all variables related to Silverstripe
Dependency

Change-Id: Ib5e6834686c4952dd8e7220a31abe71a9278e397
Signed-off-by: smarcet <smarcet@gmail.com>
2020-02-24 14:26:30 -03:00
Ian Wienand
55da1e3d06 Revert "Generate list of 404s for docs.o.o"
This reverts commit c25e91f496.

This script parses the Apache logs and writes out a local count of the
404 data to files.openstack.org, and then exports it via
files.openstack.org.

As part of the spec [1] we're trying to remove publishing from local
volumes, in general.

Since this is not widely used, there is only one link to it, it's not
discoverable from the landing page of files.openstack.org (which just
shows the afs directory listing), it has a very long latency making it
not that useful for debugging and grepping the logs there have been no
accesses in the past 2 weeks (as far back as logs go) I propose we
remove it.

If we want to retain this, we should publish the output alongside the
docs AFS volume.  That could certainly be done by distributing the
docs keytab to the host and having it write out in a similar cron job.
Another option could be to setup a keypair for remote login and keep
that as a secret in Zuul, and do the same from a periodic job
(complicated by apache logs being root only, so needs some sudo magic
or similar).  Or, we could figure out an altogether better, privacy
respecting client analytics solution.

[1] https://docs.opendev.org/opendev/infra-specs/latest/specs/retire-static.html

Depends-On: https://review.opendev.org/709036
Change-Id: Iccf24a72cf82592bae8c699f9f857aa54fc74f10
2020-02-24 14:43:11 +11:00
Zuul
30297fb10d Merge "Migrate AFS publishing to mirror-update.opendev.org" 2020-02-12 22:40:43 +00:00
Ian Wienand
97c4735129 Move afsmon to mirror-update.opendev.org
This migrates the afsmon script from puppet deploying on
mirror-update.openstack.org to ansible deploying on
mirror-update.opendev.org.

There is nothing particularly special and this just a straight install
with some minor dependencies.  Since we have log publishing running on
the opendev.org server, we publish the update logs alongside the
others.

Change-Id: Ifa3b4d59f8d0fc23a4492e50348bab30766d5779
2020-02-12 14:38:48 +11:00
Ian Wienand
6991905e52 Migrate AFS publishing to mirror-update.opendev.org
This follows-on from I62ae941e70c7d58e00bc663a50d52e79dfa5a684 to
remove the old publishing job from the afs server and enable live
publishing in the cron job on mirror-update.opendev.org.

Change-Id: Ib6cc094e6f02b513c5218264657cec9064fe867c
2020-02-11 16:14:43 +11:00
Clark Boylan
bd752a0bfe Keep only 7 days of records in ElasticSearch
We have been running out of disk recently with some indexes requiring
more than 400GB of space per index replica. Actual disk space
requirements are double that as we run with a replica. On top of that
the idea is that 5 of 6 elasticsearch nodes have enough space for all
our data so that we are resilient to losing a node.

Napkin math:

  400 * 10 * 2 = ~8TB of disk
  400 * 7 * 2 = ~5.6TB of disk

Each of the six ES nodes has 1TB of disk allocated to ES so 5.6TB should
get us just under the limit. Then for handling a node outage weekends
tend to not have as many records so our actual usage should be a little
lower.

Change-Id: Ie677bd47a9886870bc83876d2407742133299861
2020-02-06 13:50:56 -08:00
Clark Boylan
8b51cc616a Copy unminimized flot files
yui-compressor is not happy with flot's jquery.flot.js file. These files
are actually pretty small especially when compared to our input json
data. Lets just serve them as is.

Depends-On: https://review.opendev.org/704716
Change-Id: Ibfd081bb73a6c352798a7822ab781c972ace4bc3
2020-01-28 16:41:07 -08:00
Clark Boylan
89b15fec21 Flot sources moved, update our minimization to accomodate
Flot sources moved from the repo top level dir into the source dir.
Accomodate this when we minimize and copy those js files.

Change-Id: I3522271361fc43550ac1c6dc2a690c5cc5ce9c64
2020-01-27 15:38:39 -08:00
Zuul
e00dd724c6 Merge "Add mailing list for OpenInfra Labs" 2020-01-21 17:20:41 +00:00
Mohammed Naser
4b90ba9b4f Add mailing list for OpenInfra Labs
This creates a new mailing list under OpenDev for discussion
around OpenInfra Labs.

Change-Id: I6754df15b7cd205fc9a9d6bc1ace5cbcd65faa84
2020-01-17 13:03:35 -05:00
Zuul
06096940e2 Merge "Revert "Enable gerrit replication.autoReload for review"" 2020-01-15 22:56:22 +00:00
Monty Taylor
6f3a2792cc Switch to ansible on review-dev
The review-dev service playbook should do everything now that
the puppet did. Update how we're running things.

Change-Id: I70303c48328ea6713c24bf9c6f63d4808d30b95c
2020-01-14 12:04:15 -06:00
James E. Blair
447b8513ea Use LE cert for git.zuul-ci.org
The cert has been issued.

This partially reverts commit 42c0d0696c.

Change-Id: I5f6c9d648a6fdfc3f9464c01f51d95c5cd72ed49
2020-01-08 16:14:26 -08:00
Clark Boylan
42c0d0696c Fix zuul-ci.org vhost cert paths
We were setting the cert file contents to the paths rather than updating
the paths to point at the new LE certs. Fix this by setting the _file
vars which update the path.

This includes a partial revert of the previous change to not switch
git.zuul-ci.org over to LE as we haven't provisioned an LE cert for it
yet.

Change-Id: I41c2aa1d03afba4ebf6378e9abf8276154666df7
2020-01-08 10:03:05 -08:00
Zuul
44ca7193ea Merge "UCA: mirror Ussuri packages" 2020-01-08 16:26:29 +00:00
Marcin Juszkiewicz
dbdb8801f3 UCA: mirror Ussuri packages
Change-Id: Ia00b269deee91dbf681a2f6c025ff2691ff6ce3d
2020-01-08 15:58:33 +01:00
James E. Blair
9fead30442 Use LE certs for zuul-ci.org
This switches the zuul-ci.org/zuulci.org vhost to use newly issued
letsencrypt certs.  It also does the same for git.zuul-ci.org, which
is a different vhost.  Since that vhost is tied into a configuration
which can't accept cert file paths (only content), adjust it to use
the newer "website" manifest pattern which can.

Change-Id: I0cd0407754466327147917390c578da336e61269
2020-01-07 15:25:32 -08:00
Drew Walters
8ccd9bf83b lists: Add Airship VMP mailing lists
The Airship working committee is in the process of establishing a
vulnerability management process. This change adds two mailing lists,
airship-security and embargo-notice, which will provide public and
private advisories related to reported security vulnerabilities.

Change-Id: I1aa4d35cb12e4f3f45665688908af7e2cd5041a1
Signed-off-by: Drew Walters <andrew.walters@att.com>
2019-12-05 16:59:50 +00:00
Zuul
5cabb8ca07 Merge "Retire elections-committee, openstack-content MLs" 2019-12-02 19:21:20 +00:00
Zuul
7975a8648b Merge "Retire the Women-of-OpenStack list" 2019-12-02 19:19:42 +00:00
Jens Harbott
1dcba08f5b Restart apache2 on logrotate on ask.o.o
When apache2 gets reloaded multiple times in quick succession, it may
crash and fail completely. Lately this has been seen very often on our
ask.openstack.org instance, so let us use the more intrusive, but also
hopefully more stable in the end result method of restarting instead.

Change-Id: I44e4561f8696415471f65b75d683c48636fb413f
2019-11-13 15:54:50 +00:00
Jeremy Stanley
e7c6b76026 Revert "Enable gerrit replication.autoReload for review"
What we observed is that if any replication tasks are queued when
the replication.conf file is reloaded, those tasks get dropped on
the ground. It has resulted in missed refs on mirrors when a
replication change was updated by config management while a full
replication was underway:

http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2019-04-16.log.html#t2019-04-16T01:59:10

This reverts commit 02feafa962.

Change-Id: I65611d433723fbfa1965ea3f7ae4cabab8d07eca
2019-10-26 00:31:53 +00:00
Zuul
5d7ad66895 Merge "Further split Debian and Ubuntu reprepro configs" 2019-10-24 17:35:34 +00:00
Zuul
b167038b3f Merge "Mirror Ceph Nautilus for Debian based" 2019-10-23 23:48:37 +00:00
Tobias Urdin
de519d5617 Mirror Ceph Nautilus for Debian based
Ceph Nautilus is released and the official mirror
is available. This adds the Ceph Nautilus mirror
so we can sync it for Stretch and Bionic.

Based on the same change that was done when Mimic
was released [1]

[1] https://review.opendev.org/#/c/571989/

Change-Id: I9424d1f4df58acde8ea70dc16283d4de89189bae
2019-10-24 09:07:30 +11:00
Jeremy Stanley
4f0342be70 Further split Debian and Ubuntu reprepro configs
Sharing an updates file between the Debian and Ubuntu reprepro runs
causes some warnings, and is generally just unclean. They use
different release naming and repositories, so should just have
separate updates files to track them (they're already separate on
the server, they were just being copied from the same source file in
the module).

While here, remove the label and suite parameters from the Debian
reprepro distribution templates, as they're unnecessary and
potentially confusing (job nodes should never be relying on the
suite names as they change at the next release).

Also allow signatures from subkeys of the listed keys to be
sufficient to verify the debian-security mirror's release files,
like we do for the debian mirror.

Change-Id: Id0ff476864f936bbd7c4637f3dc9e2c219c6e465
2019-10-15 20:21:12 +00:00
Jeremy Stanley
ec8b0682fa Retire elections-committee, openstack-content MLs
The elections-committee and openstack-content mailing lists have
been defunct for years. They receive no new posts and their
moderators have stopped watching them. The stakeholders in the
OpenStack Foundation have given the okay to have them closed down so
only their archives remain for historical reference.

Change-Id: Ie8230265518f4b114a34849b8b1d48b6a3675700
2019-09-16 17:13:34 +00:00
Amy Marrich (spotz)
0dac2f3f87 Retire the Women-of-OpenStack list
As WoO is now part of the D&I WG this list is not needed.

Change-Id: I6af5f537357d3523aaa2d4b2e673296687e33a7d
2019-09-16 12:00:45 -05:00
Zuul
f2095676d5 Merge "Recognize DISK_FULL failure messages (review_dev)" 2019-09-09 11:37:04 +00:00
Monty Taylor
df8a7e37f0 Don't run replication on gerrit startup
Full replication is very costly and makes gerrit restarts expensive
these days. Turn off replicate_on_startup.

Depends-On: https://review.opendev.org/678486
Change-Id: I31d81821c645697e72a8702c60e2482156e01bb0
2019-08-26 10:16:08 +02:00
Jeremy Stanley
5a096f3705 Re-add the Debian 8/jessie key to reprepro
Because of a limitation in GnuPG we need to have the Jessie archive
signing key in the list of VerifyRelease key IDs for the Debian
reprepro mirror. Also some suites (currently buster-backports) are
signed by a subkey of an archive signing key, so add the "+" suffix
to all these key IDs indicating subkey signatures are also allowed.
As always, Debian signing keys are published and available here:
https://ftp-master.debian.org/keys.html

Change-Id: Iedce38318718a18ace7b2c638755a7d7d4dcd69b
2019-08-02 19:04:25 +00:00
Sorin Sbarnea
3792315db5 Recognize DISK_FULL failure messages (review_dev)
When a job is killed by zuul due to failure like DISK_FULL, a different
message ends up in as a comment.

<li>job-name
finger://ze09.openstack.org/8b6d...6f : DISK_FULL in 2h 59m 50s</li>

This adds another pattern that recognize these messages as failures,
regardess the case (DISK_FULL in this case).

Change-Id: Ib17f05a043430362b02a2826d69572f6b2dbd64a
Needed-By: https://review.opendev.org/#/c/631509/
2019-08-01 11:47:09 +01:00
James E. Blair
96aec261da Add logs.opendev.org vhost
This is a near-copy of the vhost template from puppet-openstackci.

Change-Id: I191e41b501629e2cdd82381d66daa3b850e0be81
2019-07-31 14:32:00 -07:00
Jeremy Stanley
a22df8264f Add archive signing key for Debian 10/buster
The buster-backports suite on Debian mirrors is not signed by the
old jessie signing key we have set to verify in reprepro, but also
we're not mirroring Debian 8/jessie any longer anyway. Replace that
list with the 9/stretch and 10/buster signing keys and switch to
longer key IDs which match the names used for them in the Puppet
manifest. Also add Puppetry and keyfile for the buster keys so that
they will be installed accordingly. The official list of keys can be
found here: https://ftp-master.debian.org/keys.html

Change-Id: Ia193f040b2b707329948955eb091a186eabf8096
2019-07-30 17:41:07 +00:00
Clark Boylan
36c14e4325 Remove centos mirror from openstack mirror update
This rsync'd mirror is now being managed by the opendev mirror update
server. Remove it from the older openstack server to avoid a conflict in
excludes around sclo repo.

Currently we have opendev adding sclo and openstack removing it.

Change-Id: I599ee7d0fab8c5e2a060aff86bce20f1f8d4f54b
2019-07-24 08:11:44 -07:00
Clark Boylan
cb33dba40a Increate gerrit user connection limit by 50%
Zuul has hit a scenario where a git repo update was unable to talk to
gerrit via ssh because it had reached its per user connection limit [0].
This then led to some openstack job failing [1].

The default limit (which we were using) is 64 connection per user.
Apparently this is not quite enough for a busy zuul? Increase this by
50% up to 96.

[0] http://paste.openstack.org/show/754741/
[1] http://lists.openstack.org/pipermail/release-job-failures/2019-July/001193.html

Change-Id: Ibeca2208485608f3b61aa716184165342bfcc3c9
2019-07-22 15:29:19 -07:00
Alex Schultz
2f96a248c8 Add apt-puppetlabs mirroring back
This used to be mirrored, however there were issues when upstream
dropped the PC1 repositories a few months back. The puppet openstack
jobs are still trying to leverage this mirror but it does not exist in
some regions because it was disabled on the afs content. This change
fixes the reprepo configuration to still pull down puppet5/6 for xenial
and strech and add the symlink back to the mirrors.

Change-Id: I71ad5afe086a503d75a365543ad8869e35ef873b
2019-07-11 11:12:14 -06:00
Zuul
46be2ccae6 Merge "Mailing list for Airship" 2019-07-11 15:13:28 +00:00
Roman Gorshunov
d50e17e043 Mailing list for Airship
This creates mailing list
airship-job-failures@lists.airshipit.org for Airship project.

Change-Id: Ia354c0440ababe99705041c618db2b6ea24d1450
2019-07-03 11:51:15 +00:00
Zuul
5d0d5725ec Merge "Move rsync mirror updates to new opendev.org mirror-update host" 2019-07-03 00:44:04 +00:00
Ian Wienand
b85282c046 Move rsync mirror updates to new opendev.org mirror-update host
This move was prompted by wishing to expose the mirror update logs for
the rsync updates so that debugging problems does not require a root
user (note: not actually done in this change; will be a follow-on).

Rather than start hacking at puppet, the rsync mirror scripts make a
nice delination point for starting an Ansible-first/Bionic update.

Most magic is included in the scripts, so there is not much more to do
than copy them.  The host uses the existing kerberos and openafs roles
and copies the key material into place (to be added before merge).

Note the scripts are removed from the extant puppet so we don't have
two updates happening simultaneously.  This will also require a manual
clean to remove the cron jobs as a once-off when merging.

The other part of mirror-update is the reprepro based scripts for the
various debuntu repositories.  They are left as future work for now.

Testing is added to ensure dependencies and scripts are all in place.

Change-Id: I525ac18b55f0e11b0a541b51fa97ee5d6512bf70
2019-07-02 16:42:33 +10:00
Marcin Juszkiewicz
f830202eea do backports for Debian 'buster'
Change-Id: Id818034921fdf19ccdf67ef74803f97eb04e0acf
2019-06-28 14:05:38 +02:00