8569 Commits

Author SHA1 Message Date
Kevin Carter
8b0877cb68
Add proxy for quay
This change adds a proxy config for quay which should assist
us when gating using images provided by the publically
available registry.

Change-Id: I971705e59724e70bd9d42a6920cf4f883556f673
Signed-off-by: Kevin Carter <kecarter@redhat.com>
2019-09-19 16:49:20 -05:00
Jeremy Stanley
72b3e36bf3 Add several missing ssldomains to certcheck config
There have been several Web sites added to files.o.o which missed
getting x509 SSL/TLS certificate checking added through our
certcheck cron job. Add those now so we know in advance whether
they're at risk of expiration.

Change-Id: I3eda77f165348e510d43344b172cf5b56ce2b003
2019-09-11 18:34:36 +00:00
Zuul
f2095676d5 Merge "Recognize DISK_FULL failure messages (review_dev)" 2019-09-09 11:37:04 +00:00
Zuul
97e6d1d4af Merge "Don't run replication on gerrit startup" 2019-09-05 18:55:21 +00:00
Ian Wienand
5686508716 Filter docker directories out of cacti
Docker hosts report back mounts in container directories via snmp
storage queries

 # php -q /usr/share/cacti/cli/add_graphs.php --host-id=585 --snmp-field=hrStorageDescr --list-snmp-values
 Known values for hrStorageDescr for host 585: (name)
 ...
  /var/lib/docker/containers/05ed2dc...
  /var/lib/docker/containers/7cebed4...
  /var/lib/docker/containers/f452861...

Because these can keep changing, hosts just end up getting more and
more invalid graphs in their results (see gitea0X hosts in cacti at
the moment).

Filter out docker directories from the query

Change-Id: Ia1db628975e7a67ad531438ef85735abae1ce652
2019-08-27 15:54:49 +10:00
Monty Taylor
df8a7e37f0 Don't run replication on gerrit startup
Full replication is very costly and makes gerrit restarts expensive
these days. Turn off replicate_on_startup.

Depends-On: https://review.opendev.org/678486
Change-Id: I31d81821c645697e72a8702c60e2482156e01bb0
2019-08-26 10:16:08 +02:00
Zuul
e03a97de8f Merge "files.o.o : publish .log as text/plain" 2019-08-22 00:34:44 +00:00
Jeremy Stanley
5a096f3705 Re-add the Debian 8/jessie key to reprepro
Because of a limitation in GnuPG we need to have the Jessie archive
signing key in the list of VerifyRelease key IDs for the Debian
reprepro mirror. Also some suites (currently buster-backports) are
signed by a subkey of an archive signing key, so add the "+" suffix
to all these key IDs indicating subkey signatures are also allowed.
As always, Debian signing keys are published and available here:
https://ftp-master.debian.org/keys.html

Change-Id: Iedce38318718a18ace7b2c638755a7d7d4dcd69b
2019-08-02 19:04:25 +00:00
Sorin Sbarnea
3792315db5 Recognize DISK_FULL failure messages (review_dev)
When a job is killed by zuul due to failure like DISK_FULL, a different
message ends up in as a comment.

<li>job-name
finger://ze09.openstack.org/8b6d...6f : DISK_FULL in 2h 59m 50s</li>

This adds another pattern that recognize these messages as failures,
regardess the case (DISK_FULL in this case).

Change-Id: Ib17f05a043430362b02a2826d69572f6b2dbd64a
Needed-By: https://review.opendev.org/#/c/631509/
2019-08-01 11:47:09 +01:00
James E. Blair
96aec261da Add logs.opendev.org vhost
This is a near-copy of the vhost template from puppet-openstackci.

Change-Id: I191e41b501629e2cdd82381d66daa3b850e0be81
2019-07-31 14:32:00 -07:00
Jeremy Stanley
a22df8264f Add archive signing key for Debian 10/buster
The buster-backports suite on Debian mirrors is not signed by the
old jessie signing key we have set to verify in reprepro, but also
we're not mirroring Debian 8/jessie any longer anyway. Replace that
list with the 9/stretch and 10/buster signing keys and switch to
longer key IDs which match the names used for them in the Puppet
manifest. Also add Puppetry and keyfile for the buster keys so that
they will be installed accordingly. The official list of keys can be
found here: https://ftp-master.debian.org/keys.html

Change-Id: Ia193f040b2b707329948955eb091a186eabf8096
2019-07-30 17:41:07 +00:00
Clark Boylan
36c14e4325 Remove centos mirror from openstack mirror update
This rsync'd mirror is now being managed by the opendev mirror update
server. Remove it from the older openstack server to avoid a conflict in
excludes around sclo repo.

Currently we have opendev adding sclo and openstack removing it.

Change-Id: I599ee7d0fab8c5e2a060aff86bce20f1f8d4f54b
2019-07-24 08:11:44 -07:00
Clark Boylan
cb33dba40a Increate gerrit user connection limit by 50%
Zuul has hit a scenario where a git repo update was unable to talk to
gerrit via ssh because it had reached its per user connection limit [0].
This then led to some openstack job failing [1].

The default limit (which we were using) is 64 connection per user.
Apparently this is not quite enough for a busy zuul? Increase this by
50% up to 96.

[0] http://paste.openstack.org/show/754741/
[1] http://lists.openstack.org/pipermail/release-job-failures/2019-July/001193.html

Change-Id: Ibeca2208485608f3b61aa716184165342bfcc3c9
2019-07-22 15:29:19 -07:00
Ian Wienand
983761213f files.o.o : publish .log as text/plain
This is a follow on to I67870f6d439af2d2a63a5048ef52cecff3e75275 to do
the same for files.openstack.org (as
http://files.openstack.org/mirror/logs/ is a handy central place to
point people at)

Change-Id: I07c707d45ab3e3c6f87460b3346efd7026467c56
2019-07-22 16:32:50 +10:00
Zuul
4e050d981e Merge "Complete hide logic for Zuul CI comments in Gerrit" 2019-07-16 01:58:03 +00:00
Jeremy Stanley
5a30d26f44 Complete hide logic for Zuul CI comments in Gerrit
Apply the exclusion for trusted CI comments to the hide function's
conditional case as well as the toggle function's.

Change-Id: Ia4e5ec22a097a8b8cb564c237fd0aa48ab6f8724
2019-07-16 00:47:37 +00:00
Zuul
0fc17feb9f Merge "Don't hide Zuul CI comments" 2019-07-15 22:39:57 +00:00
Jeremy Stanley
0964733556 Don't hide Zuul CI comments
When filtering CI system comments, don't hide those from Zuul, our
gating CI system. It is important to see these comments as not all
results may match the patterns used to expose them as rows in the CI
table. Rename the "Toggle CI" button to "Toggle Extra CI" so that
the name remains accurate without being too verbose.

Change-Id: Id0cd8429ee5ce914aebbbc4a24bef9ebf675e21c
2019-07-15 16:19:59 +00:00
Zuul
0f78ac2dcc Merge "Add proxy for registry.access.redhat" 2019-07-11 20:39:34 +00:00
Alex Schultz
2f96a248c8 Add apt-puppetlabs mirroring back
This used to be mirrored, however there were issues when upstream
dropped the PC1 repositories a few months back. The puppet openstack
jobs are still trying to leverage this mirror but it does not exist in
some regions because it was disabled on the afs content. This change
fixes the reprepo configuration to still pull down puppet5/6 for xenial
and strech and add the symlink back to the mirrors.

Change-Id: I71ad5afe086a503d75a365543ad8869e35ef873b
2019-07-11 11:12:14 -06:00
Zuul
46be2ccae6 Merge "Mailing list for Airship" 2019-07-11 15:13:28 +00:00
Kevin Carter
525d21a332
Add proxy for registry.access.redhat
This change adds a proxy config for registry.access.redhat which should
assist us when gating using images provided by the publically available
registry.

Change-Id: Ica7477d63659610de852d305a63f3e78d0dd8c4f
Signed-off-by: Kevin Carter <kecarter@redhat.com>
2019-07-10 07:31:08 -05:00
Roman Gorshunov
d50e17e043 Mailing list for Airship
This creates mailing list
airship-job-failures@lists.airshipit.org for Airship project.

Change-Id: Ia354c0440ababe99705041c618db2b6ea24d1450
2019-07-03 11:51:15 +00:00
Zuul
5d0d5725ec Merge "Move rsync mirror updates to new opendev.org mirror-update host" 2019-07-03 00:44:04 +00:00
Ian Wienand
b85282c046 Move rsync mirror updates to new opendev.org mirror-update host
This move was prompted by wishing to expose the mirror update logs for
the rsync updates so that debugging problems does not require a root
user (note: not actually done in this change; will be a follow-on).

Rather than start hacking at puppet, the rsync mirror scripts make a
nice delination point for starting an Ansible-first/Bionic update.

Most magic is included in the scripts, so there is not much more to do
than copy them.  The host uses the existing kerberos and openafs roles
and copies the key material into place (to be added before merge).

Note the scripts are removed from the extant puppet so we don't have
two updates happening simultaneously.  This will also require a manual
clean to remove the cron jobs as a once-off when merging.

The other part of mirror-update is the reprepro based scripts for the
various debuntu repositories.  They are left as future work for now.

Testing is added to ensure dependencies and scripts are all in place.

Change-Id: I525ac18b55f0e11b0a541b51fa97ee5d6512bf70
2019-07-02 16:42:33 +10:00
Marcin Juszkiewicz
f830202eea do backports for Debian 'buster'
Change-Id: Id818034921fdf19ccdf67ef74803f97eb04e0acf
2019-06-28 14:05:38 +02:00
Zuul
d36db889cc Merge "Revert "Move openSUSE Tumbleweed into a caching mirror instead"" 2019-06-27 01:44:50 +00:00
Jeremy Stanley
428872075e Revert "Move openSUSE Tumbleweed into a caching mirror instead"
This reverts commit b3ce1c52dc7ca455ffd94ea07d8a4fb1b6905fa8.
It removed the AFS mirror at the same time it added the proxy,
but jobs don't know to look for the proxy since it's on a
totally different TCP port.

Change-Id: I87cc03eb3322bd7b093dd6fe798aadb48f319805
2019-06-26 23:29:20 +00:00
Marcin Juszkiewicz
58023a580a UCA: mirror Train packages
Change-Id: I6aac827fedd5b17271a94340e5fd295d7f5e5a31
2019-06-20 12:32:10 +02:00
Zuul
051da8cab0 Merge "epel: mirror also aarch64" 2019-06-18 03:51:49 +00:00
Michal Nasiadka
4b1cc4b949 Add bionic to deb-docker releases in mirror_update
Change-Id: I6c7a10a8ac2cfa731700e10ab640005cb8d522f6
2019-06-11 18:00:41 +02:00
Clark Boylan
1884a22ca7 Update certcheck domains
This removes groups.openstack.org as this service was shut down. Add new
opendev services behind ssl.

Change-Id: I14c667c8fbde07c3a52778bc2c5e93abf8f053a4
2019-06-10 13:03:35 -07:00
Marcin Juszkiewicz
cf0fe03a37 epel: mirror also aarch64
We now have altarch mirrored so let us add EPEL to it.

aarch64 for start

Change-Id: I08e2dabf79c3a207d204303d217198d333730b31
2019-06-07 17:03:17 +02:00
Clark Boylan
fd97e061ae Evaluate files vhosts after we determine ssl file paths
Previously we evaluated the vhost templates before setting
ssl_cert_file_ and ssl_key_file_ and ssl_chain_file_. This made erb
unhappy because those are the three variables we use to set paths in the
vhost. Fix this by moving the vhost after the ssl file vars are set.

Change-Id: I4ba62521c9e7da104f8799d016cbcf0214cbdfc1
2019-06-06 16:28:37 -07:00
Clark Boylan
9ea8edc341 Evaluate files website vhosts in context of website not vhost
To deal with puppet scoping fun we evaluate the template for our
files.o.o website vhosts in the context of the website define and not in
the context of httpd::vhost.

Change-Id: I90bb881eb6ad78cede3a8a2548e1dfcf24e1160b
2019-06-06 15:12:15 -07:00
Zuul
0ace6c2614 Merge "Add tarballs.opendev.org vhost" 2019-06-06 20:03:33 +00:00
Ian Wienand
6b9907dcd8 Add centos/ to centos mirror path
As a follow-on to I0e110ef51c8ed301fd8280ae7fc039e3b01db92c; this
dropped the /centos/ from the base mirror, add it back.

Also switch the mirror to the only one on the altarch-mirrors page
that is in US/TX, which from the name is in Dallas, which must be
pretty close to rax.dfw where the update server lives.

Change-Id: If4d71865f4328e73a26c7b38300767ed6b790579
2019-06-06 13:36:36 +10:00
Zuul
8b98aff89c Merge "centos-mirror-update: handle non-x86 architectures" 2019-06-06 00:57:09 +00:00
James E. Blair
3c84c65614 Add tarballs.opendev.org vhost
Change-Id: I6aa85bf92b2d5726d3c86b11b103a87f11953c51
2019-06-05 14:10:19 -07:00
Marcin Juszkiewicz
ac5acbca92 centos-mirror-update: handle non-x86 architectures
CentOS keeps non-x86 architectures in /altarch/ directory (contrary to
/centos/ one for x86-64). We have aarch64 (arm64) machines in infra and
they fail due to lack of CentOS altarch mirror.

List of wanted alternative architectures is controlled by ALTARCHS
variable (aarch64 and ppc64le enabled). As CentOS has several other
architectures too they are listed in ALTARCHS_IGNORED so we do not fetch
them.

Current CentOS mirror lands in same /mirror/centos/7/ directory. Altarch
mirrors goes to /mirror/centos/altarch/7/ one.

Change-Id: I0e110ef51c8ed301fd8280ae7fc039e3b01db92c
2019-06-03 13:07:26 +02:00
Clark Boylan
7c954dba72 Add opendev service-announce list
This way we can send a single email that our users can see if subscribed
to this list instead of sending emails to all of their discuss lists.

Change-Id: I3b978a3c4e7888f14e3986628cb29a6c86bbcf61
2019-05-30 13:53:00 -07:00
Zuul
d234442dae Merge "Remove jenkins user from status.o.o" 2019-05-29 15:13:48 +00:00
Zuul
598d9106ee Merge "Remove bugday from status.o.o" 2019-05-29 15:12:00 +00:00
Zuul
d3d13152a7 Merge "Adding new ML for the NBMP group" 2019-05-29 14:17:48 +00:00
Jeremy Stanley
b1f95745f6 Exclude old distros from yum-puppetlabs mirror
The yum-puppetlabs mirror exceeded its 100GB quota as of April 26.
Rather than increase the quota, start excluding packages for old
platforms we don't provide like RHEL5-6 and Fedora F20-27. We could
probably get even more aggressive with it, but this get the
utilization back under 50% which is plenty of headroom for now.

Change-Id: I9665b3a2a89f991f9433fe7f45bc1bb0e0c7632b
2019-05-28 23:15:37 +00:00
Jean-Philippe Evrard
5568a461bf Exclude tmp files artifacts from syncing
It seems the openSUSE build process can leave artifacts behind,
in the form of .~tmp~ files in the mirror. I assume these are
wrongfully present.

This is a problem, as those ~tmp~ files prevent syncing the
repositories.

While it's most likely that openSUSE files will be cleaned in the
source repos, should this problem arise in the future, it's also
more robust to skip the syncing of those files.

This has the extra benefit of temporarily unblock mirroring of
openSUSE Leap 15.1 in infra, as of today.

Change-Id: I0124b992483cfda9f97960b43bddf94efa008030
2019-05-28 15:19:22 +02:00
Zuul
459bc1ab4f Merge "Start mirroring openSUSE Leap 15.1" 2019-05-27 07:33:25 +00:00
James E. Blair
5faf89f566 Add haproxy-statsd to haproxy server
Build a container image with the haproxy-statsd script, and run that
along with the haproxy container.

Change-Id: I18be70d339df613bf9a72e115e80a6da876111e0
2019-05-24 15:40:28 -07:00
Monty Taylor
c4b0fa1ad3 Remove jenkins user from status.o.o
This is vestigal. Removing it to make reviewing xenialification
easier.

Change-Id: I9b57336d47ee4aa18fdc3888c2ac2c75a3b23c15
2019-05-24 16:14:27 -05:00
Monty Taylor
43fdea4ac4 Remove bugday from status.o.o
It doesn't seem like this is used anymore. Let's remove it before
we update the rest of this, so that we don't have to, you know,
update abandoned things.

Change-Id: I1c3708021046a428da82eaa843961091915ba4af
2019-05-24 14:56:50 -05:00