15161 Commits

Author SHA1 Message Date
James E. Blair
6bd0d0258d Refactor gerrit master job
Make a "base" version of the gerrit master job with no file matcher
so that we can use it in other repositories (eg, gerrit, zuul).
Inherit from it with the original name to add the file matcher back.

Change-Id: I4e428b44dd82f8dba08b219cbf8407969c6436b1
2019-10-22 09:06:24 -07:00
Monty Taylor
4b4dcf37b1 Pin python images to 3.7
rehash may have an issue with python 3.8 internals.

Change-Id: I1a9c6150d74f6a646446b65a7456c73dc104fd9e
2019-10-22 03:37:23 +09:00
Monty Taylor
61b5a11bfb Always rebuild base and builder together
When we build either, it could pick up base image changes, such as
moving from stretch to buster. Make sure any time we build one we
build the other so that they stay in sync.

Change-Id: Ia28ad4f64114c88cc02289c9318a323ceb4f143d
2019-10-22 02:59:11 +09:00
David Shrewsbury
0ae912cc04 No-op change to rebuild python-builder
Apparently, an update to the python-base image caused a new image
to be uploaded to dockerhub. Because python-builder was not updated,
it is running a slightly older base image which we believe is causing
issues with python wheel packaging due to possible different python
versions.

Change-Id: I38948882131e30c7358a970292621b0280b75aac
2019-10-21 11:22:50 -04:00
Monty Taylor
8725a1ee53 Squash two ansible warnings
Add bool to use_upstream_docker conditional

This is an ansible behavior change that's coming in 2.12 but is
currently spewing warnings. The warnings make the log really hard
to read, so just fix it.

Disable group name auto-renaming

If you have group names with non-python identifier characters, it
prevents you from looking it up in jinja like "groups.group-name"
so ansible auto-transforms it so you can do "groups.group_name".
This is a confusing behavior which is going away. However, ansible
is warning everyone who has groups with characters in them as it
has no idea how you might be accessing those group names. Add
a config setting to suppress the warning about -'s in group names.

Change-Id: Ib3262025799af7c3171ed0b079cb1dd969075931
2019-10-20 08:02:43 +09:00
Monty Taylor
92fe8eae71 Add checks plugin to gerrit master build
Change-Id: Ifb7f4b84fb07c99d3c28b78ec55a1835821202f6
2019-10-20 06:36:02 +09:00
Monty Taylor
36aa77937a Add jobs to build gerrit master branch
We'll use this to test the checks plugin.

We have to add jgit as a repo because it's a submodule now.

Change-Id: Ic7e9ad0265e136a9ac6b1147998f6eb5ee398180
2019-10-20 06:35:56 +09:00
Zuul
cd402000a4 Merge "Several updates because the world is a dark place" 2019-10-19 00:58:47 +00:00
Monty Taylor
9ab25e89a9 Several updates because the world is a dark place
A few things have changed and we need to fix them in one go.

Use mirror for installing docker for buildset-registry

While, we need to make this more systemic, that's hanging off of the
mirror rework. For now, since we know all of these jobs are debian
based, just set the mirror location.

Replace use of zuul cloner with git clones

You can never be a prophet in your own hometown. This is now broken
because of the git cache rework, so just replace it.

Update libjemalloc library

python:slim is based on buster now, which has libjemalloc2 not
libjemalloc1.

Remove gerrit repo remote for submodules

A recent change to the base jobs to use prepare-workspace-git
broke the gerrit image builds by actually having the origin
remote by /dev/null as intended. This breaks submodules because
for a few of them where we don't have matching stable branches
the submodule relative path behavior is actually exactly what
we want.

Since we don't care about the remote otherwise, remove the
origin remote before doing the submodule update --init so that
the submodule will clone the refs from the zuul prepared repo.

Change-Id: Ieb5b6bc8711fe971ed3445c7c267306ac4616464
2019-10-19 07:51:29 +09:00
Zuul
6cc8598305 Merge "Expose volume for tmp dir, add dirs for hooks and static" 2019-10-16 22:53:39 +00:00
Andreas Jaeger
757aebbfc3 Stop openSUSE 42.3 mirroring
openSUSE 42.3 is removed from Infra, remove it from mirroring as well.

Change-Id: I60c77fd48e55edbacd589c2f94de977a034eb9c9
2019-10-16 11:06:21 +02:00
Monty Taylor
5a31c91623 Expose volume for tmp dir, add dirs for hooks and static
Our existing config management puts files into hooks and static.
We can bind mount those in, but having the mount-points in the
image is useful.

The tmp dir is important for gerrit to be able to write
plugin expansions and javaamelody data.

Change-Id: Idd917c268ed7bdead412620dfe3ca842736b7463
2019-10-16 14:57:46 +09:00
Zuul
74579a5e45 Merge "Use bazel 1.0.0" 2019-10-15 20:56:32 +00:00
Zuul
db44d9eacb Merge "Remove dependencies from promote" 2019-10-15 20:15:52 +00:00
Monty Taylor
d2e7da959d Use bazel 1.0.0
1.0.0 has released, which is what gerrit wants now. Use it.

Depends-On: https://review.opendev.org/688555
Change-Id: I6cd76b8cfda3656d6105f9fe96b82a388809375f
2019-10-14 16:25:30 -04:00
Zuul
f05dd90ddb Merge "Remove linaro-cn1" 2019-10-11 20:48:40 +00:00
Zuul
81aa73fd99 Merge "Remove read-only user from registry" 2019-10-11 20:48:39 +00:00
James E. Blair
95f0c32a61 Temporarily disable intermediate registry prune
Just to eliminate the possibility that a bug in this may be causing
us to erroneously delete blobs.

Change-Id: I048d9ae5ba92984c90f84f231b412050a52fcea6
2019-10-09 14:10:56 -07:00
James E. Blair
9bafd83279 Add public url to intermediate registry
This is needed for the JWT auth fix in
https://review.opendev.org/687622

Change-Id: I35f758ea6bfd3871921adc72c70766340216d654
2019-10-09 10:39:01 -07:00
James E. Blair
40cd27487b Remove read-only user from registry
This is no longer used as read access is provided to unathenticated
users with the recently added JWT support.

Depends-On: https://review.opendev.org/687422
Change-Id: I36fd28710c644be9b07d645c6b0c6092f52a7385
2019-10-08 14:20:06 -07:00
James E. Blair
dee6a8b330 Add token secret to intermediate registry
An upcoming change will add JWT authentication to the registry;
prepare for that by establishing a server-side secret for use
in signing the tokens.

Change-Id: Ibaa15dd0c4b0d797f01a1886186fdc021dc990fa
2019-10-08 14:16:43 -07:00
Zuul
3674059437 Merge "Update zuul ansible installs when restarting zuul" 2019-10-07 21:05:27 +00:00
James E. Blair
81609e6a65 Set debug logging in intermediate registry
Change-Id: Ifb997339453284c224c6fb0d3032ad6e95da9e63
2019-10-05 07:56:03 -07:00
Monty Taylor
f9e7fcab51 Remove dependencies from promote
These don't really make a ton of sense now do they?

Change-Id: Ic5f515dd50872387e3447eb70af25070257f61cd
2019-10-05 09:20:38 +02:00
Zuul
afa75226b4 Merge "Install jeepyb into the gerrit images" 2019-10-05 00:07:29 +00:00
James E. Blair
d579b997c4 Add zuul_connection_secrets to mergers and executors
These may need secrets (such as passwords) if they need to perform
git operations over http.

Change-Id: I93501ae27d246c1cccfe18fc555f3f2f4b7814d5
2019-10-04 14:11:26 -07:00
James E. Blair
1641477c95 Correct registry docker mount
We're trying to get clouds.yaml into /etc/openstack/clouds.yaml.

This should accomplish that.  The previous configuration was in
error and wiped out the /etc directory which made things not work
well.

Change-Id: I88e69b05f3e8c1688d24736fa775163fc25a07f0
2019-10-04 11:34:45 -07:00
Clark Boylan
1bd2020251 Update zuul ansible installs when restarting zuul
This change adds a task to the zuul restart playbook to update the zuul
ansible installs on the executors during the restart. This will help
ensure we keep our ansible installs there up to date.

Change-Id: I443e204eaa69dcf0c5622303201549224e893626
2019-10-04 10:12:18 -07:00
James E. Blair
5304165f7d Remove linaro-cn1
This cloud was mostly removed, except for this bit.

Change-Id: Ie37abb4fd3eb3342b66ade52ab65024c420d7264
2019-10-04 09:32:39 -07:00
Zuul
9d8cb09def Merge "Use the gerrit driver for the googlesource connection" 2019-10-04 16:10:50 +00:00
James E. Blair
8223eadf01 Switch the insecure-ci-registry to zuul-registry
This uses the new zuul-registry container image to run the
intermediate registry.  The same authentication data and certs
are used.  The new registry also writes to the same swift container,
but uses different pseudo-directories so it won't clash with the
current registry.  If there are problems, we can switch back easily.
After successful use of the new registry, we can delete the old data.

Change-Id: Ib855fb99c991411293a617b9b238d79a6bfae328
2019-10-02 14:11:33 -07:00
Ian Wienand
1ef7315ae4 mirror-update: Add EPEL 8 mirroring
Change-Id: I293fe01806743e3ec45877bdadd3e5b0fb851aee
2019-10-02 14:30:41 +10:00
James E. Blair
4a3956b9b2 Use the gerrit driver for the googlesource connection
This is the connection for Gerrit's Gerrit.  We can connect to it
directly with the Gerrit driver now that we support entirely
HTTP-based Gerrits and we have an account.  Use that so we can
set up some third-party CI testing.

Change-Id: I7b91b4e67594ce62772e3e450a1236095d4ef1c9
2019-10-01 14:19:14 -07:00
James E. Blair
086d4b638c Revert "Use the gerrit driver for the googlesource connection"
This reverts commit 01bef804de2d65f9ce4cda80e937a718c7ee7628.

This was missing an option, and also our password doesn't work.

Change-Id: I317c29034efaae62ea74a5c16ec25bed98c1e3cc
2019-10-01 14:18:08 -07:00
Zuul
3cd8cedbc9 Merge "Use the gerrit driver for the googlesource connection" 2019-10-01 16:40:30 +00:00
Zuul
199c1c5939 Merge "logrotate: better rotation options" 2019-10-01 06:49:44 +00:00
Ian Wienand
d4a96a421b logrotate: better rotation options
The extant "logrotate_daily" varaible doesn't really do anything and
isn't used.  Modify this to be able to set a range of rotation periods
or a size.

Update mirror rsync mirrors to rotate weekly, as often releases run
overnight and it's a pain to reconstruct.

Change-Id: I121dc5f4fe7f226b66d18b9ec39e7e3839be4d40
2019-10-01 09:00:53 +10:00
Ian Wienand
d59e654936 Add CentOS 8 mirror
This adds mirroring of CentOS 8.  It is somewhat simpler because the
architecture we're interested in are in the base repos, no need for
altarch.

The current mirror doesn't have a 8/ directory; possibly they require
their own mirroring filter updates?  Use an up-to-date mirror for 8
(we can switch 7 too, but leaving alone for now).

Additionally, the altarch mirror we are using appears to have gone
offline for at least a few days.  Switch to another one that is in
Texas, which should be close-ish to the DFW servers.

Change-Id: I33d95fa6b2df23fbfdb6745a3079761e228f677b
2019-09-25 06:28:23 +00:00
Zuul
f50c7379f0 Merge "Log and report infra status in #openstack-election" 2019-09-24 09:10:56 +00:00
Zuul
b6b9ff74dd Merge "Clean up logging for abandoned IRC channels" 2019-09-24 09:10:54 +00:00
Zuul
efe4f820e6 Merge "Revert to rit.edu mirror for Suse" 2019-09-24 09:10:52 +00:00
Itxaka
1087cbf433
Revert to rit.edu mirror for Suse
Looks like leaseweb, kernel.org and others are not properly
syncing the Suse mirrors as they are out of date for Leap15.0 and
missing files (deltainfo.xml.gz) which causes job breakage for
Leap15.0 based jobs.

Revert this to the original mirror from a year ago which is updated
and not broken

Change-Id: Id7184ee973bbabfec3f601fc9200ffac17322558
2019-09-23 16:25:57 +02:00
Jeremy Stanley
83cbcaa4fb Log and report infra status in #openstack-election
The #openstack-election IRC channel is used in the execution of
official OpenStack project duties involving technical elections, so
should be publicly logged. Also, its denizens would appreciate
receiving infrastructure status updates, as these may impact their
activities.

Depends-On: https://review.opendev.org/682440
Change-Id: I54a2feed3cf7265d700082f3ce04b82aa36e56e5
2019-09-20 17:51:50 +00:00
Jeremy Stanley
de8b53d32b Clean up logging for abandoned IRC channels
Analysis of IRC channel logs indicates the following have seen 5 or
fewer non-bot comments posted in the past 12 months and are likely
of no value to continue logging:

    5 #openstack-vpnaas
    2 #swift3
    2 #openstack-ko
    1 #openstack-deployment
    1 #midonet
    0 #openstack-valet
    0 #openstack-valence
    0 #openstack-swg
    0 #openstack-mogan

We're currently at Freenode's 120 channel maximum, so remove them
from logging and free up available slots for new channels which wish
to have logging added. In addition, the Neutron team has asked to
have #networking-sfc and #openstack-fwaas removed from logging as
those channels's discussions are being folded back into
the #openstack-neutron channel now:

http://lists.openstack.org/pipermail/openstack-discuss/2019-September/009480.html

While we're here, go ahead and remove the same channels from the
statusbot configuration if they're listed in it.

Change-Id: If1f65be378fa247014e175c75e1562d4755ef497
2019-09-20 17:51:30 +00:00
Kevin Carter
8b0877cb68
Add proxy for quay
This change adds a proxy config for quay which should assist
us when gating using images provided by the publically
available registry.

Change-Id: I971705e59724e70bd9d42a6920cf4f883556f673
Signed-off-by: Kevin Carter <kecarter@redhat.com>
2019-09-19 16:49:20 -05:00
Zuul
f8808d6919 Merge "Remove bazel version hack" 2019-09-19 17:22:45 +00:00
Zuul
44208d4ebc Merge "Fix gerrit java invocation" 2019-09-19 16:05:49 +00:00
Monty Taylor
d9fb5b3faf Install jeepyb into the gerrit images
We need jeepyb installed because the content of the gerrit hook scripts
we install is done via jeepyb commands. Use python-builder so that we
can just install the jeepyb wheel.

Should we maybe transition these hooks into being zuul jobs?

Depends-On: https://review.opendev.org/683146/
Change-Id: I8899885b05d1e9f48b3f354ca22b360b54d455a3
2019-09-19 15:17:14 +02:00
Monty Taylor
f0a3f0cb37 Remove bazel version hack
The upstream patch has landed, so we don't need this anymore.

Change-Id: I08a6705f189b2a24b737ab4f52bb7f449879fdf1
2019-09-19 14:18:41 +02:00
Monty Taylor
940e31f043 Fix gerrit java invocation
We need the right path to java, and we also need to tell it to run
daemon.

Change-Id: I7dfbc5a1c054953b34a02327e6bba50c486110d1
2019-09-19 14:16:30 +02:00