723 Commits

Author SHA1 Message Date
Monty Taylor
8725a1ee53 Squash two ansible warnings
Add bool to use_upstream_docker conditional

This is an ansible behavior change that's coming in 2.12 but is
currently spewing warnings. The warnings make the log really hard
to read, so just fix it.

Disable group name auto-renaming

If you have group names with non-python identifier characters, it
prevents you from looking it up in jinja like "groups.group-name"
so ansible auto-transforms it so you can do "groups.group_name".
This is a confusing behavior which is going away. However, ansible
is warning everyone who has groups with characters in them as it
has no idea how you might be accessing those group names. Add
a config setting to suppress the warning about -'s in group names.

Change-Id: Ib3262025799af7c3171ed0b079cb1dd969075931
2019-10-20 08:02:43 +09:00
Monty Taylor
36aa77937a Add jobs to build gerrit master branch
We'll use this to test the checks plugin.

We have to add jgit as a repo because it's a submodule now.

Change-Id: Ic7e9ad0265e136a9ac6b1147998f6eb5ee398180
2019-10-20 06:35:56 +09:00
Zuul
cd402000a4 Merge "Several updates because the world is a dark place" 2019-10-19 00:58:47 +00:00
Monty Taylor
9ab25e89a9 Several updates because the world is a dark place
A few things have changed and we need to fix them in one go.

Use mirror for installing docker for buildset-registry

While, we need to make this more systemic, that's hanging off of the
mirror rework. For now, since we know all of these jobs are debian
based, just set the mirror location.

Replace use of zuul cloner with git clones

You can never be a prophet in your own hometown. This is now broken
because of the git cache rework, so just replace it.

Update libjemalloc library

python:slim is based on buster now, which has libjemalloc2 not
libjemalloc1.

Remove gerrit repo remote for submodules

A recent change to the base jobs to use prepare-workspace-git
broke the gerrit image builds by actually having the origin
remote by /dev/null as intended. This breaks submodules because
for a few of them where we don't have matching stable branches
the submodule relative path behavior is actually exactly what
we want.

Since we don't care about the remote otherwise, remove the
origin remote before doing the submodule update --init so that
the submodule will clone the refs from the zuul prepared repo.

Change-Id: Ieb5b6bc8711fe971ed3445c7c267306ac4616464
2019-10-19 07:51:29 +09:00
Andreas Jaeger
757aebbfc3 Stop openSUSE 42.3 mirroring
openSUSE 42.3 is removed from Infra, remove it from mirroring as well.

Change-Id: I60c77fd48e55edbacd589c2f94de977a034eb9c9
2019-10-16 11:06:21 +02:00
Zuul
f05dd90ddb Merge "Remove linaro-cn1" 2019-10-11 20:48:40 +00:00
Zuul
81aa73fd99 Merge "Remove read-only user from registry" 2019-10-11 20:48:39 +00:00
James E. Blair
95f0c32a61 Temporarily disable intermediate registry prune
Just to eliminate the possibility that a bug in this may be causing
us to erroneously delete blobs.

Change-Id: I048d9ae5ba92984c90f84f231b412050a52fcea6
2019-10-09 14:10:56 -07:00
James E. Blair
9bafd83279 Add public url to intermediate registry
This is needed for the JWT auth fix in
https://review.opendev.org/687622

Change-Id: I35f758ea6bfd3871921adc72c70766340216d654
2019-10-09 10:39:01 -07:00
James E. Blair
40cd27487b Remove read-only user from registry
This is no longer used as read access is provided to unathenticated
users with the recently added JWT support.

Depends-On: https://review.opendev.org/687422
Change-Id: I36fd28710c644be9b07d645c6b0c6092f52a7385
2019-10-08 14:20:06 -07:00
James E. Blair
dee6a8b330 Add token secret to intermediate registry
An upcoming change will add JWT authentication to the registry;
prepare for that by establishing a server-side secret for use
in signing the tokens.

Change-Id: Ibaa15dd0c4b0d797f01a1886186fdc021dc990fa
2019-10-08 14:16:43 -07:00
Zuul
3674059437 Merge "Update zuul ansible installs when restarting zuul" 2019-10-07 21:05:27 +00:00
James E. Blair
81609e6a65 Set debug logging in intermediate registry
Change-Id: Ifb997339453284c224c6fb0d3032ad6e95da9e63
2019-10-05 07:56:03 -07:00
James E. Blair
1641477c95 Correct registry docker mount
We're trying to get clouds.yaml into /etc/openstack/clouds.yaml.

This should accomplish that.  The previous configuration was in
error and wiped out the /etc directory which made things not work
well.

Change-Id: I88e69b05f3e8c1688d24736fa775163fc25a07f0
2019-10-04 11:34:45 -07:00
Clark Boylan
1bd2020251 Update zuul ansible installs when restarting zuul
This change adds a task to the zuul restart playbook to update the zuul
ansible installs on the executors during the restart. This will help
ensure we keep our ansible installs there up to date.

Change-Id: I443e204eaa69dcf0c5622303201549224e893626
2019-10-04 10:12:18 -07:00
James E. Blair
5304165f7d Remove linaro-cn1
This cloud was mostly removed, except for this bit.

Change-Id: Ie37abb4fd3eb3342b66ade52ab65024c420d7264
2019-10-04 09:32:39 -07:00
James E. Blair
8223eadf01 Switch the insecure-ci-registry to zuul-registry
This uses the new zuul-registry container image to run the
intermediate registry.  The same authentication data and certs
are used.  The new registry also writes to the same swift container,
but uses different pseudo-directories so it won't clash with the
current registry.  If there are problems, we can switch back easily.
After successful use of the new registry, we can delete the old data.

Change-Id: Ib855fb99c991411293a617b9b238d79a6bfae328
2019-10-02 14:11:33 -07:00
Ian Wienand
1ef7315ae4 mirror-update: Add EPEL 8 mirroring
Change-Id: I293fe01806743e3ec45877bdadd3e5b0fb851aee
2019-10-02 14:30:41 +10:00
Zuul
199c1c5939 Merge "logrotate: better rotation options" 2019-10-01 06:49:44 +00:00
Ian Wienand
d4a96a421b logrotate: better rotation options
The extant "logrotate_daily" varaible doesn't really do anything and
isn't used.  Modify this to be able to set a range of rotation periods
or a size.

Update mirror rsync mirrors to rotate weekly, as often releases run
overnight and it's a pain to reconstruct.

Change-Id: I121dc5f4fe7f226b66d18b9ec39e7e3839be4d40
2019-10-01 09:00:53 +10:00
Ian Wienand
d59e654936 Add CentOS 8 mirror
This adds mirroring of CentOS 8.  It is somewhat simpler because the
architecture we're interested in are in the base repos, no need for
altarch.

The current mirror doesn't have a 8/ directory; possibly they require
their own mirroring filter updates?  Use an up-to-date mirror for 8
(we can switch 7 too, but leaving alone for now).

Additionally, the altarch mirror we are using appears to have gone
offline for at least a few days.  Switch to another one that is in
Texas, which should be close-ish to the DFW servers.

Change-Id: I33d95fa6b2df23fbfdb6745a3079761e228f677b
2019-09-25 06:28:23 +00:00
Zuul
efe4f820e6 Merge "Revert to rit.edu mirror for Suse" 2019-09-24 09:10:52 +00:00
Itxaka
1087cbf433
Revert to rit.edu mirror for Suse
Looks like leaseweb, kernel.org and others are not properly
syncing the Suse mirrors as they are out of date for Leap15.0 and
missing files (deltainfo.xml.gz) which causes job breakage for
Leap15.0 based jobs.

Revert this to the original mirror from a year ago which is updated
and not broken

Change-Id: Id7184ee973bbabfec3f601fc9200ffac17322558
2019-09-23 16:25:57 +02:00
Kevin Carter
8b0877cb68
Add proxy for quay
This change adds a proxy config for quay which should assist
us when gating using images provided by the publically
available registry.

Change-Id: I971705e59724e70bd9d42a6920cf4f883556f673
Signed-off-by: Kevin Carter <kecarter@redhat.com>
2019-09-19 16:49:20 -05:00
Zuul
f8808d6919 Merge "Remove bazel version hack" 2019-09-19 17:22:45 +00:00
Monty Taylor
f0a3f0cb37 Remove bazel version hack
The upstream patch has landed, so we don't need this anymore.

Change-Id: I08a6705f189b2a24b737ab4f52bb7f449879fdf1
2019-09-19 14:18:41 +02:00
Clark Boylan
f8bb46d810 Restore setup-org.yaml
This tasks file was removed when we rewrote the gitea management into
python. Unfortunately our gitea rename project tasks rely on this tasks
file to create new orgs if they are needed to rename a project.

Restore this file so that we can rename projects in this manner. Note we
move it into playbooks/ and rename it to gitea-rename-setup-org.yaml to
make it clear that the gitea management role doesn't use this set of
tasks, only the gitea project rename playbook does.

Change-Id: I782f6e56cad18bdcbf9504d51af15caa950a5752
2019-09-18 12:40:19 -07:00
Radosław Piliszek
dc2510aac7 Mirror OBS ha-clustering repo for CentOS 7
Kolla uses this to build hacluster images.
Direct usage is causing timeouts recently.

I changed the formatting to make it more readable
and slapped a comment note for maintainers.

Change-Id: I68d7155718c0ae0744198ca96aca1a207bab7ed6
Signed-off-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>
2019-09-17 10:25:57 +02:00
Zuul
823298b365 Merge "Fix files matcher and bazel for gerrit base image" 2019-09-17 00:41:59 +00:00
Monty Taylor
072fcca06f Fix files matcher and bazel for gerrit base image
Use latest bazel

It seems 0.27 is now too old. This is what happens when I go on vacation
apparently.

Add in a hack to override the bazelversion. We'll remove this once
https://gerrit-review.googlesource.com/c/gerrit/+/237495 lands and
has been merged up.

Change-Id: Ib7a6d33ce8bf8498fd5cd09b25087dc09acb8df4
2019-09-16 21:20:18 +02:00
Ian Wienand
3dcfe61d2f fedora mirror update : add sleep
As described inline, this should make our mirror pulses more robust
against timeouts.

This is probably ripe for turning into more of a library situation for
all the other "vos release" calls too.  But one thing at a time ... I
think we test with this for a while to see if stability returns.

Change-Id: I041a290053e4e8ceba80785598a5945e5adcf6f1
2019-09-11 13:15:29 +10:00
Ian Wienand
912dff49e7 Set zuul_work_dir for tox testing
Setting this to system-config allows us to run the base tests as 3rd
party ci for projects like testinfra.

Change-Id: I2d15df154dcdc7c5da6c3326fbecec2146201164
2019-09-09 09:44:43 +10:00
Ian Wienand
ace1c39c61 gitea: use random time for git gc run
Randomising the time of this job should help avoid a thundering herd
of I/O intensive operations in the gitea environment.

Change-Id: I035f7781a397665357b6d039b989ab9fe6a46b8a
2019-09-04 05:15:21 +10:00
Ian Wienand
c9bea2cfc4 fedora-mirror: Use rsync itemize output
We are debugging why the fedora mirror has started to take 8+ hours to
release.  There is suspicion that rsync is updating the metadata of a
lot of files, causing the incremental update to be more of a full
release.

If you read the man page carefully:

 At the single -v level of verbosity, this does not mention when a
 file gets its attributes changed.  If you ask for an itemized list of
 changed attributes (either --itemize-changes or adding "%i" to the
 --out-format setting), the output (on the client) increases to
 mention all items that are changed in any way

So we need "-i" to see these updates

Change-Id: Ia9cff1e126404bc5a3346e758a15314834aef409
2019-08-30 13:29:54 +10:00
Zuul
3eb182e551 Merge "Remove the extra bazel options" 2019-08-28 14:42:03 +00:00
Ian Wienand
b617a08989 fedora-mirror-update: update logging
We're having trouble with this volume; this makes the logging more
helpful for humans trying to establish the order of things happening.

It turns down the tracing, puts timestamps on the logs messages, makes
the messages more descriptive, and makes sure to timestamp the output
of the vos commands so we can see how long each step took.

Change-Id: Ia832b7ef86350efcbbe7d3b957e820a8a39df3e1
2019-08-27 14:09:49 +10:00
Monty Taylor
56ceaf1c40 Remove the extra bazel options
We had some extra bazel options that don't seem to be necessary
anymore now that we are using upstream bazel options appropriately.

Retry the build a couple of times if it goes south, inside of the
build image. This should allow re-use of the cache the second time,
and if there is a temporary error, it should pick up and move
forward.

Change-Id: I5f304acb21fd3a4d40701fc0414ae0c424c838e5
2019-08-26 11:26:19 +02:00
Clark Boylan
a248dbae63 Add more fedora-atomic mirror exclusions
We don't need things like efi configs or pxeboot configs or ISOs.
Exclude these items to further reduce the size of the fedora mirror.

Change-Id: I93003b2f48d79dae627026e2c7af748ea9a9e34d
2019-08-19 15:53:19 -07:00
Ian Wienand
0751b3d481 Convert nested bridge.o.o ARA report to static HTML
With the switch to swift logging, we need to convert the nested ARA
report to static HTML

Change-Id: I9e177915099598d5d48a31c15bd6db49e4d1c7e8
2019-08-19 10:28:57 +10:00
Ian Wienand
445eb7a7b2 backup: minor fixes
The ssh config file is /.ssh/config (not ssh_config)

We are accepting the ed25519 key, not the ecdsa key, so fix that in
the known_hosts stanza.

Change-Id: If3a42a7872f5d5e7a2bf9c3b5184fb14d43e6a1a
2019-08-09 14:11:41 +10:00
Zuul
788d91df1f Merge "Ansible roles for backup" 2019-08-05 08:48:41 +00:00
Ian Wienand
814e4be128 Ansible roles for backup
This introduces two new roles for managing the backup-server and hosts
that we wish to back up.

Firstly the "backup" role runs on hosts we wish to backup.  This
generates and configures a separate ssh key for running bup and
installs the appropriate cron job to run the backup daily.

The "backup-server" job runs on the backup server (or, indeed
servers).  It creates users for each backup host, accepts the remote
keys mentioned above and initalises bup.  It is then ready to receive
backups from the remote hosts.

This eliminates a fairly long-standing requirement for manual setup of
the backup server users and keys; this section is removed from the
documentation.

testinfra coverage is added.

Change-Id: I9bf74df351e056791ed817180436617048224d2c
2019-08-05 16:59:57 +10:00
Clark Boylan
3ff4bed27c Trim fedora mirror
The fedora mirror is our largest mirror (850GB about twice as big as the
next mirror). Much of this size is due to the fedora atomic images we
mirror.

On further investigation I notice that we are mirroring ppc images (for
which we do not have cpus to run them on), image for fedora 25 and 36
which are quite EOL'd, and our exclusion of the raw.xz and vagrant
images is failing.

Update the rsync excludes to ensure we don't mirror any of these images
we don't need.

Change-Id: I86856cb4e51b0e687aac45a1f014f87c5141318f
2019-08-02 14:35:21 -07:00
Clark Boylan
f686ec39f5 Switch fedora mirroring to pubmirror2.math.uh.edu
pubmirror1.math.uh.edu is currently offline and listed as an altonly.
pubmirror2 seems to work fine so switch to it.

Change-Id: I2562f8686146d17d4fad3997b9be22361fa05fca
2019-08-02 14:27:00 -07:00
James E. Blair
48cafd19f8 Add LE cert for logs.opendev.org to static
This can be used in an apache vhost later, but should be fine to
merge now.

Depends-On: https://review.opendev.org/673902
Change-Id: Ic2cb7585433351ec1bdabd88915fa1ca07da44e7
2019-07-31 13:00:50 -07:00
Jens Harbott
7df5981e12 Be explicit about fortnebula networks with nodepool
We don't want nodepool to use floating IPs in the fn cloud as it is an
ipv6 only cloud. We explicitly tell it there is no fip source and that
the tenant network routes ipv6 externally. This config is based on the
limestone configuration which is a similar cloud network wise.

Change-Id: I4a27a22a5beb9c5fc9d3e16cd2ca5b41aecbb46f
2019-07-31 08:36:23 -07:00
Clark Boylan
4b4eb02f32 Replace the fn mirror again
Networking got weird on the previous host so we rebuilt this one going
back to networking we expect to work (FIPs and all that). This updates
the inventory so that we configure the host properly.

Change-Id: I0dcdbc9efdd330d66b57da0b01d23dd3d747f79b
2019-07-30 15:15:01 -07:00
Zuul
107943e60d Merge "Build gerrit images for 2.16 and 3.0 as well" 2019-07-30 18:29:50 +00:00
Jeremy Stanley
6631b899c5 Put gitea07 and gitea08 back into service
Add the gitea07.opendev.org and gitea08.opendev.org servers into the
haproxy pools now that they've been seeded with current data. Remove
the create repos task disable list entries for them as well.

Change-Id: I69390e6a32b01cc1713839f326fa930c376282af
2019-07-29 23:35:36 +00:00
Jeremy Stanley
56a0b08aa5 Swap gitea05 into service and bring down 07 and 08
Add the gitea05.opendev.org server into the haproxy pools now that
it's been seeded with current data. Switch the create repos task
disable list from 05 to 07 and 08, and remove 07 and 08 from the
Ansible inventory and comment them out in the haproxy pools in
preparation for replacement.

To the casual observer it may appear gitea06 is being skipped, but
it was replaced first out of sequence due to filesystem corruption
during the PTG. The increased performance of the 75% of the nodes
which have already been replaced means we can get by doing the final
25% at the same time (so two servers at once).

Change-Id: Ia49157c16582b7ed0dbef3eb9d07bf7f1d4450b9
2019-07-29 16:56:39 +00:00