15161 Commits

Author SHA1 Message Date
Kevin Carter
525d21a332
Add proxy for registry.access.redhat
This change adds a proxy config for registry.access.redhat which should
assist us when gating using images provided by the publically available
registry.

Change-Id: Ica7477d63659610de852d305a63f3e78d0dd8c4f
Signed-off-by: Kevin Carter <kecarter@redhat.com>
2019-07-10 07:31:08 -05:00
James E. Blair
6d66d7ca34 Remove .zuul.yaml file matchers
The change at https://review.opendev.org/669752 will cause the
self-testing behavior we wanted from this, but will apply more
narrowly, so that jobs are run when their own configuration
changes.  Since this is no longer needed, remove it.

Change-Id: I50a863cab3bd7a3535fd0185d4ec9d1307b1b7d6
2019-07-08 21:48:30 +00:00
Zuul
aeca18cb03 Merge "Add apache restart handler for fortnebula LE setup" 2019-07-08 15:17:01 +00:00
Ian Wienand
6b22833765 Add letsencrypt documentation
Add an overview of Let's Encrypt operation, and details of requesting
a certificate for hosts.

Change-Id: I636dbeb60383edd79f8d852c52272f6a79043154
2019-07-08 15:21:44 +10:00
Zuul
fb876fcd42 Merge "Remove absentee clouds from cloud launcher" 2019-07-04 06:09:55 +00:00
Ian Wienand
23f4f3989d mirror-update: update docs for mirror-update.opendev.org
Update AFS docs to refer to the new host

Change-Id: Ib6b54729e0b186ceb7d0beffbbd68bcab0e2e1ba
2019-07-04 09:11:40 +10:00
Ian Wienand
959f0301e7 mirror-update: export mirroring logs
This adds a periodic job to copy logs to a mirror volume, and export
it via the usual mirror http.

I have precreated the log volume; just as a R/W volume because this is
expected to be very low volume access.

Change-Id: I67870f6d439af2d2a63a5048ef52cecff3e75275
2019-07-04 09:11:29 +10:00
Clark Boylan
211aac5a9b Add apache restart handler for fortnebula LE setup
THis was missed previously but we need it for LE to properly restart
apache when certs update.

Change-Id: I6cf498ce1ec8cf5b936d2fedbbfe3c9666483e07
2019-07-03 16:04:58 -07:00
Zuul
aeefe544e2 Merge "openafs-client: ensure latest package and reorder install" 2019-07-03 22:41:46 +00:00
Clark Boylan
58b5fd8022 Add fortnebula mirror LE details dict
We need to supply information to ansible about how to provision LE certs
for the new fortnebula mirror. Add this dict to host_vars for
mirror01.regionone.fortnebula.opendev.org.

Change-Id: I02218e26ab6e9fad67e634f22de207740506d9e1
2019-07-03 14:39:25 -07:00
Clark Boylan
8d6cda76ee Remove absentee clouds from cloud launcher
We've had some changes to our cloud landscape over the past little while
and cloud launcher is bailing out early when it hits clouds it can't
talk to. Fix this by removing the clouds/regions that no longer
function/exist so that we can configure the clouds that are listed
later.

Change-Id: I803655325d3a92c6d228499800b29332b5b32741
2019-07-03 14:14:27 -07:00
Zuul
efc8e17c6e Merge "Add fortnebula ci mirror" 2019-07-03 18:14:48 +00:00
Zuul
fbf3302658 Merge "Use qcow2 images on fortnebula" 2019-07-03 16:41:58 +00:00
Zuul
ba69662b4d Merge "Add fortnebula cloud to cloud launcher" 2019-07-03 16:35:57 +00:00
Clark Boylan
95f02c7aa7 Add fortnebula ci mirror
Note we depends on the DNS updates so that LE cert provisioning works
on the first pass.

Depends-On: https://review.opendev.org/668929
Change-Id: I953938b77bfce67be0cb55af5cf4bd64044100f4
2019-07-03 07:53:41 -07:00
Roman Gorshunov
d50e17e043 Mailing list for Airship
This creates mailing list
airship-job-failures@lists.airshipit.org for Airship project.

Change-Id: Ia354c0440ababe99705041c618db2b6ea24d1450
2019-07-03 11:51:15 +00:00
Zuul
681ebb22ac Merge "mirror-update: fix cron path" 2019-07-03 04:24:26 +00:00
Zuul
d56082920c Merge "mirror-update: seed cron time with unique job name" 2019-07-03 04:24:25 +00:00
Ian Wienand
92b49d6b5e mirror-update: fix cron path
We have not put /usr/local/bin into the cron path, so it does not find
the update scripts.  Since the scripts were working with this path on
the old host, restore it as a cron variable to maintain the status
quo.

Change-Id: Id9b7533720d3ccd9251055dec5b452cf5963dc85
2019-07-03 13:32:19 +10:00
Ian Wienand
89529d8dbd mirror-update: seed cron time with unique job name
Currently we start all jobs at the same time, which was not the
intent.  Switch this to seed on the unique name of the job, which
should space jobs out randomly.

Change-Id: Ib41d8ca10aefe4a29bdd02935de8a588ab881958
2019-07-03 11:36:43 +10:00
Zuul
50f698b207 Merge "Add mirror-update01.opendev.org server" 2019-07-03 00:44:07 +00:00
Zuul
daad505ca1 Merge "mirror-update: update keytab testing" 2019-07-03 00:44:06 +00:00
Zuul
5d0d5725ec Merge "Move rsync mirror updates to new opendev.org mirror-update host" 2019-07-03 00:44:04 +00:00
Clark Boylan
cd9f3cfdad Apply service-bridge.yaml in run_all.sh
Prior to https://review.opendev.org/#/c/656871/ this code was executed
by run_all.sh in every pass but seems to have been missed as part of
656871's base.yaml split up.

Add service-bridge.yaml to run_all.sh to get these updates applying to
bridge again. In particular things like clouds.yaml updates are missing
otherwise.

Note I've not merged bridge.yaml and service-bridge.yaml as it appears
we want all of the service stuff to happen after base.yaml but
bridge.yaml needs to happen before. I think this is why they were split
in the first place.

Change-Id: I0a7ce1a65cd19459bbaf244b94a23ddde360da1a
2019-07-02 15:04:55 -07:00
Clark Boylan
c27f0ab838 Use qcow2 images on fortnebula
They upload more quickly and take up less disk space.

Change-Id: Ic6069299ae32d148f7cffbc80db00b728cc6db92
2019-07-02 14:09:12 -07:00
Clark Boylan
bee327654e Add fortnebula cloud to cloud launcher
This will configure security groups and ssh keys.

Change-Id: I4cbe4d5b6fb705f85e626fe7a047aa2522ab14a0
2019-07-02 14:06:54 -07:00
Ian Wienand
439da9ec02 openafs-client: ensure latest package and reorder install
We've noticed that openafs was not getting upgraded to the PPA version
on one of our opendev.org mirrors.  Switch install of packages to
"latest" to make sure it upgrades (reboots to actually apply change
unresolved issue, but at least package is there).

Also, while looking at this, reorder this to install the PPA first,
then ensure we have the kernel headers, then build the openafs kernel
modules, then install.  Add a note about having to install/build the
modules first.

Change-Id: I058f5aa52359276a4013c44acfeb980efe4375a1
2019-07-03 06:51:09 +10:00
Zuul
2e5c074741 Merge "do backports for Debian 'buster'" 2019-07-02 10:51:07 +00:00
Ian Wienand
ece14bbfb0 Add mirror-update01.opendev.org server
Add the new mirror-update server as a follow-on to
I525ac18b55f0e11b0a541b51fa97ee5d6512bf70.

Also ensure that the new mirror server isn't in the puppet groups by
only matching the openstack.org one.

Also remove from the afsadmin group.  This group is only used for
keytabs stored on bridge.o.o.  I don't think that we need group for
the keytabs -- a keytab should only ever be in use on one host at a
time, so we are better off keeping the keytabs in a specific host_var
for the host they are used on, rather than being in a group and
possibly deployed on servers where they are not used.

Depends-On: https://review.opendev.org/668610
Change-Id: Icda92bb234adc00f6718c1c656e8f069ce2704c4
2019-07-02 17:34:09 +10:00
Ian Wienand
aa357fc19f mirror-update: update keytab testing
Keytabs are slightly longer than what is being tested; upto 100 bytes
or so.  This means the encoded data breaks over lines, which means you
need to be more careful about quoting.

Update the testing to a longer keytab (100 bytes of random data) and
fix up the quoting.  Also enable no_logging to avoid putting key
material into the logs.

Change-Id: I73c391a2ebd2c962dc9a422f9d44265160210852
2019-07-02 17:17:20 +10:00
Ian Wienand
b85282c046 Move rsync mirror updates to new opendev.org mirror-update host
This move was prompted by wishing to expose the mirror update logs for
the rsync updates so that debugging problems does not require a root
user (note: not actually done in this change; will be a follow-on).

Rather than start hacking at puppet, the rsync mirror scripts make a
nice delination point for starting an Ansible-first/Bionic update.

Most magic is included in the scripts, so there is not much more to do
than copy them.  The host uses the existing kerberos and openafs roles
and copies the key material into place (to be added before merge).

Note the scripts are removed from the extant puppet so we don't have
two updates happening simultaneously.  This will also require a manual
clean to remove the cron jobs as a once-off when merging.

The other part of mirror-update is the reprepro based scripts for the
various debuntu repositories.  They are left as future work for now.

Testing is added to ensure dependencies and scripts are all in place.

Change-Id: I525ac18b55f0e11b0a541b51fa97ee5d6512bf70
2019-07-02 16:42:33 +10:00
Zuul
784ee20501 Merge "Only backup the gitea database on gitea hosts" 2019-07-01 20:53:23 +00:00
Ian Wienand
b742bfc911 letsencrypt-install-txt-record: skip disabled hosts
We are seeing:

  fatal: [adns1.opendev.org]: FAILED! => {"msg": "The task includes an
  option with an undefined variable. The error was:
  'ansible.vars.hostvars.HostVarsVars object' has no attribute
  'acme_txt_required'

I belive this is because we have a disabled mirror host now.  So the
iad.rx.opendev.org mirror is in the "letsencrypt" group, but because
it is also disabled the prior role (letsencrypt-request-certs) has not
run and it has not populated it's "acme_txt_required" variable.

We should skip disabled hosts when inspecting the hosts for this
variable.  Add this to the "with_inventory_hostnames" match.

Change-Id: I33a1c8b6f7e8499248e370f69a9f573a2bf106a5
2019-07-01 13:06:57 +10:00
Zuul
b3776ca3b0 Merge "Add OVH GRA1 mirror" 2019-07-01 02:15:40 +00:00
Clark Boylan
3cc931b72d Add clouds.yaml entries for fortnebula cloud
Donnyd has kindly offered us access to fortnebula's test cloud. This
adds clouds.yaml entries to bridge and nodepool so that we can take
advantage of these resources.

Change-Id: I4ebc261c6f548aca0b3f37dc9b60ffac08029e67
2019-06-28 11:17:48 -07:00
Marcin Juszkiewicz
f830202eea do backports for Debian 'buster'
Change-Id: Id818034921fdf19ccdf67ef74803f97eb04e0acf
2019-06-28 14:05:38 +02:00
Ian Wienand
abf11982ce Raise callbacks for AFS server
As documented in [1]

 If the number next to "GotSomeSpaces" or any of the "GSS*" fields is
 greater than 0, then the fileserver ran out of callback space and had
 to prematurely revoke callback promises from clients in order to free
 up space.

Here's our stats on afs01:

  $ xstat_fs_test localhost -collID 3 -onceonly

  Starting up the xstat_fs service, no debugging, one-shot operation

  ------------------------------------------------------------
            13547865 DeleteFiles
          1849223729 DeleteCallBacks
            45049055 BreakCallBacks
          2098382037 AddCallBack
                 174 GotSomeSpaces
                7800 DeleteAllCallBacks
               20778 nFEs
               21184 nCBs
             1500000 nblks
            43425561 CBsTimedOut
                   0 nbreakers
                   8 GSS1
                   4 GSS2
                   5 GSS3
                 169 GSS4
                   4 GSS5

So as noted, the server ran out of callback spaces a few times.
Raising it takes only a little memory, but will help performance.

Thanks to Jeffrey Altman (auristor) for pointing this out.

[1] https://www.openafs.org/pages/newsletter/newsletter-2013-03-volume004-issue05.html

Change-Id: I2ad33dd8918cb559634d2c5b8c4e4e7f2d6d4051
2019-06-28 12:14:47 +10:00
Clark Boylan
4aac4b990a Add docs for deploying a new gitea server
We have gitea state now so deploying a new server requires a bit of
process. Document that process.

Change-Id: I946f9880b66efdfb39bc9894950cd02058ed987a
2019-06-27 12:31:48 -07:00
Clark Boylan
5727407486 Only backup the gitea database on gitea hosts
During a db recovery to rebuild a host using the existing db backups
resulted in a corrupt mysql.proc table. The issue seemed to be
attempting to restore the mysql database. Instead of dumping all
databases lets just backup the one we care about: gitea.

Change-Id: Ia2c87b62736fda1c8a9ce77126e383ec74990b4a
2019-06-27 09:53:34 -07:00
Zuul
d696d8b273 Merge "Put gitea06 back in the rotation" 2019-06-27 16:43:41 +00:00
Zuul
d36db889cc Merge "Revert "Move openSUSE Tumbleweed into a caching mirror instead"" 2019-06-27 01:44:50 +00:00
Ian Wienand
7810230408 Add OVH GRA1 mirror
This mirror will be manually configured with kafs (see
https://review.opendev.org/623974).  This should be a nice distant
geographic counterpoint to the IAD RAX server.

This will need to be manually configured with a custom kernel for now,
but fixes are making their way upstream and this host will be
converted when available.

Depends-On: https://review.opendev.org/667529
Change-Id: I6a22933029c096c781c93c33e6edf03bf59223c9
2019-06-27 10:07:44 +10:00
Clark Boylan
6baa9dca5c Put gitea06 back in the rotation
This server was replaced and has had its db restored from backup on
gitea01, repo dirs recreated via gitea admin ui function, and gerrit has
replicated all repo content to this server.

Put this back into the rotation in haproxy as well as the ansible
management of gitea git repos.

Change-Id: I424d0db0adf0787d5d46e264b6552d79b48f27ef
2019-06-26 16:36:57 -07:00
Jeremy Stanley
428872075e Revert "Move openSUSE Tumbleweed into a caching mirror instead"
This reverts commit b3ce1c52dc7ca455ffd94ea07d8a4fb1b6905fa8.
It removed the AFS mirror at the same time it added the proxy,
but jobs don't know to look for the proxy since it's on a
totally different TCP port.

Change-Id: I87cc03eb3322bd7b093dd6fe798aadb48f319805
2019-06-26 23:29:20 +00:00
Ian Wienand
f673b71466 launch-node.py : add option to skip ipv6 address checks
As noted inline, this needs to be skipped on OVH (and I always forget,
and debug this over and over when launching a mirror node there :).

Change-Id: I07780e29f5fef75cdbab3b504f278387ddc4b13f
2019-06-26 18:28:28 +10:00
Clark Boylan
3734f000f9 Enroll new gitea06 into ansible inventory
We add the new host so that it will get configured as a gitea backend
server. We exclude this server from the list of gitea hosts to configure
git repos on because we want to recover its DB from one of the other
sibling nodes first. This should preserve the http redirects for us.

Once we have the db recovered we can enable replication from gerrit then
readd this host to the haproxy load balancer.

Change-Id: Ia2a98e5ded43cad044db36ca8d0da5a96277afee
2019-06-25 15:16:59 -07:00
Clark Boylan
263ab148fe Remove gitea06 from our inventory file
Note we don't fully remove it from cacti and hiera and so on because we
are replacing this server and we just want ansible to ignore the old
gitea06 for a bit while we bootstrap the new server.

Change-Id: Iaa89e77c055d8099a7d3d511723782fead43ce74
2019-06-25 14:03:44 -07:00
Zuul
25b16b0c28 Merge "Remove dead link from 'paste' documentation" 2019-06-25 17:28:16 +00:00
Zuul
920dbd7584 Merge "Separate openafs CI mirror" 2019-06-25 06:47:32 +00:00
Zuul
e6f17aa7f8 Merge "Use openstack-ci-core PPA for openafs 1.8.3" 2019-06-25 06:29:36 +00:00