591 Commits

Author SHA1 Message Date
Zuul
9867d6c6bb Merge "Update to ansible-lint 4.1.0" 2019-06-11 01:48:18 +00:00
Zuul
88909d0a20 Merge "bridge.opendev.org: use Ansible 2.8.0 stable" 2019-06-11 01:48:16 +00:00
Jeremy Stanley
d0ff3e48d1 Suppress progress for git gc cron on Gitea servers
The stdout progress feed from `git gc` is fairly verbose and
targeted at audiences running it interactively. Since our cron for
this iterates over thoudands of repositories on our Gitea servers,
we don't need to send the progress info to all our sysadmins by
E-mail. Instead use the --quiet option to the gc subcommand so that
progress output will be suppressed.

If this still proves too verbose (as in, continues to result in
E-mail to root even when there are no failures), we can try
redirecting stdout to /dev/null.

Change-Id: Idc06e48cbf85e127a343c2a3cf51a35e6ed09685
2019-06-09 14:30:28 +00:00
James E. Blair
3199e3b225 Enable SPF checking on lists
This requires an external program and only works on Debian hosts.

Newer versions of exim (4.91) have SPF functionality built-in, but
they are not yet available to us.

Change-Id: Idfe6bfa5a404b61c8761aa1bfa2212e4b4e32be9
2019-06-07 10:34:33 -07:00
Zuul
a12de2104e Merge "mirror: rename 80/443 log files" 2019-06-07 13:21:00 +00:00
Ian Wienand
42e54e2c08 mirror: rename 80/443 log files
Having proxy_[80|443]_access.log is wrong beacuse they're not really
proxies (I think I just copied this incorrectly).  Change it to
mirror_, and update the macro that is only used on the mirror portions
too.

Change-Id: I8eca941fee9606d25dd25bc54bc552ccc7094e0f
2019-06-07 10:14:14 +10:00
Ian Wienand
52780440ff Update to ansible-lint 4.1.0
In a follow-on change (I9bf74df351e056791ed817180436617048224d2c) I
want to use #noqa to ignore an ansible-lint rule on a task; however
emperical testing shows that it doesn't work with 3.5.1.  Upgrading to
4.1.0 it seems whatever was wrong has been fixed.

This, however, requires upgrading to 4.1.0.

I've been through the errors ... the comments inline I think justify
what has been turned off.  The two legitimate variable space issues I
have rolled into this change; all other hits were false positives as
described.

Change-Id: I7752648aa2d1728749390cf4f38459c1032c0877
2019-06-06 22:13:12 +00:00
Zuul
f25deabf9c Merge "Add db backups to gitea" 2019-06-06 19:38:51 +00:00
Ian Wienand
8a06d48c84 bridge.opendev.org: use Ansible 2.8.0 stable
Testinfra works with Ansible 2.8.0 now, so we can update
bridge.opendev.org to the latest version.  This also needs an ARA
update; bring it to the latest 0.16.4 release.

Update test-requirements so that tox/ansible-lint use Ansible 2.8.0
too.  See note inline about dependencies.

Note we replace import_tasks with include_tasks in handlers to address
this porting issue:
https://docs.ansible.com/ansible/latest/porting_guides/porting_guide_2.8.html#imports-as-handlers

Change-Id: I7ed75d253857f86b68f67023af6897af4e1b4f50
2019-06-06 11:25:06 -07:00
James E. Blair
2e5291f377 Get an LE cert for tarballs.opendev.org
Depends-On: https://review.opendev.org/663424
Change-Id: I4faa12b5d241144463ccf7ec59ef2d0b11479c35
2019-06-05 13:56:34 -07:00
Clark Boylan
e832987fca Add db backups to gitea
This isn't added as a separate role because it heavily relies on the
gitea deployment specific (docker-compose, service names, etc). If we
end up running more services with docker-compose and databases we can
probably make this reconsumable.

Change-Id: I7b9084a8a90a86f73f5b24de505978d3f286850b
2019-06-04 16:07:46 -07:00
Zuul
1fe34e00d4 Merge "Add control plane clouds to nodepool builder clouds.yaml" 2019-06-04 20:15:24 +00:00
James E. Blair
2c6e1e2061 Fix gitea rename playbook org creation
The org creation task list requires a list of the existing orgs.
Copy that from the gitea creation playbook.

Change-Id: Ia21f6211004f8dde3cacf4fb549ea8418a6d2888
2019-05-31 09:11:16 -07:00
Clark Boylan
3f1d5ccdde More repo rename playbook fixes
These fixes were either missed by the omnibus or introduced by new
changes since the big opendev migration.

Change-Id: I58e2b2c93567b47b161fdbbf143ff58738a577b8
2019-05-31 09:07:26 -07:00
Clark Boylan
48945cabc2 Serve ubuntu package content on opendev mirrors
The /var/www/mirror/ubuntu -> /afs/openstack.org/mirror/ubuntu symlink
was missing so we weren't serving ubuntu mirror content from the opendev
mirror. Add this to the list of afs content symlinks we create.

Change-Id: I10b985afbaa737033cd5c1d4dd72eb8e77f8eb32
2019-05-30 15:20:10 -07:00
Jeremy Stanley
4f117bcecc Project renames include keys on zuul scheduler
Add tasks to the rename_repos utility playbook for moving the
per-project secrets and ssh keys on the zuul scheduler's filesystem,
creating new namespace parent directories if they don't already
exist.

Change-Id: Iccce53953d5829bd4eb5fe4c33c9d2f195ae825c
2019-05-30 16:17:28 +00:00
Clark Boylan
b50a748d44 Switch git lb to source balance method
We were using the leastconn method which sends new connections to the
backend with the least number of connections. Unfortunately git clients
seem to have trouble with varying backend repo state (due to GC and
packing) and the thought is sending all requests from a single client to
a single backend will alleviate this.

To do this we switch to the source balance method which hashes the
source IP and finds a stable backend to talk to. This method handles
backend outages fine as it will hash to a new backend if the older one
goes offline.

Change-Id: I2c7a4ec0809a2f4ef6556833ac6a0ff3651904dd
2019-05-28 08:17:05 -07:00
James E. Blair
5faf89f566 Add haproxy-statsd to haproxy server
Build a container image with the haproxy-statsd script, and run that
along with the haproxy container.

Change-Id: I18be70d339df613bf9a72e115e80a6da876111e0
2019-05-24 15:40:28 -07:00
Zuul
1330ed82c8 Merge "Add cron to gc on gitea servers" 2019-05-24 18:49:43 +00:00
James E. Blair
a92ac59e15 Fix new mirror system errors
Fix the reported stat name for the mirror playbook.

Run the mirror job in gate.

Set follow=false so that we're telling Ansible to set the perms
on the link rather than the target (which is the default).

Change-Id: Id594cf3f7ab1dacae423cd2b7e158a701d086af6
2019-05-24 09:42:38 -07:00
Zuul
5e908c739b Merge "Move openSUSE Tumbleweed into a caching mirror instead" 2019-05-24 08:01:38 +00:00
Dirk Mueller
b3ce1c52dc Move openSUSE Tumbleweed into a caching mirror instead
Tumbleweed is only rarely used in the openStack CI, so mirroring it
fully is not worth the time/space overhead. a caching proxy
should be good enough. Add it to the directories to clean up
and remove the older entries because they will no longer be
matching.

Change-Id: I987da098cf4a7330cdec8da9ae3cfbff2f330bf8
2019-05-24 16:19:40 +10:00
James E. Blair
b87c2d02ab Add cron to gc on gitea servers
As new change refs accumulate, replication pushes and page loads
will take longer as git stats all of the refs/ files.  To avoid
that, pack refs and gc every week to keep the number of files
and space used minimal.

Change-Id: Iff273ebbc25a512ab7e12b8418ceb30e7c722f92
2019-05-23 15:33:55 -07:00
Monty Taylor
ff1b8a94c6 Add control plane clouds to nodepool builder clouds.yaml
In order to have nodepool build images and upload them to control
plane clouds, add them to the clouds.yaml on the nodepool-builder
hosts. Keep them out of the launcher configs by splitting the config
templates. So that we can keep our copies of things to a minimum,
create a group called "control-plane-clouds" and put bridge and nb0*
in it.

There are clouds mentions in here that we no longer use, a followup
patch will clean those up.

NOTE: Requires shifting the clouds config dict from
host_vars/bridge.openstack.org.yaml to group_vars/control-plane-clouds.yaml
in the secrets on bridge.

Needed-By: https://review.opendev.org/640044
Change-Id: Id1161bca8f23129202599dba299c288a6aa29212
2019-05-23 14:34:10 -05:00
Zuul
509ec18dc9 Merge "Omnibus rename repo playbook fixes" 2019-05-23 18:17:00 +00:00
Monty Taylor
69f618d36c Disable openid login and signup
This is not a feature we're intending to support at the current
time.

Change-Id: Ie33c266c8ebcaeb471066b52ce37c56c04f93e5d
2019-05-23 10:32:01 -05:00
Adam Coldrick
e9b2ca3774 Update key for SotK
Change-Id: Ic0ca12a5036fb9025f05c2a9c267da84af62dafc
2019-05-22 20:09:08 +01:00
Zuul
0ae85ed7bb Merge "Use local fork of gitea and upgrade to 1.8.0" 2019-05-22 16:58:09 +00:00
Zuul
2c78db0146 Merge "letsencrypt : use date call for serial number" 2019-05-22 07:44:12 +00:00
Ian Wienand
93bb1d549e letsencrypt : use date call for serial number
Per [1] ansible_date_time is NOT actually the date/time -- it is the
time cached from the facts.  It seems this can not be changed because,
of course, things have started depending on this behaviour.

This is particuarly incorrect if you're using this as a serial number
for DNS and it is not incrementing across runs, and thus bind is
refusing to load the new entries in the acme.opendev.org zone during
letsencrypt runs, and the TXT authentication fails.

Use the suggested work-around in the issue which is an external call
to date.

[1] https://github.com/ansible/ansible/issues/22561

Change-Id: Ic3f12f52e8fbb87a7cd673c37c6c4280c56c2b0f
2019-05-22 16:41:51 +10:00
Zuul
afc8e507af Merge "mirror01.dfw.rax.opendev.org : use python3 for ansible" 2019-05-22 01:26:03 +00:00
Ian Wienand
2e9992af9e mirror01.dfw.rax.opendev.org : use python3 for ansible
This is a bionic host, so requires this to run as it has no
/usr/bin/python.  This is the same as the other bionic hosts, I just
forgot it.

Change-Id: Ifdd1df2fa83dd25dcc20596ce17e2f0c88279c62
2019-05-22 10:03:11 +10:00
Zuul
41c06cdf49 Merge "Bringup mirror01.dfw.rax.opendev.org" 2019-05-21 23:42:57 +00:00
Zuul
46c09946b4 Merge "Adds new key for diablo_rojo" 2019-05-21 23:01:30 +00:00
Kendall Nelson
ddc677db19 Adds new key for diablo_rojo
Change-Id: I3805ebcf613ba4459efe0bc28f6c4b0283eb12df
2019-05-22 00:01:16 +02:00
James E. Blair
70b8118ab0 Use local fork of gitea and upgrade to 1.8.0
This has a few emergency local patches while we wait for them to
appear in an upstream release.

This updates the modified templates to match the changes in 1.8.0
upstream.

This also disables the oauth2 service, which is new in 1.8.0.
Without disabling this, gitea tries to generate a JWT secret and
write it to the file, which in our case is read only. If we want
to enable it, we need to add a new JWT_SECRET setting.

Change-Id: I969682bce6ff25b7614ce9265097307ee9cbc6cb
Co-Authored-By: Monty Taylor <mordred@inaugust.com>
2019-05-21 12:16:21 -05:00
Ian Wienand
73bbc6787f Bringup mirror01.dfw.rax.opendev.org
This is an initial host for testing opendev.org mirrors

Change-Id: I26b9ed1e21e2111f48bc7ecc384880c274eed213
Depends-On: https://review.opendev.org/660235
2019-05-21 11:08:30 +10:00
Ian Wienand
670107045a Create opendev mirrors
This impelements mirrors to live in the opendev.org namespace.  The
implementation is Ansible native for deployment on a Bionic node.

The hostname prefix remains the same (mirrorXX.region.provider.) but
the groups.yaml splits the opendev.org mirrors into a separate group.
The matches in the puppet group are also updated so to not run puppet
on the hosts.

The kerberos and openafs client parts do not need any updating and
works on the Bionic host.

The hosts are setup to provision certificates for themselves from
letsencrypt.  Note we've added a new handler for mirror nodes to use
that restarts apache on certificate issue/renewal.

The new "mirror" role is a port of the existing puppet mirror.pp.  It
installs apache, sets up some modules, makes some symlinks, sets up a
cleanup cron job and installs the apache vhost configuration.

The vhost configuration is also ported from the extant puppet.  It is
simplified somewhat; but the biggest change is that we have extracted
the main port 80 configuration into a macro which is applied to both
port 80 and 443; i.e. the host will have SSL support.  The other ports
are left alone for now, but can be updated in due course.

Thus we should be able to CNAME the existing mirrors to new nodes, and
any existing http access can continue.  We can update our mirror setup
scripts to point to https resources as appropriate.

Change-Id: Iec576d631dd5b02f6b9fb445ee600be060f9cf1e
2019-05-21 11:08:25 +10:00
Zuul
2c5847dad9 Merge "Split the base playbook into services" 2019-05-20 10:04:40 +00:00
James E. Blair
8ad300927e Split the base playbook into services
This is a first step toward making smaller playbooks which can be
run by Zuul in CD.

Zuul should be able to handle missing projects now, so remove it
from the puppet_git playbook and into puppet.

Make the base playbook be merely the base roles.

Make service playbooks for each service.

Remove the run-docker job because it's covered by service jobs.

Stop testing that puppet is installed in testinfra. It's accidentally
working due to the selection of non-puppeted hosts only being on
bionic nodes and not installing puppet on bionic. Instead, we can now
rely on actually *running* puppet when it's important, such as in the
eavesdrop job. Also remove the installation of puppet on the nodes in
the base job, since it's only useful to test that a synthetic test
of installing puppet on nodes we don't use works.

Don't run remote_puppet_git on gitea for now - it's too slow. A
followup patch will rework gitea project creation to not take hours.

Change-Id: Ibb78341c2c6be28005cea73542e829d8f7cfab08
2019-05-19 07:31:00 -05:00
Zuul
8ff026ee33 Merge "letsencrypt: use a fake CA for self-signed testing certs" 2019-05-16 23:51:19 +00:00
Zuul
33e09b7ef5 Merge "Use handlers for letsencrypt cert updates" 2019-05-16 23:51:18 +00:00
Zuul
157ad6d521 Merge "Prune docker images after docker-compose up" 2019-05-16 22:55:04 +00:00
Ian Wienand
d5b321b074 Handle moved puppet repos
As per [1], it seems puppet has "cleaned up" most of the packages we
are using to install.

Install the puppet-agent packages directly as puppet's archive location
is not a valid repo. With puppet 4 at least these packages should bundle
everything we need including ruby.

[1] https://groups.google.com/forum/#!msg/puppet-users/cCsGWKunBe4/OdG0T7LeDAAJ

Depends-On: https://review.opendev.org/659384
Depends-On: https://review.opendev.org/659395
Change-Id: Ie9e2b79b42f397bddd960ccdc303b536155ce123
2019-05-15 16:03:07 -07:00
Zuul
4f92eb85a2 Merge "Force ipv4 on vexxhost nodepool nodes" 2019-05-14 21:27:09 +00:00
Zuul
91a3ce7e4d Merge "Update zuul servers to puppet 4" 2019-05-14 20:21:03 +00:00
Clark Boylan
6176978039 Force ipv4 on vexxhost nodepool nodes
Do this in an attempt to mitigate/work around the dns resolution
problems we have had in that cloud. One thoguht is that this could be
ipv6 specific.

Change-Id: I1f9ef4a031749484d06de9427943abac4de33d29
2019-05-14 11:54:43 -07:00
Ian Wienand
1992a9c1ec letsencrypt: use a fake CA for self-signed testing certs
Production letsencrypt certificate generation creates an intermediate
chain file (ca.cer); to simulate this during the self-signed tests
generate a fake CA certifcate, and use that to sign the generated
server certificate.

Tests updated to look for all these files

Change-Id: I3990529bca7ff3c6413ed0066f9c4feaf5464b1c
2019-05-14 10:24:28 +10:00
Ian Wienand
733122f0df Use handlers for letsencrypt cert updates
This change proposes calling a handler each time a certificate is
created/updated.  The handler name is based on the name of the
certificate given in the letsencrypt_certs variable, as described in
the role documentation.

Because Ansible considers calling a handler with no listeners an error
this means each letsencrypt user will need to provide a handler.

One simple option illustrated here is just to produce a stamp file.
This can facilitate cross-playbook and even cross-orchestration-tool
communication.  For example, puppet or other ansible playbooks can
detect this stamp file and schedule their reloads, etc. then remove
the stamp file.  It is conceivable more complex listeners could be
setup via other roles, etc. should the need arise.

A test is added to make sure the stamp file is created for the
letsencrypt test hosts, which are always generating a new certificate
in the gate test.

Change-Id: I4e0609c4751643d6e0c8d9eaa38f184e0ce5452e
2019-05-14 08:14:51 +10:00
Zuul
3367358fc6 Merge "Don't gather facts in set-hostnames" 2019-05-13 14:10:28 +00:00