200 Commits

Author SHA1 Message Date
James E. Blair
2e5291f377 Get an LE cert for tarballs.opendev.org
Depends-On: https://review.opendev.org/663424
Change-Id: I4faa12b5d241144463ccf7ec59ef2d0b11479c35
2019-06-05 13:56:34 -07:00
Zuul
1fe34e00d4 Merge "Add control plane clouds to nodepool builder clouds.yaml" 2019-06-04 20:15:24 +00:00
Zuul
4166abf258 Merge "Remove opendev k8s cluster from inventory" 2019-05-29 15:23:09 +00:00
Monty Taylor
ff1b8a94c6 Add control plane clouds to nodepool builder clouds.yaml
In order to have nodepool build images and upload them to control
plane clouds, add them to the clouds.yaml on the nodepool-builder
hosts. Keep them out of the launcher configs by splitting the config
templates. So that we can keep our copies of things to a minimum,
create a group called "control-plane-clouds" and put bridge and nb0*
in it.

There are clouds mentions in here that we no longer use, a followup
patch will clean those up.

NOTE: Requires shifting the clouds config dict from
host_vars/bridge.openstack.org.yaml to group_vars/control-plane-clouds.yaml
in the secrets on bridge.

Needed-By: https://review.opendev.org/640044
Change-Id: Id1161bca8f23129202599dba299c288a6aa29212
2019-05-23 14:34:10 -05:00
Clark Boylan
0848e0760b Remove the ask.openstack.org inventory entry
This trusty server has been replaced by xenial ask01.openstack.org.

Change-Id: I33090c9ce45982e19d4ef85c156e76e7583a07af
2019-05-23 12:20:09 -07:00
Clark Boylan
08152aa22f Remove groups configuration
This removes the groups servers from our inventory as well as our
manifests/modules. We don't run the groups service anymore as many
groups migrated to meetup.com independent of us and the others have
transitioned there.

Change-Id: I7cb76611e6d30e7189821923f36a38dec9ea7241
2019-05-23 12:20:04 -07:00
Zuul
41c06cdf49 Merge "Bringup mirror01.dfw.rax.opendev.org" 2019-05-21 23:42:57 +00:00
Zuul
54c72ab7b9 Merge "Create opendev mirrors" 2019-05-21 23:01:28 +00:00
Zuul
05300b6268 Merge "Update ask.openstack.org to puppet 4" 2019-05-21 19:35:04 +00:00
Zuul
82e498fb59 Merge "Remove ask-staging* from disabled list" 2019-05-21 08:39:28 +00:00
Ian Wienand
73bbc6787f Bringup mirror01.dfw.rax.opendev.org
This is an initial host for testing opendev.org mirrors

Change-Id: I26b9ed1e21e2111f48bc7ecc384880c274eed213
Depends-On: https://review.opendev.org/660235
2019-05-21 11:08:30 +10:00
Ian Wienand
670107045a Create opendev mirrors
This impelements mirrors to live in the opendev.org namespace.  The
implementation is Ansible native for deployment on a Bionic node.

The hostname prefix remains the same (mirrorXX.region.provider.) but
the groups.yaml splits the opendev.org mirrors into a separate group.
The matches in the puppet group are also updated so to not run puppet
on the hosts.

The kerberos and openafs client parts do not need any updating and
works on the Bionic host.

The hosts are setup to provision certificates for themselves from
letsencrypt.  Note we've added a new handler for mirror nodes to use
that restarts apache on certificate issue/renewal.

The new "mirror" role is a port of the existing puppet mirror.pp.  It
installs apache, sets up some modules, makes some symlinks, sets up a
cleanup cron job and installs the apache vhost configuration.

The vhost configuration is also ported from the extant puppet.  It is
simplified somewhat; but the biggest change is that we have extracted
the main port 80 configuration into a macro which is applied to both
port 80 and 443; i.e. the host will have SSL support.  The other ports
are left alone for now, but can be updated in due course.

Thus we should be able to CNAME the existing mirrors to new nodes, and
any existing http access can continue.  We can update our mirror setup
scripts to point to https resources as appropriate.

Change-Id: Iec576d631dd5b02f6b9fb445ee600be060f9cf1e
2019-05-21 11:08:25 +10:00
Zuul
695a064036 Merge "Remove grafana01.openstack.org from inventory" 2019-05-20 22:23:33 +00:00
Ian Wienand
c796021bcb Add ask01.openstack.org to inventory
Change-Id: I474c0cf7bab51d2ec73a87af0a4ecbf910109c97
2019-05-20 17:56:55 +10:00
Ian Wienand
2e83c579f6 Remove ask-staging* from disabled list
These servers have been removed

Change-Id: I26ebd650866f9a71dd8b41f889878659785e4255
2019-05-20 17:25:20 +10:00
Monty Taylor
6bc8754b87 Remove opendev k8s cluster from inventory
We're not really using/maintaining this at the moment. Before we do
put it back in production, we're likely to simply rebuild it from
scratch.

Change-Id: I469f00e90903a010f2cec45031b049556eb268a2
2019-05-19 07:36:39 -05:00
Monty Taylor
7c54c2781b Remove unreachable hosts from inventory
None of these can be reached from bridge.

Change-Id: I2f4d419a7ea9993e90dba6d25681807f98ea1db5
2019-05-19 07:36:39 -05:00
Colleen Murphy
0f1c72ef13 Update ask.openstack.org to puppet 4
Change-Id: I102e42c5964fbdeabc9fef464f803b01e33e009d
2019-05-15 09:04:36 -07:00
Colleen Murphy
20356f1bdc Update lists.openstack.org to puppet 4
Change-Id: I90dfb0481ee2f720650b9c9a09b80151182654ec
2019-05-14 13:25:23 -07:00
Zuul
91a3ce7e4d Merge "Update zuul servers to puppet 4" 2019-05-14 20:21:03 +00:00
Ian Wienand
fc988d158b Remove grafana01.openstack.org from inventory
Replaced with grafana02.openstack.org in production

Change-Id: Ieca7b56ea9d79b5642943064e37bb99dc1b43eda
2019-05-10 09:26:53 +10:00
Zuul
67df630379 Merge "Remove graphite.openstack.org" 2019-05-08 00:14:25 +00:00
Zuul
279de88246 Merge "Remove linaro-cn1 region" 2019-05-07 20:23:07 +00:00
Ian Wienand
2acfc176b0 Remove graphite.openstack.org
The server has been removed, remove it from inventory.

While we're here, s/graphite.openstack.org/graphite.opendev.org/'
... it's a CNAME redirect but we might as well clean up.

Change-Id: I36c951c85316cd65dde748b1e50ffa2e058c9a88
2019-05-08 05:55:33 +10:00
Clark Boylan
6e61cbff2e Stop ansipuppeting the old cgit farm
We have replaced the cgit farm with a gitea farm. Stop managing the cgit
farm. This removes testing for centos7 as these were our only centos7
nodes.

Depends-On: https://review.opendev.org/654549
Change-Id: Ia48ff10cb88d51f609e8b28de176c72f7a9ee24f
2019-04-22 15:50:08 +00:00
Colleen Murphy
180897e49a Update zuul servers to puppet 4
This leaves ask.o.o and lists.o.o, which are still running Trusty, and
the cgit servers, which are likely to be decommissioned soon.

Change-Id: I78e7fd9e3079cc760da0aad955f6eeb32d442fc3
2019-04-17 16:53:56 +00:00
Colleen Murphy
c7f8b298ef Update nodepool servers to puppet 4
Except nb03.openstack.org, which runs on arm64 for which there are no
puppet 4 packages.

Change-Id: Ia85d20700309a9cd886886c4d4da52fb80ac595f
2019-04-11 21:35:51 +00:00
Ian Wienand
4abd0a3184 yamlgroup: add regex match; exclude puppet4 for arm64 mirrors
Two related changes that need to go together because we test with the
production groups.yaml.

Confusingly, there are arm64 PC1 puppet repos, and it contains a bunch
of things that it turns out are the common java parts only.  The
puppet-agent package is not available, and it doesn't seem like it
will be [1].  I think this means we can not run puppet4 on our arm64
xenial ci hosts.

The problem is the mirrors have been updated to puppet4 -- runs are
now breaking on the arm mirrors because they don't have puppet-agent
packages.  It seems all we can really do at this point is contine to
run them on puppet3.

This is hard (impossible?) to express with a fnmatch in the existing
yamlgroups syntax.  We could do something like list all the mirror
hosts and use anchors etc, but we have to keep that maintained.  Add
an feature to the inventory plugin that if the list entry starts with
a ^ it is considered a full regex and passed to re.match.  This
allows us to write more complex matchers where required -- in this
case the arm64 ci mirror hosts are excluded from the puppet4 group.

Testing is updated.

[1] https://groups.google.com/forum/#!msg/puppet-dev/iBMYJpvhaWM/WTGmJvXxAgAJ

Change-Id: I828e0c524f8d5ca866786978486bc04829464b47
2019-04-11 21:34:57 +00:00
Zuul
8e4cd58b2e Merge "Update kerberos servers to puppet 4" 2019-04-11 17:34:09 +00:00
Zuul
f028966fd3 Merge "Update AFS servers to puppet 4" 2019-04-10 23:27:10 +00:00
Zuul
8f9c2aada5 Merge "Update review.openstack.org to puppet 4" 2019-04-10 22:02:31 +00:00
Ian Wienand
9cf66757bc Remove linaro-cn1 region
This region has been retired.  See also
I149e74ff5e788bb860deb4c3cc38d9c8ed5766c6

Change-Id: Ia85587a7bbe52a92d70d6cb96d843ee5f5fc84f6
2019-04-09 11:57:25 +10:00
Zuul
534f1b368d Merge "Add graphite01.opendev.org letsencrypt configuration" 2019-04-09 00:33:25 +00:00
Zuul
693fe27610 Merge "letsencrypt : minor updates" 2019-04-08 23:02:16 +00:00
Zuul
f139a81994 Merge "letsencrypt support" 2019-04-08 22:43:54 +00:00
Colleen Murphy
8ac2c91d23 Update kerberos servers to puppet 4
Change-Id: I6cdb2bb154bfe1365d2dad6c00aa17f408379609
2019-04-05 09:31:33 -07:00
Colleen Murphy
a988c9253e Update AFS servers to puppet 4
Change-Id: I02d63fe1198a8d023814820602d425f891efdb73
2019-04-05 09:31:29 -07:00
Ian Wienand
45e88482fd Add graphite01.opendev.org letsencrypt configuration
This is an initial change for deploying letsencrypt certificates on
graphite01.opendev.org.  As we are still in a testing phase, use test
mode.

Change-Id: I3e762d071cc609856950898b36f1903fe52840a6
2019-04-05 16:50:59 +11:00
Ian Wienand
6088c788f1 letsencrypt : minor updates
Minor updates from review comments for
I1f66da614751a29cc565b37cdc9ff34d70fdfd3f

Change-Id: Ie011f768345ca3d8fdcc0b833f5645a635983d64
2019-04-05 16:50:34 +11:00
Ian Wienand
afd907c16d letsencrypt support
This change contains the roles and testing for deploying certificates
on hosts using letsencrypt with domain authentication.

From a top level, the process is implemented in the roles as follows:

1) letsencrypt-acme-sh-install

   This role installs the acme.sh tool on hosts in the letsencrypt
   group, along with a small custom driver script to help parse output
   that is used by later roles.

2) letsencrypt-request-certs

   This role runs on each host, and reads a host variable describing
   the certificates required.  It uses the acme.sh tool (via the
   driver) to request the certificates from letsencrypt.  It populates
   a global Ansible variable with the authentication TXT records
   required.

   If the certificate exists on the host and is not within the renewal
   period, it should do nothing.

3) letsencrypt-install-txt-record

   This role runs on the adns server.  It installs the TXT records
   generated in step 2 to the acme.opendev.org domain and then
   refreshes the server.  Hosts wanting certificates will have
   pre-provisioned CNAME records for _acme-challenge.host.opendev.org
   pointing to acme.opendev.org.

4) letsencrypt-create-certs

   This role runs on each host, reading the same variable as in step
   2.  However this time the acme.sh tool is run to authenticate and
   create the certificates, which should now work correctly via the
   TXT records from step 3.  After this, the host will have the
   full certificate material.

Testing is added via testinfra.  For testing purposes requests are
made to the staging letsencrypt servers and a self-signed certificate
is provisioned in step 4 (as the authentication is not available
during CI).  We test that the DNS TXT records are created locally on
the CI adns server, however.

Related-Spec: https://review.openstack.org/587283

Change-Id: I1f66da614751a29cc565b37cdc9ff34d70fdfd3f
2019-04-02 15:31:41 +11:00
Colleen Murphy
db0cf87ddb Update review.openstack.org to puppet 4
Change-Id: I841bae26862d4da41849835bb9f9548a2011cc95
2019-04-01 14:54:04 -07:00
Colleen Murphy
9a7172ab8a Upgrade lists.katacontainers.io to puppet 4
Change-Id: Ic0235ffec7d65a30a44fb518414e872a44b99f37
2019-04-01 14:53:42 -07:00
Clark Boylan
fa0d4f949e Update even more servers to puppet4
Change-Id: Ice2a07e0f1914b45690455b6b7199fc8441f21be
2019-03-22 09:51:25 -07:00
Clark Boylan
a67b27edda Run static and status under puppet4
With working globs now run static and status with puppet4

Change-Id: I091cfe755a416d8193b668ffe3b550d338e1d5f0
2019-03-21 14:41:15 -07:00
Clark Boylan
948e86d3cb Run static and status under futureparser
These two services had broken globs under the futureparser group. Move
them back to futureparser with working globs before we upgrade them to
puppet 4.

Change-Id: I32a3f56407fc2542985f3be2237a41260f7155d1
2019-03-21 14:40:06 -07:00
Colleen Murphy
f52134efc3 Fix groups.openstack.org glob
Change-Id: Ibc9427d97e492e7b0cf29ec39b6f919f4be60146
2019-03-21 14:38:56 -07:00
Clark Boylan
ba0242f054 Groups-dev to puppet4
This fixes the inventory glob to ensure we run groups-dev under puppet4.

Change-Id: I4cbc911a352d2968ba650a09c2a97a767cb8bc7d
2019-03-21 14:38:56 -07:00
Colleen Murphy
1c3a530ef1 Update more servers to puppet 4
Change-Id: I6fd2172fe937e123bd3ca1f0f8fa2a905661a50b
2019-03-16 21:11:35 +01:00
Colleen Murphy
2adff3ba22 Update more servers to puppet 4
Change-Id: Idd5079c2f24c30b7cac68d51e447c82865e7e038
2019-03-16 21:10:20 +01:00
Colleen Murphy
a734f54c5c Update more servers to puppet 4
Change-Id: Ifc81cbc2b95c0cfce718a31fce3c50dff9908049
2019-03-16 21:09:37 +01:00