902 Commits

Author SHA1 Message Date
Ian Wienand
44335ab2be pip3: Add python3-distutils
This is a requirement, otherwise get-pip.py fails to install

Change-Id: If8dc87d3755056af52f7f7415f6596071ac5feef
2020-03-13 10:38:52 +11:00
Zuul
f89e89afaf Merge "Added new Fedora release to mirroring" 2020-03-12 21:36:42 +00:00
Zuul
09444f065f Merge "Add nb01.opendev.org" 2020-03-12 21:36:40 +00:00
danpawlik
b5bb9790b4 Added new Fedora release to mirroring
Some jobs are using Fedora 29 release, which is deprecated.
Create new mirror with release 31 and wait for changing job
rules.

Change-Id: I418f4c815d337c2b1edd0df1aa065536d992cb62
2020-03-12 14:46:31 +01:00
Clark Boylan
68f740faf8 Remove old 404 checker job
This is being replaced with goaccess report jobs.

Change-Id: Ia22d847bfc1a9e450bd8c8e7fab77dd08bd1dfd0
2020-03-11 15:15:00 -07:00
Clark Boylan
9e394d24d0 Return goaccess html as zuul artifact
This will give us a nice link to the goaccess reports on the zuul
dashboard build pages.

Move ansible-lint config into config file

As of 4.2.0 we can configure ansible-lint with a config file. It's
also apparently now smart enough to only find ansible yaml. Let's
see how that goes.

Add a fake zuul_return module

This should let us fake out ansible-lint without having to install
all of zuul.

Change-Id: Ib233eb577a8ca2aabfe3a49b2cd823dd4a00bd82
2020-03-11 14:28:28 -05:00
Zuul
703dd8a05a Merge "nodepool-builder: Add webserver" 2020-03-11 00:31:12 +00:00
Zuul
c0b59083df Merge "Pull goaccess report into zuul logs" 2020-03-11 00:10:08 +00:00
Zuul
08f41a8b6b Merge "Simplify goaccess command" 2020-03-11 00:10:06 +00:00
Zuul
d1af4dd321 Merge "nodepool-builder: deploy from container" 2020-03-10 23:25:33 +00:00
Clark Boylan
24ed36fa34 Pull goaccess report into zuul logs
This will be our first step at publishing these publicly.

Change-Id: I7a2eb4ace5f76b8853867e1b4a767e5f3830ce96
2020-03-10 15:50:45 -07:00
Clark Boylan
2792fa5b76 Simplify goaccess command
This switches all log file input to stdin using a subshell to cat all
the related log files. This makes the command a bit more consistent with
itself.

Change-Id: I99a8b0326f2a27c8497d616501fd9bed48678da8
2020-03-10 15:40:16 -07:00
Ian Wienand
dbe0bf1ee6 Add nb01.opendev.org
This configures an opendev nodepool-builder

Change-Id: Id8603d9d7caaac0a1ab935e1c7c80d32b02ae23e
Depends-On: https://review.opendev.org/693118
2020-03-11 09:16:31 +11:00
Ian Wienand
b1bfee423b nodepool-builder: Add webserver
This adds the webserver that serves the logs and generated images.

Change-Id: I230f5291e0bd928af2e00966d76c3f385b749cb6
2020-03-11 09:16:31 +11:00
Ian Wienand
1979d6b160 nodepool-builder: deploy from container
This deploys the nodepool-builder container and verifies it has
started in testinfra.

Change-Id: I8a717d06f1291a4112b2753641ff88f074cf0b31
2020-03-11 09:16:24 +11:00
Clark Boylan
8fb306a22b Add periodic job to run goaccess reporting
This adds a simple periodic job that runs against static.opendev.org and
produces opendev reports. To start we don't publish these reports
publicly so that we can double check their contents first. Assuming that
all goes well we'll be able to apply this to all of our
static.opendev.org hosted sites and publish these reports through the
zuul logging system.

Change-Id: I66705808d435c16bf0da6989296c896099964aaa
Story: 2007387
2020-03-10 10:42:07 -07:00
Zuul
b2b0cc1c83 Merge "Add install zookeeper role; use for nodepool-builder testing" 2020-03-10 00:12:14 +00:00
Zuul
81c158c52f Merge "Add initial Ansible for nodepool hosts" 2020-03-10 00:10:36 +00:00
Jeremy Stanley
0e2e4425b6 Fix formatting for Open Edge LE playbook
The domain names passed to the Let's Encrypt handler need to be list
elements.

Change-Id: I0122b4b86dd17e3e2fbce211be554cbda4167d05
2020-03-07 18:07:29 +00:00
Jeremy Stanley
4cbdc2fc4d Set up inventory and cert for Open Edge mirror
This adds the Open Edge (formerly Fortnebula) CI mirror.

Change-Id: I1ccf2a602f8a41e00bc64a9516a326cc07d9b254
Depends-On: https://review.opendev.org/711787
2020-03-07 00:24:20 +00:00
Jeremy Stanley
43ed9fc297 Moving FortNebula to OpenEdge
Sister change for Ia5caff34d3fafaffc459e7572a4eef6bd94422ea and
removing earlier references to the mirror server in preparation for
building and adding the new one.

Change-Id: I7d506be85326835d5e77a0c9c461f2d457b1dfd3
2020-03-06 20:43:56 +00:00
Ian Wienand
e7f1062d51 Add install zookeeper role; use for nodepool-builder testing
This adds a simple role to install Zookeeper.

Add an option to nodepool-base to use this role to install Zookeeper.

Use this in the nodepool-builder gate testing where we are just
validating that the nodepool-builder container starts and is ready to
accept connections.  It needs a zookeeper to talk to, even though it
is not going to do anything.

Change-Id: I4ae89a51e454be4ee53ad4e04407162aaa8d9f9a
2020-03-06 14:02:52 +11:00
Ian Wienand
281425a44d Add initial Ansible for nodepool hosts
This is a start at ansible-deployed nodepool environments.

We rename the minimal-nodepool element to nodepool-base-legacy, and
keep running that for the old nodes.

The groups are updated so that only the .openstack.org hosts will run
puppet.  Essentially they should remain unchanged.

We start a nodepool-base element that will replace the current
puppet-<openstackci|nodepool> deployment parts.  For step one, this
grabs project-config and links in the elements and config file.

A testing host is added for gate testing which should trigger these
roles.  This will build into a full deployment test of the builder
container.

Change-Id: If0eb9f02763535bf200062c51a8a0f8793b1e1aa
Depends-On: https://review.opendev.org/#/c/710700/
2020-03-06 14:02:52 +11:00
Ian Wienand
ac11734cf9 ansible-lint : disable 503
This has got me a number of times; I think we can tell in review if a
task firing in response to a "changed" is best in a handler or not.

Remove existing noqa flags

Change-Id: I80ad631f978eeeb9903abe230a95f23f5709d20e
2020-03-06 09:57:00 +11:00
Zuul
fd50bb63aa Merge "letsencrypt: Register email with accounts" 2020-03-05 22:23:45 +00:00
Clark Boylan
6f9c151e79 Collect docker logs as root
When testing our system-conf configuration we don't actually add zuul to
the docker group. This means the zuul user cannot access the docker
socket. This then breaks docker container log collection. Address this
by becoming root when collecting logs.

Change-Id: Ic0232f7ef458cdd07fb0853f97f2dc22ce137c71
2020-03-05 16:14:49 +11:00
Ian Wienand
3aaf87ee6d letsencrypt: Register email with accounts
Currently we don't set a contact email with our accounts.  This is an
optional feature, but would be helpful for things like [1] where we
would be notified of certificates affected by bugs, etc.

Setup the email address in the acme.sh config which will apply with
any new accounts created.  To update all the existing hosts, we see if
the account email is added/modified in the config *and* if we have
existing account details; if so we need a manual update call.

For anyone who might be poking here, we also add a note on sharing an
account based on some broadly agreed upon discussion in IRC.

[1] https://community.letsencrypt.org/t/revoking-certain-certificates-on-march-4/114864

Change-Id: Ib4dc3e179010419a1b18f355d13b62c6cc4bc7e8
2020-03-05 12:25:56 +11:00
Zuul
1a60facce5 Merge "Kill qa.o.o" 2020-02-28 21:59:30 +00:00
Zuul
d75d70b333 Merge "letsencrypt: force renewal on certificate change" 2020-02-28 13:41:25 +00:00
Andreas Jaeger
e47de667d5 Kill qa.o.o
This site was never used nor published, it can be killed according to QA
PTL.

codesearch returns no matches for it in any docs.

Keep the occurence in manifests/static.pp, this will get deleted
as part of https://review.opendev.org/710388.

Change-Id: I3c0d3b567a3eccb959dc903f169197e4581f1e13
2020-02-28 09:30:27 +01:00
Ian Wienand
a44f5acdf3 letsencrypt: force renewal on certificate change
There is a bug, or misfeature, in acme.sh using dns manual mode where
it will not renew the certificate when new domains are added to an
existing certificate.  It appears to generate the TXT record requests
correctly, but then when we renew the certificate it thinks it is not
time and skips it.  This is filed upstream with [1] however we can
work around it, and generally be better anyway.

For each letsencrypt host, during certificate request we build up the
"acme_txt_required" key which is a list of TXT record tuples.
Currently we keep the challenge domain in the first entry, which is
not useful (all our hosts have the same challenge domain,
amce.opendev.org).  Modify this to be the certificate key from the
host config.  To be clear; when a host has

letsencrypt_certs:
  hostname-cert-main:
    hostname.opendev.org
    altname.opendev.org
  hostname-cert-secondary:
    secondary.opendev.org
    secondaryalt.opendev.org

acme_txt_required when renewing all certs will end up looking like:

 [
  (hostname-cert-main, <txt1>), (hostname-cert-main, <txt2>),
  (hostname-cert-secondary, <txt3>), (hostname-cert-secondary, <txt3>>)
 ]

In the certificate creation path, we walk "acme_txt_required" and take
the unique 0-value entries; this gives us the list of keys in
"letsencrypt_certs" which were actually updated.

We can then force renewal for these certs, because we know they
changed in some way that requires reissuing them (within renewal time,
or new domains).

This isn't just a work-around, it is generically better too.
Previously if any cert on host required an update, we would try to
update them all.  This would be a no-op; acme.sh would just skip doing
anything; but now we don't even have to call into the renewal if we
know nothing has changed.

[1] https://github.com/acmesh-official/acme.sh/issues/2763

Change-Id: I1e82c64217d46d7e1acc0111dff4db2f0062c42a
2020-02-28 11:49:06 +11:00
Andreas Jaeger
03276e0c93 Update redirects for legacy sides
The content for many projects has moved but the legacy redirects were
not updated, update to current location.

Change-Id: I7030ad35378085b0c45429c272dc24f00d33b2d2
2020-02-27 10:06:25 +01:00
Ian Wienand
d961b6d0d4 static: implement legacy redirect sites
This is a slight divergence from the accepted spec, where we were
going to implement these redirects via a new haproxy instance
(I961456d44a56f2334d3c94ef27e408f27409cd65).  We've decided it's
easier to keep them on static.opendev.org

The following sites are configured to redirect to whatever they are
redirecting to now on static.opendev.org:

 * devstack.org
 * www.devstack.org
 * ci.openstack.org
 * cinder.openstack.org
 * glance.openstack.org
 * horizon.openstack.org
 * keystone.openstack.org
 * nova.openstack.org
 * qa.openstack.org
 * summit.openstack.org
 * swift.openstack.org

As a bonus, they all get a https instance too, which they didn't have
before.

testinfra coverage should be total for this change.  I have created
the _acme-challange CNAME records for all the above.

Story: #2006598
Task: #38881

Change-Id: I3f1fc108e7bb1c9500ad4d1a51df13bb4ae00cb9
2020-02-27 16:25:39 +11:00
Ian Wienand
d78f3fa8f3 static: fix git raw file redirect
When converting this from a htaccess file to run in the virtualhost
context, one instance of '^cgit' -> '^/cgit' was missed.  Fix it, and
add a coverage test for it to testinfra.

Change-Id: Icc1dae6dce232e69c5cd1cf98b594f562c60d3f2
2020-02-27 09:50:34 +11:00
Zuul
7dc6b7c89f Merge "404 periodic job: minor fixes" 2020-02-26 04:22:45 +00:00
Ian Wienand
52850ddb63 404 periodic job: minor fixes
* Remove a stray trailing ' from the key
* update the key url to use https
* fix the log path to scrape

Change-Id: I580b63f08147494a937d44f4f6637947221c8937
2020-02-26 14:28:03 +11:00
Zuul
0ea607e739 Merge "static: provide git services" 2020-02-26 03:16:27 +00:00
Zuul
a74dcb3887 Merge "404 periodic job: fix host parameter" 2020-02-26 03:01:41 +00:00
Ian Wienand
e884ebfa57 404 periodic job: fix host parameter
The documentation is wrong, this needs to be just "host:" (fixed
upstream with
cf4882e0c0
but not released yet).

Change-Id: I5110e6795fa8b5729ddc87da6aef7c9ac7ed39a4
2020-02-26 12:49:25 +11:00
Zuul
d874080237 Merge "404 periodic job : use executor only, add host key" 2020-02-26 01:31:45 +00:00
Ian Wienand
b5266ea20c static: provide git services
This creates the redirect sites

 git.airshipit.org
 git.openstack.org
 git.starlingx.io
 git.zuul-ci.org

The htaccess rules are put into the main configuration file to avoid
having to create a directory and manage another file.  We use a macro
to duplicate the rules and retain the old semantics of the http site
redirecting directly (as opposed to doing a extra 301 to
https://git.openstack.org first).  This required adding "/" to the "^"
matches as it now runs in VirtualHost context; no functional change is
intended over the old sites.

This will require _acme-challenge CNAMEs to acme.opendev.org before
being merged.

testinfra is updated to exercise some redirects matching against the
results of the extant sites.

Change-Id: Iaa9d5dc2af3f5f8abc11c2312e4308b50f5fcd2b
2020-02-26 12:27:13 +11:00
Zuul
919f817064 Merge "static: add static.openstack.org/files.openstack.org" 2020-02-26 01:25:40 +00:00
Ian Wienand
de0c28018c 404 periodic job : use executor only, add host key
This only needs to run on the executor, specify a blank nodeset.

Add the static.opendev.org host key after adding the host.

Change-Id: Iedde486ce8f3e9b415991830121fb87ba192afc6
2020-02-26 11:44:13 +11:00
Ian Wienand
56509e83a4 static: add static.openstack.org/files.openstack.org
files.openstack.org serves a view of /afs/openstack.org/, which is the
same as static.opendev.org.  Add a serveralias for it and certificate.

Make static.openstack.org be consistent with opendev by showing the
same thing.

Change-Id: I4c492e3b02554a7c736c015790bd4cd5bb435a43
2020-02-26 10:39:50 +11:00
Ian Wienand
95606e6f86 zuul-user: add role to install system-config key
This moves the creation of a zuul user with the Zuul per-project key
for system-config to a separate role from the static role, so it can
be reused on other hosts.

Change-Id: Ice605b70a2c42d9b85090406216253fec0820f50
2020-02-26 10:29:03 +11:00
Zuul
ddc6a25706 Merge "static: add a periodic 404 checker" 2020-02-25 23:17:03 +00:00
Zuul
d492b41cf2 Merge "Correct openstackid tarball URL redirection" 2020-02-25 22:30:28 +00:00
Ian Wienand
74005bb29a static: add a periodic 404 checker
This is an alternative to Iccf24a72cf82592bae8c699f9f857aa54fc74f10
which removes the 404 scraping tool.  It creates a zuul user and
enables login via the system-config per-project ssh key, and then runs
the 404 scraping script against it periodically.

Change-Id: I30467d791a7877b5469b173926216615eb57d035
2020-02-26 09:05:31 +11:00
Jeremy Stanley
5390bbf23f Correct openstackid tarball URL redirection
The source pattern for the tarballs.openstack.org openstackid
redirect incorrectly included an openstack parent directory. Remove
it, an also make the regex more properly differentiate lack of a
trailing "/" character from a directory name containing openstackid
as a substring (not that there is one, but this serves as a safer
template for future additions).

Change-Id: I705d849d1c10cf91391181aeef72a9f4b495d520
2020-02-25 21:17:46 +00:00
Monty Taylor
97a79a027a Bump letsencrypt group id to match reality
It got created in the past as 3003. Just list it here like that.

Change-Id: Ic607a80c0d775856de059dedb73065c5708f556a
2020-02-25 14:17:13 -06:00