101 Commits

Author SHA1 Message Date
Thierry Carrez
5b4c917d30 Disable global Github replication
We used to replicate every openstack/* project to GitHub,
through a global replication at Gerrit-level. Now that the job for
granular replication is in place, we can stop the global replication,
so that only active/official repositories are synced.

Depends-On: https://review.opendev.org/724310
Change-Id: Ibba02e626e33aba9779f771d5ae49920bac86b19
2020-04-29 14:37:23 +02:00
Monty Taylor
5e6aa5e70d Use python3 for ansible
We get deprecation warnings from ansible about use
of python2 on xenial hosts. Rather than setting
ansible_python_interpreter to python3 on a host by
host basis, set it globally to python3.

Set it to python for the one host that's too old,
refstack.openstack.org, which is running on trusty
which only has python3.4.

Change-Id: I4965d950c13efad80d72912911bc7099e9da1659
2020-04-28 11:54:15 -05:00
Zuul
d3360a7d60 Merge "Remove two unused ansible vars files" 2020-04-24 17:17:33 +00:00
Zuul
f6d60a638d Merge "Use python3 for ansible on etherpad" 2020-04-24 17:09:05 +00:00
Zuul
b21a8e58cf Merge "Run Zuul using Ansible and Containers" 2020-04-24 16:31:42 +00:00
Monty Taylor
275ccd9b80 Remove two unused ansible vars files
Change-Id: I415b709399df28d4da55b8fa0a204110729e70a8
2020-04-24 10:44:32 -05:00
Monty Taylor
47684ec0f4 Use python3 for ansible on etherpad
We need to override this because it's erroneously picking up
python2 which on etherpad does not have everything it needs.

Change-Id: Ic127f505740f9ac66abb8754356e04acde66dd85
2020-04-24 10:43:23 -05:00
Monty Taylor
f0b77485ec Run Zuul using Ansible and Containers
Zuul is publishing lovely container images, so we should
go ahead and start using them.

We can't use containers for zuul-executor because of the
docker->bubblewrap->AFS issue, so install from pip there.

Don't start any of the containers by default, which should
let us safely roll this out and then do a rolling restart.
For things (like web or mergers) where it's safe to do so,
a followup change will swap the flag.

Change-Id: I37dcce3a67477ad3b2c36f2fd3657af18bc25c40
2020-04-24 09:18:44 -05:00
Thierry Carrez
6935318712 No longer push refs/changes to GitHub mirrors
By default gerrit replication pushes +refs/*:refs/*, which includes
refs/changes. For large repositories that potentially means hundreds
of thousands of references.

Per-repo git mirroring does not push refs/changes, so when it runs it
ends up deleting those references, which can take a long time, blocking
the executor.

To fix that, we should:
- stop pushing refs/changes to GitHub (this change)
- delete refs/changes on GitHub repositories, asynchronously
- enable per-repo replication
- disable Gerrit-wide replication

NB: it is unclear if Gerrit replication would start deleting the
    extraneous references on remote GitHub repositories once this
    merges. If this is the case, since replication is limited to a
    single thread (default value for 'threads') and is not happening
    in an executor, this should not have negative impact, beyond
    potentially delaying GitHub mirroring.

Change-Id: I94f69c889c9b4418ef81b3b2ca436ba99696ba72
2020-04-17 11:54:01 +02:00
Monty Taylor
ebae022d07 Use project-config from zuul instead of direct clones
We use project-config for gerrit, gitea and nodepool config. That's
cool, because can clone that from zuul too and make sure that each
prod run we're doing runs with the contents of the patch in question.

Introduce a flag file that can be touched in /home/zuulcd that will
block zuul from running prod playbooks. By default, if the file is
there, zuul will wait for an hour before giving up.

Rename zuulcd to zuul

To better align prod and test, name the zuul user zuul.

Change-Id: I83c38c9c430218059579f3763e02d6b9f40c7b89
2020-04-15 12:29:33 -05:00
Monty Taylor
009068456a Fix the hostkey for gerrit
We want the hostkeys for gerrit, not for ssh on the host.

Change-Id: I45f82b70946ce63d813ab55b8f78b0c30ab7659c
2020-04-14 13:41:54 -05:00
Monty Taylor
014b3004c0 Add self host keys to known_hosts on gerrit
We run some utility scripts which ssh to ourselves, but we aren't
setting host keys for them. We should fix that.

Change-Id: I2aa5d5e65b15c5c151767377dbc5ead1e442b3ce
2020-04-13 11:33:16 -05:00
Monty Taylor
b23515c623 Make a new dockerized etherpad.opendev.org
Upstream likes building the settings file into the image, but that's
less exciting, let's bind-mount ours in.

Depends-On: https://review.opendev.org/717491/
Change-Id: Ia1894d884ef2a84e1282345b77fe07bf8898f367
2020-04-07 11:10:57 -05:00
James E. Blair
8b093dacd5 Add meetpad server
Depends-On: https://review.opendev.org/714189
Change-Id: I5863aaa805a18f9085ee01c3205b0f9ad602922d
2020-03-25 07:44:24 -07:00
Monty Taylor
e6d98f0181 Install utility scripts for running jeepyb commands
jeepyb is installed in the gerrit image because of hook scripts.
To run manage-projects and track-upstream, make wrapper scripts
in /usr/local that runs the commands from in the container image
bind-mounting the appropriate dirs and files into the container.

Change-Id: I7ef1f00e69d4c310d69d83c80ca210e8f340878d
2020-03-22 10:39:47 -05:00
Monty Taylor
906e6a72a1 Use dev subdir on review-dev for project-config things
This is in anticipation of running manage-projects, which wants to
use the dev subdir of project-config when run on dev, but we're
currently not doing anything with that. Point at root or dev as
appropriate. Then we can do a similar thing - probably just
bind-mounting the root or the dev when we docker run for
manage-projects.

Change-Id: Ia13bbb0bf8dbe1f7e7c0f378ba9b41bef3ecd5c1
2020-03-21 11:34:28 -05:00
Monty Taylor
63d8f7af48 Base 2.13 image on gerrit-base
We install jeepyb and launchpadlib in gerrit-base. Those are
important. We also need to add cgi for gitweb.

The gerrit init command does two things that we don't actually
want it to do at runtime. It extracts the plugins into the
plugins dir, and it downloads the right database library.

We can extract the plugins for it during image creation, and
then we can also download the plugin it would have downloaded.

We can also download the mysql library for it:

https://gerrit.googlesource.com/gerrit/+/refs/heads/stable-2.13/gerrit-pgm/src/main/resources/com/google/gerrit/pgm/init/libraries.config

Finally, we tell it to not download or expand anything during
init, because we're running in a container and next time we run
the process that dir isn't going to be there.

Our gerrit integration tests don't depend on our gerrit image builds.
Put in image depends between run-review and gerrit builds.

We also need to depend directly on opendev-buildset-registry.

Add java.security.egd setting to java invocation

This tells java to be secure.

https://stackoverflow.com/questions/58991966/what-java-security-egd-option-is-for

Add support for setting heap limit properly

The gerrit init script does this based on the value in
container.javaOptions. We could, but then we'd have to
run an entrypoint script. Instead, set the value via
the JAVA_OPTIONS env var setting based on a value from
ansible.

Finally, make gerrit-master image build non-voting

It looks like there might be a real issue, but debugging that
is not important for us at this moment.

Depends-On: https://review.opendev.org/714216
Change-Id: I01e94c10f470fb3c8ddfce7b0e201357e5050679
2020-03-20 16:37:18 -05:00
Ian Wienand
ba5d9547c8 Replace nb01.opendev.org with nb04
We are starting over with the container nodepool host, and also moving
it to "nb04" to avoid any possibility of conflicting with the
short-hostname of nb01.openstack.org.

Story: #2007407
Task: #39064

Depends-On: https://review.opendev.org/713575
Depends-On: https://review.opendev.org/713571

Change-Id: I18ab9834ad4da201774e0abef56f618cd7839d36
2020-03-19 07:41:56 +11:00
Zuul
b1576b6020 Merge "Set up LE certs for docs.airshipit.org static site" 2020-03-15 16:19:49 +00:00
Jeremy Stanley
69d0b85775 Set up LE certs for docs.airshipit.org static site
Start installing Let's Encrypt SSL certs on static01 for a new
docs.airshipit.org it will be hosting.

Change-Id: Ia5089515cd02d78267827840521cbee4f71ceb0b
2020-03-15 14:10:28 +00:00
Ian Wienand
dbe0bf1ee6 Add nb01.opendev.org
This configures an opendev nodepool-builder

Change-Id: Id8603d9d7caaac0a1ab935e1c7c80d32b02ae23e
Depends-On: https://review.opendev.org/693118
2020-03-11 09:16:31 +11:00
Jeremy Stanley
0e2e4425b6 Fix formatting for Open Edge LE playbook
The domain names passed to the Let's Encrypt handler need to be list
elements.

Change-Id: I0122b4b86dd17e3e2fbce211be554cbda4167d05
2020-03-07 18:07:29 +00:00
Jeremy Stanley
4cbdc2fc4d Set up inventory and cert for Open Edge mirror
This adds the Open Edge (formerly Fortnebula) CI mirror.

Change-Id: I1ccf2a602f8a41e00bc64a9516a326cc07d9b254
Depends-On: https://review.opendev.org/711787
2020-03-07 00:24:20 +00:00
Jeremy Stanley
43ed9fc297 Moving FortNebula to OpenEdge
Sister change for Ia5caff34d3fafaffc459e7572a4eef6bd94422ea and
removing earlier references to the mirror server in preparation for
building and adding the new one.

Change-Id: I7d506be85326835d5e77a0c9c461f2d457b1dfd3
2020-03-06 20:43:56 +00:00
Andreas Jaeger
e47de667d5 Kill qa.o.o
This site was never used nor published, it can be killed according to QA
PTL.

codesearch returns no matches for it in any docs.

Keep the occurence in manifests/static.pp, this will get deleted
as part of https://review.opendev.org/710388.

Change-Id: I3c0d3b567a3eccb959dc903f169197e4581f1e13
2020-02-28 09:30:27 +01:00
Ian Wienand
d961b6d0d4 static: implement legacy redirect sites
This is a slight divergence from the accepted spec, where we were
going to implement these redirects via a new haproxy instance
(I961456d44a56f2334d3c94ef27e408f27409cd65).  We've decided it's
easier to keep them on static.opendev.org

The following sites are configured to redirect to whatever they are
redirecting to now on static.opendev.org:

 * devstack.org
 * www.devstack.org
 * ci.openstack.org
 * cinder.openstack.org
 * glance.openstack.org
 * horizon.openstack.org
 * keystone.openstack.org
 * nova.openstack.org
 * qa.openstack.org
 * summit.openstack.org
 * swift.openstack.org

As a bonus, they all get a https instance too, which they didn't have
before.

testinfra coverage should be total for this change.  I have created
the _acme-challange CNAME records for all the above.

Story: #2006598
Task: #38881

Change-Id: I3f1fc108e7bb1c9500ad4d1a51df13bb4ae00cb9
2020-02-27 16:25:39 +11:00
Ian Wienand
b5266ea20c static: provide git services
This creates the redirect sites

 git.airshipit.org
 git.openstack.org
 git.starlingx.io
 git.zuul-ci.org

The htaccess rules are put into the main configuration file to avoid
having to create a directory and manage another file.  We use a macro
to duplicate the rules and retain the old semantics of the http site
redirecting directly (as opposed to doing a extra 301 to
https://git.openstack.org first).  This required adding "/" to the "^"
matches as it now runs in VirtualHost context; no functional change is
intended over the old sites.

This will require _acme-challenge CNAMEs to acme.opendev.org before
being merged.

testinfra is updated to exercise some redirects matching against the
results of the extant sites.

Change-Id: Iaa9d5dc2af3f5f8abc11c2312e4308b50f5fcd2b
2020-02-26 12:27:13 +11:00
Ian Wienand
56509e83a4 static: add static.openstack.org/files.openstack.org
files.openstack.org serves a view of /afs/openstack.org/, which is the
same as static.opendev.org.  Add a serveralias for it and certificate.

Make static.openstack.org be consistent with opendev by showing the
same thing.

Change-Id: I4c492e3b02554a7c736c015790bd4cd5bb435a43
2020-02-26 10:39:50 +11:00
Monty Taylor
97a79a027a Bump letsencrypt group id to match reality
It got created in the past as 3003. Just list it here like that.

Change-Id: Ic607a80c0d775856de059dedb73065c5708f556a
2020-02-25 14:17:13 -06:00
Zuul
8df5745c66 Merge "Rename gerrit hostvars to review01.openstack.org" 2020-02-25 17:26:49 +00:00
Monty Taylor
ef955dd270 Rename gerrit hostvars to review01.openstack.org
While the service is review.opendev.org, the server is actually
review01.openstack.org. The ansible inventory in production knows
it that way, as does the Nova in RAX DFW. Update the host_vars
entry and the zuul jobs so that it matches (And so that LE certs
apply)

Change-Id: I4c762c57f6826f2c5f9ed5c9cb0ae02644570c3d
2020-02-25 10:01:35 -06:00
Ian Wienand
3206fd02b8 static: move afs sites from files.openstack.org to static.opendev.org
This creates sites to serve

 developer.openstack.org
 docs.openstack.org
 docs.opendev.org
 docs.starlingx.io

which are all just static directories underneath /afs/openstack.org/.

This is currently done by files02.openstack.org, but will be better
served in the future by consolidating in ansible configuration on
static.opendev.org.

The following dns entries need to be made before merging to ensure the
certificates are provisioned

 _acme-challenge.developer.openstack.org
 _acme-challenge.docs.openstack.org
 _acme-challenge.docs.opendev.org
 _acme-challenge.docs.starlingx.io

Once done, we can merge and then cut-over the main DNS entries as we
like.

Since there are some follow-ons, I have not removed the puppet
configuration from files02.openstack.org.  I think it's best we
migrate everything away from that and remove it in one lot.

Change-Id: I459a36f823a8868e6cc09e2b0d85f2fe05d69002
2020-02-21 17:59:14 +01:00
Ian Wienand
047eae459d static: add releases.openstack.org site
This adds the site to publish from

 /afs/openstack.org/project/releases.openstack.org

Change-Id: Ia91deb9a51441ac9974137ed39fc5a185689a11c
Task: #37724
Story: #2006598
2020-02-21 14:35:35 +11:00
Ian Wienand
2f1b2f3eae static: Add service-types.openstack.org
Publishing changes done with https://review.opendev.org/#/c/708518/

Change-Id: I13934473aa85fce17a269f81f67c6332d51a9ab1
Story: #2006598
Task: #37723
2020-02-20 11:09:28 +11:00
Ian Wienand
738468b6ad Add specs.openstack.org
Old content is rsynced and publishing to be switched with
https://review.opendev.org/#/c/708500/

Change-Id: I797bb51970d9e7cd3ee5c2635bb5045c618b9d2c
Story: #2006598
Task: # 37721
2020-02-20 10:37:45 +11:00
Zuul
5f80e934c4 Merge "Use LE certs for Apache" 2020-02-14 19:08:51 +00:00
Monty Taylor
bbe8086726 Use LE certs for Apache
We're getting LE certs for the hosts now, use them in the apache
config. Also add the redirects.

Change-Id: I67d33b4c542182a2474ac0d2416357541b1c3a47
2020-02-13 10:31:59 -06:00
Zuul
75693aa43b Merge "Remove review-dev01.openstack.org" 2020-02-12 18:12:51 +00:00
Zuul
3308e83885 Merge "Get LE certs for review.o.o" 2020-02-12 18:12:50 +00:00
Monty Taylor
23b0667c45 Remove review-dev01.openstack.org
We've got a shiny new opendev one of these now, get rid of the
openstack.org one.

Change-Id: I9f5141a6ff23ab035ecedf100a417989453bfc1e
2020-02-12 14:40:46 +00:00
Monty Taylor
083cbf2911 Get LE certs for review.o.o
We have LE dns entries for review.o.o, but we're not actually
requesting the cert. Go ahead and request it - it'll make the
apache config easier to sort out.

Get the openstack.org certs for review-dev while we're at it.

Change-Id: I91d06c97993ba37204bd1fc326ae823e1b9c0c1a
Depends-On: https://review.opendev.org/707267
Depends-On: https://review.opendev.org/707255
2020-02-11 17:01:43 -06:00
Clark Boylan
0463609584 Add airship-citycloud mirror
This adds a mirror to the new airship citycloud region. Add the host to
the inventory and add necessary host vars for LE setup.

Depends-On: https://review.opendev.org/706573
Change-Id: I33cefe914911b4f5ce5e09e0329ba48e039ede64
2020-02-07 08:55:40 -08:00
Ian Wienand
3fd6e16077 Add tarballs.<openstack|opendev>.org to static.opendev.org
Add these hosts to static.opendev.org, serving from AFS.  Note that
tarballs.openstack.org just redirects to static.opendev.org/openstack.

This should have no effect currently, it will only become live when we
switch DNS.

For more details see the thread at:

 http://lists.openstack.org/pipermail/openstack-infra/2020-January/006584.html

Change-Id: Ie56fac17ffaa91ee55be986de636485a58125a02
2020-02-06 08:24:16 +11:00
Monty Taylor
cc619fe589 Add review-dev01.opendev.org
Add a new review-dev server on the opendev domain with LE support
enabled.

Depends-On: https://review.opendev.org/705661
Change-Id: Ie32124cd617e9986602301f230e83bb138524fdf
2020-02-05 09:58:25 -06:00
Monty Taylor
4de5f79599 Add Apache to Ansible for Gerrit
When we run gerrit, we also need to run Apache.

Change-Id: Ia2f1494808bd29d83e041e224cb2eb5fc406a93b
2020-02-03 07:57:36 -06:00
Ian Wienand
7ce0d0fb32 Add static01.opendev.org
Add this host for serving content from AFS.

The

 _acme-challenge.governance.openstack.org
 _acme-challenge.security.openstack.org

CNAMES should be in place for creating the certificates (added with
Ie1b92f06b71aa6069fe831b26ba1cc272ce4562c).

Also add a cert for the base server (static.opendev.org) since we
added the DNS entries for it.

Change-Id: I55e0ac7487b02f9a816ac486ed01b73f82b391a5
Story: #2006598
Task: #37757
Depends-On: https://review.opendev.org/704469
2020-01-28 11:30:18 +11:00
Ian Wienand
9a75422145 Add linaro-us mirror
Change-Id: I4abf249fd385872f8ba56c3e41f99d160a68efcd
Story: #2007195
Task: #38358
Depends-On: https://review.opendev.org/703746
2020-01-22 20:17:53 +11:00
Ian Wienand
f5b5ee9336 Add roles for a basic static server
Basic implementation of the opendev static server, described in

 https://docs.opendev.org/opendev/infra-specs/latest/specs/retire-static.html

Change-Id: Ie1b92f06b71aa6069fe831b26ba1cc272ce4562c
Story: #2006598
Task:  #37757
2020-01-16 14:10:08 +11:00
Clark Boylan
3deef00ba9 Manage insecure-ci-registry cert with LE
This adds a new handler to restart the zuul registry to pick up the new
cert. We may want to consider updating zuul registry to accept a reload
of ssl config without restarting the service.

Depends-On: https://review.opendev.org/702050
Change-Id: I23f6bea68285bc7cb0d12224235eaa16f0d07986
2020-01-13 15:20:20 -08:00
Clark Boylan
f30b39c769 Don't issue cert for zuul01.opendev.org
This name/host doesn't actually exist so don't try issuing a cert for
it. Instead only issue a cert for zuul.opendev.org.

Change-Id: I6c8eaa9280c3d6f070b8a1c79d850ee42e0e8d50
2020-01-10 10:13:02 -08:00