140 Commits

Author SHA1 Message Date
Monty Taylor
74b822f352 Use version 2 for docker-compose for review
Also - update the test job to use xenial nodes, since that's what
our gerrit servers are. docker-compose in xenial is too old for
version 3 file format - but we didn't get an error in the test
becuase we're running on bionic.

Change-Id: I855c3df241932098c95de84cf9f92383b335702f
2020-03-17 13:02:20 -05:00
Dr. Jens Harbott
8bdc4a8404 Increase timeout for goaccess report jobs
Seems it can take about an hour to build the report for our largest site
(docs.openstack.org), so having a timeout of 2 hours should keep us on
the safe side.

Change-Id: I848f75f6eb7bbf1986e19e6d35173fe4dddeef94
2020-03-12 15:47:49 +00:00
Clark Boylan
3acfcccd9e Add goaccess reporting to more static.o.o sites
This skips sites that redirect to other sites. This will give us data
that can be used to fix broken links (HTTP 404s), improve documents that
are retrieved often, and more.

Change-Id: I5838464c379b9f0906f1ebe3a91726838ada3508
2020-03-12 09:42:56 +00:00
Clark Boylan
68f740faf8 Remove old 404 checker job
This is being replaced with goaccess report jobs.

Change-Id: Ia22d847bfc1a9e450bd8c8e7fab77dd08bd1dfd0
2020-03-11 15:15:00 -07:00
Zuul
703dd8a05a Merge "nodepool-builder: Add webserver" 2020-03-11 00:31:12 +00:00
Zuul
d1af4dd321 Merge "nodepool-builder: deploy from container" 2020-03-10 23:25:33 +00:00
Ian Wienand
b1bfee423b nodepool-builder: Add webserver
This adds the webserver that serves the logs and generated images.

Change-Id: I230f5291e0bd928af2e00966d76c3f385b749cb6
2020-03-11 09:16:31 +11:00
Ian Wienand
1979d6b160 nodepool-builder: deploy from container
This deploys the nodepool-builder container and verifies it has
started in testinfra.

Change-Id: I8a717d06f1291a4112b2753641ff88f074cf0b31
2020-03-11 09:16:24 +11:00
Clark Boylan
8fb306a22b Add periodic job to run goaccess reporting
This adds a simple periodic job that runs against static.opendev.org and
produces opendev reports. To start we don't publish these reports
publicly so that we can double check their contents first. Assuming that
all goes well we'll be able to apply this to all of our
static.opendev.org hosted sites and publish these reports through the
zuul logging system.

Change-Id: I66705808d435c16bf0da6989296c896099964aaa
Story: 2007387
2020-03-10 10:42:07 -07:00
Ian Wienand
281425a44d Add initial Ansible for nodepool hosts
This is a start at ansible-deployed nodepool environments.

We rename the minimal-nodepool element to nodepool-base-legacy, and
keep running that for the old nodes.

The groups are updated so that only the .openstack.org hosts will run
puppet.  Essentially they should remain unchanged.

We start a nodepool-base element that will replace the current
puppet-<openstackci|nodepool> deployment parts.  For step one, this
grabs project-config and links in the elements and config file.

A testing host is added for gate testing which should trigger these
roles.  This will build into a full deployment test of the builder
container.

Change-Id: If0eb9f02763535bf200062c51a8a0f8793b1e1aa
Depends-On: https://review.opendev.org/#/c/710700/
2020-03-06 14:02:52 +11:00
James E. Blair
4e4b1b1abd Use explicit provides/requires for container jobs
Rather than rely on the implicit docker-image provides/requires,
list explicit per-image requirements for related jobs to reduce
unecessarily serialization in change queues.

Depends-On: https://review.opendev.org/711119
Change-Id: I11c1fe568d34ede294d7949a99a1eed9204a4ee0
2020-03-03 14:42:46 -08:00
Clark Boylan
fd78b58fcf Increase timeout for run static job
This job installs all of the vhosts necessary for our static hosting as
well as compiling afs kernel modules. Then it tests that they function.
Unfortunately in some cases half an hour just isn't enough time. Double
the default timeout to an hour to work around this.

Change-Id: Ief457aafa503af6c5bad83a1198b6c699d2d4983
2020-02-28 11:06:34 -08:00
Ian Wienand
de0c28018c 404 periodic job : use executor only, add host key
This only needs to run on the executor, specify a blank nodeset.

Add the static.opendev.org host key after adding the host.

Change-Id: Iedde486ce8f3e9b415991830121fb87ba192afc6
2020-02-26 11:44:13 +11:00
Zuul
ddc6a25706 Merge "static: add a periodic 404 checker" 2020-02-25 23:17:03 +00:00
Ian Wienand
74005bb29a static: add a periodic 404 checker
This is an alternative to Iccf24a72cf82592bae8c699f9f857aa54fc74f10
which removes the 404 scraping tool.  It creates a zuul user and
enables login via the system-config per-project ssh key, and then runs
the 404 scraping script against it periodically.

Change-Id: I30467d791a7877b5469b173926216615eb57d035
2020-02-26 09:05:31 +11:00
Zuul
8df5745c66 Merge "Rename gerrit hostvars to review01.openstack.org" 2020-02-25 17:26:49 +00:00
Monty Taylor
ef955dd270 Rename gerrit hostvars to review01.openstack.org
While the service is review.opendev.org, the server is actually
review01.openstack.org. The ansible inventory in production knows
it that way, as does the Nova in RAX DFW. Update the host_vars
entry and the zuul jobs so that it matches (And so that LE certs
apply)

Change-Id: I4c762c57f6826f2c5f9ed5c9cb0ae02644570c3d
2020-02-25 10:01:35 -06:00
Monty Taylor
a8e1d1496d Build gerrit images with bazelisk
We need to use bazelisk to build gerrit so that we can properly
track bazel versions in the job. Use the roles developed for
gerrit-review to do that, then simplify the dockerfile to have
it simply copy the war into the target image.

Also add polymer-bridges.

Depends-On: https://review.opendev.org/709256
Change-Id: I7c13df51d3b8c117bcc9aab9caad59687471d622
2020-02-21 17:32:01 -06:00
Monty Taylor
cc619fe589 Add review-dev01.opendev.org
Add a new review-dev server on the opendev domain with LE support
enabled.

Depends-On: https://review.opendev.org/705661
Change-Id: Ie32124cd617e9986602301f230e83bb138524fdf
2020-02-05 09:58:25 -06:00
Zuul
e038eccbc3 Merge "openafs-client: add centos8" 2020-01-23 00:59:05 +00:00
Ian Wienand
3f68936a0c openafs-client: add centos8
Add CentOS 8 support for the openafs client build

Change-Id: I8290cf1eed9ee8e4af44ac209502553944c52103
Depends-On: https://review.opendev.org/702348
2020-01-22 23:14:43 +00:00
Andreas Jaeger
9b2657527e Remove trusty testing
We do not have trusty nodes anymore, remove testing for it.

Change-Id: I1e464354b4c9e6d77a59c914d6d89f1837350b6e
2020-01-17 10:52:44 +01:00
Zuul
744a272cb2 Merge "Update python-base image upload job depends" 2020-01-17 00:11:13 +00:00
Zuul
3927db22c4 Merge "Add job dependencies to haproxy-statsd" 2020-01-17 00:11:11 +00:00
Monty Taylor
9e57b3b3b7 Update python-base image upload job depends
We've got a copy-pasta error here with upload jobs depending on
build jobs.

Change-Id: I0d3e0c359bbabc46c1d177d0369658d0b6d673f9
2020-01-17 06:50:40 +08:00
Monty Taylor
99c3fbff78 Add job dependencies to haproxy-statsd
haproxy-statsd uses opendevorg/python-base already. Add that to its
job dependencies and make sure it triggers on updates to python-base.

Update the FROM line to be fully qualified.

Change-Id: I9c8e8094f5570bf44076915610cd1be6d95ed326
2020-01-17 06:50:19 +08:00
Ian Wienand
f5b5ee9336 Add roles for a basic static server
Basic implementation of the opendev static server, described in

 https://docs.opendev.org/opendev/infra-specs/latest/specs/retire-static.html

Change-Id: Ie1b92f06b71aa6069fe831b26ba1cc272ce4562c
Story: #2006598
Task:  #37757
2020-01-16 14:10:08 +11:00
Clark Boylan
6f62b38dbc Update registry testing to use LE
This was missed when converting the registry server over to LE in
production. We need to test it this way too.

Change-Id: Ic2a05ebeae6991b69c000d5269165a45a0c72d38
2020-01-14 13:50:13 -08:00
Zuul
e68e154956 Merge "Clean up review comments" 2020-01-09 22:42:22 +00:00
Zuul
564a9d2a2a Merge "Base gitea-init on opendevorg/python-base" 2020-01-06 15:41:11 +00:00
Zuul
e6f4b1fa22 Merge "Add service playbook and test run for prod gerrit" 2019-12-18 19:46:40 +00:00
Monty Taylor
a3d91d4df3 Clean up review comments
Make image and volume list in compose file templated.

Rename the gerrit-podman directory to not be based on tool.

Make sure we run the job on changes to the playbooks.

Update the job name - it's not just review-dev anymore.

Change-Id: I0341fa95caff656a2176cc2026ec0ac8903fb24e
2019-12-17 08:13:34 -05:00
Monty Taylor
1d37be64b4 Add service playbook and test run for prod gerrit
We need to test this against production variables too.

Change-Id: I7813787506e3b70ef0960ce85dccca4eb9ec7a3f
2019-12-17 08:13:34 -05:00
Ian Wienand
4441f469ad Add arm64 mirror test
Add to the new arm64 check queue

Change-Id: I7b2f700f9326580b5424656e8fa1fd9731c8f14c
2019-12-16 10:42:41 +11:00
Zuul
29019411eb Merge "Run a gerrit container on review-dev01" 2019-12-15 19:00:21 +00:00
Ian Wienand
ce15e3bed9 mirror jobs: copy acme.sh output
We have seen failures issuing keys, but can't see the output of the
letsencrypt wrapper without capturing this logfile.  Add it.

Also, when we updated the mirror to "mirror01.openafs." (because we
have WIP for non-openafs kafs mirrors too) we didn't update the
host-vars match for the apache logs either.  Fix this.

Change-Id: I810a02d309f473e8c4aa0ce1612088aba7868c33
2019-11-27 10:46:34 +11:00
Clark Boylan
f7a305afbf Manage opendev.org with LE on all giteas
This catches up gitea02-07 with 01 managing ssl certs with LE.

Change-Id: I06228edca2204c5c57ebc5cb60b9d1308a393058
2019-11-18 12:47:08 -08:00
Clark Boylan
5392f8a27c Manage opendev.org cert with LE
This is the first step in managing the opendev.org cert with LE. We
modify gitea01.opendev.org only to request the cert so that if this
breaks the other 7 giteas can continue to serve opendev.org. When we are
happy with the results we can merge the followup change to update the
other 7 giteas.

Depends-On: https://review.opendev.org/694182
Change-Id: I9587b8c2896975aa0148cc3d9b37f325a0be8970
2019-11-18 12:07:10 -08:00
James E. Blair
4f9720e76e Run a gerrit container on review-dev01
This runs gerrit in a container on review-dev01 using podman.

Remove an unused web_server.py file that we found from copying it
from puppet to ansible.

Change-Id: I399d3cf8471bc8063022b0db0ff81718b2ee2941
2019-10-29 08:29:17 +09:00
James E. Blair
a441dddaa4 Set zuul_work_dir in gerrit master job
Set the zuul_work_dir to this project so that this job may be used
in other projects (eg gerrit itself or plugins).

Change-Id: I8662ff2a26bcff342f922c28d22225b73859929c
2019-10-24 10:40:56 -07:00
Monty Taylor
ccaf54c866 Base gitea-init on opendevorg/python-base
So that we can keep one python base image for our python things,
base jinja-init and gitea-init on python-base. Also, tie jinja-init
to python-base in the dependency graph and gitea-init to jinja-init.
This way if python-base updates, we'll rebuild our python images.

Update FROM lines to use full paths to images.

Change-Id: I554bf07fa8e458e443729cf4b8f40d7ceeaafa04
2019-10-23 17:14:02 +09:00
James E. Blair
1ce80389f1 Add system-config as a required project to gerrit-master
The build process for this job includes using content from the
system-config repo, therefore we need to make sure it is present
in case this job is used in other repos (eg, gerrit repos).

Change-Id: Id15c87b4dc1330406ab52741e1c2e2ecb62583ff
2019-10-22 13:25:35 -07:00
James E. Blair
6bd0d0258d Refactor gerrit master job
Make a "base" version of the gerrit master job with no file matcher
so that we can use it in other repositories (eg, gerrit, zuul).
Inherit from it with the original name to add the file matcher back.

Change-Id: I4e428b44dd82f8dba08b219cbf8407969c6436b1
2019-10-22 09:06:24 -07:00
Monty Taylor
61b5a11bfb Always rebuild base and builder together
When we build either, it could pick up base image changes, such as
moving from stretch to buster. Make sure any time we build one we
build the other so that they stay in sync.

Change-Id: Ia28ad4f64114c88cc02289c9318a323ceb4f143d
2019-10-22 02:59:11 +09:00
Monty Taylor
92fe8eae71 Add checks plugin to gerrit master build
Change-Id: Ifb7f4b84fb07c99d3c28b78ec55a1835821202f6
2019-10-20 06:36:02 +09:00
Monty Taylor
36aa77937a Add jobs to build gerrit master branch
We'll use this to test the checks plugin.

We have to add jgit as a repo because it's a submodule now.

Change-Id: Ic7e9ad0265e136a9ac6b1147998f6eb5ee398180
2019-10-20 06:35:56 +09:00
Monty Taylor
9ab25e89a9 Several updates because the world is a dark place
A few things have changed and we need to fix them in one go.

Use mirror for installing docker for buildset-registry

While, we need to make this more systemic, that's hanging off of the
mirror rework. For now, since we know all of these jobs are debian
based, just set the mirror location.

Replace use of zuul cloner with git clones

You can never be a prophet in your own hometown. This is now broken
because of the git cache rework, so just replace it.

Update libjemalloc library

python:slim is based on buster now, which has libjemalloc2 not
libjemalloc1.

Remove gerrit repo remote for submodules

A recent change to the base jobs to use prepare-workspace-git
broke the gerrit image builds by actually having the origin
remote by /dev/null as intended. This breaks submodules because
for a few of them where we don't have matching stable branches
the submodule relative path behavior is actually exactly what
we want.

Since we don't care about the remote otherwise, remove the
origin remote before doing the submodule update --init so that
the submodule will clone the refs from the zuul prepared repo.

Change-Id: Ieb5b6bc8711fe971ed3445c7c267306ac4616464
2019-10-19 07:51:29 +09:00
Monty Taylor
f9e7fcab51 Remove dependencies from promote
These don't really make a ton of sense now do they?

Change-Id: Ic5f515dd50872387e3447eb70af25070257f61cd
2019-10-05 09:20:38 +02:00
Monty Taylor
d9fb5b3faf Install jeepyb into the gerrit images
We need jeepyb installed because the content of the gerrit hook scripts
we install is done via jeepyb commands. Use python-builder so that we
can just install the jeepyb wheel.

Should we maybe transition these hooks into being zuul jobs?

Depends-On: https://review.opendev.org/683146/
Change-Id: I8899885b05d1e9f48b3f354ca22b360b54d455a3
2019-09-19 15:17:14 +02:00
Monty Taylor
072fcca06f Fix files matcher and bazel for gerrit base image
Use latest bazel

It seems 0.27 is now too old. This is what happens when I go on vacation
apparently.

Add in a hack to override the bazelversion. We'll remove this once
https://gerrit-review.googlesource.com/c/gerrit/+/237495 lands and
has been merged up.

Change-Id: Ib7a6d33ce8bf8498fd5cd09b25087dc09acb8df4
2019-09-16 21:20:18 +02:00