16321 Commits

Author SHA1 Message Date
Clark Boylan
e97d031d68 Couple of gerritbot role cleanups
Change restart mode to always instead of 'no' as testing shows we won't
restart in a loop in CI and we want production to restart automatically.
Also add ssh pubkey contents for completeness and simplicity if we need
to find those in the future.

Change-Id: I81573a1ad1574419194eb3088070dda95fb81fff
2020-08-10 10:42:20 -07:00
Clark Boylan
506a11f9d2 Add ansible role to manage gerritbot
This new ansible role deploys gerritbot with docker-compose on
eavesdrop.openstack.org. This way we can run it where the other bots
live.

Testing is rudimentary for now as we don't really want to connect to a
production gerrit and freenode. We check things the best we can.

We will want to coordinate deployment of this change with disabling the
running service on the gerrit server.

Depends-On: https://review.opendev.org/745240
Change-Id: I008992978791ff0a38f92fb4bc529ff643f01dd6
2020-08-07 13:20:18 -07:00
Ian Wienand
4092ef34e5 openedge mirror: remove for replacement
We are rebuilding this mirror.  Remove the old host from the
inventory.

Change-Id: Ibbc59a7ec4eae79fe4f7ae5e1b795d952d0611ea
2020-08-05 06:36:29 +10:00
Zuul
959473141b Merge "Backup inventory - match zuul01.openstack.org" 2020-08-04 01:48:27 +00:00
Ian Wienand
3cbb877d43 launch-node : add sshfp records
Add a tool to scan a host and generate the sshfp records to go into
dns.  Hook this into the DNS print out from the node launcher.

Change-Id: I686287c3c081debeb6a230e2a3e7b48e5720c65a
2020-08-04 01:04:37 +00:00
Zuul
fea231676a Merge "Use pip install -r not -f to install extras" 2020-08-03 19:03:03 +00:00
Clark Boylan
45da85124f Use pip install -r not -f to install extras
The pip install -f flag is for find-links which is a list of locations
to look for python packages. What we hvae at this path for extras
installation is a list of packages themselves not locations to find
them. We need to use the -r flag for requirements lists to specify this
instead.

This change should update our zuul and nodepool images to include useful
debugging extras.

Change-Id: I647bb835d0c85c3772e1593866a54cfc5ea1db2f
2020-08-03 10:26:30 -07:00
Clark Boylan
8f69015838 Increase gitea indexer startup timeout
The default indexer timeout is 30 seconds. During a recent gitea restart
gitea01 hit this timeout five times: 150 seconds. Increase the timeout
to double that value: 300 seconds.

This is important to ensure that our graceful restarts are in fact
graceful. We don't want the sshd container running while web is being
restarted multiple times. Doing so can lead to lost replication events
from gerrit.

Change-Id: I1f9253ccd6fbb055f848e186f478651454fee7e0
2020-07-31 14:21:50 -07:00
Zuul
872663a270 Merge "Upgrade Gitea to v1.12.3" 2020-07-31 21:01:44 +00:00
Zuul
6da79584e8 Merge "Cleanup /p/ further and add reminder comments" 2020-07-31 20:16:54 +00:00
Zuul
1bbfb007cc Merge "Deny Gerrit /p/ requests" 2020-07-31 19:19:26 +00:00
Clark Boylan
c96a80d481 Cleanup /p/ further and add reminder comments
We remove old git web server env vars from the apache config and add
comments to our /p/ handling to describe the need for further cleanup
when Gerrit is upgraded.

Change-Id: I79fc130dec0a8b00706c0ec0f8fcab4d867e34d1
2020-07-31 11:22:00 -07:00
Clark Boylan
089426420c Upgrade Gitea to v1.12.3
This is an incredibly minor update and the templates don't seem to
change at all. But its still a good idea to keep up as much as possible.

Change-Id: I7afcd741bfea056f34dacca65512c86949539340
2020-07-30 16:20:31 -07:00
Ian Wienand
6091313667 Revert "Cap pytest to <6.0.0 to fix pytest-html"
This reverts commit 9946a851833dbbb19af66829eb732196a3d32dce.

The problem was fixed with pytest 6.0.1

  https://docs.pytest.org/en/stable/changelog.html#pytest-6-0-1-2020-07-30

Change-Id: I580a96e6eb65c272d9306af118b8f27ae8cff8c7
2020-07-31 08:35:10 +10:00
Clark Boylan
9946a85183 Cap pytest to <6.0.0 to fix pytest-html
We get:

  INTERNALERROR> AttributeError: 'tuple' object has no attribute 'rsplit'

When running pytest-html and this appears related to the recent pytest
6.0.0 release. Cap pytest to <6.0.0 to fix this.

Change-Id: I1379267598099f3ab53c217f15c57f825b8889fa
2020-07-29 13:29:49 -07:00
Zuul
b4c95d08b9 Merge "Update host_vars and sync-to-review-test playbook" 2020-07-28 14:09:11 +00:00
Ian Wienand
78eadcb783 Backup inventory - match zuul01.openstack.org
The zuul01.openstack.org server is not matching the Ansible backup
group, which specifies opendev.org.  This means it is not backing up
to the "new" vexxhost server like everything else.

Change-Id: I07ac19f7cb5597950886c01806189e479e7a3724
2020-07-28 13:06:05 +10:00
Zuul
0cf20e0756 Merge "Run our etherpad prod deploy job when docker updates" 2020-07-27 20:33:24 +00:00
Clark Boylan
770dbad2dd Deny Gerrit /p/ requests
Gerrit is repurposing the /p/ path for project dashboard under
polygerrit. We use this path for Git mirrors. To resolve this let's
disable the /p/ path now then when it is used for project dashboards
users won't be as confused.

This has the added benefit of reducing the number of mirrors we need to
manage which makes managing branches in the mirrors simpler.

Change-Id: I9ebca2049a4a0707ecfbaecd92e42ebc1e6c3f87
2020-07-27 12:31:24 -07:00
Jeremy Stanley
8da9b9086f Start zuul-executor after afsd and /afs is mounted
Add an override to the systemd configuration for the docker service
unit so that it won't start until after openafs-client is started
and /afs is mounted (the latter because we don't know if the
initscript will possibly return early). Without this, it's a race to
see whether the container will have a working /afs mount, so can
lead to jobs failing to write into AFS with cryptic permissions
errors.

Change-Id: Ie00b1c1bc9c330e2af28c59b3b07a7c244c912dc
2020-07-27 18:47:52 +00:00
Clark Boylan
5fa69476c4 Continue to add_host here even though we do it in base-jobs
We need to add host (and possibly the ssh host key so its here too) in
this playbook because the add_host from the base-jobs side is only
applicable to the playbook running in base-jobs. When we start our
playbook here that state is lost. Simple fix, just add_host it again.

Change-Id: Iee60d04f0232500be745a7a8ca0eac4a6202063d
2020-07-24 15:15:52 -07:00
Zuul
3748d6af1c Merge "Use infra-prod-base in infra-prod jobs" 2020-07-24 22:01:56 +00:00
Jeremy Stanley
3d6cae8298 Run ara-report on bridge in run-base-post
We can't run ARA on the executor because that involves running
arbitrary commands, instead generate reports on the executor and put
them where the normal fetch-output will find them later.

Change-Id: I20d88a7f03872d19f6bd014bc687a1bf16e4e80e
2020-07-24 18:18:39 +00:00
James E. Blair
b9f7f5506f Use infra-prod-base in infra-prod jobs
This uses a new base job which handles pushing the git repos on to
bridge since that must now happen in a trusted playbook.

Depends-On: https://review.opendev.org/742934
Change-Id: Ie6d0668f83af801c0c0e920b676f2f49e19c59f6
2020-07-24 09:04:50 -07:00
Zuul
2f7959d5f2 Merge "Noop update to force python-builder/base to rebuild" 2020-07-18 01:13:44 +00:00
James E. Blair
aebd9fa173 Update zuul-executor stop/start playbook
These are running in docker now, update the stop/start commands.

Change-Id: I5c526863e4ef2bea94b0c2317abd443271bd12ba
2020-07-17 16:18:26 -07:00
Clark Boylan
fd6e0ec09e Noop update to force python-builder/base to rebuild
We want these to rebuild to push up multiarch images. I couldn't come up
with a good actual change so I added a . to a comment instead.

Change-Id: I287fee62075bc124ea7d9cc51baaeecdc0e73c6c
2020-07-17 16:17:22 -07:00
Zuul
b6f5d4f314 Merge "Allow setting Gitea repo branch on project creation" 2020-07-17 20:50:59 +00:00
Clark Boylan
4ebff6f9b2 Run our etherpad prod deploy job when docker updates
We want to pick up changes to our docker setup in production. Without
this we don't get the infra-prod-service-etherpad job running when we
update the etherpad docker image.

Change-Id: I25aee457b7c0547fc11439301054bb5aef799476
2020-07-17 13:20:48 -07:00
Clark Boylan
43778c9f19 Patch etherpad console logging to fix cross origin error
Etherpad 1.8.4 added console logging that breaks iframes like those used
by meetpad when logging. THis means many etherpads work fine and only
have an issue when logging is tripped.

We fix this by adding patch files based on the upstream fix,
00b6a1d9fe,
cherrypicked to 1.8.4 and then diffed. The reason we don't just use git
is that while the installation in the upstream image is a git repo there
is not git installation and adding in patch keeps our image small.

We also convert the existing css fix to using patch for consistency.

Change-Id: I7eed0d74c40141255cbff62069a83144feef6b61
2020-07-17 10:49:15 -07:00
Zuul
ae440c4fcc Merge "Fix junit error, add HTML report" 2020-07-16 23:45:53 +00:00
Zuul
fdb446f0e3 Merge "testinfra: silence yaml.load() warnings" 2020-07-16 23:45:51 +00:00
Zuul
4b9180acfa Merge "Copy generated inventory to bridge logs" 2020-07-16 23:45:49 +00:00
Zuul
33a87d7989 Merge "Revert "Revert "Add Zookeeper TLS support""" 2020-07-16 19:43:48 +00:00
Zuul
35ed33d594 Merge "Added development/rawhide image" 2020-07-16 04:39:01 +00:00
Zuul
c3b2aac1c1 Merge "Build multi-arch python-base/python-builder" 2020-07-15 23:19:31 +00:00
James E. Blair
7a32463f9d Revert "Revert "Add Zookeeper TLS support""
This reverts commit 05021f11a29a0213c5aecddf8e7b907b7834214a.

This switches Zuul and Nodepool to use Zookeeper TLS.  The ZK
cluster is already listening on both ports.

Change-Id: I03d28fb75610fbf5221eeee28699e4bd6f1157ea
2020-07-15 15:45:48 -07:00
danpawlik
7f9476f74e Added development/rawhide image
Fedora 33 is not released yet and the TripleO team would
like to perform some tests on that image.

Change-Id: I39f6bedadc12277739292cf31cc601bc3b6e30ec
2020-07-15 19:35:14 +02:00
Clark Boylan
1e2a34704a Allow setting Gitea repo branch on project creation
Note this shouldn't be used until we can configure Gerrit to do similar
with jeepyb. Otherwise we'll end up with mismatched branches between our
canonical source (Gerrit) and our mirrors (Gitea).

Change-Id: I8d353cbc90c2d354e7cdebfc4e247f3f73d97d86
2020-07-15 10:23:59 -07:00
Zuul
cd76e090c3 Merge "Update to gitea v1.12.2" 2020-07-15 17:05:41 +00:00
Monty Taylor
2302879244 Build multi-arch python-base/python-builder
In order to build multi-arch python images, we need
multi-arch python base and builder images.

Change-Id: Ifc0d6f7c16876bf55db8e1ee459a3eaa07744547
2020-07-15 09:09:35 -07:00
Zuul
1800b01bad Merge "Forward openstack-infra ML to openstack-discuss" 2020-07-15 15:22:14 +00:00
Zuul
ec22b7dac2 Merge "Stop cloning k8s-on-openstack" 2020-07-15 12:39:32 +00:00
Zuul
c1dfc78249 Merge "Backup all hosts with Ansible" 2020-07-14 23:02:52 +00:00
Ian Wienand
cacdb7f573 Backup all hosts with Ansible
The process of switching hosts to Ansible backups got a little
... backed up.  I think the idea was that we would move these legacy
hosts to an all-Ansible configuration a little faster than what has
ended up happening.

In the mean time, we have done a better job of merging our environment
so puppet hosts are just a regular host that runs a puppet step rather
than separate entities.

So there is no problem running these roles on these older servers.
This will bring consistency to our backup story with everything being
managed from Ansible.

This will currently setup these hosts to backup to the only opendev
backup server in vexxhost.  As a follow-on, we will add another
opendev backup host in another provider to provide dual-redundancy.
After that, we can remove the bup::site calls from these hosts and
retire the puppet-based backups.

Change-Id: Ieaea46d312056bf34992826d673356c56abfc87a
2020-07-15 08:33:44 +10:00
Ian Wienand
ba45f251d1 Fix junit error, add HTML report
Specifying the family stops a deprecation warning being output.

Add a HTML report and report it as an artifact as well; this is easier
to read.

Change-Id: I2bd6505c19cee2d51e9af27e9344cfe2e1110572
2020-07-15 07:03:22 +10:00
Ian Wienand
711b2493a9 testinfra: silence yaml.load() warnings
Switch to safe_load to silence warnings in output

Change-Id: If91f79a4648920999de8e6bf6e0c9fec82fde233
2020-07-15 07:03:22 +10:00
Ian Wienand
a020568ee5 Copy generated inventory to bridge logs
This is the inventory generated and used by bridge, copy it into the
logs as well.

Change-Id: I15d0ddc4c8340735c0332139ddedc06fc05b8269
2020-07-15 07:03:22 +10:00
Zuul
4b12ab8ad5 Merge "Add Zuul to backups group" 2020-07-14 20:04:57 +00:00
Monty Taylor
fca18e4776 Stop cloning k8s-on-openstack
We're not actually using this repo at the moment.

Change-Id: I765140c65e4d7b45e2258d8fc267090f982de058
2020-07-14 08:22:49 -05:00