15963 Commits

Author SHA1 Message Date
Clark Boylan
140b95a2d0 Clear LD_PRELOAD variable on zuul-web containers
We have been having memory leak issues with zuul-web on our move from
running on the host with python3.5 to running in containers with
python3.7 and python3.8. One other thing that chagned was we added
LD_PRELOAD settings to use jemalloc instead of normal libc provided
malloc. In an effort to rule this out disable jemalloc in the zuul-web
containers.

Change-Id: Icf03b60266f876dd7c322e8c8f7c207b692d3ad7
2020-05-04 13:00:52 -07:00
Andreas Jaeger
90fa9170dd Fix py3 problem in accessbot.py
Change-Id: I4e3b3683b33b600cc4c1e4f09ba02c59e6f5cbb9
Fix:
TypeError: unsupported operand type(s) for +: 'dict_items' and 'dict_items'
2020-05-02 22:11:21 +02:00
Andreas Jaeger
26f1dae81b Fix accessbot/Dockerfile
The command is broken (extra ", misses redirection) and duplicates
content from accessbot.sh. Call accessbot.sh directly and use that one
instead.

Change-Id: Ieb530ef27e5995f2848a3c23a6c04a0717716e14
2020-05-02 18:02:06 +02:00
Zuul
33e48396d1 Merge "Meetpad: redirect 80 to 443" 2020-05-01 14:34:00 +00:00
James E. Blair
a11a97e981 Meetpad: redirect 80 to 443
This should redirect meetpad connections on port 80 to port 443.

Change-Id: Ibfee9d9e6efbbac01ccf8363909a4da772a4dc42
2020-04-30 15:45:31 -07:00
Clark Boylan
6b1feb8ae6 Add logo file to docs
We are trying to use this file in our docs config but the file was
mistakently not added in that change. Add it now.

Change-Id: I8f5f9d62f96d8532477c42a7076c57aa6548c9cf
2020-04-30 13:37:55 -07:00
Zuul
9b1161e051 Merge "Set up robots.txt on lists servers" 2020-04-30 18:42:55 +00:00
Zuul
8392caaa69 Merge "Add --fail flag to zuul status backup curl" 2020-04-29 23:05:52 +00:00
Zuul
fdfbc3d0b9 Merge "Run zookeeper cluster in nodepool jobs" 2020-04-29 22:22:37 +00:00
Zuul
2be697a8e3 Merge "Don't restart the zuul scheduler in prod" 2020-04-29 22:22:00 +00:00
Monty Taylor
8d7075b02f Run zookeeper cluster in nodepool jobs
Rather than running a local zookeeper, just run a real zookeeper.
Also, get rid of nb01-test and just use nb04 - what could possibly
go wrong?

Dynamically write zookeeper host information to nodepool.yaml

So that we can run an actual zk using the new zk role on hosts in
ansible inventory, we need to write out the ip addresses of the
hosts that we build in zuul. This means having the info baked in
to the file in project-config isn't going to work.

We can do this in prod too, it shouldn't hurt anything.

Increase timeout for run-service-nodepool

We need to fix the playbook, but we'll do that after we get the
puppet gone.

Change-Id: Ib01d461ae2c5cec3c31ec5105a41b1a99ff9d84a
2020-04-29 16:18:25 -05:00
Zuul
74730d6ec6 Merge "Run test playbooks with more forks" 2020-04-29 21:08:58 +00:00
Zuul
445d0f5d52 Merge "Increase timeout on system-config-run-zuul" 2020-04-29 21:08:56 +00:00
Zuul
3c49a46fe6 Merge "Remove git*.openstack.org" 2020-04-29 19:53:43 +00:00
Zuul
e108ea1232 Merge "Disable global Github replication" 2020-04-29 19:53:41 +00:00
Zuul
961358dd8d Merge "Update to tip of master in periodic jobs" 2020-04-29 19:49:26 +00:00
Clark Boylan
eeac5467c3 Set up robots.txt on lists servers
This sets up a robots.txt on our lists servers. To start this file
prevents SEMrush bot from indexing our lists as that has been causing
lists.openstack.org to OOM with many listinfo processes started by
Apache.

We've avoided this OOM by manually configuring this robots.txt. Other
things we have ruled out are bup and input email causes qrunner's to
grow unexpectedly large. Fairly confident this bot is the trigger.

Note this fixes testing by adding 'hieradata' to set listpassword var.

Depends-On: https://review.opendev.org/724389
Change-Id: Id4f6739a8cf6a01f9796fa54c86ba1af3e31fecf
2020-04-29 17:48:13 +00:00
Monty Taylor
767e001cd6 Run test playbooks with more forks
As we add jobs that have more nodes in them, we need to make
sure we're running ansible with enough forks that the jobs
don't take forever.

Change-Id: I2b5bf55bd65eaf0fc2671f5379bd0cb5c3696f87
2020-04-29 12:04:22 -05:00
Clark Boylan
7c92589f3a Increase timeout on system-config-run-zuul
This job compiles openafs with dkms among other things that cause it run
over the default half hour timeout occasionally. Bump the timeout to an
hour to deal with that.

Change-Id: I8a56a7f42ce2ee8331befb45aceb1d511a33d9e6
2020-04-29 17:03:56 +00:00
Thierry Carrez
5b4c917d30 Disable global Github replication
We used to replicate every openstack/* project to GitHub,
through a global replication at Gerrit-level. Now that the job for
granular replication is in place, we can stop the global replication,
so that only active/official repositories are synced.

Depends-On: https://review.opendev.org/724310
Change-Id: Ibba02e626e33aba9779f771d5ae49920bac86b19
2020-04-29 14:37:23 +02:00
Zuul
fffd971c38 Merge "Use python3 for ansible" 2020-04-28 19:53:18 +00:00
Clark Boylan
5acf91a3f6 Don't restart the zuul scheduler in prod
We use the zuul_scheduler_start flag to determine if we want to start
the zuul-scheduler when new containers show up. Unfortunately we weren't
setting zuul_scheduler_start in prod so we failed with this error:

  error while evaluating conditional (zuul_scheduler_start | bool): 'zuul_scheduler_start' is undefined

Fix this by treating an unset var as equivalent to a set truthy var
value. We do this instead of always setting the var to false in prod as
it simplifies testing.

Change-Id: I1f1a86e80199601646c7f2dec2a91c5d65d77231
2020-04-28 12:24:13 -07:00
Monty Taylor
32978fe729 Update to tip of master in periodic jobs
The intent of the periodic jobs is to run with latest master. If
they get enqueued, then other patches land, they'll still run with
the value of the zuul ref from when they were enqueued. That's not
what we want for prod, as it can lead to running old versions of
config.

We don't usually like doing this, but in this case, rather than
making us remember to add a flag every time a prod job gets added
to a periodic pipeline, how's about we just calculate it.

Change-Id: Ib999731fe132b1e9f197e51d74066fa75cb6c69b
2020-04-28 12:01:51 -05:00
Monty Taylor
5e6aa5e70d Use python3 for ansible
We get deprecation warnings from ansible about use
of python2 on xenial hosts. Rather than setting
ansible_python_interpreter to python3 on a host by
host basis, set it globally to python3.

Set it to python for the one host that's too old,
refstack.openstack.org, which is running on trusty
which only has python3.4.

Change-Id: I4965d950c13efad80d72912911bc7099e9da1659
2020-04-28 11:54:15 -05:00
Zuul
81d9dc44ab Merge "Use the sync-project-config role in service-zuul" 2020-04-28 14:28:36 +00:00
Monty Taylor
686b30e71d Add --fail flag to zuul status backup curl
We don't want to save documents if they are errors.

Change-Id: I259d926605c56c2b2fd723c3783bb468144e063b
2020-04-28 08:33:05 -05:00
Zuul
2cd57c3b18 Merge "Improve zuul-web apache config" 2020-04-28 00:38:53 +00:00
Clark Boylan
6bc23598d3 Improve zuul-web apache config
Compress css and javascript content as they can be quite large for zuul.

Also, cache status json results when using the non whitelabeled api
paths for zuul.opendev.org. This should improve performance for those
status files.

Change-Id: I7b965b27a88d5fda4d43be31c39989994334989c
2020-04-27 15:08:08 -07:00
James E. Blair
913ea11a86 Clean up some job descriptions
Also remove the unused infra-prod-zuul_reconfigure job.

Change-Id: I67272b35a9c40ba70bba028cbd671faa852d1d19
2020-04-27 14:40:18 -07:00
Zuul
b0d84dc6aa Merge "Rework zuul start/stop/restart playbooks for docker" 2020-04-27 16:17:14 +00:00
Andreas Jaeger
1a1390d892 Remove git*.openstack.org
These hosts do not exist anymore, remove them from cacti list
in hiera.

Change-Id: I0b3ddd8360a78f11cab902991d171acb6ae2f750
2020-04-27 17:59:10 +02:00
Zuul
adbe2cb70a Merge "Run smart-reconfigure instead of HUP" 2020-04-27 14:59:24 +00:00
Monty Taylor
2a7c755a08 Rework zuul start/stop/restart playbooks for docker
If we need to start and stop, it's best to use playbooks.

We already have tasks files with start commands in each role,
so put the stop commands into similar task files.

Make the restart playbook import_playbook the stop and start
playbooks to reduce divergence.

Use the graceful shutdown pattern from the gerrit docker-compose
to stop the zuul scheduler.

Change-Id: Ia20124553821f4b41186bce6ba2bff6ca2333a99
2020-04-27 09:34:50 -05:00
Zuul
158f190fcd Merge "Cron module wants strings" 2020-04-27 01:39:06 +00:00
Ian Wienand
1d0d62c6a6 status.openstack.org: send zuul link to opendev zuul
Due to a configuration issue, zuul.openstack.org is currently throwing
SSL validation errors.  Update the status.openstack.org to the
canonical OpenStack tenant page directly.

Change-Id: Idf08e140de11126061cb6f9783d13dc64fefff60
2020-04-27 09:42:11 +10:00
Monty Taylor
5d1f556c2e Run smart-reconfigure instead of HUP
We don't want to HUP all the processes in the container, we just
want zuul to reconfigure. Use the smart-reconfigure command.

Also - start the scheduler in the gate job.

Change-Id: I66754ed168165d2444930ab1110e95316f7307a7
2020-04-26 11:01:08 -05:00
Monty Taylor
267d7e728d Cron module wants strings
We're getting ansible warnings about converting these to strings
for us.

Change-Id: Iba70b5e71ded01ca2e11276fb2f007c9d7aa1866
2020-04-25 11:59:29 -05:00
Zuul
2e2be9e687 Merge "Fix zuul.conf jinja2 template" 2020-04-25 15:28:37 +00:00
Zuul
ec84c8563e Merge "The package is libjemalloc1" 2020-04-25 15:28:34 +00:00
Zuul
c07d86d7ce Merge "Use pip3 to install zuul on executors" 2020-04-25 15:24:00 +00:00
Clark Boylan
18568882b8 Fix zuul.conf jinja2 template
This adds a necessary newline, removes port numbers, and sets the
executor ssh key to the correct path.

Change-Id: I6b4afa876b6cd7d8f87cc35bc51b4e9d6e31ee2b
2020-04-24 18:15:31 -07:00
Monty Taylor
b7c0c0f8ce The package is libjemalloc1
When we install packages on ubuntu, we should use their actual
package names rather than incorrect or otherwise fictional
package names.

Also, fix the hostname in the test job - because when we don't
do that, we don't run all of the roles, and thus we don't
catch these things.

Change-Id: I18e676ef0fe343513db4c8ad7e340ee45092c0a3
2020-04-24 18:15:31 -07:00
Clark Boylan
6570eda366 Use pip3 to install zuul on executors
The executors aren't using docker container images yet due to conflicts
with bubblewrap. This means we are still installing it directly on the
host using pip. Unfortunately we were using `pip` before which maybe
install to python2 and zuul doesn't run under python2. Address this by
explicitly using pip3.

Change-Id: I2ec551e8207e29ca420b09b8818154b9c32b47cf
2020-04-24 18:15:31 -07:00
Zuul
1c3a4520ef Merge "Remove an extra backslash" 2020-04-25 00:24:37 +00:00
Monty Taylor
77e848b237 Remove an extra backslash
Change-Id: Ia027a2d51ed51ece224de07ef0d45a7ee95dc485
2020-04-24 18:04:53 -05:00
Monty Taylor
05b0587871 Add nodepool node key
Change-Id: I28ccb83fc984190b1ce8e3e18c5945209fcb2387
2020-04-24 17:54:50 -05:00
Monty Taylor
c1714ef918 Use the sync-project-config role in service-zuul
We can sync the project-config repo instead of cloning it on the
hosts.

Change-Id: Ic238d743f4dda0a3c04a7e3618a9d86d3c3e5371
2020-04-24 14:43:35 -05:00
Zuul
1febd9a0f2 Merge "Add service-zookeeper to nightly jobs" 2020-04-24 17:58:21 +00:00
Zuul
8bd18e9089 Merge "Actually run service-zuul playbook in prod" 2020-04-24 17:34:26 +00:00
Zuul
d3360a7d60 Merge "Remove two unused ansible vars files" 2020-04-24 17:17:33 +00:00