15161 Commits

Author SHA1 Message Date
Jeremy Stanley
df23d48949 Reload haproxy configuration when config changes
Add an Ansible handler to send a hangup signal through
docker-compose to the running haproxy daemon any time the task to
update its configuration fires.

Change-Id: I1946c1e7eaaa8a8e2209007b5d065dba952ec6e2
2019-07-23 16:48:23 +00:00
Clark Boylan
08a113d4a8 Actually check backends are alive in haproxy
This adds the simplest form of health checking to haproxy, a tcp check
to the backends. We can do more sophisticated checks like checking ssl
negotiates or even HTTP requests but for now this is probably a good
improvement.

Change-Id: I3c6b07df4b3e0c380c757e1e5cb51ae0be655f34
2019-07-23 08:13:12 -07:00
Clark Boylan
cb33dba40a Increate gerrit user connection limit by 50%
Zuul has hit a scenario where a git repo update was unable to talk to
gerrit via ssh because it had reached its per user connection limit [0].
This then led to some openstack job failing [1].

The default limit (which we were using) is 64 connection per user.
Apparently this is not quite enough for a busy zuul? Increase this by
50% up to 96.

[0] http://paste.openstack.org/show/754741/
[1] http://lists.openstack.org/pipermail/release-job-failures/2019-July/001193.html

Change-Id: Ibeca2208485608f3b61aa716184165342bfcc3c9
2019-07-22 15:29:19 -07:00
Clark Boylan
ffcd1791bf Cleanup nodepool builder clouds.yaml
We ended up running into a problem with nodepool built control plane
images (has to do with boot from volume not allowing us to delete images
that are in use by a nova instance). We have decided to clean this up
and go back to not doing this until we can do it more properly.

Note this isn't a revert because having a group for access to control
plane clouds does seem like a good idea in general and I believe there
have been changes we'd have to resolve in the clouds.yaml files anyway.

Depends-On: https://review.opendev.org/#/c/665012/
Change-Id: I5e72928ec2dec37afa9c8567eff30eb6e9c04f1d
2019-07-22 13:55:29 -07:00
Clark Boylan
a2af942fa3 Remove gitea01 from inventory so we can replace it
The global inventory is used when launching nodes so if we want to
replace a server we have to remove it from the inventory first. This is
that step for replacing gitea01.

Note that when adding it back for the new server there are some edits to
make to the playbooks as noted in the gitea sysadmin docs.

We do also remove this instance from haproxy as well to prevent unwanted
connections while we flip things over.

Change-Id: If32405b1302353f1f262a30b7392533f86fec1e4
2019-07-22 09:20:17 -07:00
Ian Wienand
983761213f files.o.o : publish .log as text/plain
This is a follow on to I67870f6d439af2d2a63a5048ef52cecff3e75275 to do
the same for files.openstack.org (as
http://files.openstack.org/mirror/logs/ is a handy central place to
point people at)

Change-Id: I07c707d45ab3e3c6f87460b3346efd7026467c56
2019-07-22 16:32:50 +10:00
Zuul
3828e7a7cb Merge "Allow to rsync Centos Software Collections repo" 2019-07-22 06:00:40 +00:00
Ian Wienand
667e0dffa0 Add some pointers on the OpenDev PPA
The OpenStack/OpenDev PPA repositories are currently undocumented.
Add some information on where to find things.

Change-Id: Iea03c5d558b3dd6af9f7c860dfcc75a71dc59d9f
2019-07-22 10:58:54 +10:00
Ian Wienand
814b42f616 Set openafs cache sizes for mirror/mirror-update
Set the openafs cache values to the same as the puppet set values for
openafs-client role users.

Change-Id: I5a58673cad8df2a1e8dddb592c322e751d7f2ac5
2019-07-19 12:04:26 -07:00
Clark Boylan
00348a4d0d Add tool to analyze check and gate success rates
This tool scans gerrit changes for comments from zuul over the last 30
days to build out success rates for check and gate pipelines. This only
looks at changes that have merged to avoid those that never can merge
because they only fail or are expected to fail.

This tool emits information like:

  Changes: 4475
  Check Failures: 5317.0
  Check Successes: 9173.0
  Check Rate of failure: 0.3669427191166322
  Gate Failures: 687.0
  Gate Successes: 4450.0
  Gate Rate of failure: 0.13373564337161767
  Total Failures: 6004.0
  Total Successes: 13623.0
  Total Rate of failure: 0.3059051306873185

Change-Id: I759ba670c6b81f4425ce618c412db9cbd0e51401
2019-07-19 09:58:40 -07:00
Zuul
dd63186f66 Merge "Use swapfile if no extra device is present" 2019-07-18 20:55:33 +00:00
Zuul
2b46688cdd Merge "launch-node.py : add option to skip ipv6 address checks" 2019-07-18 20:54:05 +00:00
Zuul
40c53a796a Merge "mirror-update: update docs for mirror-update.opendev.org" 2019-07-18 20:54:03 +00:00
Zuul
4d25b6c3b8 Merge "Add mirror-update to run_all.sh" 2019-07-18 20:47:17 +00:00
Zuul
4e159a8ac2 Merge "Publish .log files as text/plain" 2019-07-18 20:47:15 +00:00
Zuul
1ee1e73c48 Merge "Disable cloud launcher cron job during CI" 2019-07-18 20:47:14 +00:00
Zuul
3fe88abb92 Merge "Remove apport package from ubuntu servers" 2019-07-18 13:43:05 +00:00
Zuul
876e7b0dbd Merge "Streamline documented bup setup process" 2019-07-18 13:33:08 +00:00
Zuul
5d530eb39f Merge "Add letsencrypt documentation" 2019-07-18 12:59:06 +00:00
Kien Nguyen
9b2862d5f9 Allow to rsync Centos Software Collections repo
Remove Centos sclo from exclude list.

Change-Id: I8aae878d4dbb40b537913c66912320a91a4ac99b
Needed-By: https://review.opendev.org/#/c/671178/
2019-07-18 17:06:11 +07:00
Clark Boylan
d80f4a93ab Collect haproxy logs via syslog
Haproxy wants to log to syslog (and not stdout for performance reasons,
see https://github.com/dockerfile/haproxy/issues/3). However there is no
running syslog in our haproxy container. What we can do is mount in the
host's /dev/log and have haproxy write to the hosts syslog to get
logging.

Do this via a docker compose volume bind mount.

Change-Id: Icf4a91c2bc5f5dbb0bfb9d36e7ec0210c6dc4e90
2019-07-17 13:40:53 -07:00
Clark Boylan
319c9c44f0 Use swapfile if no extra device is present
We are booting instances outside of rax and they don't always come with
extra devices that can be repurposed for swap. If in that case then
create a swapfile instead.

Note we do not use fallocate as swapon's manpage says this is suboptimal
with the linux kernel's swap implementation.

Change-Id: I8b9ce18c18e4069aba7de27bb6a9927627b15b49
2019-07-17 10:37:30 -07:00
Zuul
cb86492fa4 Merge "Silence InsecureRequestWarning and password warning" 2019-07-17 17:15:36 +00:00
Zuul
24ce1f6f8e Merge "Parallelize repo creation by org" 2019-07-17 17:01:38 +00:00
Zuul
8f9a402e57 Merge "Provide better module return info from gitea create repos" 2019-07-17 16:33:32 +00:00
Zuul
93dcd25db1 Merge "Use a thread pool to update gitea repos faster" 2019-07-17 16:28:46 +00:00
Monty Taylor
b58bc86c89 Silence InsecureRequestWarning and password warning
We're making these requests to localhost over an ssh connection.

The password warning, on the other hand, is a real thing. Let's not
log the gitea password when we run this in prod.

Change-Id: I2157e4027dce5ab9ebceb3f78dbeff22a83d9fad
2019-07-17 15:57:57 +00:00
James E. Blair
13c7c8bb7e Parallelize repo creation by org
This runs repo creation across two orgs at the same time.  It doesn't
help to parallelize more than 2 since openstack runs the entire time
in one thread (so the other thread handles all the other orgs).

Parallelizing by org avoids database contention for updating the user
table, since each org is a different user.  However, there's a weird
locking thing going on with the first update to the settings table,
so this does some extra work to serialize actions until we perform
that first update, then switches to parallel.

This is the maximum we can parallelize repo creation at the moment,
and it also maximizes settings updates (the settings updates take less
time than repo creation, so no further optimization helps).

Change-Id: I7f83dcdb4531a547ae5281434d7cda825dd50059
2019-07-16 14:24:44 -07:00
Zuul
6975e6d05f Merge "Improve idempotency of gitea-git-repos" 2019-07-16 20:51:36 +00:00
Zuul
d90bd72fdf Merge "Run actual full project creation in gitea test" 2019-07-16 20:51:34 +00:00
James E. Blair
1e18651565 Provide better module return info from gitea create repos
Be more correct about changed and failures.

Change-Id: I0b37b1bd85efc35233d864ca7801a8862806467f
2019-07-16 13:31:18 -07:00
Zuul
d6ec3a7a3e Merge "Add some logging to repo creation" 2019-07-16 20:04:45 +00:00
Zuul
36344bfcdd Merge "Translate gitea project creation to python" 2019-07-16 19:31:11 +00:00
James E. Blair
47bd535d60 Use a thread pool to update gitea repos faster
This keeps repo creation serialized (because of a bug in gitea),
but it parallelizes updating the settings.  This should reduce
our time by about half.

It also uses a requests session, though I'm not sure if that
really gets us anything.

It eliminates a couple of extraneous GET calls following 302
redirect responses from the POSTs on setting updates.

This will automatically paralellize to nproc * 5 threads.

Change-Id: I5549562d667c0939d0af1151d44b9190774196f9
2019-07-16 10:29:24 -07:00
Ian Wienand
82c6dec4fa Disable cloud launcher cron job during CI
This takes a similar approach to the extant ansible_cron_install_cron
variable to disable the cron job for the cloud launcher when running
under CI.

If you happen to have your CI jobs when the cron job decides to fire,
you end up with a harmless but confusing failed run of the cloud
launcher (that has tried to contact real clouds) in the ARA results.

Use the "disbaled" flag to ensure the cron job doesn't run.  Using
"disabled" means we can still check that the job was installed via
testinfra however.

Convert ansible_cron_install_cron to a similar method using disable,
document the variable in the README and add a test for the run_all.sh
script in crontab too.

Change-Id: If4911a5fa4116130c39b5a9717d610867ada7eb1
2019-07-16 15:01:55 +10:00
Zuul
4e050d981e Merge "Complete hide logic for Zuul CI comments in Gerrit" 2019-07-16 01:58:03 +00:00
Ian Wienand
e15735e586 Publish .log files as text/plain
Default apache mimetypes don't include .log as text/plain; add it.
Log export was added with I67870f6d439af2d2a63a5048ef52cecff3e75275 so
match the .log.1 file that logrotate creates for our rsync mirror logs
too.

Change-Id: Iaf3f19d26f3a6fda7ef3571573af219a31f1dced
2019-07-16 11:33:47 +10:00
Jeremy Stanley
5a30d26f44 Complete hide logic for Zuul CI comments in Gerrit
Apply the exclusion for trusted CI comments to the hide function's
conditional case as well as the toggle function's.

Change-Id: Ia4e5ec22a097a8b8cb564c237fd0aa48ab6f8724
2019-07-16 00:47:37 +00:00
Ian Wienand
a595d1d1d0 Add mirror-update to run_all.sh
It looks like I forgot to add this in
I525ac18b55f0e11b0a541b51fa97ee5d6512bf70 so the mirror-update
specific roles aren't running automatically.

Change-Id: Iee60906c367c9dec1143ee5ce2735ed72160e13d
2019-07-16 10:04:15 +10:00
James E. Blair
892596373f Improve idempotency of gitea-git-repos
When determining whether a project exists, we need to compare to
just the name, not the full data structure about the project.

Also, if the project exists, don't try to create it again; that
will return a 409 conflict error.

Change-Id: I0b8affac96b17fa73253082b1b87d4c00bf23463
2019-07-15 16:05:20 -07:00
Zuul
482abf3bf0 Merge "mirror-update: export mirroring logs" 2019-07-15 22:47:34 +00:00
Zuul
0fc17feb9f Merge "Don't hide Zuul CI comments" 2019-07-15 22:39:57 +00:00
Jeremy Stanley
0964733556 Don't hide Zuul CI comments
When filtering CI system comments, don't hide those from Zuul, our
gating CI system. It is important to see these comments as not all
results may match the patterns used to expose them as rows in the CI
table. Rename the "Toggle CI" button to "Toggle Extra CI" so that
the name remains accurate without being too verbose.

Change-Id: Id0cd8429ee5ce914aebbbc4a24bef9ebf675e21c
2019-07-15 16:19:59 +00:00
Zuul
0f78ac2dcc Merge "Add proxy for registry.access.redhat" 2019-07-11 20:39:34 +00:00
Monty Taylor
5c6b3411b7 Run actual full project creation in gitea test
Add the full remote_puppet_git playbook that we actually use in
production so that we can test the whole kit and caboodle. For
now don't add a review.o.o server to the mix, because we aren't
testing anything about it.

Change-Id: If1112a363e96148c06f8edf1e3adeaa45fc7271c
2019-07-11 13:39:22 -07:00
Alex Schultz
2f96a248c8 Add apt-puppetlabs mirroring back
This used to be mirrored, however there were issues when upstream
dropped the PC1 repositories a few months back. The puppet openstack
jobs are still trying to leverage this mirror but it does not exist in
some regions because it was disabled on the afs content. This change
fixes the reprepo configuration to still pull down puppet5/6 for xenial
and strech and add the symlink back to the mirrors.

Change-Id: I71ad5afe086a503d75a365543ad8869e35ef873b
2019-07-11 11:12:14 -06:00
Monty Taylor
f9358173a3 Add some logging to repo creation
So that we can verify what was done, we should emit some things to
log and return them.

Change-Id: I9c48e94fe099002335113aed296bfc9a52d4c10e
2019-07-11 11:54:47 -04:00
Zuul
46be2ccae6 Merge "Mailing list for Airship" 2019-07-11 15:13:28 +00:00
Monty Taylor
caebf387b4 Translate gitea project creation to python
Sadly, as readable as the use of the uri module to do the interactions
with gitea is, more reent ansible changed how subprocesses are forked
and this makes iterating over all the projects in projects.yaml take
an incredibly long amount of time.

Instead of doing it in yaml, make a python module that takes the list
one time and does looping and requests calls. This should make it be
possible to run the actual gitea creation playbook in integration tests.

Change-Id: Ifff3291c1092e6df09ae339c9e7dddb5ee692685
2019-07-11 08:21:35 -04:00
James E. Blair
ee3b273876 Exclude ansible_python_interpreter from write-inventory
Zuul now includes an ansible_python_interpreter hostvar in every
host in its inventory.  It defaults to python2.  The write-inventory
role, which takes the Zuul inventory and makes an inventory for
the fake bridge server in the gate passes that through.  Because it's
in /etc/ansible/inventory.yaml, it overrides any settings which may
arrive via group vars, but this is the way we set the interpreter
for all the hosts on bridge (we do not do so in the actual inventory
file).

To correct this, tell write-inventory to strip the
ansible_python_interpreter variable when it writes out the new
inventory.  This restores the behavior to match what happens on
the real bridge host.  One instance of setting the interpreter
for the fake "trusty" host used in base platform tests is moved to
a hostvars file to match the rest of the real hosts.

Change-Id: I60f0acb64e7b90ed8af266f21f2114fd598f4a3c
2019-07-10 10:10:02 -07:00