280 Commits

Author SHA1 Message Date
Clark Boylan
7ee556ca44 Have audit-users.py write out serialized data
This allows us to "query" the datastructure for different perspectives
without needing to rerun the costly queries each time we update
audit-uses.py. The script is predominantly collecting data now, then we
can use the python repl or other scripts to give us better insights.

We also do a small refactoring to simplify the collection of data.

Change-Id: Ie777ae706050b38ce294a1acf9b1b843fcf5ab41
2021-03-15 13:11:18 -07:00
Clark Boylan
112cbc6cfe Add tools being used to make sense of gerrit account inconsistencies
The first tool has been used to "retire" accounts that have preferred
email addresses without a matching external id. The second is being used
to make sense of whether or not we can do a bulk retirement of accounts
with email conflicts in their external ids. The third is a script that
can be used to remove external ids from accounts in bulk based on their
email addresses.

Change-Id: Idf22cfc9f2bac7d3921e006c40faef4585c2d977
2021-03-05 11:06:12 -08:00
Zuul
faf041540c Merge "Revert "Install older setuptools in puppet apply jobs"" 2021-02-11 01:33:50 +00:00
Clark Boylan
8d932bc706 Use sudo to move applytest results
We're wanting to more properly set permissions on the ansible puppet
role manifest dir. This ends up setting mode 0755 with ownership of
root:root on the dir. As a result sudo is necessary to move these
contents later.

Change-Id: I6b6aa79e8e8b63f4665679ab183a8551f0dd521e
2020-11-10 09:47:21 -08:00
Clark Boylan
b32d0b880c Revert "Install older setuptools in puppet apply jobs"
This reverts commit be802b319abf740e161d08e1f5a5a8c997c0278c.

A 50.1.0 release of setuptools has been made which reverts the breaking
behavior.

Change-Id: Ic32afc1466556eed9aaf3869974d85d5f779375f
2020-09-03 08:57:45 -07:00
Clark Boylan
be802b319a Install older setuptools in puppet apply jobs
We create a virtualenv to install ansible in which then runs puppet for
us in our puppet apply jobs. This is pulling in setuptools 50 which then
fails due to the problems setuptools 50 has with older pythons. Address
this by pinning back to setuptools <50.

Change-Id: I02ea466319f7cd90f73972bf5a99876d14823ac1
2020-09-01 13:59:21 -07:00
Sorin Sbarnea
b6311b5498 Switch prep-apply.sh to use python3
As part of py2 deprecation, we need to obliterate its use before
being able to drop it.

Change-Id: I35101a53265705513feaf7278e48c02a92a0c3e5
2020-06-15 14:43:25 -05:00
Monty Taylor
83ced7f6e6 Split inventory into multiple dirs and move hostvars
Make inventory/service for service-specific things, including the
groups.yaml group definitions, and inventory/base for hostvars
related to the base system, including the list of hosts.

Move the exisitng host_vars into inventory/service, since most of
them are likely service-specific. Move group_vars/all.yaml into
base/group_vars as almost all of it is related to base things,
with the execption of the gerrit public key.

A followup patch will move host-specific values into equivilent
files in inventory/base.

This should let us override hostvars in gate jobs. It should also
allow us to do better file matchers - and to be able to organize
our playbooks move if we want to.

Depends-On: https://review.opendev.org/731583
Change-Id: Iddf57b5be47c2e9de16b83a1bc83bee25db995cf
2020-06-04 07:44:36 -05:00
Monty Taylor
3e4d99b6fd Remove global variables from manifest/site.pp
We have one global variable that is used in two places.
By removing it, we can more easily split site.pp into
per-service manifest files, and ultimately we should be
deriving this from groups['elasticsearch'] anyway.

Change-Id: I1d794b269847da85778f71e816359953af9b31e0
2020-04-19 10:59:25 -05:00
Monty Taylor
00f30529e3 Make applytest files outside of system-config
We are copying system-config in parallel to a bunch of targets
and we're also creating and deleting applytest files. Instead,
do the apply test files outside of the dir that's going to
get synced in the puppet role.

While we're at it, copy don't link the openstack_project
module into /etc/puppet/modules, just to be sure.

Change-Id: I4bcd8ebd6da8395e77d673ac76f4c41568d810ec
2020-04-19 10:57:22 -05:00
Zuul
e3ad9e79eb Merge "Get rid of all-clouds.yaml" 2020-04-16 15:41:55 +00:00
Monty Taylor
ebae022d07 Use project-config from zuul instead of direct clones
We use project-config for gerrit, gitea and nodepool config. That's
cool, because can clone that from zuul too and make sure that each
prod run we're doing runs with the contents of the patch in question.

Introduce a flag file that can be touched in /home/zuulcd that will
block zuul from running prod playbooks. By default, if the file is
there, zuul will wait for an hour before giving up.

Rename zuulcd to zuul

To better align prod and test, name the zuul user zuul.

Change-Id: I83c38c9c430218059579f3763e02d6b9f40c7b89
2020-04-15 12:29:33 -05:00
Monty Taylor
8af7b47812 Get rid of all-clouds.yaml
We had the clouds split from back when we used the openstack
dynamic inventory plugin. We don't use that anymore, so we don't
need these to be split. Any other usage we have directly references
a cloud.

Change-Id: I5d95bf910fb8e2cbca64f92c6ad4acd3aaeed1a3
2020-04-09 16:44:20 -05:00
James E. Blair
9df9b7a5cb Use SafeLoader in irc_checks
This clears a python warning.

Change-Id: I79e088efb2a825a71723f97d563c96658f7f15ba
2020-04-09 06:52:32 -07:00
Zuul
927072831b Merge "Fix URLs after OpenDev rename" 2020-03-19 01:19:16 +00:00
Zuul
e3f7c8cee8 Merge "Update references to IRC channels" 2020-03-18 18:55:57 +00:00
Andreas Jaeger
173118e471 Fix URLs after OpenDev rename
As part of OpenDev rename, a lot of links were changed.
A couple of URLs point to old locations, update them.

This list was done while grepping for "openstack-infra" and fixing
locations that are wrong.

Change-Id: I313d76284bb549f1b2c636ce17fa662c233c0af9
2020-03-18 18:23:17 +01:00
Dr. Jens Harbott
c86525ccd3 Update references to IRC channels
With the move from OpenStack governance to our own OpenDev team, we
should also move to use the #opendev IRC channel in preference to
the #openstack-infra channel which will remain in use for OpenStack
specific discussions.

Update the references in our docs accordingly.

Change-Id: I448704f5d2664fd233a69a2ad12578ca24d9878a
2020-03-18 17:33:08 +01:00
Clark Boylan
9e394d24d0 Return goaccess html as zuul artifact
This will give us a nice link to the goaccess reports on the zuul
dashboard build pages.

Move ansible-lint config into config file

As of 4.2.0 we can configure ansible-lint with a config file. It's
also apparently now smart enough to only find ansible yaml. Let's
see how that goes.

Add a fake zuul_return module

This should let us fake out ansible-lint without having to install
all of zuul.

Change-Id: Ib233eb577a8ca2aabfe3a49b2cd823dd4a00bd82
2020-03-11 14:28:28 -05:00
Monty Taylor
2aebe4e09f Add quick script for cleaning boot from volume leaks
Sometimes we leak boot from volume volumes. This will clean them
up.

Change-Id: I45182c1dcad0cdcbc327aaef3a63d37947f8a66d
2020-01-10 16:55:07 -06:00
Zuul
cd402000a4 Merge "Several updates because the world is a dark place" 2019-10-19 00:58:47 +00:00
Monty Taylor
9ab25e89a9 Several updates because the world is a dark place
A few things have changed and we need to fix them in one go.

Use mirror for installing docker for buildset-registry

While, we need to make this more systemic, that's hanging off of the
mirror rework. For now, since we know all of these jobs are debian
based, just set the mirror location.

Replace use of zuul cloner with git clones

You can never be a prophet in your own hometown. This is now broken
because of the git cache rework, so just replace it.

Update libjemalloc library

python:slim is based on buster now, which has libjemalloc2 not
libjemalloc1.

Remove gerrit repo remote for submodules

A recent change to the base jobs to use prepare-workspace-git
broke the gerrit image builds by actually having the origin
remote by /dev/null as intended. This breaks submodules because
for a few of them where we don't have matching stable branches
the submodule relative path behavior is actually exactly what
we want.

Since we don't care about the remote otherwise, remove the
origin remote before doing the submodule update --init so that
the submodule will clone the refs from the zuul prepared repo.

Change-Id: Ieb5b6bc8711fe971ed3445c7c267306ac4616464
2019-10-19 07:51:29 +09:00
James E. Blair
d284333363 Remove newlines in afs server params variable
This causes newlines to appear in the config file which causes the
server to fail to start which is the opposite of what this is
supposed to do.

Change-Id: I2ff7e8835878652b3a7cdc2f633d263b37aaa7e9
2019-09-06 11:20:15 -07:00
Zuul
1b14855a45 Merge "AFS server restart and audit logging : helper script" 2019-08-29 21:03:09 +00:00
Ian Wienand
35f1321e14 AFS server restart and audit logging : helper script
This script helps restart the AFS servers, which is useful when
updating parameters.  It can also enable audit logging.

It can also stop and start the servers, although it's unlikely we'd
want all the servers offline at the same time so stopping has a
warning included.

Documentation is updated to refer to the helper script

Change-Id: Idcb3e43a3f6e614cdb787d4334e692a98bffdd15
2019-08-02 16:37:00 +10:00
Zuul
3e03b7481d Merge "Add tool to analyze check and gate success rates" 2019-07-31 00:28:06 +00:00
Clark Boylan
ffcd1791bf Cleanup nodepool builder clouds.yaml
We ended up running into a problem with nodepool built control plane
images (has to do with boot from volume not allowing us to delete images
that are in use by a nova instance). We have decided to clean this up
and go back to not doing this until we can do it more properly.

Note this isn't a revert because having a group for access to control
plane clouds does seem like a good idea in general and I believe there
have been changes we'd have to resolve in the clouds.yaml files anyway.

Depends-On: https://review.opendev.org/#/c/665012/
Change-Id: I5e72928ec2dec37afa9c8567eff30eb6e9c04f1d
2019-07-22 13:55:29 -07:00
Clark Boylan
00348a4d0d Add tool to analyze check and gate success rates
This tool scans gerrit changes for comments from zuul over the last 30
days to build out success rates for check and gate pipelines. This only
looks at changes that have merged to avoid those that never can merge
because they only fail or are expected to fail.

This tool emits information like:

  Changes: 4475
  Check Failures: 5317.0
  Check Successes: 9173.0
  Check Rate of failure: 0.3669427191166322
  Gate Failures: 687.0
  Gate Successes: 4450.0
  Gate Rate of failure: 0.13373564337161767
  Total Failures: 6004.0
  Total Successes: 13623.0
  Total Rate of failure: 0.3059051306873185

Change-Id: I759ba670c6b81f4425ce618c412db9cbd0e51401
2019-07-19 09:58:40 -07:00
Zuul
1fe34e00d4 Merge "Add control plane clouds to nodepool builder clouds.yaml" 2019-06-04 20:15:24 +00:00
Zuul
216059e134 Merge "Add opendev migration repo rename scripts" 2019-05-30 21:07:37 +00:00
Jeremy Stanley
0c0b8e3087 Add opendev migration repo rename scripts
Git repo moves based on cgit aliases from project-config, the
OpenStack TC guidance recorded in
http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004920.html
and the ethercalc used to collect input from other users of the
system. Also the results of an extensive bikeshedding session at
http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2019-04-11.log.html#t2019-04-11T14:54:09
which concluded that anything left homeless goes in a namespace
called "x" since that's short, a basic alphabetic character and
provides no particular connotation.

The opendev-migrate script, when run, provides a shareable rendering
on stdout and also writes a repos.yaml file for input into the
rename_repos playbook.

The opendev-patching script, when run, uses the repos.yaml file and
iterates over a tree of Git repositories updating their Zuul
configuration, playbooks and roles as well as .gitreview files both
for the project renames and the opendev hostname changes. It also
creates a rename commit in project-config so that manage-projects
will be in sync with the results of the rename_repos playbook.

Change-Id: Ifa9fa6896110e8a33f32dcda6325bd58846935e2
Task: #30570
Co-Authored-By: James E. Blair <jeblair@redhat.com>
2019-05-23 22:01:07 +00:00
Clark Boylan
926ba11184 Cleanup bashate errors to make them easier to understand
We ignore E006 which is line lenght longer than 79 characters. We don't
actually care about that. Fix E042 in run_all.sh this represents a
potential real issue in bash as it will hide errors.

This makes the bashate output much cleaner which should make it easier
for people to understand why it fails when it fails in check.

Change-Id: I2249b76e33003b57a1d2ab5fcdb17eda4e5cd7ad
2019-05-23 14:00:37 -07:00
Monty Taylor
ff1b8a94c6 Add control plane clouds to nodepool builder clouds.yaml
In order to have nodepool build images and upload them to control
plane clouds, add them to the clouds.yaml on the nodepool-builder
hosts. Keep them out of the launcher configs by splitting the config
templates. So that we can keep our copies of things to a minimum,
create a group called "control-plane-clouds" and put bridge and nb0*
in it.

There are clouds mentions in here that we no longer use, a followup
patch will clean those up.

NOTE: Requires shifting the clouds config dict from
host_vars/bridge.openstack.org.yaml to group_vars/control-plane-clouds.yaml
in the secrets on bridge.

Needed-By: https://review.opendev.org/640044
Change-Id: Id1161bca8f23129202599dba299c288a6aa29212
2019-05-23 14:34:10 -05:00
Zuul
01dfccbd99 Merge "Drop tools/owners.py" 2019-05-23 07:57:48 +00:00
Ian Wienand
d5b321b074 Handle moved puppet repos
As per [1], it seems puppet has "cleaned up" most of the packages we
are using to install.

Install the puppet-agent packages directly as puppet's archive location
is not a valid repo. With puppet 4 at least these packages should bundle
everything we need including ruby.

[1] https://groups.google.com/forum/#!msg/puppet-users/cCsGWKunBe4/OdG0T7LeDAAJ

Depends-On: https://review.opendev.org/659384
Depends-On: https://review.opendev.org/659395
Change-Id: Ie9e2b79b42f397bddd960ccdc303b536155ce123
2019-05-15 16:03:07 -07:00
Jeremy Stanley
4cb523cdc9 Drop tools/owners.py
Now that the tools/owners.py script is a module in the
openstack_election package within the openstack/election repository,
we can stop providing a copy here.

Change-Id: I39efbad539790687646c1d76159894e9e997ff72
Depends-On: I180ef0e5ec880b46f0427c1c952b640a780b5732
2019-05-12 11:26:39 +00:00
Monty Taylor
e69c7b7fb9 Rename review.openstack.org to review.opendev.org
There are many references to review.openstack.org, and while the
redirect should work, we can also go ahead and fix them.

Change-Id: I28f398796a6392a3dffea1d25cfe2ae3a36a3589
2019-05-09 14:38:51 +00:00
Zuul
c01a9eeccb Merge "Add script to automate GitHub organization transfers" 2019-04-23 21:52:22 +00:00
James E. Blair
8b2a4dbedd Fix logstash filter location
Change-Id: I70fbe2a9ba0ab909002704ff97b5ee149040e742
2019-04-20 09:41:49 -07:00
Monty Taylor
c6d129a108 Update some paths for opendev
There's a bunch in here. This is mostly big-ticket things and test
fixes. Also, change the README to rst - because why is it markdown?

Depends-On: https://review.opendev.org/654005
Change-Id: I21e5017011e1111b4d7a9e4bf0ea6b10f5dd8c1b
2019-04-20 09:31:14 -07:00
Ian Wienand
6a1d4da730 Add Puppet-Version: !X skip to apply tests
Setting Puppet-Version: !X (where X would usually be 3) marks a hosts
as not wanting to run the apply tests for that puppet version.  This
is helpful for puppet4 hosts that wish to bring in new modules that
are not puppet3 compatible.

Change-Id: I081d15a53bd85152e7729c4c1da094dfee6d7073
2019-04-15 12:05:26 +10:00
David Moreau Simard
c4d757da4e
Add script to automate GitHub organization transfers
This script requires GITHUB_USERNAME and the GITHUB_PASSWORD env
variables to be set and lets users with sufficient privileges initiate
a transfer from a GitHub organization to another by specifying two
arguments, for example:
  ./github-org-transfer.py oldorg/repo neworg/repo

Change-Id: I2383d256958c028efe81b235ff8641d131bbb3a7
2019-04-12 18:01:16 -04:00
James E. Blair
2db41fc488 Update hieraedit for python3
Change-Id: Ibd8991eb466416f77a2decc7b0a280d8e4124942
2019-03-26 15:32:23 -07:00
Ian Wienand
bdf8cd90f7 Replace openstack.org git:// URLs with https://
This is a mechanically generated change to replace openstack.org
git:// URLs with https:// equivalents.

This is in aid of a planned future move of the git hosting
infrastructure to a self-hosted instance of gitea (https://gitea.io),
which does not support the git wire protocol at this stage.

This update should result in no functional change.

For more information see the thread at

 http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003825.html

Change-Id: I6c126f7e724249741403a87733f546c1642f7f25
2019-03-25 09:40:52 -07:00
Monty Taylor
9cac3c6b63 Run k8s-on-openstack to manage k8s control plane
The k8s-on-openstack project produces an opinionated kubernetes
that is correctly set up to be integrated with OpenStack. All of the
patches we've submitted to update it for our environment have been
landed upstream, so just consume it directly.

It's possible we might want to take a more hands-on forky approach in
the future, but for now it seems fairly stable.

Change-Id: I4ff605b6a947ab9b9f3d0a73852dde74c705979f
2019-02-05 18:50:31 +00:00
Monty Taylor
ac602f9d14 Add a script to generate the static inventory
We moved from dynamic to static inventory. When creating a new host with
launch-node, a script isn't really needed, the inventory is yaml, the
new host can just be added. However, generating a new inventory by
hitting the APIs of all of our clouds might be useful, so add a utility
script to help in case such a thing is needed.

Change-Id: Iae1be8e9cfe19533005e9f0395d1ef7a6427bc83
2018-12-12 07:20:16 +11:00
Zuul
38b9e983c5 Merge "Run puppet apply test serially on centos" 2018-10-08 16:38:39 +00:00
Zuul
59f03951f7 Merge "Use zuul-sphinx README.rst detection" 2018-09-19 23:54:54 +00:00
James E. Blair
11238771d5 Write ansible log to fileout in apply test
So that ansible output lives with the puppet output.

Change-Id: Ia3022096a8a006b1b1eba864df3809b3e66cf7fb
2018-09-06 15:15:40 -07:00
Clark Boylan
f9e18bc348 Run puppet apply test serially on centos
There appears to be a race running the ansible synchronize (rsync under
the hood) top copy puppet modules for multiple puppet applies at the
same time on CentOS7. Running this in parallel appears safe on Ubuntu
and does save quite a bit of job runtime.

Workaround this by running the apply test serially on CentOS only.

Change-Id: Icd0836db215c0b417989d38994a378a705bbc62b
2018-09-06 15:06:27 -07:00