Commit Graph

227 Commits

Author SHA1 Message Date
Ian Wienand
4402201d56 roles: Add README.rst and lint
These role docs aren't exactly War and Peace, but I think longer term
as we fiddle about making things generic or not and moving them
around, we'll be better off having kept ourselves to writing
*something*.

Add terse README.rst files for all existing roles, and add simple
linter check to ensure new roles get them too.

Change-Id: Ibc836310fb8a45e12c2e31f112d92509ac350413
2018-08-23 21:34:42 +10:00
Zuul
744d7fb737 Merge "Remove infracloud references" 2018-08-20 20:21:40 +00:00
Monty Taylor
0625c289c8
Remove infracloud references
We don't run a cloud anymore and don't use these. With the cfg
management update effort, it's unlikely we'd use them in the form they
are in even if we did get more hardware and decide to run a cloud again.

Remove them for clarity.

Change-Id: I88f58fc7f2768ad60c5387eb775a340cac2c822a
2018-08-20 11:03:55 -05:00
Monty Taylor
a634593a05
Set mgmt_hieradata in puppet group_vars
This is not a variable describing the system-under-management
bridge.openstack.org - it's a variable that is always true for all
systems in the puppet group.

As a result, update the puppet apply test to figure out which directory
we should be copying modules _from_ - since the puppet4 tests will be
unhappy otherwise.

Change-Id: Iddee83944bd85f69acf4fcfde83dc70304386baf
2018-08-17 14:25:50 -05:00
Monty Taylor
92c9a7c869
Clean up puppet variables and playbooks
The puppet playbooks were some of the first we wrote, so they're
slightly wonky.

Remove '---' lines that are completely unnecessary.

Fix indentation.

Move some variables that are the same everywhere into
ansible variables.

Put puppet related variables into the puppet group_vars.

Stop running puppet on localhost in the git playbook.

Change-Id: I2d2a4acccd3523f1931ebec5977771d5a310a0c7
2018-08-17 09:41:12 -05:00
Monty Taylor
1a8c2f66da
Move /opt/system-config/production to /opt/system-config
The production directory is a relic from the puppet environment concept,
which we do not use. Remove it.

The puppet apply tests run puppet locally, where the production
environment is still needed, so don't update the paths in the
tools/prep-apply.sh.

Depends-On: https://review.openstack.org/592946
Change-Id: I82572cc616e3c994eab38b0de8c3c72cb5ec5413
2018-08-17 09:41:02 -05:00
Monty Taylor
2e6e67e8d2
Add base.yaml to kick.sh
There are things in base that are needed. Add them to kick.sh.

Change-Id: I94a92cd98fa62552df40b73976bcec764c66ebec
2018-08-17 05:44:37 -05:00
Zuul
b0343f35f9 Merge "Add a clouds.yaml checker" 2018-06-26 23:33:57 +00:00
Zuul
61eff41bac Merge "tools/sorted_modules.py : make python3 safe" 2018-06-26 23:12:57 +00:00
Ian Wienand
19ed6637b6 Add a clouds.yaml checker
Load our various cloud configuration files into openstacksdk for a
basic sanity check of the configuration.

Change-Id: Ieedb3439533f3162d2b5cecca4b7fc07b631019e
2018-06-22 11:47:39 +10:00
Ian Wienand
c807245d8b tools/sorted_modules.py : make python3 safe
Trivial updates to print statements only

Change-Id: I633c21a226364d045fc1650cb41b79b83988d7db
2018-06-22 11:47:39 +10:00
Colleen Murphy
6168de161f Fix acceptance test setup script for puppet 4
Map the modules to the correct path depending on whether this is running
under puppet 4. Currently, acceptance tests for puppet 4 will always
explicitly set PUPPET_VERSION.

Also clean up an unused variable to avoid confusion.

Change-Id: I9bd36a277411c72ef77ce157f8364845acc2e923
2018-06-15 11:09:43 +02:00
Clark Boylan
8b3c5fc255 Don't run against localhost by default with kick.sh
Running against localhost by default with kick.sh means that if you have
disabled puppet crons on the puppet master to test things you will
reenable them automatically creating potentially bad side effects. Avoid
this problem by forcing people to explicitly puppet localhost if this is
what they want.

Change-Id: Id4212005735cac13e6d26439f309569d7567206c
2018-04-25 13:26:03 -07:00
Colleen Murphy
f10f116f60 Fix puppet config for puppet 4
There are some assumptions made about the puppet config paths that
aren't applicable for puppet 4. Configuration, including modules,
belongs under /etc/puppetlabs. It's also no longer necessary to fix up
the templatedir or server configs in puppet.conf.

Change-Id: I3b544b6ce4a96a7a2478522a78402f77ff86a5a5
2018-04-20 14:24:17 +02:00
Ian Wienand
53ac2cfeae Remove per-image logging from nodepool-builder logging config
Change Ia7415d2fbbb320f8eddc4e46c3a055414df5f997 removed per-image
logging from nodepool; it now keeps a series of named log files.  The
"image" logger is removed, along with all the specific build logs.

I972a66f9c153f52df934d25ab2f77596bb2e5429 removed the
image_log_document_root argument; which wasn't doing anything in
nodepool::builder, but was actually being used to fill-out the
builder-logging.conf file.  Thus we need this to not have a bad path
in the config file.

Change-Id: I6fdc3b622454b069f3ad2cf42da584d8df23a110
2018-02-26 14:12:35 +11:00
Paul Belanger
4ca2a7a598
Remove support for snapshot images in nodepool
Update our nodepool_log_config.py script and remove support for
snapshots.

Generate latest version of nodepool-builder logging.

Change-Id: Ib87e8c352b56f63ca44668f43fa440645fab9f37
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-01-21 18:20:00 -05:00
Paul Belanger
a3a2a84d8a
Remove puppetdb / puppetboard server
Currently puppetdb and puppetboard have been broken for some time (+1
year) and with ubuntu precise becoming EOL it is prime for deleting.

This leaves openstack-infra with a gap in reporting for non-root
users.  As such, as proposal is in the works to maybe use ARA.

Change-Id: Ifc73a2dba3b37ebe790a29c0daa948d6bad0aa33
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-12-11 10:11:11 -05:00
Clark Boylan
7867989dd5 Set origin on /etc/project-config in rsepc tests
We use zuul cloner + zuul to set up /etc/project-config for integration
testing with rsprec. As of zuulv3 this no longer results in an origin
remote being set which the puppet vcsrepo resource expects.

Address this expectation by manually configuring an origin remote after
we clone the repo.

Change-Id: I83e7088c36b1b57279bb32e44c9f74730da582cf
2017-11-29 14:57:00 -08:00
Jeremy Stanley
d263d0ce45 Use cgit more in contributor listing
The change which just merged to incorporate recent retired
deliverables and removed projects in active contributor analysis was
written prior to the gitweb->cgit transition, so fix up that feature
to also use the new cgit-based function call.

Change-Id: Ifde87147399967c43f3e4c93de4347acd31e73cc
2017-11-01 18:35:00 +00:00
Zuul
ec0908b08c Merge "Use foundation member IDs in invitations list" 2017-11-01 17:29:12 +00:00
Zuul
0e3d1ee751 Merge "Include retired projects/deliverables in elections" 2017-11-01 17:29:10 +00:00
Zuul
fa8d21205b Merge "Remove unneeded encoding change." 2017-10-31 01:06:20 +00:00
Zuul
d15a91c978 Merge "Switch to cgit from gitweb" 2017-10-31 01:03:58 +00:00
Tony Breeds
6a979feb6b Remove unneeded encoding change.
Change-Id: Ia83f8aadcc5dca4adc54a1fa253e38f4aa0275b4
2017-10-27 10:19:40 +11:00
Paul Belanger
7797b5927a
Set SETUP_PIP=false for tools/prep-apply.sh
Because we are running tests in the gate, we can skip downloading pip,
as our images have already been setup to use the latest version.

Change-Id: I3d235334e0457b70bb69a8fe4b475e69d84c3407
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-10-15 21:31:08 -04:00
Tony Breeds
4d2ff114c6 Switch to cgit from gitweb
Change-Id: I06a65df0140530df0c414565f78de437a5f26ac1
2017-10-14 18:37:14 +11:00
Jeremy Stanley
bf8e2b9ad0 Limit PTL rolls to foundation members
Missing the original I920e1bc134d1283e2019f4389a5491e9b746cca0
implementation, also filter the per-team PTL electorates to only
include OpenStack Foundation Individual Members. The previous change
did this correctly for the full (Technical Committee) electorate but
failed to apply a similar conditional constraint to Project Team
Lead electoral rolls.

Change-Id: I03e90e5c512ee463cfcc49f147ff523806ae05c7
Story: #2001094
Task: #4877
2017-08-10 00:20:45 +00:00
Andreas Jaeger
983f097743 Remove tools/clean-fips.py
This was only used for OSIC which is decommissioned now, so we can
remove the script.

Change-Id: I85b70f77685b0563c508ea2c0fa1841a892b9072
2017-08-05 18:40:33 +02:00
Jeremy Stanley
9341118edd Use foundation member IDs in invitations list
Since the foundation sometimes needs to deduplicate subsequent
invitations for the same foundation member against invitations sent
by another system, use the member ID as the first field of the
invite instead of using the Gerrit username now that we have that
information available. Use a default member ID of 0 to indicate
non-member invitees who may require additional manual deduplication
or vetting. Also log the member ID in the output of the invite mass
mailer script.

Change-Id: I86d879a5f06144a0889eb852f2cf3d555a12a7aa
2017-08-03 20:20:30 +00:00
Jeremy Stanley
b2ad9b4f50 Include retired projects/deliverables in elections
If a deliverable or entire project is retired within the qualifying
period for an election, those contributions still count toward
defining the electorate. Merge them into the list of official
projects under those circumstances, but check for any possible
duplicates in case of partial deliverable retirement for a
multi-repo deliverable.

Change-Id: If626373cdbc6ca9ad7d016b2e18b4d82a4ceaa23
Depends-On: I7ca3c9294d325085ce9139f3a86b55e5b31c79c7
2017-08-03 20:19:19 +00:00
Jeremy Stanley
fb3e76576b Check change owners against OSF Individual Members
According to the Bylaws of the OpenStack Foundation Appendix 4
Technical Committee Member Policy §3.b[1] along with the OpenStack
Technical Committee Charter definitions for APC[2] and ATC[3], we
limit the voter rolls for technical elections to Foundation
Individual Members.

Use the OpenStack Foundation's member directory lookup API to record
the member ID of each change owner. If an owner does not have any
E-mail address in Gerrit which matches at least one E-mail address
of an OSF Individual Membership record, then exclude them from CIVS
ballots and event discount invitations.

[1] https://www.openstack.org/legal/technical-committee-member-policy/
[2] https://governance.openstack.org/tc/reference/charter.html#voters-for-ptl-seats-apc
[3] https://governance.openstack.org/tc/reference/charter.html#voters-for-tc-seats-atc

Change-Id: I920e1bc134d1283e2019f4389a5491e9b746cca0
Story: #2001094
Task: #4813
2017-08-02 23:39:45 +00:00
Jeremy Stanley
5b8df42959 Refactor requests usage in owners script
In an effort to be more consistent about encoding parameters for GET
requests made in the script which generates electoral rolls and some
event invite lists, centralize calls to the requests module. This
has the added benefit of encapsulating retry logic for potential
reuse in later additions to the script.

Also generalize the query_gerrit() function to flexibly handle both
Gerrit REST API which return neutered JSON and Gerrit Gitweb queries
for retrieving YAML with its own special character encoding
challenges.

Further extract out the JSON decoder error handling so it can be
reused for other APIs than Gerrit's in future feature additions.

Change-Id: Ibda65d41c17416eb28eb326e2cdd28c90153f108
2017-08-02 21:19:58 +00:00
Jesse Keating
1d41039356 Remove bluebox cloud
This cloud is being decommissioned.

Change-Id: I26b51425d21e3ee86eb36763e1dedfd65e78d2b7
2017-06-29 17:14:35 -07:00
Paul Belanger
dc0cf8acbb Fix puppet-apply tests
Puppet is now throwing an error about this for centos-7 unit tests.

Add empty SSH key for krotscheck.

We also need to make sure puppet has been properly installed, this is
because we have recently removed puppet from our diskimages.

Change-Id: I8bcaddd045be3f81de5c2481e0de6a8526c97a91
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-05-01 16:35:03 +00:00
Jenkins
13a530c9a4 Merge "Ignore release bot account in owners example" 2017-04-25 17:26:27 +00:00
Paul Belanger
e72ed2fe58
Create hosts file for ansible-playbook
Be explicity with our inventory file, to avoid ansible-playbook from
skipping hosts.

Change-Id: I29260bd97432a443e5989fa4a883e8e8d37307fd
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-04-18 13:40:51 -04:00
Colleen Murphy
2a20fbb52b Fix puppet-apply test for Debian Jessie
The version of puppet 3.x in Debian Jessie is 3.7.2, which seems to be
unable to read the modulepath setting from environment.conf. Since we
only have the one environment, the quick fix is to link the module into
the standard module path.

Change-Id: I6f4fa87fa4941c29abf37c277f5ce895a51945bc
2017-03-26 00:32:19 +01:00
Jeremy Stanley
327f444818 Add human name to sample of invites from address
Add a human name in the invites sample configuration's from address
value, since this will cause messages to appear with a friendlier
from name than just a raw E-mail address.

Change-Id: Idf16904bcd56692f3a82e41cb52201ed2223fdfd
2017-03-06 21:01:47 +00:00
Jeremy Stanley
182ef2d2e0 Eat the extra leading newline in invites
In the sample settings file, backslash-continue the string literal
opening to avoid introducing an extra leading newline in the
resulting message body.

Change-Id: Ief61f69acb5299ba324af8a897ea4f96fa47711d
2017-03-06 21:00:04 +00:00
Jeremy Stanley
6bfb87071e Use PYTHONIOENCODING=utf-8 for invites invocation
Since the send.py script for invites uses print to emit progress
including the names of contacts to which messages are sent,
redirecting stdout of the script to a local file can result in
encoding errors if some of the names include non-ASCII codepoints
(but not when stdout is your interactive terminal). Setting
PYTHONIOENCODING=utf-8 in the calling environment works around this
nicely.

Change-Id: I52b70c64f8b3a20da5143b31c5090192b0c75c99
2017-03-06 20:57:26 +00:00
Jeremy Stanley
97b50088dd Remove extra sample ATCs file
There only needs to be one sample ATCs file for the invites
mass-mailer, so remove the redundant one which isn't referenced in
the README file. This should help prevent people editing the wrong
sample and then performing a dry-run with the unmodified one.

Change-Id: I61e90ad085e89c466053858518db09a414a07792
2017-03-06 20:18:21 +00:00
Jeremy Stanley
53bb44da81 Ignore release bot account in owners example
The owners.py script is used to generate electoral rolls. It has
example usage comments which previously skipped reporting the
OpenStack Proposal Bot account used to push general changes into
code review, so add the OpenStack Release Bot account there as well
since it also pushes some release-oriented changes.

Change-Id: I98607b9adb8d2c6535beab2879e24097d32434e7
2017-02-28 17:17:16 +00:00
Tristan Cacqueray
31b9fc7ba3 Allow importing the owner script.
This change modifies the usage of the script so that it can be imported by
the election module. It also fixes maketrans support for python3.

Change-Id: Ie3b602237625e11651825abc8feb38caa73e9632
2017-01-30 01:59:28 +00:00
Jenkins
6969569a38 Merge "Fix logstash-filters test" 2017-01-19 19:11:05 +00:00
Paul Belanger
cac41d1314 Fix typo in nodepool-builder.logging.conf
Sadly, I broke things.

Change-Id: I7f1aa09d0a9bfddea710f571203a26c0f943f4df
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2016-11-29 10:18:09 -05:00
Paul Belanger
f46e0e4c01 Add kazoo logger for nodepool-builder
Kazoo client is very verbose for debug output, default to INFO.

Change-Id: Iff2682d98738e2df2acda11ac2dbcb2706262256
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2016-11-28 22:08:53 -05:00
Jenkins
f724dab4a3 Merge "Check number of meetbot channels" 2016-11-23 08:22:21 +00:00
Ian Wienand
82773ab56c Add image upload filters for nodepool
This adds loggers and handlers for each provider in the nodepool
config file for separate image upload logs on the nodepool host.

This doesn't regenerate the file yet.  It won't do anything until the
dependent change to nodepool starts generating upload logs with logger
names that match these.

Change-Id: Ic80e532891f039e0f835847ddfbae38f1ce3cd8a
2016-11-15 03:55:49 +00:00
Andreas Jaeger
4139c901d8 Check number of meetbot channels
Check that we do not have more than 120 channels setup for meetbot.

Change-Id: Ib5b948c68691ec959d4826a2cdf28f3e50631e60
2016-11-02 20:45:06 +01:00
K Jonathan Harker
74a92c4645 Fix logstash-filters test
In order to run the script directly in the test, the script needs to
have the executable bit set. Currently, the test is failing with a
"Permission denied" error when attempting to run the script.

We also need to run install_modules.sh in the logstash-filters test, so
move that invocation to the shared prep-apply.sh script.

The vcsrepo puppet define doesn't understand revisions of the form
"refs/zuul/foo", so instead parse the HEAD sha and pass that to vcsrepo.

Finally, we need to ensure that $elasticsearch_nodes is populated with
an array, otherwise the function @elasticsearch_nodes.map in the
output.conf template will fail.

Change-Id: Ic1ab6d04232abb3cf347f0d27afbf9172f62b0e4
2016-10-12 09:43:26 -07:00