172 Commits

Author SHA1 Message Date
Jeremy Stanley
d12740116f Be more careful checking the change submitted date
While rare, it _is_ possible for changes to be in a merged state in
Gerrit while lacking a submitted datestamp. For example,
https://review.openstack.org/274113 was merged in a strange way that
left it with no record of merge submission. We had fallback error
handling in owners.py to report this condition, but neglected an
exception path in the previous implementation which would cause the
script to bail early rather than falling through to that error
handler. Now it should use the submitted date if one is present, and
report an explicit warning (including change and owner IDs) for any
where it is not.

Change-Id: I1a40959fdd14dfef720c32e3c87ff5bbde7432df
2016-09-12 23:05:49 +00:00
Jenkins
71016bb569 Merge "Add a script to list change owner statistics" 2016-08-30 16:02:33 +00:00
Jeremy Stanley
791769fbba Add a script to list change owner statistics
With the recent upgrade to Gerrit 2.11, changes in its database
schema require updates to our "ATC" generator tools. However, the
REST API in 2.11 is complete enough that it can provide all the
required data, so replace the old MySQL query and SSH API based
tooling which required database access with something simpler to
run.

This new implementation retains the data model of its predecessor
(in other words, it tracks "owners" of merged changes in Gerrit as a
best-effort proxy for identifying contributors who are also
foundation individual members). The new script should be extensible
enough that we can further change this model at a later time, with
consensus from the technical committee. Further, unlike or earlier
method, anyone with a Gerrit account can now generate technical
election rolls, contact lists for Summit registration discount code
mail merges, and so on.

Change-Id: I6df798fc9e7fc4ff59972aff39c2e37f8c28d776
Signed-off-by: Jeremy Stanley <fungi@yuggoth.org>
2016-08-16 02:53:41 +00:00
Jenkins
e929803e23 Merge "Add support for logstash-filter integration tests" 2016-08-10 22:17:42 +00:00
Adrien Cunin
a62a7ec2ae Handle non-ASCII characters in body and subject
Change-Id: I522482c9d8cca1493206988694c8d497bc9d32d5
2016-08-02 16:12:32 +02:00
Mikhail S Medvedev
0e9a9b9f6e Fix shebang for modules.env sort check script
It is obviously missing "python" which makes it not possible to run on
its own. It works with tox because tox.ini explicitly invokes it with
"python {toxinidir}/tools/sorted_modules_env.py".

Change-Id: I6eab942660d69eacda95572c7d2077687087b7c7
2016-07-21 11:52:40 -05:00
Jenkins
74d6d75251 Merge "Add simple script to help debug shade image uploads" 2016-07-19 00:44:47 +00:00
James E. Blair
b7550ab637 Commit transaction in delete-gerrit-spam
Also, print an indication when the DB is modified.

And get rid of a silly loop.

Change-Id: Ib0b25d51f389413ab255fd2332fcf211aeabec78
2016-07-13 10:51:48 -07:00
James E. Blair
852bc14bce Add a script to delete Gerrit spam comments
Change-Id: I1e79a0a50a2166660b601dfe2d886f00d7469c75
2016-07-13 09:37:26 -07:00
Jenkins
e06b365bb9 Merge "Filter nonexistent groups in who-approves.py" 2016-07-01 13:43:38 +00:00
James E. Blair
21ebdae7ff Remove jenkins playbooks and tools
Change-Id: Ida59078c8b4d8af5057fddbbfde7e3249b48839b
2016-06-15 15:00:40 -07:00
K Jonathan Harker
166b00c17f Add support for logstash-filter integration tests
Add a script for integration testing with
openstack-infra/logstash-filters and refactor the apply-test script to
leverage much of the prep work done there.

Also add some needed parameters to install arbitrary versions of the
logstash-filters repo.

Change-Id: Ifcc7fb13c4fb21986e33327487c739966e4bb107
2016-05-25 14:06:15 -07:00
Monty Taylor
cada3ba846 Add FIP cleaning script
While it's hopefully not useful for much longer, this is the script that
safely cleans FIPs from nodepool clouds that use neutron FIPs.

Change-Id: I6f6f2c5f119a40060f46ea1f244a65601f8b7bb9
2016-05-06 19:43:35 -07:00
Jeremy Stanley
31eab4a00d Filter nonexistent groups in who-approves.py
More recent versions of Gerrit (somewhere between the 2.8 we were
running and the 2.11 to which we upgraded) switched some common
groups to be virtual routines, for example the Registered Users
group, so they no longer appear in the groups list. However we
should also not be counting members of these virtual groups as
approvers anyway (even though in the case of the sandbox repo they
are), so filter them out of the assembly loop in who-approves.py to
prevent it from breaking when it encounters them in an ACL.

Change-Id: Ibb73b68e6338cd2fe2e18c09993d6e02b03c312a
2016-04-12 18:21:05 +00:00
Monty Taylor
ee2eebc23d
Add simple script to help debug shade image uploads
A useful step when debugging image uploads is do to it directly using
shade. This is a script to facilitate that.

Change-Id: I8ae44cb5bf1eb90180afee1dac70fbbc4be4a639
2016-04-11 09:03:21 -05:00
Jenkins
ab4953d9f8 Merge "Cleanup of nodepool builder logging" 2016-03-17 01:07:11 +00:00
Paul Belanger
ae2b68c4b7 Run puppet-apply on debian jessie jenkins slaves
Change-Id: I4e876f997324bdb43732bec26067d88531b587d5
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2016-03-16 20:15:18 -04:00
Ian Wienand
968f09bfa9 Cleanup of nodepool builder logging
We can use textwrap to break up the ridiculously long lines in the
logging configuration (with less images, it's less ridiculous than it
used to be, but it's still easier to parse).  Only trick is to put
spaces between the arguments to give it somewhere to split (and better
for readability) and stop it splitting on words/hyphens.

Change-Id: I04840e98b0564dc531b74b96e8595f9d18b284af
2016-03-17 09:37:23 +11:00
Jenkins
1818e513c6 Merge "Add a simple LP OpenId reverse lookup utility" 2016-03-11 11:03:50 +00:00
Monty Taylor
e8e201cc75 Add a playbook and script for doing adhoc puppet
Running puppet remotely in an ad-hoc manner on disabled hosts is mildly
complex. To facilitate, have a wide open playbook that we always run
with --limit - and a shell script to help us type less.

Change-Id: I629072dcada38d0465d351b1b99828466405372f
2016-03-09 14:23:49 -06:00
Jeremy Stanley
933bd0406c Add a simple LP OpenId reverse lookup utility
Change-Id: Ibe1705bc53a48aa6c23f095ae9118d2d367aea78
2016-02-29 21:09:51 +00:00
Jenkins
478eaa8053 Merge "Add Date and Message-ID to summit invites" 2016-02-26 11:09:11 +00:00
Spencer Krum
16e56fce9d Script to generate dns for infracloud
This script makes management of 100+ dns names more realistic
It only outputs commands to rackdns, it does not run them

Change-Id: I48ecaca79eb1cd274f102da6eb3592a292b7a5fb
2016-02-25 11:26:48 +11:00
Jenkins
70e94d8606 Merge "Add helper script to delete Gerrit groups" 2016-02-12 21:24:02 +00:00
Clark Boylan
8ed863ad9d Provide separate nodepool builder log config
Now that the nodepool builder is running as a separate daemon it needs
its own log config file. Move the auto generated nodepool logging config
stuff over to the new builder logging config as we can manage the main
daemon's logging config by hand trivially now.

Depends-On: I013835621dfbc311a0f7bd7c957b7d4656dfa628
Change-Id: Ic1da30eab949876e5bd6c88e83979bdedc6dd50a
2016-01-29 10:20:40 -08:00
Khai Do
61dfce7390 Add helper script to delete Gerrit groups
Gerrit is missing a feature to remove Gerrit groups. This is a
helper script to remove the group from the database.

Reference:
  https://code.google.com/p/gerrit/issues/detail?id=44

Change-Id: I1cb6cf852622fa5697c64666b132ac2205120595
2016-01-21 16:06:57 -08:00
Jeremy Stanley
192414e957 Add Date and Message-ID to summit invites
The mail merger for summit invites includes Date and Message-ID
headers, because Crocker said so in 1977 that's why.

Change-Id: I2105fe4cd04b7b2042c9e32888cb1f094697d7bf
2016-01-14 00:07:47 +00:00
Jenkins
37e10bd3de Merge "Add script to downgrade gerrit from 2.10" 2016-01-07 16:17:26 +00:00
Spencer Krum
691897db6f Use In-Tree Public Hiera data
Also make the apply-test public-hiera aware

Change-Id: I761bf49ed6279a492cabeca878ec5e7c0fac3d0e
2015-12-30 13:31:17 -08:00
Jenkins
7b2787cac1 Merge "Add script to cleanup Gerrit 2.8 data" 2015-12-11 20:12:53 +00:00
Jenkins
6133978f41 Merge "Add rollback script for 2.11" 2015-12-11 17:55:32 +00:00
Khai Do
6ff434b77a Add script to cleanup Gerrit 2.8 data
This script is very specific to the current state of our
Gerrit 2.8 DB and repository.  I'm not sure that it's useful
for anyone else or in any other situation.

Change-Id: Ifb8303b2a2841e2de1ee9f357db4b8341122655d
2015-12-11 09:34:53 -08:00
Clark Boylan
ff17732828 Remove centos6 support
This removes centos6 support from install_puppet.sh because centos6 is
no longer supported so this code is dead.

Change-Id: If59f10a6a9c576b1299b0e49a2e82d2a1a1d7ecf
2015-12-04 13:05:10 -08:00
Monty Taylor
1bbd21a3d5 Add rollback script for 2.11
In case we need to rollback from 2.11 to 2.8, this is a stab at
fixing things. It doesn't seem like there are as many relevant
data transformations from 2.10 to 2.11.

Not including table drops, because having extra tables is not
destructive.

Change-Id: I96a638eeeb832fcc697cde7d94d981b5cbca6234
2015-11-30 15:42:29 -08:00
James E. Blair
5e6c18ef69 Add shade logging to nodepool
At warning level.

Change-Id: Iac65988477a0c0c80e0b2f3c8829e09c8e4e50f9
2015-11-19 18:14:04 -08:00
Jenkins
f739370e92 Merge "Re-home migrate-to-phab.sql" 2015-11-13 06:57:53 +00:00
Ramy Asselin
14fd503aaa Migrate to puppet-openstackci openstackci::nodepool
Manage nodepool configurations using the common-ci solution
in puppet-openstackci

Remove nodepool.yaml.erb from this repo as openstackci::nodepool
will pull it in from project-config/nodepool/nodepool.yaml

Remove the tox nodepool environment and test dependency as it
has been migrated to project-config

The nodepool logging template file and associated tool that generates the file
will remain in this repo. In the short term, updates to nodepool.yaml in
project-config repo may require a related change in this repo to update the logging
configuration. In the longer term, nodepool will be updated to automatically
log image creations without needing a customized logging configuration.

Depends-on: I89207d100eb4b6bbb502a6ed38831f49e4b29096
Depends-on: I473a1b78acdb035eb379394a7ed5f771434dc942
Depends-on: I6b01ab7260a41927fff34b9b81b631ea2c933f22
Change-Id: I2b45a7145805368b1598d3a3e8a94f0e4eb8cf2d
2015-11-10 15:13:50 -08:00
Spencer Krum
8cdaf92609 Treat openstack_health normally
This is a partial revert of 05c16e2e that removes special casing around
the openstack-health/openstack_health repo.

Change-Id: I832f1b764a34c128d737a467fa568cc064226c5a
2015-11-06 10:00:14 -08:00
Craige McWhirter
8df9be2332 Re-home migrate-to-phab.sql
This used to live in puppet-phabricator but as it has nothing to
do with the puppet module, it is being moved here.

Change-Id: I17d7cbcd3f7bb0147305bcd1d2976dd7c409fec8
Implements: spec maniphest migration
2015-11-01 07:41:30 +10:00
Monty Taylor
5e64f03f52 Add script to downgrade gerrit from 2.10
A while back we had an emergency downgrade of gerrit and developed
this script to do it.

Change-Id: I95c121b5c7c6f3f8946919e712c09af47762b11a
2015-10-29 10:05:43 +09:00
Jenkins
ddcb60ac19 Merge "Remove common.yaml default for hieredit" 2015-10-28 07:49:16 +00:00
James E. Blair
6a8e2e8881 Remove common.yaml default for hieredit
We have split the hiera file, so there is no sensible default for
this command now.

Change-Id: I9befdff8aabb8c8e84653dbd57b1b5d339623ddc
2015-10-28 15:27:08 +10:00
Monty Taylor
b381b6e9bd Dump hiera values properly in remap
These changes were made live before the file split.

Change-Id: If7ac51fa2ffda5190ff2e04a3dc18161c3d30633
2015-10-28 12:31:11 +09:00
Jenkins
36ac4f52ed Merge "Make hiera default values optional" 2015-10-23 22:35:56 +00:00
Matthew Treinish
05c16e2e35
Add openstack-health api server
This commit adds the puppet config for running an openstack-health
api instance. It'll use the subunit2sql trove db node as a data
source. The puppet-openstack-health module is incorrectly named which
is causing issues, a workaround to rename the repo is added to
install_modules.sh. This is a temporary measure until the gerrit
rename is completed. There is also a workaround in the apply test.

Change-Id: I7e6d9664d087e7bdc21d92624991d0d5f86c0c99
2015-10-19 15:27:00 -04:00
K Jonathan Harker
cd00179976 Make hiera default values optional
Currently, all hiera calls need to have default values in order for the
apply test to work. This is done by using sed to replace the hiera call
with the default value in the call, i.e. 'hiera("foo", "bar")' is
replaced with '"bar"'.

This adds another sed command that replaces a hiera call that has no
default value specified with the name of the key appended with
NoDefault, i.e. 'hiera("foo")' will become '"fooNoDefault"'.

By making the hiera default values optional, we no longer need to pass
in a default value to every hiera call, and so we can remove the
placeholder XXX that were previously needed by the apply tests but not
by puppet itself.

Change-Id: Ibdeb0c2e49245ea05116e8043936d5b012f46666
2015-10-16 14:03:35 -07:00
Jenkins
bd44a946cc Merge "Turn image logging level up to DEBUG" 2015-10-13 21:05:34 +00:00
Jenkins
5a7c74c6d4 Merge "Test that modules.env is complete" 2015-10-07 07:10:31 +00:00
Ian Wienand
4de87db276 Turn image logging level up to DEBUG
In I54db78b028017500b58693ec717c23eaf6a220b6 I added some handy values
that are intendend to go into public log files to help you establish
when an image-build has started or stopped (for example, [1] is
currently ~300MiB of looping build log data and it's really hard to
pick out where things are going wrong).

The problem is, I put them in at DEBUG level and the image loggers are
only set to INFO level.  So the messages aren't making it into the
public logfiles as hoped, and thus not available to help people
diagnose issues with their builds.

I think the best idea is to turn these up to DEBUG level to capture
these, and any other low-level messages from this logger, in the
public output.

[1] http://nodepool.openstack.org/image.log

Change-Id: Ib9ac5fc81ccef73f1b1e20b8381bdd1afac4bcae
2015-10-01 13:14:45 +10:00
Colleen Murphy
324e5c4f57 Make apply test more configurable
The Infra Cloud project uses a different set of puppet modules than the
standard infra ones in order to avoid possible dependency conflicts[1].
Currently the puppet-apply test hard-codes the entry-point puppet
manifest and module list. This patch makes the manifest, currently
manifests/site.pp, and the module file, currently modules.env,
configurable, so that we can add a new job to test just these nodes
with the correct modules installed.

[1] https://review.openstack.org/#/c/209617/

Change-Id: I744123befef0bf27a1eb2a79f5d814e0cefe81f5
2015-09-24 15:52:03 -07:00