157 Commits

Author SHA1 Message Date
K Jonathan Harker
948aff1f02 Clean up bashate failures
Fix all of the bashate failures.  Also, bashate will install its negative
tests in its virtualenv, so we need to generate a list of files to pass to
bashate that excludes .tox/

Change-Id: I55559bf6137f705aec9a7f277928ad8d4cadc2ca
2014-09-30 12:40:59 -07:00
Steve Martinelli
8cf6e680f4 Remove projects_alphabetized from tox.ini
Since the split to project-config, there is no longer a need to
run 'tox' for projects_alphabetized, since those resources no
longer exist. Also removing the shell scripts that were running
the tests.

Change-Id: Ib7a762d872da9c5d945206807c405e7452b49638
2014-09-30 15:15:10 -04:00
Jeremy Stanley
58829de5bc Allow a git ref to pin the governance repo
Change-Id: I230c22fb83447b336eda14a7f10cf33e52e3d321
2014-09-26 16:28:53 +00:00
Jeremy Stanley
dfb806c896 Ignore extra-atcs lines which look like comments
Change-Id: I8054f4582ad93a5dd31a2d5de288250780ebbf3f
2014-09-26 16:27:53 +00:00
Ian Wienand
a88fed3c9e Remove d-i-b tools
These are ported into the new repo with change
I087a2396581bf4415c9676f70b4c739a62c84b23

Change-Id: I9378cef3555f29857a6869a273b3904c636bb7b0
2014-09-26 13:37:27 +10:00
Jenkins
e30f37376d Merge "Removes content split into project-config" 2014-09-26 00:45:16 +00:00
Monty Taylor
8387e739fe Add project-config to zuul cloner
Make sure that the puppet apply test gets the project-config repo.

Change-Id: I3a772e84a1bc6028caeabcc8e09a0942f44b8612
2014-09-25 15:58:51 -07:00
Anita Kuno
80a80a6a6b Removes content split into project-config
Content from the config repo was split into a new repo,
project-config. This patch deletes the moved content from the
config repo.

Change-Id: Ifa1ae2cb7c23ae48f388b7d57a4c0d0799ef1dc2
Story: 167
2014-09-25 14:12:51 -04:00
Jenkins
95080d8703 Merge "Add support for disk-image-builder in nodepool" 2014-09-19 21:35:55 +00:00
Monty Taylor
19e278dfb7 Add support for disk-image-builder in nodepool
In order to use disk-image-builder to build nodepool images, we need
a few things. First, we need additional debs to be installed for
disk-image-builder to work. Then we need the image elements themselves
to describe the images we want to create. Also, a few helper scripts
for locally working with the disk images have been included.

Tested with build-image.sh script; devstack-gate-trusty.qcow2 created
correctly

Change-Id: I539743147341dc5b387d103fecd3e9ff0f01fdfc
2014-09-17 12:25:01 -07:00
James E. Blair
a963d32efe Move puppetmaster_boostrap to tools dir
We don't really need both tools and utils directories.

Change-Id: I25509b436b721590225736b54cd0fbebbe2a6b14
2014-09-15 16:50:07 -07:00
James E. Blair
542e170b81 Keep environment for cloner in apply test sudo
The zuul cloner needs the ZUUL_* env vars when setting up projects
for the new apply integration test.

Also, add some more debugging to both versions of the test.

Change-Id: I486b19cecd034250068e4a59604e1e0ca90ed83d
2014-09-15 13:19:37 -07:00
James E. Blair
9d23185ac5 Add integration apply test
Make module installation conditional in install_modules.sh so that
we can specify that we _do not_ want it to install the modules
that will be participating in the integration test.

Add a variant of the apply test script (currently test.sh) that
uses zuul-cloner to check out the appropriate versions of modules
that are participating in the integration test.

Add jobs that run this to the experimental pipeline.

Change-Id: I16055cdd45f6e05aa8ebceb500c5709f265268df
2014-09-03 15:03:09 -07:00
Monty Taylor
764ee8c74f Delete select-mirror and all the args
It turns out that we pass a lot of args around simply for the
purpose of doing mirror selection. This goes away if everything
gets a mirror.

This change should only be merged after the parent change has
propagated to all puppeted slaves and updated nodepool images.

Co-Authored-By: Jeremy Stanley <fungi@yuggoth.org>
Change-Id: I4a721f6559d7160c8213268a8c1e35c667a67a4f
2014-08-06 02:04:43 +00:00
Jenkins
dcde93b7f1 Merge "Modernize ATCs script to Gerrit 2.8, programs.yaml" 2014-07-01 14:38:23 +00:00
Jenkins
db39a17c49 Merge "Make ATC lists sort E-mails with preferred first" 2014-07-01 14:36:24 +00:00
Jeremy Stanley
9b83466230 Combine infra/ci core/admin gerrit groups
This is the result of running:

    find modules/openstack_project/files/gerrit/acls/ -type f \
        -name "*.config" -exec ./tools/normalize_acl.py {} 6 \;

Change-Id: I7aa27b859529b2bc8a990d6272334222996cbbc4
2014-06-30 17:30:45 +00:00
Jeremy Stanley
3b70e55683 Add a script to normalize Gerrit ACLs
* tools/normalize_acl.py: Script which can perform one or more of a
list of normalizing transformations to an ACL file.

Change-Id: I063ab91b6e786eccaee61c669f0e840c7af6be14
2014-06-30 17:22:40 +00:00
Jeremy Stanley
19c0ee2d3c Modernize ATCs script to Gerrit 2.8, programs.yaml
* tools/atc/README: Simplify instructions to use mysqlclient
tab-separated-values stdout, and remove obsolete manual steps which
have been automated.

* tools/atc/email-stats.sh: The functionality this shell script
previously provided has now been translated to Python and folded
into the inner script.

* tools/atc/email_stats.py: Implement the old email-stats.sh outer
loop, make it possible to run in a virtualenv, and make previously
hard-coded constants into command-line parameters with sane and
dynamically-determined defaults. Update field order and contents to
cope with differences in Gerrit 2.8, use a proper YAML parser on the
programs.yaml which has also changed format since the previous run,
and adjust for the improved extra-atcs file format as well.

Change-Id: I2deef2e6766ff998190e66737e7cec9cdcd5459c
2014-06-26 02:35:29 +00:00
Jeremy Stanley
69e247d873 Make ATC lists sort E-mails with preferred first
* tools/atc/email-stats.py: Adjust the order in which Gerrit E-mail
addresses are returned so that the one listed as the preferred
address is sorted to the beginning of the set.

Change-Id: I95b5817ada58f4059df4f5c0b406183f1c1e5b74
2014-06-24 20:37:40 +00:00
Jeremy Stanley
876ba7a23b ATC listing script is importable
* tools/atc/email_stats.py: Move procedural code out of the global
scope to make this file safely importable, and clean up a couple of
unused variables flake8 noticed in the newly importable main
codeblock.

Change-Id: I1a29c84e067eea875ba57a16196723f65f60ff01
2014-06-24 19:38:10 +00:00
Jeremy Stanley
1c608c8143 Rename ATC listing script
Rename tools/atc/email-stats.py to tools/atc/email_stats.py in
preparation for making it an importable module. This is being
separated from the subsequent change to make it more reviewable.

Change-Id: I1907de0715c96f54edb44910746fbb24a5ab8665
2014-06-24 19:38:03 +00:00
Doug Hellmann
08de5694b6 Make it possible to run zuul layout test locally
Add a tox environment to run the zuul layout tests locally.

Modify run-layout.sh to ignore a missing mirror selection script
and assume that it is running with a valid mirror configuration.
This allows the script to be run locally without breaking a
dev environment.

Change-Id: I5b0c94ae51e7f41b8b26b3ef4f78b830031d263e
2014-05-28 12:52:43 -07:00
Jenkins
cc7b0418fc Merge "Add tool to parse puppet reports" 2014-05-08 00:11:25 +00:00
Doug Hellmann
3fa3be7ede Add check job for upstream URLs on imported repos
Add a check job to require upstream URLs to use a scheme
that we are confident won't cause connection errors during
the import.

Add a tox configuration for this new job, as well as the
existing alphabetical check script, to make it easier
for developers to run those tests locally just by running
"tox".

Change-Id: I894ac0f8afb234404bb6e80e64b90aa25e052f10
2014-05-06 12:03:16 -07:00
Monty Taylor
751d49446e Add tool to parse puppet reports
puppetboard does not always tell us what we need to know. This tool
is the beginning of trying to figure that out.

Change-Id: I495986e40cdb2c062da9591a29feef5288f1bc57
2014-05-05 15:16:43 -07:00
K Jonathan Harker
bfd19bdfff Check for new files as well as changed files
Currently, the run-compare-xml.sh script will only report changed or
removed files, it does not report new files being created. This change
replaces diff in a for loop with a recursive diff that treats missing
files as empty files. The result is that the output is presented as a
single unified diff that includes new files as well as changed and
removed files.

Change-Id: I0bb5a1ee93e689666ca5080ec5ccbe036c73d682
2014-04-28 16:08:32 -07:00
Jeremy Stanley
9822dd517e Retrieve ATC project list from governance repo
* tools/atc/email-stats.sh: We now have an official list of projects
in the openstack/governance project, so use it instead of the old
hard-coded list.

Change-Id: If93c71174687ac7e4e46d514642c48edb841adfe
2014-02-28 16:50:29 +00:00
Jenkins
7e63b56768 Merge "Modernize ATC list format" 2014-02-24 17:46:09 +00:00
Jeremy Stanley
d654b45b60 Modernize ATC list format
* tools/atc/README: When generating the initial data dump, MySQL on
review.openstack.org creates output files with DOS line endings and
ISO-8859-1 encoding. Recommend using fromdos and iconv to normalize
them.

Change-Id: Ie16a56c024b0eee97b0e6bd5abe8ed3b0e0a9f7e
2014-02-13 16:24:22 +00:00
Sergey Lukjanov
36c5828df4 Add storyboard-webclient to ATC lists generator
Storyboard is already added, so, let's add storyboard-webclient folks too.

Change-Id: Ic5ec6de4c393f42d215e71920022223cd5d42a5b
2014-02-11 15:52:21 +04:00
Monty Taylor
bc794da31e Split config from projects list
The projects list is a common list for devs to interact with. The config
in the list is not, but the config in the list means the file needs to
be in an erb template.

Split the two concerns, similar to zuul. Put the config in a config file
and the project data in a yaml file.

Change-Id: I708b8655b4b1ce377f3b7369e987418c1d72d977
2014-01-28 09:32:09 -08:00
Jenkins
9e28b0608a Merge "Move tuskar-ui to horizon program." 2014-01-28 04:12:34 +00:00
Monty Taylor
da1465db09 Make the alphabetized check self-gating
When putting the alphabetize script into slave_scripts, we run the risk
of bugs entering the script without being tested before they are rolled
live and become part of the gate. This just recently happened. Instead,
move it to the tools dir, which makes it easy for a dev to run, and also
which makes changes to the script self-gating.

Change-Id: Ia5e1870f84b0cbc0c6f172b6982263440af5575e
2014-01-26 09:32:44 -08:00
Robert Collins
2ac82bed8b Move tuskar-ui to horizon program.
Where it is moving :).

Change-Id: I403834570f646cc78f879c9ca0d9d9bbe470fa3c
2014-01-22 10:01:31 +13:00
Jenkins
48ed1f98d5 Merge "Fix pep8 errors in modules dir" 2013-12-02 16:55:12 +00:00
Jeremy Stanley
2f8a91ef22 Stop running stable/folsom jobs
The OpenStack Havana release heralds end of support for Folsom, so
stop testing it.

* modules/jenkins/files/slave_scripts/docbook-properties.sh: Remove
folsom-specific settings.

* .../openstack_project/files/gerrit/acls/openstack/ceilometer.config:
Get rid of the stable/folsom ACL override.

* .../jenkins_job_builder/config/devstack-gate.yaml
* .../manuals.yaml
* .../projects.yaml
* .../python-bitrot-jobs.yaml
* modules/openstack_project/files/zuul/layout-dev.yaml
* .../layout.yaml: Remove all folsom-only jobs and clean up
configuration and comments associated with them.

* tools/invite2summit/settings.py.sample: Adjust usage comment to no
longer refer to Folsom.

Change-Id: Ieb38a74e6945abe2e40cd3e1cd23b52fbd826933
2013-11-13 20:35:34 +00:00
Monty Taylor
38f8d43ffc Fix pep8 errors in modules dir
There weren't that many of them - just fix them.

Change-Id: I6aa6d056746c4a6804146b89c12e6313f9d22d9a
2013-11-12 11:30:34 -05:00
Stefano Maffulli
80875944b7 Adds the script used to send summit invitations to ATC
Change-Id: If67bdf6099067e96dc168027db6e8f5ee7e4b4cb
Closes-Bug: #1243930
2013-10-23 15:57:48 -07:00
Jeremy Stanley
455f05b0ec Update official list of ATC-qualifying projects
* tools/atc/email-stats.sh: Based on a review of current
repositories and subsequent discussion, this catches up the list of
projects used to determine ATC status to reflect the present state
of OpenStack's development. Also, confirmed non-code contributors
will now automatically be added to the output.

Change-Id: I509f4991a664f45cc6c3428e3d8d405668ebc609
2013-10-12 19:20:02 +00:00
James E. Blair
96f2f6ddd9 Initial checkin of ATC scripts
Change-Id: Ibffab01bc835d380ca9f0c18521c8a6d808de761
2013-09-24 09:34:10 -07:00
Monty Taylor
eef4f76011 Clean up flake8 issues before switch
In anticipation of swtiching to flake8 as part of the removal of
pyflakes jobs, clean up a few flake8 errors so that we don't have to
ignore as many things.

Change-Id: Idb8466bb3194904d84e17dd3f2a7c32aff13788c
2013-09-20 14:37:30 -07:00
James E. Blair
6611b0dc9e Validate that zuul-referenced jobs exist
Create a list of jobs defined by the current JJB configuration
and use it to validate that every job referenced in the Zuul
layout.yaml exists.

Also, fix all of the errors currently identified by this.

Requires Iceb74440cb004e9ebe6fc08a4eedf7715de2d485 in Zuul.

Change-Id: Iae8f612836a269520ce7448b9a656c8df183e126
2013-07-30 16:02:39 -07:00
James E. Blair
3fb03c7328 Add hieraedit.py
A handy script to edit hieradata's common.yaml.

Change-Id: Ib7aa9ebd89cdcd95bd36690a32bb7f7736aafca5
2013-07-20 09:48:37 -07:00
James E. Blair
8c6df9f19c Select a pypi mirror before running tox.
Change-Id: I62a86b6a0ad30ff8cb387ddcc05668128248c6d0
Reviewed-on: https://review.openstack.org/23513
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
2013-03-18 20:29:07 +00:00
James E. Blair
f2016a6615 Remove rfc.sh.
Change-Id: I52d02a0b116432b0996f08de86ce914d7d06832c
Reviewed-on: https://review.openstack.org/23782
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Approved: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
2013-03-07 20:36:05 +00:00
James E. Blair
8eea39dad6 Add a script to test the Jenkins API.
Using the extended versions of python-jenkins that are used
by Zuul and devstack-gate, exercise all of the Jenkins API
calls that are used by by Zuul, devstack-gate, and Jenkins
Job Builder.

(JJB doesn't need any extended methods.  The extended methods
used by devstack-gate have been upstreamed, but not released.)

Change-Id: I9c3a4d7cfc96965ceb83b8c5ea4b12d4f2e07e12
Reviewed-on: https://review.openstack.org/22580
Reviewed-by: Paul Belanger <paul.belanger@polybeacon.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Khai Do <zaro0508@gmail.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-02-21 22:34:09 +00:00
James E. Blair
67aeba3668 Add zuul layout validation gate job.
Non-voting for now to make sure it works.

Change-Id: Iac29eba2320c8a972a03df81d10feb0183d98fd7
Reviewed-on: https://review.openstack.org/19719
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
2013-01-16 17:41:56 +00:00
Monty Taylor
74460853d1 Rename CI to Infra in MANY MANY places.
Change-Id: I409bd50ae374e0288531f07cfeea34856c5f8067
Reviewed-on: https://review.openstack.org/17319
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: James E. Blair <corvus@inaugust.com>
2012-12-16 17:30:47 +00:00
James E. Blair
f355bc2417 Add script to run jenkins job xml comparison.
Change-Id: Ied3addbe4e43a08ea2ac51e0f9b5560eb46823f6
Reviewed-on: https://review.openstack.org/12453
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
2012-09-05 20:51:28 +00:00