131 Commits

Author SHA1 Message Date
Andreas Jaeger
2a6bf90fbc Block stable/feature branch for new project imports
When importing projects, check that they do not contain a simple
"stable" branch as that will block creation of "stable/RELEASE" branches
like "stable/mitaka".
Similar for "feature".

Change-Id: Ia190fdbddf9029149b54e686aa0e82fe040b6733
2016-04-21 17:00:39 +02:00
Jenkins
eedeb8d0d0 Merge "Revert "Add script to facilitate project renaming in project-config repo"" 2016-04-20 08:38:46 +00:00
K Jonathan Harker
ced395e7ce Add a few more layout checks
Create three new checks for zuul/layout.yaml:
* Ensure every project has a check job
* Ensure every project has a gate job
* Ensure that no check or gate pipeline mixes noop jobs with real jobs

vinz-webclient did not have any gate jobs defined, so add noop to its
gate pipeline.

Change-Id: I5935cfbb0e99270af22ca144382b35f6a1b08fee
2016-04-15 09:21:36 -07:00
Andreas Jaeger
6ef07db860 Add checks for gerrit ACLs
Check the following common problems with gerrit ACL lines:
* Wrong path
* Non-existence of a referenced file
* Redundant setting of acl-config with default value

Remove redundant entries that the script found.
Add acl-config for freezer-specs since no ACL existed.
Add acl-config for vinz-webclient since no ACL existed,
this was in original change I1706332f911ad9c8887d1e40613b06cbeb5e1143.

Change-Id: I6b9fc6d2ff1cd5a1f871975608947d6fe763b500
2016-04-11 16:52:26 +02:00
Jim Rollenhagen
26641b6561 Update build-image.sh for new images, add EXTRA_ELEMENTS
This changes the base element for build-image.sh to ubuntu-minimal, and
adds the simple-init element, to make this match the images used in the
gate.

It also adds an EXTRA_ELEMENTS environment variable that users can use
to add elements to the image. For example, local-config is useful to put
the current user's ssh keys in the image.

Last, it documents the things I needed to install to get it to run
correctly on Ubuntu; this is likely not an exhaustive list.

Change-Id: I2506c6722b77bb2f861d08e7a6da453dc2bbbfc2
2016-03-28 16:19:59 -07:00
Andreas Jaeger
d91225a30a Revert "Add script to facilitate project renaming in project-config repo"
The mass rename is done, we do not need these anymore in our repository, remove it again.

This reverts commit 246bbfe03bb5a04ee904e9d09f199dab615fd244.

Change-Id: I5caeabdde4988a987d3fd828605bf5cb73b1aefa
2016-03-28 21:56:24 +02:00
Andreas Jaeger
3498848692 Report number of changed jobs
Check how many jobs have been changed and report that number.
Do not abort if the diff or grep fails to find anything.

Change-Id: I7f655f65cbe4d5cd084af2bda645ba57697ba3be
2016-03-11 18:56:25 +01:00
Jenkins
d003569acd Merge "Unshadow All-Projects in exclusiveGroupPermissions" 2016-02-19 15:10:05 +00:00
Jenkins
a32acadfdc Merge "Keep Gerrit ACL lines deduplicated" 2016-02-18 23:12:03 +00:00
Jeremy Stanley
f62e441f24 Unshadow All-Projects in exclusiveGroupPermissions
Whenever a project-specific ACL declares exclusiveGroupPermissions
on some permission, it can block other valid uses of that permission
which would otherwise be inherited from the All-Projects pseudoACL.
Make sure that Project Bootstrappers retains access to abandon,
-2..+2 on label-Code-Review and -1..+1 on label-Workflow. Also make
sure Change Owners can still abandon and add -1..0 on
label-Workflow, and that Registered Users can always -1..+1 on
label-Code-Review.

This change corrects existing ACLs to meet the above criteria, and
also introduces a normalization rule to prevent regression.

Change-Id: I2eecb7028bcab7d5d82ad4155a775a9b2daa441f
2016-02-17 22:39:03 +00:00
Jeremy Stanley
cb63263510 Keep Gerrit ACL lines deduplicated
Gerrit ACLs can have multiple duplicate option keys in a section,
but completely duplicate lines (key and value together) have no use
so make sure they're collapsed into at most 1 copy.

Change-Id: I6bf43e860dcc8c3d7b2846d4e058b6c8ac7243eb
2016-02-17 22:31:54 +00:00
Andreas Jaeger
03d0680e41 Check for astra misspelling
Add common missspellings, fix also the one issue it caught.

Change-Id: Iaa6a94e3f1e3bff5f4d2986c4926078bd9090b42
2016-02-11 16:51:14 +01:00
Angus Lees
65916bfdb4 Report actual error in error message
While debugging a YAML change just now, I found it impossible to work
out what jenkins-projects-checks.py was actually complaining about
without this additional information.  I thought others might benefit
from it too.

Change-Id: I79b8286a3ef74008e5b0927bddb60f9c98b9b629
2016-01-15 11:11:19 +11:00
James E. Blair
9ed42604d0 Validate grafyaml based on master
Rather than installing grafyaml from pypi, install it with
zuul-cloner so that it may be validated with the contents of
master (or even changes in review).  We run grafyaml continuously
deployed from source rather than installing from pypi, so this
more closely matches what we actually want to validate.

Change-Id: I26663e5c70883b910fced6c32ae364734542b5a5
2016-01-07 17:23:27 -08:00
James E. Blair
0e327b9092 Ignore E011 in bashate
Change-Id: Id735a21cda1fc46e40b4f9ea51410d89bbf000a1
2016-01-07 16:52:39 -08:00
Jenkins
ca4e56665c Merge "Basic validation for gerritbot/channels.yaml" 2016-01-04 18:22:49 +00:00
Andreas Jaeger
71aa9e00aa Fix check for alphabetical sorting in layout.yaml
Previously z/tempest was in the middle of the file and needed special
treatment for sorting. Nowadays it's at the end and thus no special
treatment is needed, remove it.

With this change additions to the end of the file will now be flagged.

Change-Id: I337017b4a17e8aa086d658649613ddc61212adc8
2015-12-26 09:26:30 +00:00
Andreas Jaeger
0b4b988d9f Basic validation for gerritbot/channels.yaml
Recently a change merged that had not branches line and gerritbot
refused to start again. This could have been avoided by validation of
the file.

Test some basic validation of the yaml file to avoid breakage.

Change-Id: I2dc4a28aeac07d3fc73d3954d8fbe8f2448334e5
2015-12-23 17:53:43 +01:00
Andreas Jaeger
e617d1ffce Disable bashate long line warning
We still have 79 bashate warnings for lines longer than 79 characters.
bashate also warns about here documents, so cutting down on all of
them is not worth it. Let's disable these bashate warnings so that we can
see the real problems.

Warnings disabled are of the form:
[W] E006: Line too long: ...

Change-Id: I3abfae4432f9432af6e0d8b43bfc340673f06802
2015-12-23 13:57:05 +01:00
Andreas Jaeger
1db085d5ec Wrap some overly long lines
To cut down on noise when running bashate, wrap some overlong lines.

Change-Id: I23dcffbae2539d3d6b11e5527f3d4cbf08a18064
2015-12-23 13:56:53 +01:00
Andreas Jaeger
17439f6409 Add description label to jenkins checker
Allow description in more entries for JJB checks.

Change-Id: Id7ab6c274049d435eb5d429f56bc988bb3578eb7
2015-12-21 21:11:25 +01:00
Jenkins
c03fb66894 Merge "Run regex match over jobs in memory" 2015-12-16 21:40:16 +00:00
Jenkins
fe99d3f31b Merge "Test that job regexes in zuul/layout match" 2015-12-16 21:38:53 +00:00
Ian Wienand
6ee8bfcf1d Run regex match over jobs in memory
The jobs file isn't that big, so it makes sense to read it in once and
use it, rather than re-read.  Saves a very small amount of time in the
test.

Change-Id: Id20a2d6926a250f9117108fe32b04fc70b7bfe28
2015-12-16 09:39:29 +00:00
Andreas Jaeger
4b7fd514d8 Test that job regexes in zuul/layout match
Enhance our zuul/layout.yaml checks to check that the names in the jobs
section actually match an existing job. A mismatch implies that the
pattern does not work as intented.

Add note to run-layout.sh that the job files is used elsewhere as well,
so that it does not get deleted by accident.

Change-Id: I5c4cd97829acf90bb8f72dde32e5fe8063933cdb
2015-12-16 10:38:49 +01:00
Jenkins
46b0a26c6d Merge "Start validating JJB job-templates" 2015-12-16 09:07:05 +00:00
Paul Belanger
8ccd1d987e Start validating JJB job-templates
Initial commit to validate some JJB files. This validates
for existence of required sections and where possible that no additional
ones are available.

It will catch some errors that the JJB parser misses, since JJB ignores
unknown entries.

Change-Id: I51606d7066af7d5761f09655eead48883f7f56dc
Co-Authored-By: Andreas Jaeger <aj@suse.com>
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2015-12-16 08:46:46 +01:00
Andreas Jaeger
2304cb56b3 Validate options in gerrit/projects
Check that only known options are used in gerrit/projects.yaml.

Change-Id: I4ba62547e71b21d8c32297ba06b4aa937f9b637e
2015-12-13 16:49:12 +01:00
Ian Wienand
d793c97904 Pull latest changes in run-layout.sh
Some of my local tox tests were failing in odd ways; eventually I
chased it down to out-of-date versions of tools left in ".test"
directory.

This tries to pull the latest changes if the repos exist, and adds a
few helpful output points to let you know what's going on.

Change-Id: I30b7ddf84520e8895466a49ed6481812ea8875a8
2015-12-03 15:32:12 +11:00
Paul Belanger
b544a361ef Run gate-project-config-bashate on .sh files only
Like pep8 before we are trying to reduce the amount of jobs launched
in the check / gate pipelines. Over the last 200 commits only 2 or 3
files have been touched.

We are also reducing run-bashate.sh and remove unsourced files.

Change-Id: Ie77fad9a272fb41fbb48f94ceee910954e903939
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2015-11-11 11:27:39 -05:00
Jenkins
83818ce335 Merge "Remove global vars from check_valid_gerrit_projects.py" 2015-10-23 11:15:21 +00:00
Jenkins
1b9a935b64 Merge "Check for valid upstream git projects" 2015-10-22 09:08:04 +00:00
Andreas Jaeger
10b18599e3 Remove global vars from check_valid_gerrit_projects.py
Replace global var with local variable usage.

Change-Id: I44640a274947f9be84232fb172f8247c15dbaee5
2015-10-22 11:00:02 +02:00
Jenkins
3ee06b2c0d Merge "Add script to facilitate project renaming in project-config repo" 2015-10-22 00:41:12 +00:00
Andreas Jaeger
106f6b80e2 Check for valid upstream git projects
This checks out the upstream git projects (unless we use track-upstream)
and prints some information about it.

Note this checks for all upstream repositories but since those are
cleaned up regularly, this should be fine.

It fails if the there is no remote master branch.

Reindent the original code as main function.

Change-Id: Ie469905f48e4bc1a408fb9e731f95b2b735a6a02
2015-10-21 20:34:45 +02:00
Jenkins
29dc96d915 Merge "Sort zuul/layout.yaml alphabetical" 2015-10-19 00:25:25 +00:00
Andreas Jaeger
ba7ad70283 Sort zuul/layout.yaml alphabetical
Sort zuul/layout.yaml project entries in alphabetical
order by "namespace/project".

Reenable check for sorted order of layout.yaml that has been disabled
during mass project rename.

Change-Id: I3749f88edd1984286f9cfc840bfe37722b2f28be
2015-10-18 19:13:54 +02:00
Andreas Jaeger
67f162671d Resort jenkins/jobs/projects.yaml
Sort the file in alphabetical order and update
tools/jenkins-projects-checks.py for this sort order.

Use single blank line as separator between entries everywhere.

Change-Id: I2ff09483191260a9ae5d3460046b50d212ceb1ef
2015-10-18 18:54:28 +02:00
Clint Adams
246bbfe03b Add script to facilitate project renaming in project-config repo
This takes a list of projects from stdin in the format of
aeromancer
anvil
blazar

and renames stackforge/aeromancer, stackforge/anvil, and stackforge/blazar
to openstack/aeromancer, openstack/anvil, and openstack/blazar respectively
in gerrit/projects.yaml, gerritbot/channels.yaml, zuul/layout.html,
and every file in jenkins/jobs/ , as well as renaming acl files.

A side effect of this process is that comments will be stripped from
gerritbot/channels.yaml ; if you are committing the results of a run of
this script, please re-add the comments manually unless we come up with
a more elegant solution.

This script also has test coverage for verification and documentation.

Also included is a WIP list of renames in data/stackforge-renames, as
per
http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2015-10-13.log.html#t2015-10-13T20:10:17
and a WIP list of retirements in data/stackforge-retirements, which
is not used by the script in this change.

Co-Authored-By: Augustina Ragwitz <aragwitz+lp@pobox.com>
Change-Id: Iaf2a13d8b9b9a840d9fde07ec1ef65d1c9e90e8e
2015-10-17 15:50:14 +00:00
Clint Adams
d0fc16cd6b Disable alphabetical ordering check for zuul/layout.yaml
In order to facilitate the mass stackforge renames, disable sorting
check.

Once the renames have been completed successfully and the sections
have been reordered, this check should be re-enabled.

Change-Id: I460081a42c42d0dfd2ffd8af7281018128f264fe
2015-09-30 17:34:48 -04:00
Clint Adams
952bc590ba Test formatting of gerritbot/channels.yaml
Ensure that the output of tools/normalize_channels_yaml.py matches
gerritbot/channels.yaml modulo any comments.

Change-Id: I3147cb094a795aeeffcceda182d065ed5145c2dd
Depends-On: Ia710997f37e8fba4a18b26577ecac42dd3137e2a
2015-09-29 09:56:18 -04:00
Clint Adams
983626a715 Factor out yaml emitter/dumper functions from normalize_projects_yaml.py
This moves the functions used by normalize_projects_yaml.py out for the
purposes of being imported by multiple scripts.

Change-Id: I76a93124fdbe13bce34a0c5073fa66bc86806c0a
2015-09-28 19:19:13 -04:00
Ian Wienand
3e78b33c31 Check indent in zuul/layout.yaml
I noticed this when looking at
Ieea93753e352ccb09f65aac12c8c81c1d96593ac (but the misalignment there
seems to have been subsequently fixed).

Anyway, this is just a quick port of
I92ffda01a2e766954c18fe795862186863f8182f and fixes two existing
cases.

Change-Id: Ia3e654ed32e1edb9d841984df427c764d220b908
2015-09-25 13:06:06 +10:00
Ian Wienand
1942b394d6 Add a simple description check for gerrit/projects.yaml
Add a very simple check for some common typos that might appear in the
descriptions.

Correct existing typos as found by the check

Change-Id: I6edd833de8544e44750c3a3fb49a065f99a18499
2015-08-27 12:38:52 +10:00
Ian Wienand
b472603903 Check indenting of jenkins/jobs/projects.yaml
I just noticed when reviewing the file that a couple of indents were
off.  This isn't major, but we can catch that pretty easily with the
existing jenkins-projects-checks.py.

Change-Id: I92ffda01a2e766954c18fe795862186863f8182f
2015-07-17 12:10:39 +10:00
Andreas Jaeger
cb89bb5354 Fix Normalize projects.yaml proposal
The proposals bot run on a tree that is in detached state. Current
git review fails to push from this any changes.

Create instead directly a branch and work on that.

This is the same change as done for other proposal jobs in
I487337cb1def5f5995d776c831ea61b658652677.

Change-Id: I3ca3cbb0c21efde28a43164c518405c0dea87a86
2015-07-01 08:39:07 +02:00
K Jonathan Harker
344bdf1faf Fix logic error in gate-project-config-projects-yaml
Previously labels were only being checked for validity if an upstream
was given due to a premature 'continue' if no upstream is given before
checking labels.

Change-Id: I7dd564d59a30ca14bbad4044a985ba08fbe55ca5
2015-06-19 13:37:03 -07:00
Steve Kowalik
85a0f4f44f Switch to using $() for subshells, part 1
Cleanup every use of `` for subshells in the nodepool and tools
directory , replacing them with $(), and finally making the scripts
consistent.

Change-Id: I2b05cd20f9c9a30ab88f8db235aa81da93b1fad3
2015-06-04 15:30:55 +10:00
Jenkins
8c7eed19e5 Merge "Use zuul-cloner instead of git clone" 2015-06-02 19:21:53 +00:00
Steve Kowalik
f4a2c9802d Switch to using test -n
Using the construct of x$var != x is unnecessary, and has been
unnecessary for quite some time. We trust our shell, so switch to
using test -n.

Change-Id: I8eea943e5b641ba3d8781212877da7ddec167a6c
2015-05-21 13:39:55 -07:00