We have run into a case where we need to store a secret longer
than 3760 bits. We may eventually support a hybrid encryption
scheme, but for now, let's also support the alt.zuul.secrets protocol
where we split the secret into 3760 bit chunks and recombine it.
The encrypt_secret utility is updated to output a copy-pastable
YAML data structure to simplify dealing with long secrets.
Change-Id: Ied372572e5aa29fddfb7043bf07df4cd3e39566c
It always downloads the GPG key from the interwebs. Stop doing that
because depending on the network is bad.
Change-Id: Ia7c1a5e06d7698c15b22e80c328b9bb368a687e4
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
This fixes the encrypt_secret tool for use with python3. This needs
some minor changes to imports, encodings and base64 encoding.
Change-Id: Id29ebedab2115d0d5d47049f2a0412e8c75aa8ef
It exists only for py2/py3 compat. We do not need it any more.
This will explicitly break Zuul v3 for python2, which is different than
simply ceasing to test it and no longer declaring we support it. Since
we're not testing it any longer, it's bound to degrade overtime without
us noticing, so hopefully a clean and explicit break will prevent people
from running under python2 and it working for a minute, then breaking
later.
Change-Id: Ia16bb399a2869ab37a183f3f2197275bb3acafee
Bubblewrap is a setuid wrapper for unprivileged chroot and namespace
manipulation. We'll be using this to run ansible-playbook from.
Change-Id: I7c6414281724d066812094d02a726df357978076
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Add a check to confirm mysql and zookeeper are running.
Change-Id: I5c46b0c33bfd6c474793900b296dedf07ca28164
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
This will allow us to enter results from all jobs for
use with the openstack-health dashboard.
Depends-On: I08dbbb64b3daba915a94e455f75eef61ab392852
Change-Id: I28056d84a3f6abcd8d9038a91a6c9a3902142f90
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
We no longer expect this script to run the test, but rather, just
setup the host. Rename it accordingly.
Also, remove the temporary sleep and log cat as the job now does
something useful.
Change-Id: I0236d36b0c531777e61422de4273c5e8fb33323a
Depends-On: I12143ac0841e5fc646448eae4171594579fa0086
Zuul mergers create a vast number of git references under /refs/zuul
which are never garbage collected.
With hundred of thousands of references, that makes git fetch operations
very slow since git uploads all references to Gerrit to synchronize the
Zuul maintained repository. On one of Wikimedia busy repository
(mediawiki/core) we had 55000 such references and it can take up to 18
seconds for a fetch to complete. I have seen occurences of a merge
taking 2 minutes to complete.
As such, this tiny script clears out references for which the commit date
of the pointed commit object is older than 360 days (the default).
It is not perfect since a recent reference can well point to an old
object. That would be the case on repositories that are barely active.
In such case the ref will be gone despite it being recently created.
A better way would be to vary Zuul references by using month/day which
will let one easily garbage collect them. But I am being lazy and that
would not let us clear out references using the current scheme.
Example usage:
zuul-clear-refs.py --verbose --dry-run --until 90 /srv/zuul/git/project
Would show a list of references pointing to commit dates older than 90
days and output a message whenever the script would delete them.
Hint about the utility in our merger documentation.
Reference:
https://phabricator.wikimedia.org/T70481
Change-Id: Id4e55f5d571ebd5e8271e516f53f8e05c1f78c1a
* tools/trigger-job.py: Add a --url option which passes the required
ZUUL_URL parameter with a sane default for OpenStack Infra's
environment.
Change-Id: I2d8aa73b11b0069e31ffa599c1c9d9059919fb7a
* tools/trigger-job.py: Add a logpath arg and make it required. This
allows us to upload logs to appropriate dirs even when manually
triggering jobs.
Change-Id: I59144a2d5443fb6396af45f302f67bc8eec70780
Change-Id: Ic396c845e5f61a98d3b0e8c3fc08c6f9dbfffc6a
Reviewed-on: https://review.openstack.org/34437
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
Change-Id: I0385ac9f8336a8c2e45b0fa6ed062b6b23a182aa
Reviewed-on: https://review.openstack.org/33351
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
Change-Id: I734f0f8237fb603ee41a39f06e63c007e79825a9
Reviewed-on: https://review.openstack.org/33350
Reviewed-by: James E. Blair <corvus@inaugust.com>
Reviewed-by: Doug Hellmann <doug.hellmann@dreamhost.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
And less than 0.4.0 for safety.
Change-Id: I539a2cc7816c15491db55975ac07ee7a148535dc
Reviewed-on: https://review.openstack.org/32973
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
The internal data structure of the server queue changed.
Change-Id: I280a671577d2247b315174f543dfb2ff8b7d51f4
Reviewed-on: https://review.openstack.org/31997
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Change-Id: I14dbe148f0ff57e5d0c50cc712120a2708f158b1
Reviewed-on: https://review.openstack.org/31727
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Khai Do <zaro0508@gmail.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
Remove the Jenkins launcher and add a new Gearman launcher (designed
to be compatible with Jenkins) in its place.
See the documentation for how to set up the Gearman Plugin for
Jenkins.
Change-Id: Ie7224396271d7375f4ea42eebb57f883bc291738
We used to pin statsd to version 1.0.0. James E. Blair has written a
few tests that now let us allow the current v2.0.1 version. To prevent
unexpected upgrades, we disallow v3.0.0.
Zuul scheduler only do very simple statsd operations. Testing with
statsd 2.0.1:
I have set up a basic listener:
listener$ netcat -l 127.0.0.1 -p 666 --udp
Then wrote the python script:
import extras
statsd = extras.try_import('statsd.statsd')
statsd.incr('zuul.counter')
statsd.timing('zuul.timer', 1234)
statsd.timing('zuul.timer', 1)
statsd.gaug:('zuul.gauge', 1234)
Running it as:
STATSD_HOST='127.0.0.1' STATSD_PORT=666 python statsd-test.py
The listener gets:
zuul.counter:1|czuul.timer:1234|mszuul.timer:1|mszuul.gauge:1234|g
Sounds legit enough.
Fix bug #1169599 'Zuul should depends on statsd 2 or later'
Change-Id: I41b797d606d55eeda300e673b05c2ae4442ce029
Reviewed-on: https://review.openstack.org/27008
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
tools/pip-requires list WebOb in lower case which confuse pip freeze and
gives a false positive:
Before patch:
$ pip freeze -r tools/pip-requires 2>&1|grep -i webob
Requirement file contains webob, but that package is not installed
WebOb==1.1.1
$
The second line comes from another packages which list 'WebOb' as a
dependency.
After patch:
$ pip freeze -r tools/pip-requires 2>&1|grep -i webob
WebOb==1.1.1
$
Change-Id: I73d01c734c25fc65022de146264e5b2c3cc2e67f
Reviewed-on: https://review.openstack.org/26786
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
* tools/pip-requires: A new voluptuous (0.7.0) was released today,
introducing backward-incompatible changes to schema objects and
breaking zuul's layout validation. This is a brown bag fix to keep
zuul functional until voluptuous 0.7 support is implemented.
Change-Id: I5818d3cd529e8682877b368edbc10280a5c167df
Reviewed-on: https://review.openstack.org/24454
Reviewed-by: Anita Kuno <akuno@lavabit.com>
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
The script prints a gerrit review command for every change in
a given zuul pipeline.
Change-Id: Ibdee87ea64c977e7589cbc3ee9567e866cb78341
Reviewed-on: https://review.openstack.org/23784
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
Voluptuous has a version 0.6 now which deprecates the any, match, and
required predicates and replaces them with Any, Match, and Required.
Update the minimum required version of Voluptuous and stop using the
deprecated predicates.
Change-Id: I554a6fda2c622a0b63f9247b1eae5eb494af225b
Reviewed-on: https://review.openstack.org/22167
Reviewed-by: Khai Do <zaro0508@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Based on voluptuous library.
Basic validation should catch typos, missing or extra attributes.
Can be expanded to do more serious validation (ie, specifying
a comment in a trigger should require the event be comment-added).
Adds a command line option to validate a named layout file and
exit.
(Also add dist/ to .gitignore.)
Change-Id: Ia864ebde1765141d4d1a52bc77033689b6210e81
Reviewed-on: https://review.openstack.org/19443
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Reports statistics on job runtime, number/rate of jobs run, and
number/rate of individual gerrit events to statsd.
Change-Id: I0db9352610b4c268c95da98fd282d7f463945f90
Reviewed-on: https://review.openstack.org/18628
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
We're starting to use this elsewhere in OpenStack... avoids having to
do pyflakes workaround tricks.
Change-Id: Ic767677e87f30794d49baf100e3ded7555272df6
Reviewed-on: https://review.openstack.org/16851
Reviewed-by: James E. Blair <corvus@inaugust.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: Monty Taylor <mordred@inaugust.com>
Tested-by: Jenkins
zuul also need the 'lockfile' and 'python-daemon' python packages.
Change-Id: I16ed4fba9705796f83aa980687de8b9d262eb715
Reviewed-on: https://review.openstack.org/13523
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Adding nosehtmloutput as a test dependency allows nose to output its
results to an html file. This will be used by Jenkins to save logs on
a different server.
Change-Id: Ia335969d830fba1dcfbc9a68a6deeb24828b14d0
Reviewed-on: https://review.openstack.org/11753
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Add git repo management and merging. When collecting changes to
be tested together, merge or cherry-pick those changes into the
zuul-managed repos, and create a unique ref for that configuration.
Pass the ref to Jenkins instead of the string description of the
changes, so that Jenkins only needs to checkout that one ref.
This moves the complexity of merging and managing multiple commits
out of Jenkins and into Zuul.
The GERRIT_CHANGES variable is deprecated (along with the rest of
the GERRIT_* variables) and will be removed in a future patch
(which will contain a documentation update).
Change-Id: I126c9030223c07a30f7092e2273ebd7605d9f3df
Reviewed-on: https://review.openstack.org/11349
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
The zuul-coverage job depends on the coverage tool. Add it to
tools/test-requires list.
Change-Id: I4c5609f7992dd6dae649f5c059a69bb06b021831
Reviewed-on: https://review.openstack.org/10684
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Needed for the gate-zuul-docs job.
Change-Id: Ib330aeaf322543ab1f8b92e48672b0b0b4a4f692
Reviewed-on: https://review.openstack.org/10681
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Add support for testing, and some initial functional tests.
Some threads in Zuul now have proper shutdown methods, since
that's needed for testing. The threads that are faked in the
testing infrastructure still lack shutdown methods.
Fix some bugs uncovered by tests:
Reverse the order of commits that are given to Jenkins (the
commits that preceded the one being tested were listed in
revers order, this corrects that).
Return 0 in reportChange if the change has already been
reported. The old return value of True was misinterpreted
as an error.
Change-Id: I4979d0efd8581061a2b856b892d4ecdb75284a1b
Reviewed-on: https://review.openstack.org/10572
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
James E. Blair