213 Commits

Author SHA1 Message Date
Jenkins
19247d92c3 Merge "Make wheel mirror builders normal zuul nodes" 2017-09-13 23:22:39 +00:00
James E. Blair
2b464b1782 Return gearman job handles on log job submission
For easier debugging.  Also, override the invocation kwarg so that
ansible doesn't automatically log the arguments to this module
(since they include all of the hostvars which include way too much
data).

Change-Id: I415da15dd1dc154e08e159453235448c90a5e5d2
2017-09-13 14:55:41 -06:00
Monty Taylor
02c84560b8
Make wheel mirror builders normal zuul nodes
Including puppet madness!

We don't need the long-lived wheel builders - we build from scratch
every time. So all we need is AFS running on a node.

AFS, however, is tricky on CentOS. Rewriting all that right now is meh.
We can do that later.

This gets us to not needing special nodes for wheel builders. Yay!

Change-Id: I6d92c47f4f7a26ca290cd400452cb0a73872b24d
NOTE: It may be inefficient to run this on our low-bandwidth nodes.
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-09-13 14:25:37 -04:00
James E. Blair
6ffab6f745 Fix import errors in subunit job module
Change-Id: I0da368c933e8fb6e3de7e6c4e81c934636b0317e
2017-09-13 10:12:30 -06:00
James E. Blair
fad3ecd79b Update success to zuul_success
Change-Id: I59eb5e4c76b6de76028f0303bd4ffd4143c0da18
2017-09-13 09:19:40 -06:00
James E. Blair
47ae93c061 Add log processing roles
These roles replace the jenkins-log-client portion of the
logstash/subunit queue processing system.  Much of the code is
derived from the jenkins-log-client.

Rather than subscribing to ZMQ events emitted from jenkins (and
later zuul v2.5), these roles are intended to be run in a late base
post playbook (right after log uploading).  They examine the logs
directory which contains the files that were just uploaded and any
matching files have jobs submitted to the gearman queue.  Currently
the module attempts to maintain compatability as much as possible,
but eventually we should represent some of the data in a more v3-like
manner, as well as consider adding additional data.  Some suggestions
are noted in comments.

Change-Id: I10ea613d9278465b90d891371d2626b9e99f8f31
2017-09-11 10:12:28 -06:00
Monty Taylor
622f04f214
Treat wheel builders more like build nodes
Pass the wheel keytab across and aklog with it.

Also, split the build role into a build and a copy so that we only have
the token when we need it, and not while we're running the wheel builds
themselves. And split release from build so that we can release the
mirror once for a consistent state across python2 and python3.

Change-Id: Id30a99f5b515dd520d013fabbd356e32c82030a0
2017-09-06 14:44:06 -04:00
Monty Taylor
8a7456ed90
Ansiblify the wheel build a little more
Change-Id: I669e7b5ca9eac4b00bf0284c1cd4f9ad8c3f9d09
2017-09-06 14:44:04 -04:00
James E. Blair
64da87d225 Zuulv3: fix reference to stat results in use-cached-repos
Change-Id: I2fda509eda5012075e2092ab8511625741f604ed
2017-08-30 13:07:57 -07:00
James E. Blair
28b2169a0a Fix errors in use-cache-repos
These options need to go in the 'args' dict.

Change-Id: I3981706d7f1da71061698163123b96a24f844035
2017-08-29 17:55:33 -07:00
James E. Blair
696dbb5fae Remove whitespace from use-cached-repos role
Change-Id: I9229feedb87c4ba9352c876ac2f4e63b97c76817
2017-08-29 17:54:21 -07:00
James E. Blair
0d94dbf838 Zuul v3: Add use-cached-repos role
This adds a role to clone the cached repos into the workspace,
and alters base-test to use it as well as follow-up roles to
synchronize their states and start zuul_console, instead of the
default prepare-workspace role which uses rsync and is less
efficient.

Change-Id: Ifb0fedf1df1d50444ecb1f3fb03f21b112c9dad9
Depends-On: Ib50ddff0198411e7cd5857d5bfbb915f02baadf1
2017-08-29 16:44:01 -07:00
Monty Taylor
a2ccdf2d08
Extract scripts copying into a role
We'll need it for more than just proposal jobs, but it's clearly very
project-config specific, so keep it here.

While refactoring that, consume add-sshkey from zuul-jobs.

Depends-On: I569b93ba0e775896b5a4f59b544fe32d1e2dce9b
Change-Id: I6b16acdcf55bcc6ff28ad496c044fe7db4599cd8
2017-08-28 11:11:06 -05:00