Commit Graph

510 Commits (f88b81781397bf38c6be1d544378b6d10d711328)

Author SHA1 Message Date
James E. Blair f88b817813 Add nodepool integration test
This uses the python unittest framework to run a test similar to
the existing zuul-nodepool unit test, but expecting an actual
running nodepool.

Change-Id: I769e1421c146cc3545dce606487c8c72e3d3a4c5
2017-01-24 13:37:21 -08:00
James E. Blair 2a8e0fa5f1 Move tests into test/unit
This makes room for a sibling directory for nodepool functional tests.

Change-Id: Iace94d313edb04192ac23a533ed967f076410980
2017-01-24 10:18:38 -08:00
Jenkins 4ec7ca8e9b Merge "Re-enable zuultrigger test" into feature/zuulv3 2017-01-23 21:38:59 +00:00
Jenkins 9672473d87 Merge "Reorganize connections into drivers" into feature/zuulv3 2017-01-23 21:38:02 +00:00
James E. Blair a80ef4e7c6 Re-enable zuultrigger test
This gets both running again, but the second fails because of a
behavior change in v3 which should be addressed separately.

Story: 2000773
Task: 3422

Change-Id: I3d0e7268304e97fa611f70706e15212952aa3c62
2017-01-20 05:43:28 -08:00
James E. Blair e511d2f6c4 Reorganize connections into drivers
This change, while substantial, is mostly organizational.
Currently, connections, sources, triggers, and reporters are
discrete concepts, and yet are related by virtue of the fact that
the ConnectionRegistry is used to instantiate each of them.  The
method used to instantiate them is called "_getDriver", in
recognition that behind each "trigger", etc., which appears in
the config file, there is a class in the zuul.trigger hierarchy
implementing the driver for that trigger.  Connections also
specify a "driver" in the config file.

In this change, we redefine a "driver" as a single class that
organizes related connections, sources, triggers and reporters.

The connection, source, trigger, and reporter interfaces still
exist.  A driver class is responsible for indicating which of
those interfaces it supports and instantiating them when asked to
do so.

Zuul instantiates a single instance of each driver class it knows
about (currently hardcoded, but in the future, we will be able to
easily ask entrypoints for these).  That instance will be
retained for the life of the Zuul server process.

When Zuul is (re-)configured, it asks the driver instances to
create new connection, source, trigger, reporter instances as
necessary.  For instance, a user may specify a connection that
uses the "gerrit" driver, and the ConnectionRegistry would call
getConnection() on the Gerrit driver instance.

This is done for two reasons: first, it allows us to organize all
of the code related to interfacing with an external system
together.  All of the existing connection, source, trigger, and
reporter classes are moved as follows:

  zuul.connection.FOO -> zuul.driver.FOO.FOOconnection
  zuul.source.FOO -> zuul.driver.FOO.FOOsource
  zuul.trigger.FOO -> zuul.driver.FOO.FOOtrigger
  zuul.reporter.FOO -> zuul.driver.FOO.FOOreporter

For instance, all of the code related to interfacing with Gerrit
is now is zuul.driver.gerrit.

Second, the addition of a single, long-lived object associated
with each of these systems allows us to better support some types
of interfaces.  For instance, the Zuul trigger maintains a list
of events it is required to emit -- this list relates to a tenant
as a whole rather than individual pipelines or triggers.  The
timer trigger maintains a single scheduler instance for all
tenants, but must be able to add or remove cron jobs based on an
individual tenant being reconfigured.  The global driver instance
for each of these can be used to accomplish this.

As a result of using the driver interface to create new
connection, source, trigger and reporter instances, the
connection setup in ConnectionRegistry is much simpler, and can
easily be extended with entrypoints in the future.

The existing tests of connections, sources, triggers, and
reporters which only tested that they could be instantiated and
have names have been removed, as there are functional tests which
cover them.

Change-Id: Ib2f7297d81f7a003de48f799dc1b09e82d4894bc
2017-01-20 05:43:21 -08:00
Jesse Keating 2dfc651202 Correct comments in requirement test cases
The comments were misleading which led to a lot of head scratching. This
corrects the comments to reflect the realities of the test.

Change-Id: Id83243140558fb76e91f83843da8285fd17f384b
2017-01-19 14:31:44 -08:00
Jenkins 7f238a2991 Merge "Spell NodeSet consistently" into feature/zuulv3 2017-01-18 22:14:58 +00:00
Jenkins aa62bc204c Merge "Fix copyright date in test_nodepool" into feature/zuulv3 2017-01-18 22:14:52 +00:00
Jenkins fa8e36a8b4 Merge "Handle nodepool allocation failure" into feature/zuulv3 2017-01-18 22:14:18 +00:00
Jenkins 09d6e4a9e0 Merge "Verify nodes and requests are not leaked" into feature/zuulv3 2017-01-18 21:24:07 +00:00
Jenkins 808ee14a91 Merge "Remove excess printing from stats test" into feature/zuulv3 2017-01-18 21:23:59 +00:00
Jenkins 5c63371b48 Merge "Cancel/return nodepool requests on job cancel" into feature/zuulv3 2017-01-18 20:10:29 +00:00
Jenkins 932f49b7f9 Merge "Return nodes after use" into feature/zuulv3 2017-01-18 19:34:20 +00:00
Jenkins 8b43b722f3 Merge "Mark nodes as 'in-use' before launching jobs" into feature/zuulv3 2017-01-18 19:23:02 +00:00
Jenkins 379ac33d2f Merge "Lock nodes when nodepool request is fulfilled" into feature/zuulv3 2017-01-18 19:22:56 +00:00
Jenkins 2874abe908 Merge "Re-submit node requests on ZooKeeper disconnect" into feature/zuulv3 2017-01-18 19:20:59 +00:00
Jenkins 67a125ff27 Merge "Add nodepool test class" into feature/zuulv3 2017-01-18 17:45:08 +00:00
James E. Blair 1511bc38a3 Spell NodeSet consistently
It is intended to match the existing pattern used by BuildSet.
Usage is currently evenly split between NodeSet and Nodeset.

Change-Id: Iab96ed90a9ada0cb4709c0bb8b04923ab0c765e8
2017-01-18 09:26:10 -08:00
Jenkins d733484924 Merge "Add a test printHistory function" into feature/zuulv3 2017-01-18 17:13:00 +00:00
Jenkins ece8e15d89 Merge "Improve test output" into feature/zuulv3 2017-01-18 17:12:54 +00:00
Jenkins ca43352365 Merge "Fix race condition in test_success_url" into feature/zuulv3 2017-01-18 17:12:34 +00:00
Jenkins 869456c297 Merge "Add FakeNodepool test fixture" into feature/zuulv3 2017-01-18 17:11:10 +00:00
James E. Blair 15d91cc388 Fix copyright date in test_nodepool
This file was recently added (by me) with the wrong copyright date.

Change-Id: I39ed8ff1b29c48d2079338691a00ef6b4153b27b
2017-01-18 09:05:17 -08:00
Jenkins edfe0d0a30 Merge "Add Zookeeper to tests" into feature/zuulv3 2017-01-11 11:55:59 +00:00
Jenkins f84b1c232d Merge "Fix test_rerun_on_abort" into feature/zuulv3 2017-01-09 18:40:05 +00:00
James E. Blair 6ab79e0637 Handle nodepool allocation failure
When a request is either fulfilled or failed, pass it through to
the scheduler which will accept the request (which means deleting
it in the case of a failure) and pass it on to the pipeline manager
which will set the result of the requesting job to NODE_FAILURE
and cause any sub-jobs to be SKIPPED.

Adjust the request algorithm to only request nodes for jobs that
are ready to run.  The current behavior requests all jobs for a
build set asap, but that has two downsides: it may request and
return nodes more aggressively than necessary (if you have chosen
to create a job tree, you *probably* don't want to tie up nodes
until they are actually needed).  However, that's a grey area,
and we may want to adjust or make that behavior configurable later.
More pressing here is that it makes the logic of when to return
nodes *very* complicated (since SKIPPED jobs are represented by
fake builds, there is no good opportunity to return their nodes).

This seems like a good solution for now, and if we want to make
the node request behavior more aggressive in the future, we can
work out a better model for knowing when to return nodes.

Change-Id: Ideab6eb5794a01d5c2b70cb87d02d61bb3d41cce
2017-01-06 16:47:02 -08:00
James E. Blair e18d460e47 Verify nodes and requests are not leaked
Check that at the end of every test, there are no outstanding
nodepool requests and no locked nodes.

Move final state assertions into the tearDown method so that
they run right after the end of the test but before any
cleanup handlers are called (which can interfere with the
assertion checking by, say, deleting the zookeeper tree we
are trying to check).  Move the cleanup in test_webapp to
tearDown so that it ends the paused job that the tests in
that class use before the assertion check.

Fix some bugs uncovered by this testing:

* Two typos.
* When we re-launch a job, we need a new nodeset, so make sure
  to remove the nodeset from the buildset after the build
  completes if we are going to retry the build.
* Always report build results to the scheduler even for non-current
  buildsets so that it can return used nodes for aborted builds.
* Have the scheduler return the nodeset for a completed build rather
  than the pipeline manager to avoid the edge case where a build
  result is returned after a configuration that removes the pipeline
  (and therefore, there is no longer a manager to return the nodeset).
* When canceling jobs, return nodesets for any jobs which do not yet
  have builds (such as jobs which have nodes but have not yet
* Return nodes for skipped jobs.

Normalize the debug messages in

Change-Id: I32f6807ac95034fc2636993824f4a45ffe7c59d8
2017-01-05 17:28:35 -08:00
James E. Blair ce001e11a8 Remove excess printing from stats test
This isn't necessary and makes the output of this test noisy.

Change-Id: I214d40fada9567ac6b9cee5ff9cdc748a472cbbb
2017-01-05 16:14:18 -08:00
James E. Blair 01695c35e5 Cancel/return nodepool requests on job cancel
When canceling a running or requested job, cancel outstanding
nodepool requests or return unused nodes.

Change-Id: I77f8869b9d751ccd6c9f398ed03ef5ac482cc204
2017-01-04 17:29:25 -08:00
James E. Blair 62295d322e Return nodes after use
This returns the nodes after use (or in some cases, before use)
and unlocks them.

Change-Id: I84e998a6099217f0a0924caf1872f33dce28ba01
2017-01-04 16:09:33 -08:00
James E. Blair cacdf2b659 Mark nodes as 'in-use' before launching jobs
While we immediately lock a node given to us by nodepool, we delay
setting the node to 'in-use' until we actually request that the job
be launched so that if we end up canceling the job before it is
run, we might return the node unused to nodepool.

Change-Id: I2d2c0f9cdb4c199f2ed309e7b0cfc62e071037fa
2017-01-04 16:09:24 -08:00
James E. Blair a38c28efa3 Lock nodes when nodepool request is fulfilled
This is continuing work on implementing the Zuul<->Nodepool protocol
from the Zuulv3 spec.

Change-Id: Ic8477e607fd09b85a37f47cbee7da905c017c534
2017-01-04 16:08:43 -08:00
James E. Blair 15be0e1e11 Re-submit node requests on ZooKeeper disconnect
Change-Id: I689bf812c713fa6f5f37958b7001b0d5fb0a254b
2017-01-04 09:11:35 -08:00
James E. Blair 3897a13743 Add nodepool test class
To exercise the Zuul<->Nodepool interface.

Change-Id: Ie57146e2ab7221d4abf9021055d601b67a6efedf
2016-12-22 18:23:42 -08:00
James E. Blair 6ac368c57b Add a test printHistory function
I frequently add something like this ad-hoc when debugging a test.
Make it a convenient function that's easy to add when needed, and
also run it at the completion of every test so a developer can
easily survey the logs to see what happened.

Change-Id: I3d3810f51245d92855f086b875edfd52bdd86983
2016-12-22 18:07:20 -08:00
James E. Blair 10fc1eb487 Improve test output
* When a test timeout occurs, output the state debug information at
  error level so that it shows up in all logs.
* Add some more info to that output.
* Further restrict the (often not useful) chatty gear logs by default.

Change-Id: Ib275441172c5b1598593d0931cef0168d02e521d
2016-12-21 16:22:16 -08:00
James E. Blair e75768068b Fix race condition in test_success_url
The builds may run in any order.

Change-Id: I705be7a32d0534ae06ea4778bf2bc76d630388d8
2016-12-21 16:15:00 -08:00
James E. Blair dce6ceac8e Add FakeNodepool test fixture
Add a fake nodepool that immediately successfully fulfills all
requests, but actually uses the Nodepool ZooKeeper API.

Update the Zuul Nodepool facade to use the Nodepool ZooKeeper API.

Change-Id: If7859f0c6531439c3be38cc6ca6b699b3b5eade2
2016-12-21 14:16:51 -08:00
James E. Blair 498059ba28 Add Zookeeper to tests
Add a requirement on kazoo and add a Zookeeper chroot to the test

This is based on similar code in Nodepool.

Change-Id: Ic05386aac284c5542721fa3dcb1cd1c8e52d4a1f
2016-12-20 14:14:15 -08:00
Clint Byrum 77c184aa3a Re-enable test_webapp_find_change
This makes the tenant name required in the URL, which I believe is desired
for now, but maybe there will be an all-tenant view or a default tenant
in the future.

Note that the setUp method for this test suite actually had a bug that
was only approving change A twice, rather than approving B.

Change-Id: I25bd11088719e8b3465563863123d93eff16030e
Story: 2000773
Task: 3433
2016-12-20 12:27:21 -08:00
Jenkins 93573e5e42 Merge "Remove now-unused ZuulTestCase.resetGearmanServer" into feature/zuulv3 2016-12-19 18:37:16 +00:00
Jenkins ba5c05204f Merge "Remove TestScheduler.test_nonexistent_job" into feature/zuulv3 2016-12-19 18:36:24 +00:00
James E. Blair 5aed111523 Fix test_rerun_on_abort
Change I6d7d8d0f7e49a11e926667fbe772535ebdd35e89 erroneously altered
test_rerun_on_abort to match the observed behavior.

Change I6e64ef03cbb10ce858b22d6a4590f58ace0a5332 restored the values
in the test but then erroneously changed the accounting system to
match the observed behavior.

The actual problem is that this test also exercises the 'attempts' job
attribute, and does so by specifying it in a custom test configuration.
The test was failing because that attribute, which instructs zuul to
attempt to run a job for a non-default number of retries, was not being

It was not being read because the test was still using the old configuration
loading scheme.  It updated the "layout_file" which isn't a thing anymore
and asked the scheduler to reload.  The scheduler *did* reload, but it
simply reloaded the same configuration.  The solution to this is to either
create a new configuration, or, in this case, since the additional
configuration needed is compatible with the configuration used by the tests
siblings, simply add it to the active config file for the test.

Once the test is loading the correct configuration, one can observe that
the 'attempts' attribute was not added to the validator.  That is corrected
in this change as well.

With all of this complete, the test passes in its original form and no
modifications to the job retry accounting system.

Change-Id: Icf6d697cbae0166bc516faf5b7e60cac05885ab0
2016-12-15 09:25:39 -08:00
Jenkins 9b8d8079cd Merge "Re-enable test_no_job_project" into feature/zuulv3 2016-12-15 13:03:07 +00:00
Jenkins bc9c475df5 Merge "Re-enable test_nonvoting_job" into feature/zuulv3 2016-12-15 13:01:38 +00:00
Jenkins 68fd340802 Merge "Fix retry accounting off-by-one bug" into feature/zuulv3 2016-12-15 12:49:34 +00:00
Clint Byrum ee0786dd8b Remove now-unused ZuulTestCase.resetGearmanServer
This was used in only one test and it isn't in use anymore and is thus
untested, dead code.

Change-Id: Iff8c235583424a45926f273a88838d908381e237
2016-12-14 11:39:54 -08:00
Clint Byrum 8953efd281 Remove TestScheduler.test_nonexistent_job
This test was added to verify a fix involving a deadlock in the old
jenkins launcher, which has been removed. I don't believe it adds
coverage otherwise.

Change-Id: I9ba5a24af5accff41057a7634baf4320a4afca48
Story: 2000773
Task: 3430
2016-12-13 23:33:58 -08:00
Clint Byrum 883c30e947 Re-enable test_no_job_project
Minimal changes are needed, just translation of the old layout to new
and tenant API differences.

Change-Id: I3563fd1998dcc16426d665d50e26644b45198be0
Story: 2000773
Task: 3429
2016-12-13 12:08:47 -08:00