Commit Graph

1489 Commits (f88b81781397bf38c6be1d544378b6d10d711328)

Author SHA1 Message Date
James E. Blair f88b817813 Add nodepool integration test
This uses the python unittest framework to run a test similar to
the existing zuul-nodepool unit test, but expecting an actual
running nodepool.

Change-Id: I769e1421c146cc3545dce606487c8c72e3d3a4c5
6 years ago
James E. Blair 2a8e0fa5f1 Move tests into test/unit
This makes room for a sibling directory for nodepool functional tests.

Change-Id: Iace94d313edb04192ac23a533ed967f076410980
6 years ago
James E. Blair ac8df45bc5 Add nodepool integration test script
Change-Id: I197df80280d4454de7566549930fa6e6348e65c4
Depends-On: Icc58521f520f719f24f59132c424b3a71432285f
6 years ago
Jenkins 287d092c96 Merge "Add Drivers to documentation" into feature/zuulv3 6 years ago
Jenkins 473bd3cef6 Merge "Separate driver interfaces and make abstract" into feature/zuulv3 6 years ago
Jenkins 4ec7ca8e9b Merge "Re-enable zuultrigger test" into feature/zuulv3 6 years ago
Jenkins 9672473d87 Merge "Reorganize connections into drivers" into feature/zuulv3 6 years ago
James E. Blair e4de4f44dd Add Drivers to documentation
Re-organize the "internals" section into a "Developer's Guide" with
sub-pages and add the drivers documentation as one of them.

Change-Id: Ibdf0287a7a077dfeaac6c63d1c8beca8d5f3f4be
6 years ago
James E. Blair 2f001c05a1 Separate driver interfaces and make abstract
Separate the 4 driver interfaces into abstract interface classes,
and also add an abstract driver class.  Make the existing driver
implementations inherit from these as appropriate.

This should help clearly express which methods a given driver
needs to implement.

Change-Id: I918d35eced7b0581a65d53d2ebcd7124f3b3f0a6
6 years ago
James E. Blair a80ef4e7c6 Re-enable zuultrigger test
This gets both running again, but the second fails because of a
behavior change in v3 which should be addressed separately.

Story: 2000773
Task: 3422

Change-Id: I3d0e7268304e97fa611f70706e15212952aa3c62
6 years ago
James E. Blair e511d2f6c4 Reorganize connections into drivers
This change, while substantial, is mostly organizational.
Currently, connections, sources, triggers, and reporters are
discrete concepts, and yet are related by virtue of the fact that
the ConnectionRegistry is used to instantiate each of them.  The
method used to instantiate them is called "_getDriver", in
recognition that behind each "trigger", etc., which appears in
the config file, there is a class in the zuul.trigger hierarchy
implementing the driver for that trigger.  Connections also
specify a "driver" in the config file.

In this change, we redefine a "driver" as a single class that
organizes related connections, sources, triggers and reporters.

The connection, source, trigger, and reporter interfaces still
exist.  A driver class is responsible for indicating which of
those interfaces it supports and instantiating them when asked to
do so.

Zuul instantiates a single instance of each driver class it knows
about (currently hardcoded, but in the future, we will be able to
easily ask entrypoints for these).  That instance will be
retained for the life of the Zuul server process.

When Zuul is (re-)configured, it asks the driver instances to
create new connection, source, trigger, reporter instances as
necessary.  For instance, a user may specify a connection that
uses the "gerrit" driver, and the ConnectionRegistry would call
getConnection() on the Gerrit driver instance.

This is done for two reasons: first, it allows us to organize all
of the code related to interfacing with an external system
together.  All of the existing connection, source, trigger, and
reporter classes are moved as follows:

  zuul.connection.FOO -> zuul.driver.FOO.FOOconnection
  zuul.source.FOO -> zuul.driver.FOO.FOOsource
  zuul.trigger.FOO -> zuul.driver.FOO.FOOtrigger
  zuul.reporter.FOO -> zuul.driver.FOO.FOOreporter

For instance, all of the code related to interfacing with Gerrit
is now is zuul.driver.gerrit.

Second, the addition of a single, long-lived object associated
with each of these systems allows us to better support some types
of interfaces.  For instance, the Zuul trigger maintains a list
of events it is required to emit -- this list relates to a tenant
as a whole rather than individual pipelines or triggers.  The
timer trigger maintains a single scheduler instance for all
tenants, but must be able to add or remove cron jobs based on an
individual tenant being reconfigured.  The global driver instance
for each of these can be used to accomplish this.

As a result of using the driver interface to create new
connection, source, trigger and reporter instances, the
connection setup in ConnectionRegistry is much simpler, and can
easily be extended with entrypoints in the future.

The existing tests of connections, sources, triggers, and
reporters which only tested that they could be instantiated and
have names have been removed, as there are functional tests which
cover them.

Change-Id: Ib2f7297d81f7a003de48f799dc1b09e82d4894bc
6 years ago
Jesse Keating 2dfc651202 Correct comments in requirement test cases
The comments were misleading which led to a lot of head scratching. This
corrects the comments to reflect the realities of the test.

Change-Id: Id83243140558fb76e91f83843da8285fd17f384b
6 years ago
Jenkins 7f238a2991 Merge "Spell NodeSet consistently" into feature/zuulv3 6 years ago
Jenkins aa62bc204c Merge "Fix copyright date in test_nodepool" into feature/zuulv3 6 years ago
Jenkins fa8e36a8b4 Merge "Handle nodepool allocation failure" into feature/zuulv3 6 years ago
Jenkins 0338b76917 Merge "Use constants for state names" into feature/zuulv3 6 years ago
Jenkins 09d6e4a9e0 Merge "Verify nodes and requests are not leaked" into feature/zuulv3 6 years ago
Jenkins 808ee14a91 Merge "Remove excess printing from stats test" into feature/zuulv3 6 years ago
Jenkins cb3e13bc22 Merge "Remove unused clasess from" into feature/zuulv3 6 years ago
Jenkins 5c63371b48 Merge "Cancel/return nodepool requests on job cancel" into feature/zuulv3 6 years ago
Jenkins 932f49b7f9 Merge "Return nodes after use" into feature/zuulv3 6 years ago
Jenkins 8b43b722f3 Merge "Mark nodes as 'in-use' before launching jobs" into feature/zuulv3 6 years ago
Jenkins 379ac33d2f Merge "Lock nodes when nodepool request is fulfilled" into feature/zuulv3 6 years ago
Jenkins 2597ff7777 Merge "Copy nodeset when making node requests" into feature/zuulv3 6 years ago
Jenkins 2874abe908 Merge "Re-submit node requests on ZooKeeper disconnect" into feature/zuulv3 6 years ago
Jenkins 67a125ff27 Merge "Add nodepool test class" into feature/zuulv3 6 years ago
James E. Blair 1511bc38a3 Spell NodeSet consistently
It is intended to match the existing pattern used by BuildSet.
Usage is currently evenly split between NodeSet and Nodeset.

Change-Id: Iab96ed90a9ada0cb4709c0bb8b04923ab0c765e8
6 years ago
Jenkins d733484924 Merge "Add a test printHistory function" into feature/zuulv3 6 years ago
Jenkins ece8e15d89 Merge "Improve test output" into feature/zuulv3 6 years ago
Jenkins ca43352365 Merge "Fix race condition in test_success_url" into feature/zuulv3 6 years ago
Jenkins 869456c297 Merge "Add FakeNodepool test fixture" into feature/zuulv3 6 years ago
James E. Blair 15d91cc388 Fix copyright date in test_nodepool
This file was recently added (by me) with the wrong copyright date.

Change-Id: I39ed8ff1b29c48d2079338691a00ef6b4153b27b
6 years ago
Jenkins edfe0d0a30 Merge "Add Zookeeper to tests" into feature/zuulv3 6 years ago
Jenkins f84b1c232d Merge "Fix test_rerun_on_abort" into feature/zuulv3 6 years ago
James E. Blair 6ab79e0637 Handle nodepool allocation failure
When a request is either fulfilled or failed, pass it through to
the scheduler which will accept the request (which means deleting
it in the case of a failure) and pass it on to the pipeline manager
which will set the result of the requesting job to NODE_FAILURE
and cause any sub-jobs to be SKIPPED.

Adjust the request algorithm to only request nodes for jobs that
are ready to run.  The current behavior requests all jobs for a
build set asap, but that has two downsides: it may request and
return nodes more aggressively than necessary (if you have chosen
to create a job tree, you *probably* don't want to tie up nodes
until they are actually needed).  However, that's a grey area,
and we may want to adjust or make that behavior configurable later.
More pressing here is that it makes the logic of when to return
nodes *very* complicated (since SKIPPED jobs are represented by
fake builds, there is no good opportunity to return their nodes).

This seems like a good solution for now, and if we want to make
the node request behavior more aggressive in the future, we can
work out a better model for knowing when to return nodes.

Change-Id: Ideab6eb5794a01d5c2b70cb87d02d61bb3d41cce
6 years ago
James E. Blair 803e94f986 Use constants for state names
Change-Id: I7f471799b7c04354361914a4bbde7aea2b9449a8
6 years ago
James E. Blair e18d460e47 Verify nodes and requests are not leaked
Check that at the end of every test, there are no outstanding
nodepool requests and no locked nodes.

Move final state assertions into the tearDown method so that
they run right after the end of the test but before any
cleanup handlers are called (which can interfere with the
assertion checking by, say, deleting the zookeeper tree we
are trying to check).  Move the cleanup in test_webapp to
tearDown so that it ends the paused job that the tests in
that class use before the assertion check.

Fix some bugs uncovered by this testing:

* Two typos.
* When we re-launch a job, we need a new nodeset, so make sure
  to remove the nodeset from the buildset after the build
  completes if we are going to retry the build.
* Always report build results to the scheduler even for non-current
  buildsets so that it can return used nodes for aborted builds.
* Have the scheduler return the nodeset for a completed build rather
  than the pipeline manager to avoid the edge case where a build
  result is returned after a configuration that removes the pipeline
  (and therefore, there is no longer a manager to return the nodeset).
* When canceling jobs, return nodesets for any jobs which do not yet
  have builds (such as jobs which have nodes but have not yet
* Return nodes for skipped jobs.

Normalize the debug messages in

Change-Id: I32f6807ac95034fc2636993824f4a45ffe7c59d8
6 years ago
James E. Blair ce001e11a8 Remove excess printing from stats test
This isn't necessary and makes the output of this test noisy.

Change-Id: I214d40fada9567ac6b9cee5ff9cdc748a472cbbb
6 years ago
James E. Blair 20c39d5c5b Remove unused clasess from
We ended up using classes from instead to better interact
with the rest of Zuul, so remove these.

Change-Id: I4fa4a06b27d9ef6cc7f7878f29a92aafd7ffe9d1
6 years ago
James E. Blair 01695c35e5 Cancel/return nodepool requests on job cancel
When canceling a running or requested job, cancel outstanding
nodepool requests or return unused nodes.

Change-Id: I77f8869b9d751ccd6c9f398ed03ef5ac482cc204
6 years ago
James E. Blair 62295d322e Return nodes after use
This returns the nodes after use (or in some cases, before use)
and unlocks them.

Change-Id: I84e998a6099217f0a0924caf1872f33dce28ba01
6 years ago
James E. Blair cacdf2b659 Mark nodes as 'in-use' before launching jobs
While we immediately lock a node given to us by nodepool, we delay
setting the node to 'in-use' until we actually request that the job
be launched so that if we end up canceling the job before it is
run, we might return the node unused to nodepool.

Change-Id: I2d2c0f9cdb4c199f2ed309e7b0cfc62e071037fa
6 years ago
James E. Blair a38c28efa3 Lock nodes when nodepool request is fulfilled
This is continuing work on implementing the Zuul<->Nodepool protocol
from the Zuulv3 spec.

Change-Id: Ic8477e607fd09b85a37f47cbee7da905c017c534
6 years ago
James E. Blair cbf4367837 Copy nodeset when making node requests
Nodesets are collections of nodes, but they, and the Node objects
they contain, have a dual role representing both the configuration
of a job and the actual nodes returned from nodepool.  So that
multiple jobs sharing the same nodeset configuration don't end up
sharing the same actual nodes, create a copy of a job's nodeset
for use in interacting with nodepool/zk.

Change-Id: I83f503ba22fc3f92b8c90b15ccfb6b07dc3c4709
6 years ago
James E. Blair 15be0e1e11 Re-submit node requests on ZooKeeper disconnect
Change-Id: I689bf812c713fa6f5f37958b7001b0d5fb0a254b
6 years ago
James E. Blair 3897a13743 Add nodepool test class
To exercise the Zuul<->Nodepool interface.

Change-Id: Ie57146e2ab7221d4abf9021055d601b67a6efedf
7 years ago
James E. Blair 6ac368c57b Add a test printHistory function
I frequently add something like this ad-hoc when debugging a test.
Make it a convenient function that's easy to add when needed, and
also run it at the completion of every test so a developer can
easily survey the logs to see what happened.

Change-Id: I3d3810f51245d92855f086b875edfd52bdd86983
7 years ago
James E. Blair 10fc1eb487 Improve test output
* When a test timeout occurs, output the state debug information at
  error level so that it shows up in all logs.
* Add some more info to that output.
* Further restrict the (often not useful) chatty gear logs by default.

Change-Id: Ib275441172c5b1598593d0931cef0168d02e521d
7 years ago
James E. Blair e75768068b Fix race condition in test_success_url
The builds may run in any order.

Change-Id: I705be7a32d0534ae06ea4778bf2bc76d630388d8
7 years ago
James E. Blair dce6ceac8e Add FakeNodepool test fixture
Add a fake nodepool that immediately successfully fulfills all
requests, but actually uses the Nodepool ZooKeeper API.

Update the Zuul Nodepool facade to use the Nodepool ZooKeeper API.

Change-Id: If7859f0c6531439c3be38cc6ca6b699b3b5eade2
7 years ago