When jobs share gates, Zuul has a nice feature which is to creates reference of the Change in every involved repositories. That was sadly undocumented. Reference: James E. Blair on openstack-infra mailing list: http://lists.openstack.org/pipermail/openstack-infra/2014-January/000605.html Change-Id: Ia3bd640f6a90e12192af1686547a5f4146192dbe
8.7 KiB
- title
-
Project Gating
Project Gating
Traditionally, many software development projects merge changes from developers into the repository, and then identify regressions resulting from those changes (perhaps by running a test suite with a continuous integration system such as Jenkins), followed by more patches to fix those bugs. When the mainline of development is broken, it can be very frustrating for developers and can cause lost productivity, particularly so when the number of contributors or contributions is large.
The process of gating attempts to prevent changes that introduce regressions from being merged. This keeps the mainline of development open and working for all developers, and only when a change is confirmed to work without disruption is it merged.
Many projects practice an informal method of gating where developers with mainline commit access ensure that a test suite runs before merging a change. With more developers, more changes, and more comprehensive test suites, that process does not scale very well, and is not the best use of a developer's time. Zuul can help automate this process, with a particular emphasis on ensuring large numbers of changes are tested correctly.
Zuul was designed to handle the workflow of the OpenStack project, but can be used with any project.
Testing in parallel
A particular focus of Zuul is ensuring correctly ordered testing of changes in parallel. A gating system should always test each change applied to the tip of the branch exactly as it is going to be merged. A simple way to do that would be to test one change at a time, and merge it only if it passes tests. That works very well, but if changes take a long time to test, developers may have to wait a long time for their changes to make it into the repository. With some projects, it may take hours to test changes, and it is easy for developers to create changes at a rate faster than they can be tested and merged.
Zuul's DependentPipelineManager allows for parallel execution of test jobs for gating while ensuring changes are tested correctly, exactly as if they had been tested one at a time. It does this by performing speculative execution of test jobs; it assumes that all jobs will succeed and tests them in parallel accordingly. If they do succeed, they can all be merged. However, if one fails, then changes that were expecting it to succeed are re-tested without the failed change. In the best case, as many changes as execution contexts are available may be tested in parallel and merged at once. In the worst case, changes are tested one at a time (as each subsequent change fails, changes behind it start again). In practice, the OpenStack project observes something closer to the best case.
For example, if a core developer approves five changes in rapid succession:
A, B, C, D, E
Zuul queues those changes in the order they were approved, and notes that each subsequent change depends on the one ahead of it merging:
- blockdiag foo {
-
node_width = 40; span_width = 40; A <- B <- C <- D <- E;
}
Zuul then starts immediately testing all of the changes in parallel. But in the case of changes that depend on others, it instructs the test system to include the changes ahead of it, with the assumption they pass. That means jobs testing change B include change A as well:
Jobs for A: merge change A, then test
Jobs for B: merge changes A and B, then test
Jobs for C: merge changes A, B and C, then test
Jobs for D: merge changes A, B, C and D, then test
Jobs for E: merge changes A, B, C, D and E, then test
Hence jobs triggered to tests A will only test A and ignore B, C, D:
- blockdiag foo {
-
node_width = 40; span_width = 40; master -> A -> B -> C -> D -> E; group jobs_for_A { label = "Merged changes for A"; master -> A; } group ignored_to_test_A { label = "Ignored changes"; color = "lightgray"; B -> C -> D -> E; }
}
The jobs for E would include the whole dependency chain: A, B, C, D, and E. E will be tested assuming A, B, C, and D passed:
- blockdiag foo {
-
node_width = 40; span_width = 40; group jobs_for_E { label = "Merged changes for E"; master -> A -> B -> C -> D -> E; }
}
If changes A and B pass tests (green), and C, D, and E fail (red):
- blockdiag foo {
-
node_width = 40; span_width = 40;
A [color = lightgreen]; B [color = lightgreen]; C [color = pink]; D [color = pink]; E [color = pink];
master <- A <- B <- C <- D <- E;
}
Zuul will merge change A followed by change B, leaving this queue:
- blockdiag foo {
-
node_width = 40; span_width = 40;
C [color = pink]; D [color = pink]; E [color = pink];
C <- D <- E;
}
Since D was dependent on C, it is not clear whether D's failure is the result of a defect in D or C:
- blockdiag foo {
-
node_width = 40; span_width = 40;
C [color = pink]; D [label = "Dn?"]; E [label = "En?"];
C <- D <- E;
}
Since C failed, Zuul will report its failure and drop C from the queue, keeping D and E:
- blockdiag foo {
-
node_width = 40; span_width = 40;
D [label = "Dn?"]; E [label = "En?"];
D <- E;
}
This queue is the same as if two new changes had just arrived, so Zuul starts the process again testing D against the tip of the branch, and E against D:
- blockdiag foo {
-
node_width = 40; span_width = 40; master -> D -> E; group jobs_for_D { label = "Merged changes for D"; master -> D; } group ignored_to_test_D { label = "Skip"; color = "lightgray"; E; }
}
- blockdiag foo {
-
node_width = 40; span_width = 40; group jobs_for_E { label = "Merged changes for E"; master -> D -> E; }
}
Cross projects dependencies
When your projects are closely coupled together, you want to make sure changes entering the gate are going to be tested with the version of other projects currently enqueued in the gate (since they will eventually be merged and might introduce breaking features).
Such dependencies can be defined in Zuul configuration by registering
a job in a DependentPipeline of several projects. Whenever a change
enters such a pipeline, it will create references for the other projects
as well. As an example, given a main project acme
and a
plugin plugin
you can define a job acme-tests
which should be run for both projects:
pipelines:
- name: gate
manager: DependentPipelineManager
projects::
- name: acme
gate:
- acme-tests
- name: plugin
gate:
- acme-tests # Register job again
Whenever a change enters the gate
pipeline queue, Zuul
creates a reference for it. For each subsequent change, an additional
reference is created for the changes ahead in the queue. As a result,
you will always be able to fetch the future state of your project
dependencies for each change in the queue.
Based on the pipeline and project definitions above, three changes
are inserted in the gate
pipeline with the associated
references:
Change Project Branch Zuul Ref. Change 1 acme master master/Z1 Change 2 plugin stable stable/Z2 Change 3 plugin master master/Z3
Since the changes enter a DependentPipelineManager pipeline, Zuul creates additional references:
Change Project Zuul Ref. Description 1 acme master/Z1 acme master + change 1 ------ ------- --------- ----------------------------- 2 acme master/Z2 acme master + change 1 2 plugin stable/Z2 plugin stable + change 2 ------ ------- --------- ----------------------------- 3 acme master/Z3 acme master + change 1 3 plugin stable/Z3 plugin stable + change 2 3 plugin master/Z3 plugin master + change 3
In order to test change 3, you would clone both repositories and
simply fetch the Z3 reference for each combination of project/branch you
are interested in testing. For example, you could fetch
acme
with master/Z3 and plugin
with master/Z3
and thus have acme
with change 1 applied as the expected
state for when Change 3 would merge. When your job fetches several
repositories without changes ahead in the queue, they may not have a Z
reference in which case you can just check out the branch.