Each scheduler in tests needs its own connection registries.
This change only affects tests.
Change-Id: I2ad188cf5a72c46f7486c23ab653ff574123308b
Story: 2007192
The boolean "held" attribute is set to True if a build triggered
a autohold request, and its nodeset was held.
Allow filtering builds by "held" status.
Change-Id: I6517764c534f3ba8112094177aefbaa144abafae
Currently the mqtt reporter uses the report url as log_url. This is
fine as long as report-build-page is disabled. As soon as
report-build-page is enabled on a tenant it reports the url to the
result page of the build. As mqtt is meant to be consumed by machines
this breaks e.g. log post processing.
Fix this by reporting the real log url as log_url and add the field
web_url for use cases where really the human url is required.
This fixes also a wrong indentation in the mqtt driver documentation,
resulting in all buildset.builds.* attributes being listed as buildset.*
attributes.
Change-Id: I91ce93a7000ddd0d70ce504b70742262d8239a8f
Since we added those to the MQTT reporter, we should also store them in
the SQL database. They are stored in the zuul_build table and can be
identified via the new "final" column which is set to False for those
builds (and True for all others).
The final flag can also be used in the API to specifically filter for
those builds or remove them from the result. By default, no builds are
filtered out.
The buildset API includes these retried builds under a dedicated
'retry_builds' key to not mix them up with the final builds. Thus, the
JSON format is equally to the one the MQTT reporter uses.
For the migration of the SQL database, all existing builds will be set
to final.
We could also provide a filter mechanism via the web UI, but that should
be part of a separate change (e.g. add a checkbox next to the search
filter saying "Show retried builds").
Change-Id: I5960df8a997c7ab81a07b9bd8631c14dbe22b8ab
To improve consistency and remove `self.sched` and
use `self.scheds.first.sched` in all tests.
This change only touches tests.
Change-Id: I0a8c3f8ad634d1d88f3c68a11e4d5587993f4a0d
Story: 2007192
As a preparation for scale-out-scheduler the scheduler in tests
were extracted in order to start multiple instances in a previous
change.
This change continues on by introducing a manager to create
additional scheduler instances and the ability to call certain
methods on some or all of those instances.
This change only touches tests.
Change-Id: Ia05a7221f19bad97de1176239c075b8fc9dab7e5
Story: 2007192
This should be stored in the SQL database so that the build page
can present the reason why a particular build failed, instead of
just the result "ERROR".
Change-Id: I4dd25546e27b8d3f3a4e049f9980082a3622079f
Having the zuul event id available in the database and also in the build
and buildset detail page makes debugging a lot easier.
Change-Id: Ia1e4aaf50fb28bb27cbcfcfc3b5a92bba88fc85e
When trying to trace logs of builds it is often useful to search for log
messages via the event id of a specific build.
The event id is printed in (nearly) all log messages but is not provided
by the MQTT reporter, so one has to look it up first based on the build
id. To circumvent this extra step and make searching the logs more
straight forward, this patch makes sure the event id is provided in the
JSON message by the MQTT reporter.
Change-Id: I908dd7eca250825eed97bf8261fd33b69cc5f543
The build page needs the actual log_url returned by the job (without
any modification from success_url or failure_url) in order to create
links to the log site.
The reported success/failure URL isn't as important in this context,
and I suspect their days are numbered once we require the SQL
reporter and report the link to the build page instead. So we just
won't record those in the DB. If we feel that they are important,
we can add new columns for them.
Also, ensure it has a trailing / so that API users (including the JS
pages) are not annoyed by inconsistent data.
Change-Id: I5ea98158d204ae17280c4bf5921e2edf4483cf0a
The query for builds-joined-with-buildsets is currently optimized
for the case where little additional filtering is performed. E.g.,
the case where a user browses to the builds tab and does not enter
any search terms. In that case, mysql needs a hint supplied in
order to choose the right index.
When search terms are entered which can, due to the presense of
other indexes, greatly reduce the working set, it's better to let
the query planner off the leash and it will make the right choices.
This change stops adding the hint in the cases where a user supplies
a search term that matches one of the indexes on the build or
buildset table (notable exception: job_name because it is difficult
to generalize about that one).
It also adds an additional index for build and buildset uuids,
which should provide excellent performance when searching for
only those terms.
Change-Id: I0277be8cc4ba7555c5e6a9a7eb3eed988a24469c
Sometimes, e.g. during reconfiguration, it can take quite some time
between the trigger event and when a change is enqueued.
This change allows tracking the time it takes from receiving the event
until it is processed by the scheduler.
Change-Id: I347acf56bc8d7671d96f6be444c71902563684be
We are using the MQTT reporter to create metrics about our job runtimes,
queueing times, etc. Knowing if a job has a dependency or not is a
valuable information for those calculations that was missing so far.
Change-Id: I4133487f98d458be65495b71c271316360dd982b
Reporting the execute_time helps to distinguish between the preparation
phase and the actual execution phase.
Change-Id: Ib578efb39cede37996c3516e1a9a251d6ed8a4b0
Having the enqueue and report timestamp allows for better reporting
regarding the livecycle of a change.
Change-Id: Ia3915ea35853f007181d5660c361417175c29507
This change fixes reporting of Tag event where the branch attribute was
expected:
File "zuul/scheduler.py", line 383, in onBuildCompleted
branchname = (build.build_set.item.change.branch.
AttributeError: 'Tag' object has no attribute 'branch'
File "zuul/driver/sql/sqlreporter.py", line 55, in report
branch=item.change.branch,
AttributeError: 'Tag' object has no attribute 'branch'
Change-Id: I5dbe4663c4d1e341b08a32eedbbcfb775330e881
Some deployments use postgres as database backend. In order to have
official support for that we need to add tests for it.
Change-Id: I36408c27ccb36fd0826d97f34d98b211ee075f4e
Depends-On: https://review.openstack.org/535718
In some database engines the index names are scoped within the
database instead of the table name. Thus we need to also prefix the
index names. This change adds a migration which renames already
existing indexes.
Change-Id: I08b6ba4d54a53a6f8d4ef1ed113cb24946b9f3e9
The previous change I26ff56159c2710af1515955d27bf4e9ebfcf76e9 had a
small bug which rendered it basically a noop. Further it missed some
important parts and test changes.
Change-Id: I6cfb8a6af05e589140c2f7c8b1d7228f6d4d8fcb
In some environments zuul operators may have to rely on external
database providers. In this case it can be cumbersome to get extra
databases for each test environment. Adding an optional prefix to the
table names makes it possible to gracefully run several zuul
deployments against the same database and ensure they're still
isolated against each other.
Change-Id: Ib9948d6d74f4dc2453738f5d441e233e39e7f944
We have deployed our Gerrit with cgit so the old gitweb urls provided by
Zuul's gerrit connection no longer work. Add in a new config option on
Gerrit connections to specify a url template string which we can modify
to point at our cgit instance. This should in theory also support github
users too.
The default is to continue pointing at Gerrit's built in gitweb
instance.
Change-Id: I91d77e309cfeea0e90a85f926aca9b8c347b0385
* Always send a build start event for noop jobs so that we get
start and end times for them
* Handle builds without start or end times in the sql reporter.
These should no longer include noop builds, but may still include
SKIPPED builds.
Test both.
Change-Id: I73eb6bda482ebb515d231492c0769d49cf6ff28a
This change adds oldrev/newrev column to the zuul_buildset sql table to
be able to query post job for a specific commit.
Change-Id: Ic9c0b260560123d080fa47c47a76fcfa31eb3607
This change adds a ref_url column to the zuul_buildset sql table to be able
to render change/ref url when querying buildset.
Change-Id: I91a9a3e5e3b362885e36fa0993b07c750adc69d3
After upgrading Gerrit to 2.13 our gate stopped working. The reason
for this is that after a successful gate run zuul does something like
'gerrit review --label verified=2 --submit'. The verified label in
Gerrit by default is configured as 'Verified'. The newer version of
gerrit behaves different now. It accepts the +2 vote on verified but
doesn't submit the patch anymore if the casing is not correct. This
forces us to specify the label in the same casing as gerrit
expects. In that case the tolower() in canMerge prevents the patch
from entering the gate.
In order to avoid confusion and be consistent, avoid any case
conversions and use the labels exactly as defined in Gerrit.
Note that this patch requires changes to the pipelines such that the
labels are spelled exactly as defined in Gerrit.
Change-Id: I9713a075e07b268e4f2620c0862c128158283c7c
This change adds a tenant column to the zuul_buildset sql table to be able
to filter results based on tenant.
Change-Id: I4eec8e9a32883c54da953b7279e0f8a8c86b943b
zuul now provides socket-based console streaming, which is super cool.
In order to have jenkins parity with web streaming, we need to provide a
websocket (javascript in browsers can't really connect to random ports
on servers)
After surveying the existing python websocket options, basically all of
them are based around twisted, eventlet, gevent or asyncio. It's not
just a thing we can easily deal with from our current webob/paste
structure, because it is a change to the fundamental HTTP handling.
While we could write our own websocket server implementation that was
threaded like the rest of zuul, that's a pretty giant amount of work.
Instead, we can run an async-based server that's just for the
websockets, so that we're not all of a sudden putting async code into
the rest of zuul and winding up frankensteined. Since this is new code,
using asyncio and python3 seems like an excellent starting place.
aiohttp supports running a websocket server in a thread. It also
supports doing other HTTP/REST calls, so by going aiohttp we can set
ourselves up for a single answer for the HTTP tier.
In order to keep us from being an open socket relay, we'll expect two
parameters as the first message on the websocket - what's the zuul build
uuid, and what log file do we want to stream. (the second thing,
multiple log files, isn't supported yet by the rest of zuul, but one can
imagine a future where we'd like to support that too, so it's in the
protocol) The websocket server will then ask zuul over gearman for the
IP and port associated with the build and logfile and will start
streaming it to the socket.
Ultimately we'll want the status page to make links of the form:
/console.html?uuid=<uuid>&logfile=console.log
and we'll want to have apache map the websocket server to something like
/console.
Co-Authored-By: Monty Taylor <mordred@inaugust.com>
Change-Id: Idd0d3f9259e81fa9a60d7540664ce8d5ad2c298f
The new upcoming log streaming will be based on a protocol nearly
identical to finger. So much so, that it's actually finger. Isn't that
cool?
Change-Id: I51ef51ee236227e7816effe6683733ae3f29750a
When we have multiple sources in the same pipeline we can end up with a
pipeline trying to report to gerrit when the result came from a
different source. Only report to gerrit when the source came from
gerrit.
Change-Id: I26e92002517a86af0d5482fbc73c5dd5b308b363
Signed-off-by: Jamie Lennox <jamielennox@gmail.com>
Co-Authored-By: Tobias Henkel <tobias.henkel@bmw.de>
Change I453754052b75de3d4fac266dc73921c7ce9c075f had the effect that
no source connection was configured in the merger. This can be fixed
by changing the type check from issubclass to isinstance. Further also
the zuul and timer drivers don't need to be configured by the merger.
This also adds a test case for that.
Change-Id: If437123c3e8a0d63c5f852619e214cefa0892ce3
To avoid confusion with nodepool-launcher, we've decided to rename
zuul-launcher to zuul-executor.
Change-Id: I7d03cf0f0093400f4ba2e4beb1c92694224a3e8c
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Re-enable the test with multiple gerrits affecting the same projects.
Extends the test a bit as the last test never showed actual interaction
between the two builds.
Story: 2000773
Task: 3412
Change-Id: Ia8621ea481057cc7602dd01ef149b2ebfbf0aa58
This change, while substantial, is mostly organizational.
Currently, connections, sources, triggers, and reporters are
discrete concepts, and yet are related by virtue of the fact that
the ConnectionRegistry is used to instantiate each of them. The
method used to instantiate them is called "_getDriver", in
recognition that behind each "trigger", etc., which appears in
the config file, there is a class in the zuul.trigger hierarchy
implementing the driver for that trigger. Connections also
specify a "driver" in the config file.
In this change, we redefine a "driver" as a single class that
organizes related connections, sources, triggers and reporters.
The connection, source, trigger, and reporter interfaces still
exist. A driver class is responsible for indicating which of
those interfaces it supports and instantiating them when asked to
do so.
Zuul instantiates a single instance of each driver class it knows
about (currently hardcoded, but in the future, we will be able to
easily ask entrypoints for these). That instance will be
retained for the life of the Zuul server process.
When Zuul is (re-)configured, it asks the driver instances to
create new connection, source, trigger, reporter instances as
necessary. For instance, a user may specify a connection that
uses the "gerrit" driver, and the ConnectionRegistry would call
getConnection() on the Gerrit driver instance.
This is done for two reasons: first, it allows us to organize all
of the code related to interfacing with an external system
together. All of the existing connection, source, trigger, and
reporter classes are moved as follows:
zuul.connection.FOO -> zuul.driver.FOO.FOOconnection
zuul.source.FOO -> zuul.driver.FOO.FOOsource
zuul.trigger.FOO -> zuul.driver.FOO.FOOtrigger
zuul.reporter.FOO -> zuul.driver.FOO.FOOreporter
For instance, all of the code related to interfacing with Gerrit
is now is zuul.driver.gerrit.
Second, the addition of a single, long-lived object associated
with each of these systems allows us to better support some types
of interfaces. For instance, the Zuul trigger maintains a list
of events it is required to emit -- this list relates to a tenant
as a whole rather than individual pipelines or triggers. The
timer trigger maintains a single scheduler instance for all
tenants, but must be able to add or remove cron jobs based on an
individual tenant being reconfigured. The global driver instance
for each of these can be used to accomplish this.
As a result of using the driver interface to create new
connection, source, trigger and reporter instances, the
connection setup in ConnectionRegistry is much simpler, and can
easily be extended with entrypoints in the future.
The existing tests of connections, sources, triggers, and
reporters which only tested that they could be instantiated and
have names have been removed, as there are functional tests which
cover them.
Change-Id: Ib2f7297d81f7a003de48f799dc1b09e82d4894bc
The addFailTest method needed to be updated to catch up with some
test infrastructure changes. Rather than consulting the merger's
git repo for the ZUUL_REF to find out if a job which is supposed to
fail is present for a given build, use the new FakeBuild.hasChanges
method to inspect the launcher's prepared git repo.
Also rename the method as 'Job' is more descriptive than 'Test' and
document it.
Change-Id: I3224b8a01d49cfa06b799a8028c1bf0d455d25b1
Use the newer form of spelled-out category names in tests since
that's what we're writing in config files now.
Change-Id: Ib679e54b53131280956cbda20b84f5602a4953c8
When connections are set up in tests, multiple Gerrit connections which
are configured to connect to the same fake Gerrit server share a change
database so that changes sent to Gerrit via one connection are reflected
back to Zuul on another. They also share an event queue so that events
injected on one are seen by another.
Unfortunately, that part doesn't work, and in fact, events are only seen
by one of the gerrit connections. This happens to work since it doesn't
matter which gerrit connection receives an event, which is why we haven't
noticed the problem in tests.
Where we do see the problem in Zuulv3 is in shutting down the connections.
When a GerritConnection is stopped, a sentinal object (None) is added to
the event queue. When the GerritConnection gets an event from the queue,
it first checks whether it has stopped before processing that event.
Because in tests (but not in production) multiple GerritConnections share
an event queue, the connection that adds the None object to the queue
may not be the one that receives it, which causes the test to raise an
exception and not stop correctly.
We did not notice this in v2 because the order in which the Queue.Queue
class decides to awaken a thread is deterministic enough that the thread
which submitted the sentinel was always the one that received it. In
v3, the thread order is sufficiently different that the thread for the
*other* connection is reliably the one which receives it.
To correct this, stop using a shared queue between the differing
GerritConnection objects, and instead add a helper method to the testcase
class which will add an event to every connection for a given server.
Change-Id: Idd3238f5ab8f5e09e295c0fa028e140c089a2a3f
Test reporting back to gerrit as a different user.
Test multiple gerrit instances.
Squashed with: Assign fake connections to test class dynamically
As multiple connections can be configured for testing purposes, it makes
more sense to be able to access them all comfortably.
Drawback of this change is, that the dynamicism makes the code less
readable and less obvious.
Previously: If50567dd5a087e9fe46cd4e30f9e0562cda794d7
Co-Authored-By: Jan Hruban <jan.hruban@gooddata.com>
Change-Id: I4652799c8b9a626bd2f7f6996d09c95da2953936
Jan Kubovy
vass
Matthieu Huin
Tobias Henkel
Benedikt Loeffler
James E. Blair
Simon Westphahl
Benjamin Schanzel
Felix Schmidt
Tristan Cacqueray
Clark Boylan
Monty Taylor
Tobias Henkel
Jamie Lennox
Paul Belanger
Morgan Fainberg
James E. Blair
Joshua Hesketh