It looks like we need 2GB of ram; mention that as a requirement.
Also, add a "sudo apt-get update" command for Ubuntu/Debian users;
with this, a user should get a working system from a pristine VM
using only and exactly the commands specified.
Change-Id: Ife48afbd6326b8d8e774b4227fbe6523d3aed81e
We missed some cases where builds might be aborted and the results
not reported to the database. This updates the test framework to
assert that tests end with no open builds or buildsets in the
database.
To fix the actual issues, we need to report some build completions
in the scheduler instead of the pipeline manager. So to do that,
we grab a SQL reporter object when initializing the scheduler
(and we therefore no longer need to do so when initializing the
pipeline manager). The SQL reporter isn't like the rest of the
reporters -- it isn't pipeline specific, so a single global instance
is fine.
Finally, initializing the SQL reporter during scheduler init had
some conflicts with a unit test which tested that the merger could
load "source-only" connections. That test actually verified that
the *scheduler* loaded source-only connections. So to correct this,
it now verifies that the executor (which has a merger and is under
the same constraints as the merger for this purpose) can do so. We
no longer need the source_only flag in tests.
Change-Id: I1a983dcc9f4e5282c11af23813a4ca1c0f8e9d9d
We don't want tenant validation to change the production config in
Zookeeper. Therefore we use a temporary path in Zookeeper for the config
cache when validating tenants.
Change-Id: I41a22df69232d5551ab27aa014fecbc6a5d16a56
Since we are now explicitly alpha-sorting tenants for display
purposes and their internal ordering for iteration is undefined,
let's remove the OrderedDict constructors from the Abide so that
it doesn't look like we expect them to be ordered.
Change-Id: If8b0a1699331bbf63b2af4e8c3eeb8dbccb21606
This extracts the event data from the queue entries and stores
it in a separate location, sharded and compressed. This can
reduce ZK traffic and avoid disconnects with very large event
and result data.
We only shard when necessary, as sharding all events has a
significant performance impact.
Change-Id: I121399cd40f7bc5eeed899064198927efda2e485
The serialized nodeset is now supplied as a build parameter,
which makes the synthetic hosts and groups parameters which are
derived from it redundant.
Update the executor to rely entirely on the deserialized nodeset.
We also rename the method which creates the parameters since they
are not used for gearman any more.
A subsequent change can remove the hosts and nodes parameters.
Change-Id: Ied7f78c332485e5c66b5721c1007c25660d4238e
Since we no longer create a new abide instance during reconfigurations
and just replace the tenant instance, the tenants are not longer sorted
in the order they appear in the config but depending on which one was
recently reconfigured.
From a users perspective it makes sense that the list of tenants is
sorted alphabetically.
Change-Id: Ie9ae0700fd3928cc1f8f93d1de77537ce97518cd
Avoid modifying the abide's tenant dictionary in order to prevent
exceptions in other threads that are concurrently iterating over the
tenants dictionary.
2021-07-17 06:36:33,742 ERROR zuul.Scheduler: Error in periodic stats:
Traceback (most recent call last):
File "/opt/zuul/lib/python3.8/site-packages/zuul/scheduler.py", line 292, in runStats
self._runStats()
File "/opt/zuul/lib/python3.8/site-packages/zuul/scheduler.py", line 420, in _runStats
for tenant in self.abide.tenants.values():
RuntimeError: OrderedDict mutated during iteration
Change-Id: I497a8bbf2c9443aed90e6cb7cf0cd05adbf82b15
We're currently storing all of (trigger, result, management) events
in a single queue per tenant or per-pipeline. This means that in
order to iterate over just the management events, we need to
filter the list by name.
By splitting the event queues, we will be able to deal with smaller
numbers of events at a time, and can more quickly calculate event
queue length.
Change-Id: I2c01e51a6f9c55ba521d26416db16dfba068246d
We're currently recording a lot of NO_JOBS buildsets in the db,
and it's likely that no one is interested in that info. Instead,
only add a buildset entry if we know we're going to run jobs.
Change-Id: Ib89c3513a23908befaaea4f09933e846c6477aaa
The main purpose of the build and buildset listings is to find
completed builds, not in-progress builds (that's the status page),
so when we query the Zuul API, always set "completed=true".
In the future, it might be nice to list in-progress builds on these
pages, but that will require significantly more UI work to add
it to the filter bar.
Additionally, update the buildset page to show in-progress builds.
It currently doesn't because it assumes all non-final builds are
retry attemtps and therefore stores them under a secondary key.
That's no longer true, and additionally, completed retried builds
now have an explicit "RETRY" result which makes them easy for
users to distinguish.
It's more useful for a user to be able to see that a buildset is
still in progress and also the retried builds (now that we store
all of them), so this change returns all the builds of a buildset.
Change-Id: I83d89c2903e6fd0aaf67c66f61d106b2c5db3797
This adds a tri-state parameter to the build and buildset queries,
both in the internal API and via the web API. True means return
builds with results, False means only in-progress builds,
None (or omitted) means both.
Also render "In Progress" builds as such in the web UI.
Change-Id: Ib021e6a2c7338c08deae1aef4dbb5f0d9154daa0
We have a local unparsed branch cache on each scheduler and the global
config cache in Zookeeper. In order to check if the local unparsed
branch cache is up-to-date, we need to also store the logical timestamp
similar to the Zookeeper config cache.
Depending on the event that triggers (re-)loading of a tenant we must
make sure that those caches are considered valid or invalid correctly.
Instead of clearing the cache for a project(-branch) we will use the
event ltime to detected an outdated project-branch cache. This way
parallel tenant reconfigurations will not all clear out the cache in
Zookeeper even though it might be valid again because of a previously
processed reconfiguration.
There are four scenarios to consider when loading a tenant.
1. Processing a tenant reconfig event:
- The min. ltime for the changed project(-branches) will be set to
the event's ``zuul_event_ltime`` (to establish a happened-before
relation in respect to the config change). The min. ltime for all
other project-branches will be -1.
- Config for needed project-branch(es) is updated via cat job if the
cache is not valid (cache ltime < min. ltime).
- Cache in Zookeeper and local unparsed branch cache is updated. The
ltime of the cache will be the timestamp created shortly before
requesting the config via the mergers (only for outdated items).
2. Processing a full reconfiguration event:
- The min. ltime for all project-branches is given as the
``zuul_event_ltime`` of the reconfiguration event.
- Config for needed project-branch(es) is updated via cat job if the
cache is not valid (cache ltime < min. ltime). Otherwise the local
unparsed branch cache or the global config cache in Zookeper is
used.
- Cache in Zookeeper and local unparsed branch cache is updated, with
the ltime shortly before requesting the config via the mergers
(only for outdated items).
3. Processing a smart reconfiguration event:
- The min. ltime for all project-branches is given as -1 in order to
use cached data wherever possible.
- Config for new project-branch(es) is updated via cat job if the
project is not yet cached. Otherwise the local unparsed branch
cache or the global config cache in Zookeper is used.
- Cache in Zookeeper and local unparsed branch cache is updated, with
the ltime shortly before requesting the config via the mergers
(only for new items).
4. (Re-)loading a tenant due to a changed layout (happens after an event
according to one of the other scenarios was processed on another
scheduler):
- The min. ltime for all project-branches is given as -1 in order to
only use cached config.
- Local unparsed branch cache is updated if needed.
Change-Id: I45ec6dd794413870f85e4f9e9de8122eddce222e
Now that the Zuul web server and database are both required, provide
a consistent user experience by always reporting the build page.
This means that success-url and failure-url are no longer useful,
so remove them.
Update the quick-start to reflect that the build page is always
reported.
Change-Id: I4ff108df3917c9b6f44e2f5b0ccc4a7adbda1677
When we cancel a node request, we delete the request from ZK. We
might get the callback from ZK to update the node request object
(due to the delete event) in a seprate thread while the first thread
is between the lines where we delete the request and set the internal
flag indicating it was canceled.
That would cause the update callback to think that the request was
externally deleted (not by us) and resubmit it.
To correct this, set the internal canceled flag before performing the
ZK delete.
Change-Id: I1b4771b5840cb168b01939bd8590534ef618d878
In order to establish a proper happens-before relationship we need to
create the update ltimestamp before requesting the files from the
mergers. For that we will touch the /zuul/ltime node and use the
returned modified transaction id (mzxid).
Before we were using the mzxid of the files cache Znode. But since this
happend AFTER the cat job there is a small chance that the cache would
be considered valid later on even though the content of the repo has
changed in the meantime.
It would also be possible to use the zuul_event_ltime e.g. of the tenant
reconfiguration event for the update ltime. However, this is inefficient
for reconfigurations when a project is part of multiple tenants. In that
case we would trigger ~N cat jobs (where N is the number of tenants the
project is part of) since the cache would most likely be considered
outdated.
Change-Id: I9403bca6710c1c50460fcad3fd26ae48dc558ef9
In addition to the tenant read lock, schedulers need to also acquire an
exclusive pipeline lock. In case a pipeline is already locked by another
scheduler, it is skipped and processing will continue with the next
pipeline in the tenant.
Tenant management and trigger queues are protected by a queue specific
lock and need to acquire other locks depending on the processed event
(e.g. tenant write lock for tenant reconfigurations).
This also fixes an issue with the locked context manager when an
exception is thrown inside the context. In this case the lock was not
unlocked.
Change-Id: I2e3f0feca7119908d1d13010d2e4de791f95137e
This puts the NodesProvisionedEvents into ZooKeeper. With this, all
result events are now in ZooKeeper.
To make the NodesProvisionedEvent serializable, we cannot store the
whole NodeRequest object in the event anymore. Instead we are using the
request.id, job name and the buildset UUID to look up the corresponding
NodeRequest object from the buildset when the NodesProvisionedEvent is
handled.
As a result of this we have to find a way to return the NodeSet even in
case the NodeRequest or the buildset cannot be found anymore (e.g. due
to a gate reset). In this case we look up the NodeRequest directly from
ZooKeeper and provide a faked NodeSet which allows us to retrieve the
node information from Zookeeper (via the update mechanism of the
NodeRequest in the Nodepool client).
Finally, we can get rid of the local result event queue in the scheduler
as now all result events are in ZooKeeper. Together with that the
`zuul.scheduler.eventqueues.result` gauge was also removed.
Change-Id: Ib5e0f13d25a21ebad908d38f0201e92b704a1c85
Due to the wrong if condition (should be elif) an error was logged for
every processed result event that didn't match the second if block.
Change-Id: Ia6c65706fa507c3f4f8f5bc3cf37a5a8fa015202
This release appears to have broken the docs build. Unfortunately it
looks like an issue in the sphinx-contrib/httpdomain project related
to [1] so will require some external releases to fix.
[1] https://github.com/sphinx-doc/sphinx/pull/9155
Change-Id: I1d3a2bf7c399e9e008f1739237fe5455edde57ba
Zuul
Jeremy Stanley
James E. Blair
Simon Westphahl
Tristan Cacqueray
Felix Edel
Ian Wienand