Merge "Add a spec for global semaphores"
This commit is contained in:
commit
f2297cadb0
|
@ -0,0 +1,127 @@
|
||||||
|
Global Semaphores
|
||||||
|
=================
|
||||||
|
|
||||||
|
.. warning:: This is not authoritative documentation. These features
|
||||||
|
are not currently available in Zuul. They may change significantly
|
||||||
|
before final implementation, or may never be fully completed.
|
||||||
|
|
||||||
|
Semaphores are useful for limiting access to resources, but their
|
||||||
|
implementation as a per-tenant configuration construct may be limiting
|
||||||
|
if they are used for real-world resources that span tenants.
|
||||||
|
|
||||||
|
This is a proposal to address that by adding global semaphores.
|
||||||
|
|
||||||
|
Background
|
||||||
|
----------
|
||||||
|
|
||||||
|
Semaphores may be used for a variety of purposes. One of these is to
|
||||||
|
limit access to constrained resources. Doing so allows Zuul to avoid
|
||||||
|
requesting nodes and scheduling jobs until these resources are
|
||||||
|
available. This makes the overall system more efficient as jobs don't
|
||||||
|
need to wait for resources during their run phase (where they may be
|
||||||
|
idling test nodes which could otherwise be put to better use).
|
||||||
|
|
||||||
|
A concrete example of this is software licenses. If a job requires
|
||||||
|
software which uses a license server to ensure that the number of
|
||||||
|
in-use seats does not exceed the available seats, a semaphore with a
|
||||||
|
max value equal to the number of available seats can be used to help
|
||||||
|
Zuul avoid starting jobs which would otherwise need to wait for a
|
||||||
|
license.
|
||||||
|
|
||||||
|
If only one Zuul tenant uses this piece of software, the existing
|
||||||
|
implementation of semaphores in Zuul is satisfactory. But if the
|
||||||
|
software licenses are shared across Zuul tenants, then a Zuul
|
||||||
|
semaphore can't be used in this way since semaphores are per-tenant
|
||||||
|
constructs.
|
||||||
|
|
||||||
|
The general solution to sharing Zuul configuration objects across
|
||||||
|
tenants is to define them in a single git repo and include that git
|
||||||
|
repo in multiple tenants. That works as expected for Jobs, Project
|
||||||
|
Templates, etc. But semaphores have a definition as well as a
|
||||||
|
run-time state (whether they are aquired and by whom). Including a
|
||||||
|
semaphore in multiple tenants essentially makes copies of that
|
||||||
|
semaphore, each with its own distinct set of holders.
|
||||||
|
|
||||||
|
Proposed Change
|
||||||
|
---------------
|
||||||
|
|
||||||
|
A new global semaphore configuration would be added to the tenant
|
||||||
|
configuration file. Note this is the global configuration file (where
|
||||||
|
tenants are defined); not in-repo configuration where semaphores are
|
||||||
|
currently defined.
|
||||||
|
|
||||||
|
The definition would be identical to the current in-repo semaphore
|
||||||
|
configuration. In order to grant access to only certain tenants, each
|
||||||
|
tenant will also need to specify whether that semaphore should be
|
||||||
|
available to the tenant. This scheme is similar to the way that
|
||||||
|
authorization rules are defined in this file and then attached to
|
||||||
|
tenants.
|
||||||
|
|
||||||
|
For example:
|
||||||
|
|
||||||
|
.. code-block:: yaml
|
||||||
|
|
||||||
|
- semaphore:
|
||||||
|
name: licensed-software
|
||||||
|
max: 8
|
||||||
|
|
||||||
|
- tenant:
|
||||||
|
name: example-tenant
|
||||||
|
semaphores:
|
||||||
|
- licensed-software
|
||||||
|
source:
|
||||||
|
gerrit:
|
||||||
|
config-projects:
|
||||||
|
...
|
||||||
|
|
||||||
|
The existing in-repo semaphores will remain as they are today -- they
|
||||||
|
will not be deprecated (they are still very useful on their own for
|
||||||
|
most other use cases).
|
||||||
|
|
||||||
|
If an in-repo semaphore is defined with the same name as a global
|
||||||
|
semaphore, that will become a configuration error. The global
|
||||||
|
semaphore will take precedence.
|
||||||
|
|
||||||
|
Implementation
|
||||||
|
--------------
|
||||||
|
|
||||||
|
The user-visible configuration is described above.
|
||||||
|
|
||||||
|
Current semaphores are stored in the ZooKeeper path
|
||||||
|
``/zuul/semaphores/<tenant>/<semaphore>``. Global semaphores will use
|
||||||
|
a similar scheme without the tenant name:
|
||||||
|
``/zuul/global-semaphores/<semaphore>``.
|
||||||
|
|
||||||
|
Locking, releasing, and leak cleanup will all behave similarly to the
|
||||||
|
current per-tenant semaphores. On release, a per-tenant semaphore
|
||||||
|
broadcasts a PipelineSemaphoreReleaseEvent to all pipelines in order
|
||||||
|
to trigger a pipeline run and start any jobs which may be waiting on
|
||||||
|
the semaphore. A global semaphore will do the same, but for every
|
||||||
|
pipeline of every tenant which includes the semaphore.
|
||||||
|
|
||||||
|
Alternatives
|
||||||
|
------------
|
||||||
|
|
||||||
|
We could add a field to the in-repo definitions of semaphores which
|
||||||
|
indicates that the semaphore should be global. As this has the
|
||||||
|
ability to affect other tenants, we would need to restrict this to
|
||||||
|
config-projects only. However, that still opens the possibility of
|
||||||
|
one tenant affecting another via the contents of a config-project.
|
||||||
|
Some method of allowing the administrator to control this via the
|
||||||
|
tenant config file would still likely be necessary. As long as that's
|
||||||
|
the case, it seems simpler to just define the semaphores there too.
|
||||||
|
|
||||||
|
We could outsource this to Nodepool. In fact, having nodepool manage
|
||||||
|
resources like this seems like a natural fit. However, the current
|
||||||
|
Nodepool implementation doesn't permit more than one provider to
|
||||||
|
satisfy a node request, so a hypothetical semaphore provider wouldn't
|
||||||
|
be able to be combined with a provider of actual test nodes.
|
||||||
|
Addressing this is in-scope and a worthwhile change for Nodepool, but
|
||||||
|
it is potentially a large and complex change. Additionally, the idea
|
||||||
|
of waiting for a semaphore before submitting requests for real
|
||||||
|
resources adds a new dimension to even that idea -- Nodepool would
|
||||||
|
need to know whether to run the semaphore provider first or last
|
||||||
|
depending on the desired resource aquisition order. Meanwhile, since
|
||||||
|
Zuul does have the concept of semaphores already and they almost fit
|
||||||
|
this use case, this seems like a reasonable change to make in Zuul
|
||||||
|
regardless of any potential Nodepool changes.
|
|
@ -23,3 +23,4 @@ documentation instead.
|
||||||
enhanced-regional-executors
|
enhanced-regional-executors
|
||||||
tenant-resource-quota
|
tenant-resource-quota
|
||||||
community-matrix
|
community-matrix
|
||||||
|
global-semaphores
|
||||||
|
|
Loading…
Reference in New Issue