When executing the unit tests with multiple schedulers in parallel a lot
of them are failing becasue there are too many open SQL connections.
However, executing those tests one after another doesn't end up in the
connection limit being exhausted. So this doesn't look like a shutdown
problem in Zuul.
Having a look at the MySQL server when it's under load during the tests
revealed that the default connection limit of 151 on the server side is
exhausted very quickly when a lot of tests are running in parallel (each
test running with two schedulers).
Therefore, this change increases the default MySQL connection limit to
300. We should keep in mind that this connection limit has to go in hand
with the number of schedulers used in the unit tests and the number of
tests executed in parallel. Maybe on the long term it might make sense
to cap the latter parameter somehow in the tox.ini file.
The same applies to the ZooKeeper connection limit which is configured
in tools/zoo.cfg.
Change-Id: Iff76e99ec82edc8e8bc110a22a096bb689d8dd1f
The number of schedulers can be defined via the ZUUL_SCHEDULER_COUNT
environment variable (also forwarded by tox).
Disclaimer: running the tests with multiple schedulers will still fail
as not all test support multiple schedulers yet.
Change-Id: Ie11467dcf0fc36a5e167adcbd29ac4f256e0b3c5
Instead of sending the required job variables via the build request we
will give the executor the job path so it can load the frozen job from
the pipeline state.
Change-Id: Ie1b7ea0a2bc5dfc2d44bcafbc9eb8c227bbe7de2
Check every second for an authenticated user's token expiration.
If the token has expired, display a modal providing a link to
log in again.
Closing the modal properly logs the user out.
Change-Id: I0baa4f415bdc61735fc42f9fd80a58309976096f
Add a "create autohold request" button on the autoholds page for admin
users.
Refactor autohold form modal so that it can be reused in autoholds and
build pages.
Improve error messages when using admin actions: if the API call returns
an error description, display it in the toast. Otherwise return the HTTP
error code.
Change-Id: I9d1cc8a1d751e7b4ce32e7abecc2ad9f3e96f676
Add a "promote" button in the actions menu of a change, if the
currently logged in user is an admin on the tenant and if the pipeline
is dependent.
Clicking the button opens a modal, so that the user can confirm her decision.
Change-Id: I8262888aef9ba1a106e0b321cc4cf2e14465b90c
Add specific icons next to pipelines names on the tenant
status page to help differentiate their characteristics. A mouseover
text provides a brief summary for each type of pipeline.
Type <-> Icon mapping:
* independent: FlaskIcon (it is for experiments)
* dependent: CodeBranchIcon (integration pipeline)
* serial: SortAmountDownIcon (deploy all and in order)
* supercedent: BundleIcon (deploy latest)
* default: StreamIcon (icon used for pipelines in the build/buildset page)
Change-Id: Ia16993af70d133f15aad9a212737805c3623b76e
Since not all job data attributes of a FrozenJob are of type dict we
need to remove the isinstance check and just store the value.
Change-Id: I61ac5c766604fa9c3b8d4a1225b39f1bb17a9f89
A user can list active autoholds for a given tenant. If the user is a
tenant admin, she can also discard a autohold request from the autoholds
page.
Add a autohold page holding information about a single autohold request,
in particular links to held builds if any.
Change-Id: I4f5f2cfdf8e46ce8fb7ac69e330d6e51bd8b19fe
Add a "autohold" command on a build page, displayed only if the currently
logged in user is an admin on the tenant. By clicking the command, the
user can display a form and set custom parameters for an autohold
request.
Change-Id: I0f9f069d4ad36d4961eb6925146f67fe4910cb2e
Add a "re-enqueue" action command on a Buildset page, displayed only
if the currently logged in user is an admin on the tenant. By
clicking this command, the user can re-enqueue the change with
the same parameters as the buildset.
Redux: turned the API error notifications into a more generic
notification system, that can now include success notifications.
Change-Id: I05b6b3deb912b121df8de207944d9ec26e7c92d1
Add a togglable "Actions" kebab menu on the right of a change, if the
currently logged in user is an admin on the tenant.
Show available actions (currently "dequeue" only) when toggled.
Clicking the "Dequeue" item opens a modal, so that the user
can confirm her decision.
Change-Id: I7c97509d62ef167ee5904a816998049d88c6b3cb
By default the UserManager uses session storage for its authentication
credentials. That is restricted to a single tab. In order to support
using the same auth token in multiple tabs, we could switch that to
localStorage which is shared by all tabs of the same domain. But then
if a user exited the browser, they might be surprised to find that they
were still logged in when restarting. The typically short lifetime of
OIDC tokens mitigates that somewhat, but it's probably best not to
subvert that expectation anyway.
Instead, we can continue to use session storage by using a BroadcastChannel
to notify other tabs of login/out events and transfer the token info as
well. This is a standard feature of modern browsers, but we're using
a library that wraps it for two reasons: it supports older browsers
with compatability workarounds if required, and it implements a leader
election protocol. More on that in a minute.
We would also like to automatically renew tokens shortly before they
expire. The UserManager has an automatic facility for that, but it
isn't multi-tab aware, so every tab would try to renew at the same time
if we used it. Instead, we hook into the UserManager timer that fires
about one minute before token expiration and use the leader election to
decide which tab will renew the token.
We renew the token silently in the background with a hidden iframe. In
this case, instead of using our normal auth callback page, we use a much
simpler "silent callback" which does not render the rest of our application.
This avoids confusion and reduces resource usage.
This also moves any remaining token lifecycle handling out of the Auth
component and into ZuulAuthProvider, so the division of responsibilities
is much simpler.
Change-Id: I17af1a98bf8d704dd7650109aa4979b34086e2fa
Under the hood, this uses AuthProvider as supplied by oidc-react.
Most of the theory is explained in the comment in ZuulAuthProvider.jsx
The benefit of doing this is that we allow the AuthProvider and
userManager to handle the callback logic, so we don't need to
handle the callback logic ourselves. A callback page is still required
though in order to deal with the parameters passed in a successful
redirection from the Identity Provider.
The challenge in using these classes as-is is that our authority
endpoints (eg, the IDP itself) may change from one tenant to
the next; these classes aren't set up for that. So we need to be
careful about how and when we change those authority URLs.
In terms of functionalities: if the default realm's authentication driver
is set to "OpenIDConnect", display a "Sign in" button. If the the user
is logged in, redirect to the last page visited prior to logging in;
fetch user authorizations and add them to the redux store; display the
user's preferred username in the upper right corner. Clicking on the
user icon in the right corner displays a modal with user information
such as the user's zuul-client configuration, and a sign out button.
Clicking on the sign out button removes user information from the
store (note that it does not log the user out from the Identity Provider).
Add some basic documentation explaining how to configure Zuul with
Google's authentication, and with a Keycloak server.
(This squashes https://review.opendev.org/c/zuul/zuul/+/816208 into
https://review.opendev.org/c/zuul/zuul/+/734082 )
Co-authored-by: James E. Blair <jim@acmegating.com>
Change-Id: I31e71f2795f3f7c4253d0d5b8ed309bfd7d4f98e
This updates react-router-dom to not quite the latest version, but
a new enough version that it supports hooks.
This also changes how React Refs are handled in such a way that
it breaks some tests which rely on on reaching through the redux
store into child nodes. To resolve this, some tests are updated
to use the react-test-renderer instead.
The tests in App.test.jsx were not asserting anything past the
initial toEqual assertion in the path, which is why they are only
now being updated to match links added since the test inception.
Change-Id: Ia80fbfe3cf2d2d275fd8422111ec193c467bf606
Instead of sending the repo state via the build request we only provide
the path to the merge/extra repo state that we already as part of the
pipeline state. The executor will then load the repo states from the
given paths.
This way we reduce the amount of duplicate data we store in Zookeeper
which is especially important for the repo state, as it can become quite
large.
Change-Id: I7dc5d854dbee93af52d25b4462293c85eb7a1a8e
In case we encounter an issue during pipeline processing we don't want
to let that exception bubble up to the run handler exception block as
this might starve out following pipelines in case the issue persists.
Instead we will log the exception at the pipeline level and move on to
the next pipeline.
Change-Id: Ie00444e9094b01b31e91f0660290921b887deb94
It seems that we check for the reported_enqueue flag only when
initially enqueueing an item. This we can remove it and save a
zookeeper operation when enqueueing a change.
Change-Id: Ic869698e882377217410ea66825c5c999f9ae322
We didn't include the branch cache ltimes when loading the initial
config in zuul-web, so if zuul-web starts before the branch cache
is updated, it won't be told to refresh it when loading the layout.
This matches what we do in zuul-scheduler, and adds a skipped test.
The test does unusual things with the startup sequence and does not
appear to be reliable when run upstream, so it is disabled for now.
This also adds a debug line and adjust another to remove an apostraphe
to make grepping easier.
Change-Id: Ib184f4cadb21124fd3dedeea3797b01bb9c431cc
This adds a new component state: INITIALIZING and the scheduler and
web components use it when they are creating their initial config.
It is safe for the scheduler to start processing tenant and pipeline
events as soon as it starts because it only processes those for
the tenants that it has already loaded.
However it is not safe for drivers to move events from their
incoming queue into the scheduler since that requires the full
tenant list. The 4 drivers to which this applies are updated
to wait on config priming.
Zuul-web is already structured to wait until config priming
so does not need a corresponding change.
Change-Id: I36dd4927e583328434e66553aa3ff0cd7469b488
In our override of construct_mapping, we don't handle one of the
error cases that the base class does. If the user supplies
"foo: {{ bar }}" we fail before the superclass method runs and
therefore we don't benefit from its error. This change corrects
that by testing for that case and allowing the superclass method
to run.
Change-Id: I0b92053b08cb4991309163c94846bd34c4cb1393
When we create a new layoutstate for a tenant, we can create a
zero-byte entry which can result in a json-decode error from
another component watching the state.
Update it to atomically write the state.
Change-Id: Idc62dbdc2299405b6c0e9948dc7ccc50830750a8
Zuul
Felix Edel
Simon Westphahl
James E. Blair
Matthieu Huin
Tobias Henkel