Something (?) changed which is causing ssh-keyscan to try to use
IPv6 lo on our test nodes which does not work. Use ssh-keyscan -4
to work around this.
There is no user impact since we expect the user to interactively
accept the keys in the tutorial.
Change-Id: Ibf033de7a3ed6cb41993c6f4adbddeeb53f4c09c
Ib9e095430fa82be2327dcf7fd01ee4275b17415f introduced loops in "_run"
methods which have the side effect of events no longer being set.
Some tests are running until timeout expiration.
Test id Runtime
-------------------------------------------------------------- --------
tests.unit.test_gerrit.TestPolling.test_config_update 123.778
tests.unit.test_gerrit.TestPolling.test_post 123.729
tests.unit.test_gerrit.TestChecksApi.test_new_patchset 123.635
tests.unit.test_gerrit.TestPolling.test_tag 123.628
By changing the subclass method used to set event to one inside the
loop, notifications are restored.
Test id Runtime
-------------------------------------------------------------- --------
tests.unit.test_gerrit.TestPolling.test_config_update 5.246
tests.unit.test_gerrit.TestPolling.test_post 5.252
tests.unit.test_gerrit.TestChecksApi.test_new_patchset 5.276
tests.unit.test_gerrit.TestPolling.test_tag 5.247
Change-Id: I8c9ad6577d43b9b4866f83e4f8727615abf239c4
We discovered a regression in the global repo state that can lead to
wrong commits checked out on required projects. Further a fix for this
needs a slight re-design of the reconfiguration process. In order to
have some more time to do this revert it for now.
This reverts commit 175990ec42.
Change-Id: Ibcf3758ab886a01468095a8c588cf78db209529e
We don't like to make releases without release notes, so add one that
indicates why we are making a release without any significant user-
visible changes.
Change-Id: Iba48ec74435faca58d26372536c6c495c3afd923
In case of active event gathering we need to exit the event receiver
loop if we lose the connection to Zookeeper. This is to avoid duplicate
events.
Change-Id: I23b8416fff1c0ce6d3a67059283d41fdf29d7f96
Zookeeper docker 'latest' image was updated from 3.6.2 to 3.7.0
Some exceptions java.nio.file.NoSuchFileException about files
- /var/certs/keystores/examples_zk_1.examples_default.pem
- /var/certs/certs/cacert.pem
appeared.
This change adds a check on the last file created by tools/zk-ca.sh
before running zookeeper service.
Change-Id: I15b67977a8b14bb83474390786ab47000e7be07c
Change the base class from ZooKeeperBase to ZooKeeperSimpleBase for all
components that don't use the onConnect/Disconnect hooks.
Change-Id: Id81c528e7b9ff6ccd6303eed010858cd3315cbf8
If there happens to be a bug in Zuul, or if the scheduler crashes,
then we may leak semaphores, which, now that they are held in ZK,
is a permanent situation. To avoid the necessity of operators
manually editing or removing ZK nodes, introduce a periodic semaphore
cleanup.
This identifies any semaphores which are held by jobs which are not
running and therefore may be removed. It runs once an hour, outside
the main scheduler thread. It also runs immediately after the first
reconfiguration, so that if a scheduler crashes and is restarted,
leaked semaphores are cleaned up quickly.
The general algorithm should be safe with multiple schedulers, though
the details about how we should traverse the entire abide are unclear
at this point, so this may need to change. But it is as forward-
compatible as I can make it right now. We may also decide that once
all running builds are in ZK, we no longer need this. Regardless,
it is needed now, and should be adaptible to future situtaions if we
want it.
Change-Id: I023ddabb13047a1ed95a26a8f5793de36817c191
A recent change pushed semaphore handling into zookeeper. To do this a
new ZooKeeperBase class called SemaphoreHandler was written. Previously
the ZookeeperBase class expected instances of it to be few in number and
long lived, but semaphore handlers are replaced every time a new layout
is created. Unfortunately this expectation causes problems because
references to onConnect and onDisconnect methods in the SemaphoreHanlder
are held by the ZooKeeper Client. This causes the SemaphoreHandlers to
not be garbage collectable when they are replaced.
To make this worse the SemaphoreHandlers also keep a reference to tenant
layouts. Since the SemaphoreHandler cannot be GC'd the layout cannot be
either and we create a big enough to notice over time memory leak.
We fix this by changing the ZooKeeper client component hierarchy
slightly to introduce the idea that some components need state handling
on connection state changes and others do not. For the ones that do not
like SemaphoreHandler we can simply avoid registering any methods with
the client. The class previously didn't make use of this so no change in
behavior there.
Credit to corvus for doing the debugging and tracking this down. I'm
just trying to get a head start on a mergeable change.
Change-Id: I198769f3b7f3efe3c9d9321e0c12c0771dea0281
With the introduction of type checking, many of the Optional attributes
must be checked in various places before accessing. Making the buildset
mandatory on the build removes some of those otherwise necessary
NoneType checks and will avoid more in future changes.
Change-Id: I70409551f0f786ad6ea9c4f2cf0ab0f9eb2d1555
Active event gathering uses leader election to ensure that there is
only one instance at a time receiving events.
Change-Id: Ib9e095430fa82be2327dcf7fd01ee4275b17415f
This change is an alternative to I6f6d7a02ca8358c26ca1afa1d1bfaf9341ca2c3f.
At a high level, our plan for ZooKeeper queues is that anyone can append to
a queue without a lock, but a queue would only have one processor at a time
which is able to remove items from the queue. In the case of the github
driver, that is generally true, except that we have two threads doing that
work; the first processes queue items, and the second actually removes them.
The two threads in some respects act like two simultaneous "owners" of the
queue. This cas cause a problem within a single process (in that the second
thread can change the queue out from under the first) but also potentially
in multiple processes if we don't drain events from the second thread before
stopping the first (the second thread could continue to remove events
out from under a *different* process in that case).
To address both of these, we move all of the event handling into a single
thread. This maintains our "single-owner" paradigm, while also maintaining
the benefits of the concurrent processing. We start each cycle of the
dispatch loop by reading events from the queue and beginning new pre-processing
of those events, and we finish each cycle by collecting and acking any events
whose pre-processing is complete.
The events_in_progress set no longer has to contend with a second thread, so
we don't need to make a copy of it anymore.
Change-Id: I42f49a60ac3494c39f62cf63e35b78fb5fc0d804
In order to store project secrets keys and ssh keys in Zookeeper we need
to move the generation and loading of those keys to the key store. This
way we have a clear interface for getting project keys.
Change-Id: I17dcf9553a39e5554288ee4ee2f43c99af2bf5ac
EnqueueEvents and DequeueEvents are duck-typed as TriggerEvents
when they are passed to source.getChange. TriggerEvents are always
created in the context of a connection, so they have a hostname
field and a name field, and name should never have the hostname.
However, our current EnqueueEvents are created directly from
client-side user input and accept any form of project name. This
means when we get a change from them, we may end up with a Project
that has a canonical name for it's "name" field. This could not
only cause the enqueue event to be rejected, it could leak project
objects.
To correct this, perform some pre-processing on the input data to
get separate host/name data for the project, and then use that to
construct an Enqueue/DequeueEvent with the same field usage as
a TriggerEvent.
Change-Id: Ib596c282eba3a01c10d28593545e5d238d09de0c
We are concerned about the case where an external client submits
a management event which requires a result, but then disconnects
from ZK before retrieving the result. Currently, the client is
responsible for deleting the result node once it is populated; in
this case, the result node could leak.
To avoid this, let's have the client pre-create the result node as
an ephemeral node, and then the processor of the event can write to
that node (if it exists, or log a warning if it doesn't). If the
client happens to disconnect before the result is written, the node
will still be cleaned up.
An ephemeral node makes sense in this case since no other component
of the system needs the results of the management event other than
the originating client.
Change-Id: I420235ca748c4c28b40aaf4453e7805f73dfd5ba
Zuul
James E. Blair
Albin Vass
Guillaume Chauvel
Tobias Henkel
Simon Westphahl
Clark Boylan
Felix Edel
Kenny Ho