This adds a modified kazoo.DataWatch class which does not set
watches on paths that don't exist. This is mostly so that when
a DataWatched path is deleted, we don't leave a watch in place.
Without that, the executor api will leak watches on the ZK cluster.
We vendor the entire watches.py file from kazoo (ASL2 licensed) to
avoid any issues with API changes. Separately we will see if they
are interested in this class upstream.
This updates the tests to use the wchp 4lw to test that we don't
This mirrors the configuration in Nodepool for using TLS-enabled
ZooKeeper in tests. We use the ensure-zookeeper role in order
to get a newer ZooKeeper than is supplied in bionic.
This change is a common root for other
Zookeeper related changed regarding
scale-out-scheduler. Zookeeper becoming
a central component requires to increase
Since the ZooKeeper class is expected to grow
significantly (ZooKeeper is becoming a central part
of Zuul) a split of the ZooKeeper class (zk.py) into
zk module is done here to avoid the current god-class.
Also the zookeeper log is copied to the "zuul_output_dir".