acbd318084
The TreeCache can be a bit of a bottleneck since it is centralizing in a single thread fetch operations which may have previously happened in different threads. It also may in some cases fetch more data than we did previously since it will update all objects in memory as they are updated in ZK regardless of whether nodepool is interested in them at the moment. Or at least, it may concentrate those fetches in a smaller timespan. To address the bottleneck, this change uses asynchronous fetches from ZooKeeper. The background worker which previously updated the cache in response to events from ZK is split into two threads. The first will decide whether or not a fetch for a given event is necessary, and then fire off the async fetch operation if so. It then pushes the future for that operation, along with other event information into a second queue. A second background thread (called the "playback" worker) will pop events from that queue, still in order, and await the future. This ensures all cache operations are still ordered, while multiple fetch operations can be outstanding in parallel. This relies even more heavily on the path parsing implemented by the three caches to be correct. In particular, we want to make sure that we don't accidentally fetch a znode if we're just dealing with a lock node event. The previous code sequenced the checks so that we would handle the lock node event and short circuit before deciding whether to fetch. The new code reverses that, which means that it is paramount that we don't accidentally have the path parsing return a match on a lock node. The cache would function correctly, but it would silently perform far more operations than necessary, so such an error would likely go unnoticed for some time. To prevent this, explicit testing for path parsing is added. Change-Id: I6ef10c724c3993ee565510ab917dce64d2e3d3f9
Nodepool
Nodepool is a system for managing test node resources. It supports launching single-use test nodes from cloud providers as well as managing access to pre-defined pre-existing nodes. Nodepool is part of a suite of tools that form a comprehensive test system, including Zuul.
The latest documentation for Nodepool is published at: https://zuul-ci.org/docs/nodepool/
The latest documentation for Zuul is published at: https://zuul-ci.org/docs/zuul/
Getting Help
There are two Zuul-related mailing lists:
- zuul-announce
-
A low-traffic announcement-only list to which every Zuul operator or power-user should subscribe.
- zuul-discuss
-
General discussion about Zuul, including questions about how to use it, and future development.
You will also find Zuul developers in the #zuul channel on Freenode IRC.
Contributing
To browse the latest code, see: https://opendev.org/zuul/nodepool To clone the latest code, use git clone https://opendev.org/zuul/nodepool
Bugs are handled at: https://storyboard.openstack.org/#!/project/zuul/nodepool
Code reviews are handled by gerrit at https://review.opendev.org
After creating a Gerrit account, use git review to submit patches. Example:
# Do your commits
$ git review
# Enter your username if promptedJoin #zuul on Freenode to discuss development or usage.
License
Nodepool is free software, licensed under the Apache License, version 2.0.
Python Version Support
Nodepool requires Python 3. It does not support Python 2.