Zuul has a content-addressable blobstore in ZooKeeper. We use it for
storing secrets, because they can be quite large, and many of them are
the same for many (or every) queue item, so we rely on the
deduplication inherent in content-addressable storage to keep ZK
traffic and storage minimal.
The same (with a few tweaks) could be true for repo state objects as
well. The repo state is a dictionary of connection -> project > refs.
Queue items with exactly the same set of involved projects should end
up with identical repo states (assuming they were started around the
same time and a branch has not since advanced). But queue items may
not have exactly the same projects; they may share a common set, but
then one might have one extra project. To promote reuse while
accommodating this, we will store per-project repo states in the blob
store. The queue items will then store a list of blobstore keys.
When we need to deal with the repo state for a queue item, we will
load in all of the project-repo-states specified by that list of keys
and combine them.
Likewise, when we update a repo state with the results of a merger
operation, we will split the repo state when storing it in the blob
store.
Change-Id: I2b276a072b48b91dec66d2f8e601a2b5c9128429
3102b75a48