We originally wrote the change list to be a best-effort service for
the scheduler check for whether a change is in a pipeline (which
must be fast and can't lock each of the pipelines to read in the full
state). To make it even simpler, we avoided sharding and instead
limited it to only the first 1024 changes. But scope creep happened,
and it now also serves to provide the list of relevant changes to the
change cache. If we have a pipeline with 1025 changes and delete
one of them from the cache, that tenant will break, so this needs to
be corrected.
This change uses sharding to correct it. Since it's possible to
attempt to read a sharded object mid-write, we retry reads in the case
of exceptions until they succeed.
In most cases this should still only be a single znode, but we do
truncate sharded znodes, so there is a chance even in the case of
a small number of changes of reading incorrect data.
To resolve this for all cases, we retry reading until it succeeds.
The scheduler no longer reads the state at the start of pipeline
processing (it never needed to anyway), so if the data become corrupt,
a scheduler will eventually be able to correct it. In other words,
the main pipeline processing path only writes this, and the other
paths only read it.
(An alternative solution would be to leave this as it was and instead
load the full pipeline state for maintaining the change cache; that
runs infrequently enough that we can accept the cost. This method is
chosen since it also makes other uses of this object more correct.)
Change-Id: I132d67149c065df7343cbd3aea69988f547498f4
61eea2169b