Reference change dependencies by key

In order to cache changes in Zookeeper we need to make change objects
JSON serializable. This means that we can no longer reference other
change objects directly. Instead we will use a cache key consisting of
the connection name and a connection specific cache key.

Those cache keys can be resolved by getting the source instance using
the connection name and then retrieving the change instance via the new
`getChangeByKey()` interface.

The pipeline manager provides a helper method for resolving a list of
cache keys. Cache keys that where resolved once are also cached by the
manager as long as the reference is needed by any change in the
pipeline. The cache will be cleaned up at the end of a run of the queue
processor.

Until we can run multiple schedulers the change cache in the pipeline
manager will avoid hitting Zookeeper every time we resolve a cache key.

Later on when we have the pipeline state in Zookeeper we probably want
to clear the change cache in the pipeline manager at the end of the
queue processor. This way we make sure the change is recent enough when
we start processing a pipeline.

Change-Id: I09845d65864edc0e54af5f24d3c7be8fe2f7a919
This commit is contained in:
Simon Westphahl
2021-08-24 14:49:50 +02:00
parent 22c379bf80
commit 88f84bc5d5
16 changed files with 123 additions and 34 deletions

View File

@@ -223,8 +223,10 @@ class BaseReporter(object, metaclass=abc.ABCMeta):
return msg
def _formatItemReportOtherBundleItems(self, item):
related_changes = item.pipeline.manager.resolveChangeKeys(
item.change.needs_changes)
return "Related changes:\n{}".format("\n".join(
c.url for c in item.change.needs_changes if c is not item.change))
c.url for c in related_changes if c is not item.change))
def _getItemReportJobsFields(self, item):
# Extract the report elements from an item