Currently, we cache unparsed YAML-dicts of the configuration on
the global project objects. During a dynamic reconfiguration,
we use that YAML-dict cache, subsituting any content from changes
being tested, and create completely new configuration objects.
The resulting memory usage is large. We clear part of that cache
when performing a tenant reconfiguration after a change lands,
and we clear the entire cache when performing a full reconfiguration.
An apparent longstanding bug is that we also cleared the entire
cache again for each tenant during a full reconfig.
This change alters the cache situation to the following:
We maintain a global cache of all YAML-dicts on the abide object.
During a full reconfiguration, we clear that cache (once), by virtue
of the fact that a full reconfig creates a new abide.
When a tenant (re-)configuration starts, it begins by identifying all
of the project-branches involved in its configuration. It issues
merger cat jobs to fetch anything not in the cache, and caches the
resulting YAML-dicts in the abide. Note at this point, we do associate
the SourceContext of all of those YAML-dicts, but since these are going
in a global cache, we don't know whether the SourceContext should be
trusted or not, so they all get marked as untrusted.
Then, as a second step, we fetch all of the cached YAML-dicts and make
a copy of them; on the copy, we update the SourceContext to indicate
whether the source is trusted or not in this tenant. We also check
to make sure that pipelines don't show up in untrusted config.
Then we take the collected YAML-dicts and parse them into configuration
objects. These we then cache on the TenantProjectConfig objects in the
tenant. We cache them there because the objects themselves are
tenant-specific. The objects are then filtered according to the include/
exclude filters of the tenant, and then added to a layout.
During a dynamic reconfiguration, we use the cached objects from the
TenantProjectConfig where possible, and for changes under test, we load
new YAML-dicts and parse them into objects (like above, but we don't
cache the results). We then combine the previously-cached and new objects
into a layout.
Because none of the config objects reference each other by value, only by
name, it is safe to share them across multiple layouts within the same
tenant.
Change-Id: Ia10abc1ea3fce3e65d986b4988803e2b38254da2
081f921ba1