Reuse queue items after reconfiguration

When we reconfigure, we create new Pipeline objects, empty the values
in the PipelineState and then reload all the objects from ZK.  We then
re-enqueue all the QueueItems to adjust and correct the object
pointers between them (item_ahead and items_behind).  We can avoid
reloading all the objects from ZK if we keep queue items from the
previous layout and rely on the re-enqueue method correctly resetting
any relevant object pointers.

We already defer this re-enqueue work to the next pipeline processing
after a reconfiguration (so the reconfiguration itself doesn't take
very long, but now the first pipeline run after a reconfiguration must
perform a complete refresh).  With this change, that first refresh
is no longer be a complete refresh but a normal refresh, so we will get
the benefits of previous reductions in refresh times.

The main risk of this change is that it could introduce a memory leak.
During development, additional debugging was performed to verify that
after a re-enqueue, there are no obsolete layout or pipeline objects
reachable from the pipeline state object.

On schedulers where a re-enqueue does not take place (these schedulers
would simply see the layout update and re-create their PipelineState
python objects and refresh them after another scheduler has already
performed the re-enqueue), we need to ensure that we update any
internal references to Pipeline objects (which then lead to Layout
objects and can cause memory leaks).

To address that, we update the pipeline references in the ChangeQueue
instances underneath a given PipelineState when that state is being
reset after a reconfiguration.

This change also removes the pipeline reference from the QueueItem,
replacing it with a property that uses the pipeline reference
on the ChangeQueue instead.  This removes one extra place where
an incorrect reference could cause a memory leak.

Change-Id: I7fa99cd83a857216321f8d946fd42abd9ec427a3

zuul/manager/__init__.py

@@ -100,11 +100,14 @@ class PipelineManager(metaclass=ABCMeta):
             # Make sure we have state and change list objects, and
             # ensure that they exist in ZK.  We don't hold the
             # pipeline lock, but if they don't exist, that means they
-            # are new, so no one else will either.  These will not
-            # automatically refresh now, so they will be out of date
-            # until they are refreshed later.
+            # are new, so no one else will either, so the write on
+            # create is okay.  If they do exist and we have an old
+            # object, we'll just reuse it.  If it does exist and we
+            # don't have an old object, we'll get a new empty one.
+            # Regardless, these will not automatically refresh now, so
+            # they will be out of date until they are refreshed later.
             self.pipeline.state = PipelineState.create(
-                self.pipeline, layout.uuid)
+                self.pipeline, layout.uuid, self.pipeline.state)
             self.pipeline.change_list = PipelineChangeList.create(
                 self.pipeline)