While we are careful to ensure there are no race conditions in updating
the software deployment metadata in the Heat database, there was no such
protection in place for the actual update of the metadata provided to
the server via Swift. In legacy Heat, this race could only occur when
software deployments were created from the REST API or from separate
stacks deploying software to the same server, because resources in the
same stack were handled by a single thread making synchronous calls to
create the software deployments. However, with convergence, resources in
the same stack not only can be processed concurrently but will tend to
be synchronised if they have the same dependencies (since they'll be
triggered by consecutive asynchronous RPC messages).
This handles the race by retrying if a concurrent update is detected by
the time the data has been provided to Swift.
Note that this is only guaranteed to result in the latest data being
available on the assumption that a later write to Swift cannot overwrite
an earlier one that has been accepted. This is probably *not* guaranteed
by Swift in the presence of clock skew. Nonetheless, this should
drastically reduce the failure rate.
Similar caveats would probably apply to Zaqar, although Zaqar nominally
provides FIFO queues (in practice, it likely depends on the backend
used). However, with Zaqar clients receive all updates, not just the
latest one, and os-collect-config stores the deployments contained in
every message. Thus, Zaqar is not affected even assuming out-of-order
arrival.
Change-Id: Ic9a447f27e8c51f91f47f93b0fd1e9710341ec38
Closes-Bug: #1731032
46132999c0