Wait for the stack lock to be released

Sometimes, when trying to update a stack in a *_COMPLETE/FAILED status,
it can fail since the stack lock is not released yet. As a short-term
workaround, I've put in place an ugly hack that will simply retry the
update if the operation failed because of the stack lock. The downside of
this is that we can't now add functional tests which will test updates on
Stack Lock itself since we now simply just ignore the HTTPConflict Exception
on updates.

I couldn't think of another solution that doesn't include a giant refactoring
that will solve this problem.

PS: Convergence doesn't have the notion of a Stack-Lock, so that should
take care of this by default.

Partially-Closes Bug: #1450314

Change-Id: Ib1a9d5c425285ebcffcb9ff8a362a56fd8f3574a

heat_integrationtests/common/test.py

@@ -328,15 +328,32 @@ class HeatIntegrationTest(testscenarios.WithScenarios,
         env_files = files or {}
         parameters = parameters or {}
         stack_name = stack_identifier.split('/')[0]
-        self.client.stacks.update(
-            stack_id=stack_identifier,
-            stack_name=stack_name,
-            template=template,
-            files=env_files,
-            disable_rollback=disable_rollback,
-            parameters=parameters,
-            environment=env
-        )
+
+        build_timeout = self.conf.build_timeout
+        build_interval = self.conf.build_interval
+        start = timeutils.utcnow()
+        while timeutils.delta_seconds(start,
+                                      timeutils.utcnow()) < build_timeout:
+            try:
+                self.client.stacks.update(
+                    stack_id=stack_identifier,
+                    stack_name=stack_name,
+                    template=template,
+                    files=env_files,
+                    disable_rollback=disable_rollback,
+                    parameters=parameters,
+                    environment=env
+                )
+            except heat_exceptions.HTTPConflict as ex:
+                # FIXME(sirushtim): Wait a little for the stack lock to be
+                # released and hopefully, the stack should be updatable again.
+                if ex.error['error']['type'] != 'ActionInProgress':
+                    raise ex
+
+                time.sleep(build_interval)
+            else:
+                break
+
         kwargs = {'stack_identifier': stack_identifier,
                   'status': expected_status}
         if expected_status in ['ROLLBACK_COMPLETE']: