Fix race in worker-based executor result processing

When worker-based engine receives event it should first remove
request from cache and then call request.set_result. Otherwise,
another thread waiting the request to complete might schedule
another request with same uuid (e.g. reversion of failed task),
which will replace current request in cache and then will
be deleted, thus leading to engine hang.

Related-bug: #1297668
Change-Id: Icbbca58067ee22fe45c63e1f69d189ea56697d20

taskflow/engines/worker_based/executor.py

@@ -105,8 +105,11 @@ class WorkerTaskExecutor(executor.TaskExecutorBase):
                 elif response.state == pr.PROGRESS:
                     request.on_progress(**response.data)
                 elif response.state in (pr.FAILURE, pr.SUCCESS):
-                    request.set_result(**response.data)
+                    # NOTE(imelnikov): request should not be in cache when
+                    # another thread can see its result and schedule another
+                    # request with same uuid; so we remove it, then set result
                     self._requests_cache.delete(request.uuid)
+                    request.set_result(**response.data)
                 else:
                     LOG.warning("Unexpected response status: '%s'",
                                 response.state)