Fix incorrect handling of storage exceptions

Since [0] _save_flow_detail and _save_atom_detail use tenacious.retry,
it means that they no longer raise StorageException but a
tenacious.RetryError which is not caught by the callers.

Some jobs may have been incorrectly consumed (instead of being
rescheduled) when StorageException was raised.
Now tenacious reraises only the last exception (StorageException), which
is correctly handled.

[0] Ic6b0a78d20124cc027468ecc6aeff189c25d1a8a

Closes-Bug: #2037050
Change-Id: I030dd8fc5b65833243cf0948af53dc1aeabf41d9
This commit is contained in:
Gregory Thiemonge 2023-09-22 03:22:07 -04:00
parent cd80fc96de
commit 07a1a3f417
2 changed files with 10 additions and 2 deletions

View File

@ -0,0 +1,6 @@
---
fixes:
- |
Fixed an issue with the handling of exceptions when running a flow, some
jobs may have been incorrectly consumed (and not rescheduled) during
outages.

View File

@ -454,7 +454,8 @@ class Storage(object):
@tenacity.retry(retry=tenacity.retry_if_exception_type(
exception_types=exceptions.StorageFailure),
stop=tenacity.stop_after_attempt(RETRY_ATTEMPTS),
wait=tenacity.wait_fixed(RETRY_WAIT_TIMEOUT))
wait=tenacity.wait_fixed(RETRY_WAIT_TIMEOUT),
reraise=True)
def _save_flow_detail(self, conn, original_flow_detail, flow_detail):
# NOTE(harlowja): we need to update our contained flow detail if
# the result of the update actually added more (aka another process
@ -491,7 +492,8 @@ class Storage(object):
@tenacity.retry(retry=tenacity.retry_if_exception_type(
exception_types=exceptions.StorageFailure),
stop=tenacity.stop_after_attempt(RETRY_ATTEMPTS),
wait=tenacity.wait_fixed(RETRY_WAIT_TIMEOUT))
wait=tenacity.wait_fixed(RETRY_WAIT_TIMEOUT),
reraise=True)
def _save_atom_detail(self, conn, original_atom_detail, atom_detail):
# NOTE(harlowja): we need to update our contained atom detail if
# the result of the update actually added more (aka another process