Handle NotFound exception when declaring a queue

Catch and handle the NotFound exception when declaring a queue.
This exception occurs when attempting to declare a queue that already
exists and is non-durable, non-HA, and hosted on a node that has
recently gone down.

Closes-Bug: #2068630
Change-Id: I637d931242c64465701266149d5e301f9b5ec4de
(cherry picked from commit 0d3338fd1d)
(cherry picked from commit 58124491ba)
(cherry picked from commit 9c36c27087)

oslo_messaging/_drivers/impl_rabbit.py

@@ -424,6 +424,29 @@ class Consumer(object):
                 # from the one used first.
                 LOG.warning(err)
                 self._declare_fallback(err, conn, consumer_arguments)
+            except amqp_ex.NotFound as ex:
+                # NOTE(viktor.krivak): This exception is raised when
+                # non-durable and non-ha queue is hosted on node that
+                # is currently unresponsive or down. Thus, it is
+                # not possible to redeclare the queue since its status
+                # is unknown, and it could disappear in
+                # a few moments. However, the queue could be explicitly
+                # deleted and the redeclaration will then succeed.
+                # Queue will be then scheduled on any of the
+                # running/responsive nodes.
+                # This fixes bug:
+                # https://bugs.launchpad.net/oslo.messaging/+bug/2068630
+
+                LOG.warning("Queue %s is stuck on unresponsive node. "
+                            "Trying to delete the queue and redeclare it "
+                            "again, Error info: %s", self.queue_name, ex)
+                try:
+                    self.queue.delete()
+                except Exception as in_ex:
+                    LOG.warning("During cleanup of stuck queue deletion "
+                                "another exception occurred: %s. Ignoring...",
+                                in_ex)
+                self.queue.declare()
 
         except conn.connection.channel_errors as exc:
             # NOTE(jrosenboom): This exception may be triggered by a race

releasenotes/notes/bug-2068630-6ff92f213bc4eca0.yaml

@@ -0,0 +1,6 @@
+---
+fixes:
+  - |
+    Force queue deletion when it is not possible de redeclare a queue.
+    See `bug 2068630 <https://launchpad.net/bugs/2068630>`__
+    for details.