Octavia is struggling with proper handling of DB connectivity issues
bringing down all running loadbalancers. Octavia tries to failover
amphorae and can fail in one of the following stages:
1. Octavia can't create new amphora because Nova isn't ready yet after
DB outage. Nova-API throws 500, Octavia nukes amphora instance and
won't try to recreate it again.
2. Octavia tries to recreate amphora instance but it gets stuck in
PENDING_CREATE forever.
3. Octavia fails completely reporting DB connection issues, leaving some
amphoras in error, some in pending_delete as bellow: It affects also
HA deployments.
This patch fixes that by wrapping the DB check for health, waiting for
the connection to be re-established and sleeping off the full
"heartbeat_timeout" interval.
Story: 2003575
Task: 24871
Change-Id: I7b30cd31e1ce0cf9dab61484f4404f1c6ccddd5e
92473ce210