octavia/octavia/db
Carlos Goncalves 37c86aa794 Make health checks resilient to DB outages
Octavia is struggling with proper handling of DB connectivity issues
bringing down all running loadbalancers. Octavia tries to failover
amphorae and can fail in one of the following stages:

1. Octavia can't create new amphora because Nova isn't ready yet after
   DB outage. Nova-API throws 500, Octavia nukes amphora instance and
   won't try to recreate it again.
2. Octavia tries to recreate amphora instance but it gets stuck in
   PENDING_CREATE forever.
3. Octavia fails completely reporting DB connection issues, leaving some
   amphoras in error, some in pending_delete as bellow: It affects also
   HA deployments.

This patch fixes that by wrapping the DB check for health, waiting for
the connection to be re-established and sleeping off the full
"heartbeat_timeout" interval.

Story: 2003575
Task: 24871

Change-Id: I7b30cd31e1ce0cf9dab61484f4404f1c6ccddd5e
(cherry picked from commit 92473ce210)
2018-09-13 20:00:42 +00:00
..
migration Extend api to accept qos_policy_id 2017-12-20 09:23:16 +08:00
__init__.py Initial directory skeleton 2014-09-04 15:28:10 -06:00
api.py Make health checks resilient to DB outages 2018-09-13 20:00:42 +00:00
base_models.py Fix fields translation on filtering 2018-07-03 11:47:52 +00:00
models.py Merge "Improve user error messages for duplicate objects" 2018-01-05 03:15:48 +00:00
prepare.py Add support for single-create for APIv2 2017-05-05 20:56:22 -07:00
repositories.py Merge "Improve the error logging for zombie amphora" into stable/queens 2018-07-18 13:39:03 +00:00