Don't heartbeat manually in the cloudkitty orchestrator

This fixes cloudkitty-processor randomly crashing when releasing locks.

Until now, heartbeating was done manually after each collect period,
once all tenants have been processed. However, the first collect period
is generally much longer than the following ones, given that the tenant workers
need to catch up with the current date.
Because of that, too much times passes between the moment where the coordinator
is created and the first heartbeat. This causes locks to be automatically
released, which leads to an error when we try to release them manually.

This issue is fixed by letting the tooz coordinator taking care of
heartbeating.

Change-Id: If82f5981645032dcc649ea6e29cfdf8b1054b01e
Story: 2004270
Task: 27817
This commit is contained in:
Luka Peschke 2018-11-06 15:33:40 +01:00
parent 7778986c3a
commit 641b501699
2 changed files with 6 additions and 2 deletions

View File

@ -253,7 +253,7 @@ class Orchestrator(object):
self.coord = coordination.get_coordinator(
CONF.orchestrator.coordination_url,
uuidutils.generate_uuid().encode('ascii'))
self.coord.start()
self.coord.start(start_heart=True)
def _lock(self, tenant_id):
lock_name = b"cloudkitty-" + str(tenant_id).encode('ascii')
@ -303,7 +303,6 @@ class Orchestrator(object):
lock.release()
self.coord.heartbeat()
# NOTE(sheeprine): Slow down looping if all tenants are
# being processed
eventlet.sleep(1)

View File

@ -0,0 +1,5 @@
---
fixes:
- |
cloudkitty-processor crashes which happened when using distributed
tooz locks have been fixed.