Add warning message about slow volume backend

If a volume driver takes an excessively long amount
of time to return volume stats info, the only indication
if this is a generic "Function outlasted interval" message
that doesn't really explain what is going on.

Introduce a specific log message when this happens so
that this situation is more clear for someone looking
at cinder-volume logs.

The message is set to trigger if the driver takes more
than half of the stats polling interval to complete.

Change-Id: Id6b27eb7b90c8a8c91fb46de69aa94b8210da18d
This commit is contained in:
Eric Harney 2020-12-11 09:51:00 -05:00
parent 85975fb2b6
commit 6d85d05f78
2 changed files with 21 additions and 1 deletions

View File

@ -2646,7 +2646,20 @@ class VolumeManager(manager.CleanableManager,
resource={'type': 'driver',
'id': self.driver.__class__.__name__})
else:
volume_stats = self.driver.get_volume_stats(refresh=True)
slowmsg = "The " + self.driver.__class__.__name__ + " volume " \
"driver's get_volume_stats operation ran for " \
"%(seconds).1f seconds. This may indicate a " \
"performance problem with the backend which can lead " \
"to instability."
@timeutils.time_it(
LOG, log_level=logging.WARN, message=slowmsg,
min_duration=CONF.backend_stats_polling_interval / 2)
def get_stats():
return self.driver.get_volume_stats(refresh=True)
volume_stats = get_stats()
if self.extra_capabilities:
volume_stats.update(self.extra_capabilities)
if volume_stats:

View File

@ -0,0 +1,7 @@
---
features:
- |
Log a warning from the volume service when a volume driver's
get_volume_stats() call takes a long time to return. This can help
deployers troubleshoot a cinder-volume service misbehaving due to a
driver/backend performance issue.