Browse Source

Add warning message about slow volume backend

If a volume driver takes an excessively long amount
of time to return volume stats info, the only indication
if this is a generic "Function outlasted interval" message
that doesn't really explain what is going on.

Introduce a specific log message when this happens so
that this situation is more clear for someone looking
at cinder-volume logs.

The message is set to trigger if the driver takes more
than half of the stats polling interval to complete.

Change-Id: Id6b27eb7b90c8a8c91fb46de69aa94b8210da18d
changes/51/766751/3
Eric Harney 7 months ago
parent
commit
6d85d05f78
  1. 15
      cinder/volume/manager.py
  2. 7
      releasenotes/notes/slow-get-volume-stats-91b84c6e661dc605.yaml

15
cinder/volume/manager.py

@ -2646,7 +2646,20 @@ class VolumeManager(manager.CleanableManager,
resource={'type': 'driver',
'id': self.driver.__class__.__name__})
else:
volume_stats = self.driver.get_volume_stats(refresh=True)
slowmsg = "The " + self.driver.__class__.__name__ + " volume " \
"driver's get_volume_stats operation ran for " \
"%(seconds).1f seconds. This may indicate a " \
"performance problem with the backend which can lead " \
"to instability."
@timeutils.time_it(
LOG, log_level=logging.WARN, message=slowmsg,
min_duration=CONF.backend_stats_polling_interval / 2)
def get_stats():
return self.driver.get_volume_stats(refresh=True)
volume_stats = get_stats()
if self.extra_capabilities:
volume_stats.update(self.extra_capabilities)
if volume_stats:

7
releasenotes/notes/slow-get-volume-stats-91b84c6e661dc605.yaml

@ -0,0 +1,7 @@
---
features:
- |
Log a warning from the volume service when a volume driver's
get_volume_stats() call takes a long time to return. This can help
deployers troubleshoot a cinder-volume service misbehaving due to a
driver/backend performance issue.
Loading…
Cancel
Save