A powerful metric to watch for a swift cluster is the
number of handoff partitions on a drive on a storage node.
A build up of handoff nodes on a particular server could
indicate a disk problem somewhere in the cluster. A bottleneck
somewhere. Or better, when would be a good time to rebalance
the ring (as you'd want to do it when existing backend data
movement is at a minimum.
So it turns out to be a great visualisation of the health of
a cluster.
That's what this check plugin does. Each instance check takes
the following values:
ring: <path to a Swift ring file>
devices: <path to the directory of mountpoints>
granularity: <either server or device>
To be able to determine primary vs handoff partitions on a drive
the swift ring needs to be consulted. If a storage node stores
more then 1 ring, and an instance would be defined for each.
You give swift a bunch of disks. These disks are placed in what
swift calls the 'devices' location. That is a directory where a
mount point for each mounted swift drive is located.
Finally, you can decide on the granularity, which defaults to
`server` if not defined. Only 2 metrics are created from this
check:
swift.partitions.primary_count
swift.partitions.handoff_count
But with the hostname dimension a ring dimension will also be set.
Allowing the graphing of the handoff vs partitions of each ring.
When the granularity is set to device, then an additional
dimension to the metric is added, the device name (the name of
the devices mount point). This allows the graphing and monitoring
of each device in a server if a finer granularity is required.
Because we need to consult the Swift ring there is a runtime
requirement on the Python Swift module being installed. But
this isn't required for the unit tests. Making it a runtime
dependency means when the check is loaded it'll log an error
and then exit if it can't import the swift module.
This is the second of two Swift check plugins I've been working on.
For more details see my blog post[1]
[1] - https://oliver.net.au/?p=358
Change-Id: Ie91add9af39f2ab0e5b575390c0c6355563c0bfc
833e5946fe