Allow for override of statsd/influxdb settings per cloud
There is currently no way to separate out statsd settings per cloud. While you can set a prefix other than 'openstack.api', this applies to all clouds and groups all the stats together. While this may be what you want, it may also be not what you want :) For example in nodepool, we have multiple cloud providers who each have their own grafana page, so we'd like them all to log themselves to different stats buckets (i.e. set individual prefixes). This allows setting a "metrics" field in each individual cloud entry that overrides the global settings. TBH I don't know if this is important for InfluxDB as well, but I've implemented it for that too for consistency. Test cases are added for the global and merged settings. I've also updated the documentation page a bit. I've separated it into subsections for the three types of stats available. I removed some of the in-depth stuff about logging types which wasn't that clear (I think we can just present what we support and let people decide). Change-Id: I9c3342161a257603f3cfd26bd03f6b71ffbfdd0d
This commit is contained in:
parent
e13b59c7bd
commit
2395651429
@ -2,28 +2,23 @@
|
||||
Statistics reporting
|
||||
====================
|
||||
|
||||
`openstacksdk` offers possibility to report statistics on individual API
|
||||
requests/responses in different formats. `Statsd` allows reporting of the
|
||||
response times in the statsd format. `InfluxDB` allows a more event-oriented
|
||||
reporting of the same data. `Prometheus` reporting is a bit different and
|
||||
requires the application using SDK to take care of the metrics exporting, while
|
||||
`openstacksdk` prepares the metrics.
|
||||
`openstacksdk` can report statistics on individual API
|
||||
requests/responses in several different formats.
|
||||
|
||||
Due to the nature of the `statsd` protocol lots of tools consuming the metrics
|
||||
do the data aggregation and processing in the configurable time frame (mean
|
||||
value calculation for a 1 minute time frame). For the case of periodic tasks
|
||||
this might not be very useful. A better fit for using `openstacksdk` as a
|
||||
library is an 'event'-recording, where duration of an individual request is
|
||||
stored and all required calculations are done if necessary in the monitoring
|
||||
system based required timeframe, or the data is simply shown as is with no
|
||||
analytics. A `comparison
|
||||
<https://prometheus.io/docs/introduction/comparison/>`_ article describes
|
||||
differences in those approaches.
|
||||
Note that metrics will be reported only when corresponding client
|
||||
libraries (`statsd` for 'statsd' reporting, `influxdb` for influxdb,
|
||||
etc.). If libraries are not available reporting will be silently
|
||||
ignored.
|
||||
|
||||
Simple Usage
|
||||
------------
|
||||
statsd
|
||||
------
|
||||
|
||||
To receive metrics add a following section to the config file (clouds.yaml):
|
||||
`statsd` can be configured via configuration entries or environment
|
||||
variables.
|
||||
|
||||
A global `metrics` entry defines defaults for all clouds. Each cloud
|
||||
can specify a `metrics` section to override variables; this may be
|
||||
useful to separate results reported for each cloud.
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
@ -31,12 +26,26 @@ To receive metrics add a following section to the config file (clouds.yaml):
|
||||
statsd:
|
||||
host: __statsd_server_host__
|
||||
port: __statsd_server_port__
|
||||
prefix: __statsd_prefix__ (default 'openstack.api')
|
||||
clouds:
|
||||
..
|
||||
a-cloud:
|
||||
auth:
|
||||
...
|
||||
metrics:
|
||||
statsd:
|
||||
prefix: 'openstack.api.a-cloud'
|
||||
|
||||
If the `STATSD_HOST` or `STATSD_PORT` environment variables are set,
|
||||
they will be taken as the default values (and enable `statsd`
|
||||
reporting if no other configuration is specified).
|
||||
|
||||
In order to enable InfluxDB reporting following configuration need to be done
|
||||
in the `clouds.yaml` file
|
||||
InfluxDB
|
||||
--------
|
||||
|
||||
`InfluxDB <https://www.influxdata.com/>`__ is supported via
|
||||
configuration in the `metrics` field. Similar to `statsd`, each cloud
|
||||
can provide it's own `metrics` section to override any global
|
||||
defaults.
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
@ -53,11 +62,6 @@ in the `clouds.yaml` file
|
||||
clouds:
|
||||
..
|
||||
|
||||
Metrics will be reported only when corresponding client libraries (
|
||||
`statsd` for 'statsd' reporting, `influxdb` for influxdb reporting
|
||||
correspondingly). When those libraries are not available reporting will be
|
||||
silently ignored.
|
||||
|
||||
InfluxDB reporting allows setting additional tags into the metrics based on the
|
||||
selected cloud.
|
||||
|
||||
@ -69,3 +73,16 @@ selected cloud.
|
||||
...
|
||||
additional_metric_tags:
|
||||
environment: production
|
||||
|
||||
prometheus
|
||||
----------
|
||||
..
|
||||
NOTE(ianw) 2021-04-19 : examples here would be great; this is just terse
|
||||
description taken from
|
||||
https://review.opendev.org/c/openstack/openstacksdk/+/614834
|
||||
|
||||
The prometheus support does not read from config, and does not run an
|
||||
http service since OpenstackSDK is a library. It is expected that an
|
||||
application that uses OpenstackSDK and wants request stats be
|
||||
collected will pass a `prometheus_client.CollectorRegistry` to
|
||||
`collector_registry`.
|
||||
|
@ -1145,6 +1145,21 @@ class OpenStackConfig:
|
||||
if not prefer_ipv6:
|
||||
force_ipv4 = True
|
||||
|
||||
# Override global metrics config with more specific per-cloud
|
||||
# details.
|
||||
metrics_config = config.get('metrics', {})
|
||||
statsd_config = metrics_config.get('statsd', {})
|
||||
statsd_host = statsd_config.get('host') or self._statsd_host
|
||||
statsd_port = statsd_config.get('port') or self._statsd_port
|
||||
statsd_prefix = statsd_config.get('prefix') or self._statsd_prefix
|
||||
influxdb_config = metrics_config.get('influxdb', {})
|
||||
if influxdb_config:
|
||||
merged_influxdb = copy.deepcopy(self._influxdb_config)
|
||||
merged_influxdb.update(influxdb_config)
|
||||
influxdb_config = merged_influxdb
|
||||
else:
|
||||
influxdb_config = self._influxdb_config
|
||||
|
||||
if cloud is None:
|
||||
cloud_name = ''
|
||||
else:
|
||||
@ -1167,10 +1182,10 @@ class OpenStackConfig:
|
||||
cache_class=self._cache_class,
|
||||
cache_arguments=self._cache_arguments,
|
||||
password_callback=self._pw_callback,
|
||||
statsd_host=self._statsd_host,
|
||||
statsd_port=self._statsd_port,
|
||||
statsd_prefix=self._statsd_prefix,
|
||||
influxdb_config=self._influxdb_config,
|
||||
statsd_host=statsd_host,
|
||||
statsd_port=statsd_port,
|
||||
statsd_prefix=statsd_prefix,
|
||||
influxdb_config=influxdb_config,
|
||||
)
|
||||
# TODO(mordred) Backwards compat for OSC transition
|
||||
get_one_cloud = get_one
|
||||
|
@ -47,6 +47,22 @@ USER_CONF = {
|
||||
'client': {
|
||||
'force_ipv4': True,
|
||||
},
|
||||
'metrics': {
|
||||
'statsd': {
|
||||
'host': '127.0.0.1',
|
||||
'port': '1234'
|
||||
},
|
||||
'influxdb': {
|
||||
'host': '127.0.0.1',
|
||||
'port': '1234',
|
||||
'use_udp': True,
|
||||
'username': 'username',
|
||||
'password': 'password',
|
||||
'database': 'database',
|
||||
'measurement': 'measurement.name',
|
||||
'timeout': 10,
|
||||
}
|
||||
},
|
||||
'clouds': {
|
||||
'_test-cloud_': {
|
||||
'profile': '_test_cloud_in_our_cloud',
|
||||
@ -172,6 +188,25 @@ USER_CONF = {
|
||||
'domain-id': '12345',
|
||||
},
|
||||
},
|
||||
'_test-cloud-override-metrics': {
|
||||
'auth': {
|
||||
'auth_url': 'http://example.com/v2',
|
||||
'username': 'testuser',
|
||||
'password': 'testpass',
|
||||
},
|
||||
'metrics': {
|
||||
'statsd': {
|
||||
'host': '127.0.0.1',
|
||||
'port': 4321,
|
||||
'prefix': 'statsd.override.prefix'
|
||||
},
|
||||
'influxdb': {
|
||||
'username': 'override-username',
|
||||
'password': 'override-password',
|
||||
'database': 'override-database',
|
||||
}
|
||||
},
|
||||
},
|
||||
},
|
||||
'ansible': {
|
||||
'expand-hostvars': False,
|
||||
|
@ -313,7 +313,7 @@ class TestConfig(base.TestCase):
|
||||
def test_get_cloud_names(self):
|
||||
c = config.OpenStackConfig(config_files=[self.cloud_yaml],
|
||||
secure_files=[self.no_yaml])
|
||||
self.assertEqual(
|
||||
self.assertCountEqual(
|
||||
['_test-cloud-domain-id_',
|
||||
'_test-cloud-domain-scoped_',
|
||||
'_test-cloud-int-project_',
|
||||
@ -323,8 +323,9 @@ class TestConfig(base.TestCase):
|
||||
'_test_cloud_hyphenated',
|
||||
'_test_cloud_no_vendor',
|
||||
'_test_cloud_regions',
|
||||
'_test-cloud-override-metrics',
|
||||
],
|
||||
sorted(c.get_cloud_names()))
|
||||
c.get_cloud_names())
|
||||
c = config.OpenStackConfig(config_files=[self.no_yaml],
|
||||
vendor_files=[self.no_yaml],
|
||||
secure_files=[self.no_yaml])
|
||||
@ -513,6 +514,60 @@ class TestConfig(base.TestCase):
|
||||
'openstacksdk', region._auth.get_cache_id(),
|
||||
region._auth.get_auth_state())
|
||||
|
||||
def test_metrics_global(self):
|
||||
c = config.OpenStackConfig(config_files=[self.cloud_yaml],
|
||||
vendor_files=[self.vendor_yaml],
|
||||
secure_files=[self.secure_yaml])
|
||||
self.assertIsInstance(c.cloud_config, dict)
|
||||
cc = c.get_one('_test-cloud_')
|
||||
statsd = {
|
||||
'host': '127.0.0.1',
|
||||
'port': '1234',
|
||||
}
|
||||
# NOTE(ianw) we don't test/call get_<stat>_client() because we
|
||||
# don't want to instantiate the client, which tries to
|
||||
# connect / do hostname lookups.
|
||||
self.assertEqual(statsd['host'], cc._statsd_host)
|
||||
self.assertEqual(statsd['port'], cc._statsd_port)
|
||||
self.assertEqual('openstack.api', cc.get_statsd_prefix())
|
||||
influxdb = {
|
||||
'use_udp': True,
|
||||
'host': '127.0.0.1',
|
||||
'port': '1234',
|
||||
'username': 'username',
|
||||
'password': 'password',
|
||||
'database': 'database',
|
||||
'measurement': 'measurement.name',
|
||||
'timeout': 10
|
||||
}
|
||||
self.assertEqual(influxdb, cc._influxdb_config)
|
||||
|
||||
def test_metrics_override(self):
|
||||
c = config.OpenStackConfig(config_files=[self.cloud_yaml],
|
||||
vendor_files=[self.vendor_yaml],
|
||||
secure_files=[self.secure_yaml])
|
||||
self.assertIsInstance(c.cloud_config, dict)
|
||||
cc = c.get_one('_test-cloud-override-metrics')
|
||||
statsd = {
|
||||
'host': '127.0.0.1',
|
||||
'port': '4321',
|
||||
'prefix': 'statsd.override.prefix'
|
||||
}
|
||||
self.assertEqual(statsd['host'], cc._statsd_host)
|
||||
self.assertEqual(statsd['port'], cc._statsd_port)
|
||||
self.assertEqual(statsd['prefix'], cc.get_statsd_prefix())
|
||||
influxdb = {
|
||||
'use_udp': True,
|
||||
'host': '127.0.0.1',
|
||||
'port': '1234',
|
||||
'username': 'override-username',
|
||||
'password': 'override-password',
|
||||
'database': 'override-database',
|
||||
'measurement': 'measurement.name',
|
||||
'timeout': 10
|
||||
}
|
||||
self.assertEqual(influxdb, cc._influxdb_config)
|
||||
|
||||
|
||||
class TestExcludedFormattedConfigValue(base.TestCase):
|
||||
# verify https://storyboard.openstack.org/#!/story/1635696
|
||||
|
Loading…
Reference in New Issue
Block a user