openstack/openstacksdk
Code Issues Proposed changes

Allow for override of statsd/influxdb settings per cloud

Browse Source 
There is currently no way to separate out statsd settings per cloud.
While you can set a prefix other than 'openstack.api', this applies to
all clouds and groups all the stats together.

While this may be what you want, it may also be not what you want :)
For example in nodepool, we have multiple cloud providers who each
have their own grafana page, so we'd like them all to log themselves
to different stats buckets (i.e. set individual prefixes).

This allows setting a "metrics" field in each individual cloud entry
that overrides the global settings.  TBH I don't know if this is
important for InfluxDB as well, but I've implemented it for that too
for consistency.

Test cases are added for the global and merged settings.

I've also updated the documentation page a bit.  I've separated it
into subsections for the three types of stats available.  I removed
some of the in-depth stuff about logging types which wasn't that clear
(I think we can just present what we support and let people decide).

Change-Id: I9c3342161a257603f3cfd26bd03f6b71ffbfdd0d
This commit is contained in:
Ian Wienand 2021-04-19 14:24:02 +10:00
parent e13b59c7bd
commit 2395651429
4 changed files with 155 additions and 33 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

71
doc/source/user/guides/stats.rst
View File

@ -2,28 +2,23 @@
Statistics reporting
====================
`openstacksdk` offers possibility to report statistics on individual API
requests/responses in different formats. `Statsd` allows reporting of the
response times in the statsd format. `InfluxDB` allows a more event-oriented
reporting of the same data. `Prometheus` reporting is a bit different and
requires the application using SDK to take care of the metrics exporting, while
`openstacksdk` prepares the metrics.
`openstacksdk` can report statistics on individual API
requests/responses in several different formats.
Due to the nature of the `statsd` protocol lots of tools consuming the metrics
do the data aggregation and processing in the configurable time frame (mean
value calculation for a 1 minute time frame). For the case of periodic tasks
this might not be very useful. A better fit for using `openstacksdk` as a
library is an 'event'-recording, where duration of an individual request is
stored and all required calculations are done if necessary in the monitoring
system based required timeframe, or the data is simply shown as is with no
analytics. A `comparison
<https://prometheus.io/docs/introduction/comparison/>`_ article describes
differences in those approaches.
Note that metrics will be reported only when corresponding client
libraries (`statsd` for 'statsd' reporting, `influxdb` for influxdb,
etc.). If libraries are not available reporting will be silently
ignored.
Simple Usage
------------
statsd
------
To receive metrics add a following section to the config file (clouds.yaml):
`statsd` can be configured via configuration entries or environment
variables.
A global `metrics` entry defines defaults for all clouds. Each cloud
can specify a `metrics` section to override variables; this may be
useful to separate results reported for each cloud.
.. code-block:: yaml
@ -31,12 +26,26 @@ To receive metrics add a following section to the config file (clouds.yaml):
statsd:
host: __statsd_server_host__
port: __statsd_server_port__
prefix: __statsd_prefix__ (default 'openstack.api')
clouds:
..
a-cloud:
auth:
...
metrics:
statsd:
prefix: 'openstack.api.a-cloud'
If the `STATSD_HOST` or `STATSD_PORT` environment variables are set,
they will be taken as the default values (and enable `statsd`
reporting if no other configuration is specified).
In order to enable InfluxDB reporting following configuration need to be done
in the `clouds.yaml` file
InfluxDB
--------
`InfluxDB <https://www.influxdata.com/>`__ is supported via
configuration in the `metrics` field. Similar to `statsd`, each cloud
can provide it's own `metrics` section to override any global
defaults.
.. code-block:: yaml
@ -53,11 +62,6 @@ in the `clouds.yaml` file
clouds:
..
Metrics will be reported only when corresponding client libraries (
`statsd` for 'statsd' reporting, `influxdb` for influxdb reporting
correspondingly). When those libraries are not available reporting will be
silently ignored.
InfluxDB reporting allows setting additional tags into the metrics based on the
selected cloud.
@ -69,3 +73,16 @@ selected cloud.
...
additional_metric_tags:
environment: production
prometheus
----------
..
NOTE(ianw) 2021-04-19 : examples here would be great; this is just terse
description taken from
https://review.opendev.org/c/openstack/openstacksdk/+/614834
The prometheus support does not read from config, and does not run an
http service since OpenstackSDK is a library. It is expected that an
application that uses OpenstackSDK and wants request stats be
collected will pass a `prometheus_client.CollectorRegistry` to
`collector_registry`.

23
openstack/config/loader.py
View File

@ -1145,6 +1145,21 @@ class OpenStackConfig:
if not prefer_ipv6:
force_ipv4 = True
# Override global metrics config with more specific per-cloud
# details.
metrics_config = config.get('metrics', {})
statsd_config = metrics_config.get('statsd', {})
statsd_host = statsd_config.get('host') or self._statsd_host
statsd_port = statsd_config.get('port') or self._statsd_port
statsd_prefix = statsd_config.get('prefix') or self._statsd_prefix
influxdb_config = metrics_config.get('influxdb', {})
if influxdb_config:
merged_influxdb = copy.deepcopy(self._influxdb_config)
merged_influxdb.update(influxdb_config)
influxdb_config = merged_influxdb
else:
influxdb_config = self._influxdb_config
if cloud is None:
cloud_name = ''
else:
@ -1167,10 +1182,10 @@ class OpenStackConfig:
cache_class=self._cache_class,
cache_arguments=self._cache_arguments,
password_callback=self._pw_callback,
statsd_host=self._statsd_host,
statsd_port=self._statsd_port,
statsd_prefix=self._statsd_prefix,
influxdb_config=self._influxdb_config,
statsd_host=statsd_host,
statsd_port=statsd_port,
statsd_prefix=statsd_prefix,
influxdb_config=influxdb_config,
)
# TODO(mordred) Backwards compat for OSC transition
get_one_cloud = get_one

35
openstack/tests/unit/config/base.py
View File

@ -47,6 +47,22 @@ USER_CONF = {
'client': {
'force_ipv4': True,
},
'metrics': {
'statsd': {
'host': '127.0.0.1',
'port': '1234'
},
'influxdb': {
'host': '127.0.0.1',
'port': '1234',
'use_udp': True,
'username': 'username',
'password': 'password',
'database': 'database',
'measurement': 'measurement.name',
'timeout': 10,
}
},
'clouds': {
'_test-cloud_': {
'profile': '_test_cloud_in_our_cloud',
@ -172,6 +188,25 @@ USER_CONF = {
'domain-id': '12345',
},
},
'_test-cloud-override-metrics': {
'auth': {
'auth_url': 'http://example.com/v2',
'username': 'testuser',
'password': 'testpass',
},
'metrics': {
'statsd': {
'host': '127.0.0.1',
'port': 4321,
'prefix': 'statsd.override.prefix'
},
'influxdb': {
'username': 'override-username',
'password': 'override-password',
'database': 'override-database',
}
},
},
},
'ansible': {
'expand-hostvars': False,

59
openstack/tests/unit/config/test_config.py
View File

@ -313,7 +313,7 @@ class TestConfig(base.TestCase):
def test_get_cloud_names(self):
c = config.OpenStackConfig(config_files=[self.cloud_yaml],
secure_files=[self.no_yaml])
self.assertEqual(
self.assertCountEqual(
['_test-cloud-domain-id_',
'_test-cloud-domain-scoped_',
'_test-cloud-int-project_',
@ -323,8 +323,9 @@ class TestConfig(base.TestCase):
'_test_cloud_hyphenated',
'_test_cloud_no_vendor',
'_test_cloud_regions',
'_test-cloud-override-metrics',
],
sorted(c.get_cloud_names()))
c.get_cloud_names())
c = config.OpenStackConfig(config_files=[self.no_yaml],
vendor_files=[self.no_yaml],
secure_files=[self.no_yaml])
@ -513,6 +514,60 @@ class TestConfig(base.TestCase):
'openstacksdk', region._auth.get_cache_id(),
region._auth.get_auth_state())
def test_metrics_global(self):
c = config.OpenStackConfig(config_files=[self.cloud_yaml],
vendor_files=[self.vendor_yaml],
secure_files=[self.secure_yaml])
self.assertIsInstance(c.cloud_config, dict)
cc = c.get_one('_test-cloud_')
statsd = {
'host': '127.0.0.1',
'port': '1234',
}
# NOTE(ianw) we don't test/call get_<stat>_client() because we
# don't want to instantiate the client, which tries to
# connect / do hostname lookups.
self.assertEqual(statsd['host'], cc._statsd_host)
self.assertEqual(statsd['port'], cc._statsd_port)
self.assertEqual('openstack.api', cc.get_statsd_prefix())
influxdb = {
'use_udp': True,
'host': '127.0.0.1',
'port': '1234',
'username': 'username',
'password': 'password',
'database': 'database',
'measurement': 'measurement.name',
'timeout': 10
}
self.assertEqual(influxdb, cc._influxdb_config)
def test_metrics_override(self):
c = config.OpenStackConfig(config_files=[self.cloud_yaml],
vendor_files=[self.vendor_yaml],
secure_files=[self.secure_yaml])
self.assertIsInstance(c.cloud_config, dict)
cc = c.get_one('_test-cloud-override-metrics')
statsd = {
'host': '127.0.0.1',
'port': '4321',
'prefix': 'statsd.override.prefix'
}
self.assertEqual(statsd['host'], cc._statsd_host)
self.assertEqual(statsd['port'], cc._statsd_port)
self.assertEqual(statsd['prefix'], cc.get_statsd_prefix())
influxdb = {
'use_udp': True,
'host': '127.0.0.1',
'port': '1234',
'username': 'override-username',
'password': 'override-password',
'database': 'override-database',
'measurement': 'measurement.name',
'timeout': 10
}
self.assertEqual(influxdb, cc._influxdb_config)
class TestExcludedFormattedConfigValue(base.TestCase):
# verify https://storyboard.openstack.org/#!/story/1635696