Add cAdvisor host plugin
Queries a given cAdvisor instance for host metrics. This is useful when the agent in running in a container and does not have access to all host level metrics Taken from original review https://review.openstack.org/#/c/391559/ Change-Id: Ifc3d514e5256bdbc6089ec653065348a488f7c57
This commit is contained in:
Kaiyan Sheng authored and Michael James Hoppal committed
1 parent
ab62bcd394
commit
059d7eedd5
4 files changed
+237
-2
No files matched your search
@@ -0,0 +1,10 @@
|
||||
# (C) Copyright 2017 Hewlett Packard Enterprise Development LP
|
||||
init_config:
|
||||
# Timeout on cAdvisor requests
|
||||
# connection_timeout: 3
|
||||
|
||||
instances:
|
||||
# URL of cAdvisor to connect to.
|
||||
- cadvisor_url: "http://127.0.0.1:4194"
|
||||
# You can also set the plugin to determine cadvisor url from running within a kubernetes container
|
||||
# kubernetes_detect_cadvisor: True
|
||||
+82
-1
@@ -23,8 +23,10 @@
|
||||
- [Network](#network)
|
||||
- [Monasca Agent](#monasca-agent)
|
||||
- [Limiting System Metrics](#limiting-system-metrics)
|
||||
- [A10](#a10)
|
||||
- [Apache](#apache)
|
||||
- [Cacti](#cacti)
|
||||
- [cAdvisor_host](#cadvisor_host)
|
||||
- [Check_MK_Local](#check_mk_local)
|
||||
- [Certificate Expiration (HTTPS)](#certificate-expiration-https)
|
||||
- [Couch](#couch)
|
||||
@@ -47,6 +49,12 @@
|
||||
- [HTTP Metrics](#http-metrics)
|
||||
- [IIS](#iis)
|
||||
- [Jenkins](#jenkins)
|
||||
- [JsonPlugin](#jsonplugin)
|
||||
- [Simple Reporting](#simple-reporting)
|
||||
- [Writing and Locking the Metrics File](#writing-and-locking-the-metrics-file)
|
||||
- [Additional Directives](#additional-directives)
|
||||
- [Custom JSON file locations](#custom-json-file-locations)
|
||||
- [The monasca.json_plugin.status Metric](#the-monascajson_pluginstatus-metric)
|
||||
- [Kafka Checks](#kafka-checks)
|
||||
- [KyotoTycoon](#kyototycoon)
|
||||
- [Libvirt VM Monitoring](#libvirt-vm-monitoring)
|
||||
@@ -56,6 +64,7 @@
|
||||
- [MK Livestatus](#mk-livestatus)
|
||||
- [Mongo](#mongo)
|
||||
- [MySQL Checks](#mysql-checks)
|
||||
- [Note](#note)
|
||||
- [Nagios Wrapper](#nagios-wrapper)
|
||||
- [Nginx](#nginx)
|
||||
- [NTP](#ntp)
|
||||
@@ -121,6 +130,7 @@ The following plugins are delivered via setup as part of the standard plugin che
|
||||
| a10 | | |
|
||||
| apache | /root/.apache.cnf | Apache web server |
|
||||
| cacti | | |
|
||||
| cAdvisor_host | | |
|
||||
| cert_check | | |
|
||||
| check_mk_local | | |
|
||||
| couch | | |
|
||||
@@ -502,6 +512,77 @@ The Apache checks return the following metrics:
|
||||
## Cacti
|
||||
See [the example configuration](https://github.com/openstack/monasca-agent/blob/master/conf.d/cacti.yaml.example) for how to configure the Cacti plugin.
|
||||
|
||||
## cAdvisor_host
|
||||
This plugin collects metrics about a host from a given cAdvisor instance. This is useful in a container environment where
|
||||
the agent is running in a container but still wants to monitor the underlying hosts.
|
||||
|
||||
It connects to the cAdvisor instance and queries the stats API about the host.
|
||||
|
||||
There are two ways to configure the plugin.
|
||||
* Set cAdvisor url
|
||||
* Set kubernetes detect url to True. If true, the assumption is that the Agent is running in a Kubernetes
|
||||
container. The agent will obtain the cAdvisor url by first querying the Kubernetes API to ask which node it is running on and
|
||||
then from there hit the local cAdvisor on that node that is included in the kubelet.
|
||||
|
||||
As a result the plugin only supports getting data from one cAdvisor endpoint. So the config yaml file must
|
||||
only have one instance defined under instances. (Example shown below)
|
||||
|
||||
Sample config (passing in cAdvisor url):
|
||||
|
||||
```
|
||||
init_config:
|
||||
# Timeout on GET requests to the cAdvisor endpoints
|
||||
connection_timeout: 3
|
||||
instances:
|
||||
# Set to the url of the cAdvisor instance you want to connect to
|
||||
- cadvisor_url: "127.0.0.1:4194"
|
||||
```
|
||||
|
||||
Sample config (setting Kubernetes detect cAdvisor url):
|
||||
```
|
||||
init_config:
|
||||
# Timeout on GET requests to the cAdvisor endpoints
|
||||
connection_timeout: 3
|
||||
instances:
|
||||
# Set to the url of the cAdvisor instance you want to connect to
|
||||
- kubernetes_detect_cadvisor: True
|
||||
```
|
||||
|
||||
**Note if both a url and detect cAdvisor are both set it will by default use the url**
|
||||
|
||||
|
||||
The cAdvisor host check returns the following metrics:
|
||||
|
||||
| Metric Name | Dimensions | Semantics |
|
||||
| ----------- | ---------- | --------- |
|
||||
| cpu.system_time | hostname, unit | Cumulative system CPU time consumed in core seconds
|
||||
| cpu.system_time_sec | hostname, unit | Rate of system CPU time consumed in core seconds per second
|
||||
| cpu.total_time | hostname, unit | Cumulative CPU time consumed in core seconds
|
||||
| cpu.total_time_sec | hostname, unit | Rate of CPU time consumed in core seconds per second
|
||||
| cpu.user_time | hostname, unit | Cumulative user cpu time consumed in core seconds
|
||||
| cpu.user_time_sec | hostname, unit | Rate of user CPU time consumed in core seconds per second
|
||||
| fs.total_bytes | hostname, device, unit | Number of bytes available
|
||||
| fs.usage_bytes | hostname, device, unit | Number of bytes consumed
|
||||
| mem.cache_bytes | hostname, unit | Number of bytes of page cache memory
|
||||
| mem.swap_bytes | hostname, unit | Swap usage in memory in bytes
|
||||
| mem.used_bytes | hostname, unit | Current memory in use in bytes
|
||||
| net.in_bytes | hostname, interface, unit | Total network bytes received
|
||||
| net.in_bytes_sec | hostname, interface, unit | Number of network bytes received per second
|
||||
| net.in_dropped_packets | hostname, interface, unit | Total inbound network packets dropped
|
||||
| net.in_dropped_packets_sec | hostname, interface, unit | Number of inbound network packets dropped per second
|
||||
| net.in_errors | hostname, interface, unit | Total network errors on incoming network traffic
|
||||
| net.in_errors_sec | hostname, interface, unit | Number of network errors on incoming network traffic per second
|
||||
| net.in_packets | hostname, interface, unit | Total network packets received
|
||||
| net.in_packets_sec | hostname, interface, unit | Number of network packets received per second
|
||||
| net.out_bytes | hostname, interface, unit | Total network bytes sent
|
||||
| net.out_bytes_sec | hostname, interface, unit | Number of network bytes sent per second
|
||||
| net.out_dropped_packets | hostname, interface, unit | Total outbound network packets dropped
|
||||
| net.out_dropped_packets_sec | hostname, interface, unit | Number of outbound network packets dropped per second
|
||||
| net.out_errors | hostname, interface, unit | Total network errors on outgoing network traffic
|
||||
| net.out_errors_sec | hostname, interface, unit | Number of network errors on outgoing network traffic per second
|
||||
| net.out_packets | hostname, interface, unit | Total network packets sent
|
||||
| net.out_packets_sec | hostname, interface, unit | Number of network packets sent per second
|
||||
|
||||
## Check_MK_Local
|
||||
The [Check_MK](http://mathias-kettner.com/check_mk.html) [Agent](http://mathias-kettner.com/checkmk_linuxagent.html) can be extended through a series of [local checks](http://mathias-kettner.com/checkmk_localchecks.html). This plugin parses the `<<<local>>>` output of `check_mk_agent` and converts them into Monasca metrics. It is installed by `monasca-setup` automatically when the `check_mk_agent` script is found to be installed on the system.
|
||||
|
||||
@@ -2088,4 +2169,4 @@ The following Freezer processes are monitored, if they exist when the monasca-se
|
||||
=======
|
||||
|
||||
# License
|
||||
(C) Copyright 2015-2016 Hewlett Packard Enterprise Development LP
|
||||
(C) Copyright 2015-2017 Hewlett Packard Enterprise Development LP
|
||||
@@ -0,0 +1,143 @@
|
||||
# (C) Copyright 2017 Hewlett Packard Enterprise Development LP
|
||||
import requests
|
||||
import six
|
||||
|
||||
from monasca_agent.collector.checks import AgentCheck
|
||||
from monasca_agent.collector.checks import utils
|
||||
|
||||
DEFAULT_TIMEOUT = "3"
|
||||
|
||||
# format for METRICS: (cadvisor metric name, [metric types], [metric units])
|
||||
METRICS = {
|
||||
"cpu_metrics": {
|
||||
"system": ("system_time", ["gauge", "rate"],
|
||||
["core_seconds", "core_seconds_per_second"]),
|
||||
"total": ("total_time", ["gauge", "rate"],
|
||||
["core_seconds", "core_seconds_per_second"]),
|
||||
"user": ("user_time", ["gauge", "rate"],
|
||||
["core_seconds", "core_seconds_per_second"])
|
||||
},
|
||||
"memory_metrics": {
|
||||
"swap": ("swap_bytes", ["gauge"], ["bytes"]),
|
||||
"cache": ("cache_bytes", ["gauge"], ["bytes"]),
|
||||
"usage": ("used_bytes", ["gauge"], ["bytes"]),
|
||||
},
|
||||
"filesystem_metrics": {
|
||||
"capacity": ("total_bytes", ["gauge"], ["bytes"]),
|
||||
"usage": ("usage_bytes", ["gauge"], ["bytes"])
|
||||
},
|
||||
'network_metrics': {
|
||||
"rx_bytes": ("in_bytes", ["gauge", "rate"],
|
||||
["bytes", "bytes_per_second"]),
|
||||
"tx_bytes": ("out_bytes", ["gauge", "rate"],
|
||||
["bytes", "bytes_per_second"]),
|
||||
"rx_packets": ("in_packets", ["gauge", "rate"],
|
||||
["packets", "packets_per_second"]),
|
||||
"tx_packets": ("out_packets", ["gauge", "rate"],
|
||||
["packets", "packets_per_second"]),
|
||||
"rx_dropped": ("in_dropped_packets", ["gauge", "rate"],
|
||||
["packets", "packets_per_second"]),
|
||||
"tx_dropped": ("out_dropped_packets", ["gauge", "rate"],
|
||||
["packets", "packets_per_second"]),
|
||||
"rx_errors": ("in_errors", ["gauge", "rate"],
|
||||
["errors", "errors_per_second"]),
|
||||
"tx_errors": ("out_errors", ["gauge", "rate"],
|
||||
["errors", "errors_per_second"])
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
class CadvisorHost(AgentCheck):
|
||||
"""Queries given cAdvisor API for node metrics
|
||||
"""
|
||||
|
||||
def __init__(self, name, init_config, agent_config, instances=None):
|
||||
AgentCheck.__init__(self, name, init_config, agent_config, instances)
|
||||
if instances is not None and len(instances) > 1:
|
||||
raise Exception('cAdvisor host check only supports one configured instance.')
|
||||
self.connection_timeout = int(init_config.get('connection_timeout', DEFAULT_TIMEOUT))
|
||||
self.cadvisor_url = None
|
||||
|
||||
def check(self, instance):
|
||||
if not self.cadvisor_url:
|
||||
cadvisor_url = instance.get("cadvisor_url", None)
|
||||
detect_cadvisor_url = instance.get("kubernetes_detect_cadvisor", False)
|
||||
if not cadvisor_url:
|
||||
if detect_cadvisor_url:
|
||||
kubernetes_connector = utils.KubernetesConnector(self.connection_timeout)
|
||||
host = kubernetes_connector.get_agent_pod_host()
|
||||
cadvisor_url = "http://{}:4194".format(host)
|
||||
else:
|
||||
exception_message = "Either cAdvisor url or kubernetes detect cAdvisor must be set when " \
|
||||
"monitoring a Kubernetes Node."
|
||||
self.log.error(exception_message)
|
||||
raise Exception(exception_message)
|
||||
self.cadvisor_url = "{}/{}".format(cadvisor_url, "api/v2.0/stats?count=1")
|
||||
dimensions = self._set_dimensions(None, instance)
|
||||
try:
|
||||
host_metrics = requests.get(self.cadvisor_url, self.connection_timeout).json()
|
||||
except Exception as e:
|
||||
self.log.error("Error communicating with cAdvisor to collect data - {}".format(e))
|
||||
else:
|
||||
self._parse_send_metrics(host_metrics, dimensions)
|
||||
|
||||
def _send_metrics(self, metric_name, value, dimensions, metric_types,
|
||||
metric_units):
|
||||
for metric_type in metric_types:
|
||||
if metric_type == 'rate':
|
||||
dimensions.update({'unit': metric_units[metric_types.index('rate')]})
|
||||
self.rate(metric_name + "_sec", value, dimensions)
|
||||
elif metric_type == 'gauge':
|
||||
dimensions.update({'unit': metric_units[metric_types.index('gauge')]})
|
||||
self.gauge(metric_name, value, dimensions)
|
||||
|
||||
def _parse_memory(self, memory_data, dimensions):
|
||||
memory_metrics = METRICS['memory_metrics']
|
||||
for cadvisor_key, (metric_name, metric_types, metric_units) in six.iteritems(memory_metrics):
|
||||
if cadvisor_key in memory_data:
|
||||
self._send_metrics("mem." + metric_name, memory_data[cadvisor_key], dimensions,
|
||||
metric_types, metric_units)
|
||||
|
||||
def _parse_filesystem(self, filesystem_data, dimensions):
|
||||
filesystem_metrics = METRICS['filesystem_metrics']
|
||||
for filesystem in filesystem_data:
|
||||
file_dimensions = dimensions.copy()
|
||||
file_dimensions['device'] = filesystem['device']
|
||||
for cadvisor_key, (metric_name, metric_types, metric_units) in six.iteritems(filesystem_metrics):
|
||||
if cadvisor_key in filesystem:
|
||||
self._send_metrics("fs." + metric_name, filesystem[cadvisor_key], file_dimensions,
|
||||
metric_types, metric_units)
|
||||
|
||||
def _parse_network(self, network_data, dimensions):
|
||||
network_interfaces = network_data['interfaces']
|
||||
network_metrics = METRICS['network_metrics']
|
||||
for interface in network_interfaces:
|
||||
network_dimensions = dimensions.copy()
|
||||
network_dimensions['interface'] = interface['name']
|
||||
for cadvisor_key, (metric_name, metric_types, metric_units) in six.iteritems(network_metrics):
|
||||
if cadvisor_key in interface:
|
||||
self._send_metrics("net." + metric_name, interface[cadvisor_key], network_dimensions,
|
||||
metric_types, metric_units)
|
||||
|
||||
def _parse_cpu(self, cpu_data, dimensions):
|
||||
cpu_metrics = METRICS['cpu_metrics']
|
||||
cpu_usage = cpu_data['usage']
|
||||
for cadvisor_key, (metric_name, metric_types, metric_units) in six.iteritems(cpu_metrics):
|
||||
if cadvisor_key in cpu_usage:
|
||||
# Convert nanoseconds to seconds
|
||||
cpu_usage_sec = cpu_usage[cadvisor_key] / 1000000000.0
|
||||
self._send_metrics("cpu." + metric_name, cpu_usage_sec, dimensions, metric_types, metric_units)
|
||||
|
||||
def _parse_send_metrics(self, metrics, dimensions):
|
||||
for host, cadvisor_metrics in six.iteritems(metrics):
|
||||
host_dimensions = dimensions.copy()
|
||||
# Grab first set of metrics from return data
|
||||
cadvisor_metrics = cadvisor_metrics[0]
|
||||
if cadvisor_metrics['has_memory'] and cadvisor_metrics['memory']:
|
||||
self._parse_memory(cadvisor_metrics['memory'], host_dimensions)
|
||||
if cadvisor_metrics['has_filesystem'] and cadvisor_metrics['filesystem']:
|
||||
self._parse_filesystem(cadvisor_metrics['filesystem'], host_dimensions)
|
||||
if cadvisor_metrics['has_network'] and cadvisor_metrics['network']:
|
||||
self._parse_network(cadvisor_metrics['network'], host_dimensions)
|
||||
if cadvisor_metrics['has_cpu'] and cadvisor_metrics['cpu']:
|
||||
self._parse_cpu(cadvisor_metrics['cpu'], host_dimensions)
|
||||
+2
-1
@@ -2,7 +2,7 @@
|
||||
# of appearance. Changing the order has an impact on the overall integration
|
||||
# process, which may cause wedges in the gate later.
|
||||
|
||||
requests>=2.10.0 # Apache-2.0
|
||||
requests>=2.10.0,!=2.12.2,!=2.13.0 # Apache-2.0
|
||||
oslo.concurrency>=3.8.0 # Apache-2.0
|
||||
oslo.config>=3.14.0,!=3.18.0 # Apache-2.0
|
||||
oslo.i18n>=2.1.0 # Apache-2.0
|
||||
@@ -18,6 +18,7 @@ pymongo>=3.0.2,!=3.1
|
||||
python-memcached>=1.56 # PSF
|
||||
python-monascaclient>=1.1.0 # Apache-2.0
|
||||
redis>=2.10.0 # MIT
|
||||
six>=1.9.0 # MIT
|
||||
supervisor>=3.1.3,<3.4
|
||||
stevedore>=1.17.1 # Apache-2.0
|
||||
tornado>=4.3
|
||||
Reference in new issue
Block a user