kolla-ansible/bug-2048223-bb66fa11c6b36c5e.yaml at a1eec2498ab27367a729cfb07e7e614073d1415a - kolla-ansible - OpenDev: Free Software Needs Free Tools

openstack/kolla-ansible

Mark Goddard 97e5c0e9b1 cadvisor: Set housekeeping interval to Prometheus scrape interval

The prometheus_cadvisor container has high CPU usage. On various
production systems I checked it sits around 13-16% on controllers,
averaged over the prometheus 1m scrape interval. When viewed with top we
can see it is a bit spikey and can jump over 100%.

There are various bugs about this, but I found
https://github.com/google/cadvisor/issues/2523 which suggests reducing
the per-container housekeeping interval. This defaults to 1s, which
provides far greater granularity than we need with the default
prometheus scrape interval of 60s.

Reducing the housekeeping interval to 60s on a production controller
reduced the CPU usage from 13% to 3.5% average. This still seems high,
but is more reasonable.

Change-Id: I89c62a45b1f358aafadcc0317ce882f4609543e7
Closes-Bug: #2048223

2024-01-05 11:02:41 +00:00

8 lines

269 B

YAML

Raw Blame History

 ---
 fixes:
   - |
     Fixes an issue with high CPU usage of the cAdvisor container by setting the
     per-container housekeeping interval to the same value as the Prometheus
     scrape interval. `LP#2048223
     <https://bugs.launchpad.net/kolla-ansible/+bug/2048223>`__