6cd7433086
When we migrated this to ansible I missed that we didn't bring across the storage-aggregation.conf file. This has had the unfortunate effect of regressing the xFilesFactor set for every newly created graphite stat since the migration. This setting is a percentage (0-1 float) of how much of a "bucket" needs to be non-null to keep the value when rolling up changes. We want this to be zero due to the sporadic nature of data (see the original change I5f416e798e7abedfde776c9571b6fc8cea5f3a33). This only affected newly created statistics, as graphite doesn't modify this setting once it creates the whisper file. This probably helped us overlook this for so long, as longer-existing stats were operating correctly, but newer were dropping data when zoomed out. Restore this setting, and double-check it in testinfra for the future. For simplicity and to get this back to the prior state I will manually update the on-disk .wsp files to this when this change applies. Change-Id: I57873403c4ca9783b1851ba83bfba038f4b90715
85 lines
2.1 KiB
YAML
85 lines
2.1 KiB
YAML
- name: Ensure docker-compose directories exists
|
|
file:
|
|
state: directory
|
|
path: '{{ item }}'
|
|
loop:
|
|
- /etc/graphite-docker
|
|
- /etc/graphite-docker/graphite
|
|
- /etc/graphite-docker/graphite/conf
|
|
|
|
- name: Write settings file
|
|
template:
|
|
src: docker-compose.yaml.j2
|
|
dest: /etc/graphite-docker/docker-compose.yaml
|
|
|
|
- name: Write storage config file
|
|
template:
|
|
src: storage-schemas.conf.j2
|
|
dest: /etc/graphite-docker/graphite/conf/storage-schemas.conf
|
|
|
|
- name: Write storage aggregation config file
|
|
template:
|
|
src: storage-aggregation.conf.j2
|
|
dest: /etc/graphite-docker/graphite/conf/storage-aggregation.conf
|
|
|
|
- name: Write nginx override config
|
|
template:
|
|
src: graphite-statsd.conf.j2
|
|
dest: /etc/graphite-docker/graphite-statsd.conf
|
|
|
|
- name: Write statsd override config
|
|
template:
|
|
src: statsd.js.j2
|
|
dest: /etc/graphite-docker/statsd.js
|
|
|
|
- name: Ensure storage directory exists
|
|
file:
|
|
state: directory
|
|
path: /opt/graphite/storage
|
|
|
|
- name: Ensure log directory exists
|
|
file:
|
|
state: directory
|
|
path: /var/log/graphite
|
|
|
|
- name: Run docker-compose pull
|
|
shell:
|
|
cmd: docker-compose pull
|
|
chdir: /etc/graphite-docker/
|
|
|
|
- name: Run docker-compose up
|
|
shell:
|
|
cmd: docker-compose up -d
|
|
chdir: /etc/graphite-docker/
|
|
|
|
- name: Run docker prune to cleanup unneeded images
|
|
shell:
|
|
cmd: docker image prune -f
|
|
|
|
# net is handy to have on the host for checking stat ingestion
|
|
# whisper utils are handy for inspecting .wsp files for debugging
|
|
- name: Install netcat
|
|
package:
|
|
name:
|
|
- netcat
|
|
- python3-whisper
|
|
state: present
|
|
|
|
# Removes files not updated for ~9 months, and clears out empty directories
|
|
- name: Run periodic cleanup
|
|
cron:
|
|
name: "Graphite cleanup"
|
|
minute: "0"
|
|
hour: "2"
|
|
job: >
|
|
bash -c 'echo "--- $(date) ---";
|
|
find /opt/graphite/storage/whisper -type f -mtime +270 -name "*.wsp" -delete -print;
|
|
find /opt/graphite/storage/whisper -depth -type d -empty -delete -print'
|
|
>> /var/log/graphite-cleanup.log 2>&1
|
|
|
|
- name: Rotate cleanup logs
|
|
include_role:
|
|
name: logrotate
|
|
vars:
|
|
logrotate_file_name: '/var/log/graphite-cleanup.log'
|