Elastic: Add elasticsearch-curator for index management

The elasticsearch-curator chart is already part of
the helm/stable charts pulled in as part of monitor-helm.
The chart is patched up to the latest version which includes
the k8s 1.16 support.

Three actions are set in the manifest to delete old filebeat,
metricbeat and collectd indices according to sizes set through
environment variables which can be overridden.

Story: 2005733
Task: 37148

Change-Id: Ifad8a859583c898e47c62589381119b5fbc0b50f
Signed-off-by: Kevin Smith <kevin.smith@windriver.com>
This commit is contained in:
Kevin Smith 2019-10-16 08:18:42 -04:00
parent 97576c7dc3
commit cfb72e5681
3 changed files with 517 additions and 0 deletions

View File

@ -26,6 +26,7 @@ Patch05: 0005-decouple-config.patch
Patch06: 0006-add-system-info.patch
Patch07: 0007-three-masters.patch
Patch08: 0008-Update-stx-monitor-for-kubernetes-API-1.16.patch
Patch09: 0009-add-curator-as-of-2019-10-10.patch
BuildRequires: helm
@ -42,6 +43,7 @@ Monitor Helm charts
%patch06 -p1
%patch07 -p1
%patch08 -p1
%patch09 -p1
%build
# initialize helm and build the toolkit
@ -76,6 +78,7 @@ make kube-state-metrics
make kibana
make nginx-ingress
make logstash
make elasticsearch-curator
# terminate helm server (the last backgrounded task)
kill %1

View File

@ -0,0 +1,397 @@
From 21cd4d9720064f89843551e7da4c1e0528b6cbf5 Mon Sep 17 00:00:00 2001
From: Kevin Smith <kevin.smith@windriver.com>
Date: Thu, 10 Oct 2019 15:43:20 -0400
Subject: [PATCH 1/1] add curator as of 2019-10-10
---
stable/elasticsearch-curator/Chart.yaml | 6 +--
stable/elasticsearch-curator/OWNERS | 6 +--
stable/elasticsearch-curator/README.md | 34 ++++++++++---
.../ci/initcontainer-values.yaml | 9 ++++
.../elasticsearch-curator/templates/_helpers.tpl | 22 +++++++++
.../elasticsearch-curator/templates/cronjob.yaml | 10 ++++
stable/elasticsearch-curator/templates/psp.yml | 35 +++++++++++++
stable/elasticsearch-curator/templates/role.yaml | 23 +++++++++
.../templates/rolebinding.yaml | 21 ++++++++
.../templates/serviceaccount.yaml | 12 +++++
stable/elasticsearch-curator/values.yaml | 57 ++++++++++++++++++++--
11 files changed, 218 insertions(+), 17 deletions(-)
create mode 100644 stable/elasticsearch-curator/ci/initcontainer-values.yaml
create mode 100644 stable/elasticsearch-curator/templates/psp.yml
create mode 100644 stable/elasticsearch-curator/templates/role.yaml
create mode 100644 stable/elasticsearch-curator/templates/rolebinding.yaml
create mode 100644 stable/elasticsearch-curator/templates/serviceaccount.yaml
diff --git a/stable/elasticsearch-curator/Chart.yaml b/stable/elasticsearch-curator/Chart.yaml
index 24a37ce..7a8e0a7 100644
--- a/stable/elasticsearch-curator/Chart.yaml
+++ b/stable/elasticsearch-curator/Chart.yaml
@@ -2,7 +2,7 @@ apiVersion: v1
appVersion: "5.5.4"
description: A Helm chart for Elasticsearch Curator
name: elasticsearch-curator
-version: 1.3.2
+version: 2.0.2
home: https://github.com/elastic/curator
keywords:
- curator
@@ -12,7 +12,7 @@ sources:
- https://github.com/kubernetes/charts/elasticsearch-curator
- https://github.com/pires/docker-elasticsearch-curator
maintainers:
- - name: tmestdagh
- email: mestdagh.tom@gmail.com
+ - name: desaintmartin
+ email: cedric.dsm@gmail.com
- name: gianrubio
email: gianrubio@gmail.com
diff --git a/stable/elasticsearch-curator/OWNERS b/stable/elasticsearch-curator/OWNERS
index d8c0ba0..89df1c0 100644
--- a/stable/elasticsearch-curator/OWNERS
+++ b/stable/elasticsearch-curator/OWNERS
@@ -1,6 +1,6 @@
approvers:
- - tmestdagh
+ - desaintmartin
- gianrubio
reviewers:
- - tmestdagh
- - gianrubio
\ No newline at end of file
+ - desaintmartin
+ - gianrubio
diff --git a/stable/elasticsearch-curator/README.md b/stable/elasticsearch-curator/README.md
index 0a9f311..2057b85 100644
--- a/stable/elasticsearch-curator/README.md
+++ b/stable/elasticsearch-curator/README.md
@@ -23,6 +23,17 @@ To install the chart, use the following:
$ helm install stable/elasticsearch-curator
```
+## Upgrading an existing Release to a new major version
+
+A major chart version change (like v1.2.3 -> v2.0.0) indicates that there is an
+incompatible breaking change needing manual actions.
+
+### To 2.0.0
+
+v2.0.0 uses docker image from `elasticsearch-curator` author, which differs in its way to install curator.
+
+If you have a hardcoded `command` value, please update it to follow the new `curator` executable path: `/curator/curator` (which is not in PATH).
+
## Configuration
The following table lists the configurable parameters of the docker-registry chart and
@@ -31,8 +42,8 @@ their default values.
| Parameter | Description | Default |
| :----------------------------------- | :---------------------------------------------------------- | :------------------------------------------- |
| `image.pullPolicy` | Container pull policy | `IfNotPresent` |
-| `image.repository` | Container image to use | `quay.io/pires/docker-elasticsearch-curator` |
-| `image.tag` | Container image tag to deploy | `5.5.4` |
+| `image.repository` | Container image to use | `untergeek/curator` |
+| `image.tag` | Container image tag to deploy | `5.7.6` |
| `hooks` | Whether to run job on selected hooks | `{ "install": false, "upgrade": false }` |
| `cronjob.schedule` | Schedule for the CronJob | `0 1 * * *` |
| `cronjob.annotations` | Annotations to add to the cronjob | {} |
@@ -43,15 +54,22 @@ their default values.
| `dryrun` | Run Curator in dry-run mode | `false` |
| `env` | Environment variables to add to the cronjob container | {} |
| `envFromSecrets` | Environment variables from secrets to the cronjob container | {} |
-| `envFromSecrets.*.from.secret` | - `secretKeyRef.name` used for environment variable | |
-| `envFromSecrets.*.from.key` | - `secretKeyRef.key` used for environment variable | |
-| `command` | Command to execute | ["curator"] |
-| `configMaps.action_file_yml` | Contents of the Curator action_file.yml | See values.yaml |
-| `configMaps.config_yml` | Contents of the Curator config.yml (overrides config) | See values.yaml |
+| `envFromSecrets.*.from.secret` | - `secretKeyRef.name` used for environment variable | |
+| `envFromSecrets.*.from.key` | - `secretKeyRef.key` used for environment variable | |
+| `command` | Command to execute | ["/curator/curator"] |
+| `configMaps.action_file_yml` | Contents of the Curator action_file.yml | See values.yaml |
+| `configMaps.config_yml` | Contents of the Curator config.yml (overrides config) | See values.yaml |
| `resources` | Resource requests and limits | {} |
| `priorityClassName` | priorityClassName | `nil` |
| `extraVolumeMounts` | Mount extra volume(s), | |
| `extraVolumes` | Extra volumes | |
-| `securityContext` | Configure PodSecurityContext |
+| `extraInitContainers` | Init containers to add to the cronjob container | {} |
+| `securityContext` | Configure PodSecurityContext | `false` |
+| `rbac.enabled` | Enable RBAC resources | `false` |
+| `psp.create` | Create pod security policy resources | `false` |
+| `serviceAccount.create` | Create a default serviceaccount for elasticsearch curator | `true` |
+| `serviceAccount.name` | Name for elasticsearch curator serviceaccount | `""` |
+
+
Specify each parameter using the `--set key=value[,key=value]` argument to
`helm install`.
diff --git a/stable/elasticsearch-curator/ci/initcontainer-values.yaml b/stable/elasticsearch-curator/ci/initcontainer-values.yaml
new file mode 100644
index 0000000..578becf
--- /dev/null
+++ b/stable/elasticsearch-curator/ci/initcontainer-values.yaml
@@ -0,0 +1,9 @@
+extraInitContainers:
+ test:
+ image: alpine:latest
+ command:
+ - "/bin/sh"
+ - "-c"
+ args:
+ - |
+ true
diff --git a/stable/elasticsearch-curator/templates/_helpers.tpl b/stable/elasticsearch-curator/templates/_helpers.tpl
index c786fb5..8018c5d 100644
--- a/stable/elasticsearch-curator/templates/_helpers.tpl
+++ b/stable/elasticsearch-curator/templates/_helpers.tpl
@@ -12,6 +12,17 @@ Return the appropriate apiVersion for cronjob APIs.
{{- end -}}
{{/*
+Return the appropriate apiVersion for podsecuritypolicy.
+*/}}
+{{- define "podsecuritypolicy.apiVersion" -}}
+{{- if semverCompare "<1.10-0" .Capabilities.KubeVersion.GitVersion -}}
+{{- print "extensions/v1beta1" -}}
+{{- else -}}
+{{- print "policy/v1beta1" -}}
+{{- end -}}
+{{- end -}}
+
+{{/*
Expand the name of the chart.
*/}}
{{- define "elasticsearch-curator.name" -}}
@@ -42,3 +53,14 @@ Create chart name and version as used by the chart label.
{{- define "elasticsearch-curator.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}}
{{- end -}}
+
+{{/*
+Create the name of the service account to use
+*/}}
+{{- define "elasticsearch-curator.serviceAccountName" -}}
+{{- if .Values.serviceAccount.create -}}
+ {{ default (include "elasticsearch-curator.fullname" .) .Values.serviceAccount.name }}
+{{- else -}}
+ {{ default "default" .Values.serviceAccount.name }}
+{{- end -}}
+{{- end -}}
diff --git a/stable/elasticsearch-curator/templates/cronjob.yaml b/stable/elasticsearch-curator/templates/cronjob.yaml
index d0388f4..37274f6 100644
--- a/stable/elasticsearch-curator/templates/cronjob.yaml
+++ b/stable/elasticsearch-curator/templates/cronjob.yaml
@@ -53,6 +53,16 @@ spec:
imagePullSecrets:
- name: {{ .Values.image.pullSecret }}
{{- end }}
+{{- if .Values.extraInitContainers }}
+ initContainers:
+{{- range $key, $value := .Values.extraInitContainers }}
+ - name: "{{ $key }}"
+{{ toYaml $value | indent 12 }}
+{{- end }}
+{{- end }}
+ {{- if .Values.rbac.enabled }}
+ serviceAccountName: {{ template "elasticsearch-curator.serviceAccountName" .}}
+ {{- end }}
containers:
- name: {{ .Chart.Name }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
diff --git a/stable/elasticsearch-curator/templates/psp.yml b/stable/elasticsearch-curator/templates/psp.yml
new file mode 100644
index 0000000..5f62985
--- /dev/null
+++ b/stable/elasticsearch-curator/templates/psp.yml
@@ -0,0 +1,35 @@
+{{- if .Values.psp.create }}
+apiVersion: {{ template "podsecuritypolicy.apiVersion" . }}
+kind: PodSecurityPolicy
+metadata:
+ labels:
+ app: {{ template "elasticsearch-curator.name" . }}
+ chart: {{ template "elasticsearch-curator.chart" . }}
+ release: {{ .Release.Name }}
+ heritage: {{ .Release.Service }}
+ name: {{ template "elasticsearch-curator.fullname" . }}-psp
+spec:
+ privileged: true
+ #requiredDropCapabilities:
+ volumes:
+ - 'configMap'
+ - 'secret'
+ hostNetwork: false
+ hostIPC: false
+ hostPID: false
+ runAsUser:
+ rule: 'RunAsAny'
+ seLinux:
+ rule: 'RunAsAny'
+ supplementalGroups:
+ rule: 'MustRunAs'
+ ranges:
+ - min: 1
+ max: 65535
+ fsGroup:
+ rule: 'MustRunAs'
+ ranges:
+ - min: 1
+ max: 65535
+ readOnlyRootFilesystem: false
+{{- end }}
diff --git a/stable/elasticsearch-curator/templates/role.yaml b/stable/elasticsearch-curator/templates/role.yaml
new file mode 100644
index 0000000..8867f67
--- /dev/null
+++ b/stable/elasticsearch-curator/templates/role.yaml
@@ -0,0 +1,23 @@
+{{- if .Values.rbac.enabled }}
+kind: Role
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+ labels:
+ app: {{ template "elasticsearch-curator.name" . }}
+ chart: {{ template "elasticsearch-curator.chart" . }}
+ heritage: {{ .Release.Service }}
+ release: {{ .Release.Name }}
+ component: elasticsearch-curator-configmap
+ name: {{ template "elasticsearch-curator.name" . }}-role
+rules:
+- apiGroups: [""]
+ resources: ["configmaps"]
+ verbs: ["update", "patch"]
+{{- if .Values.psp.create }}
+- apiGroups: ["extensions"]
+ resources: ["podsecuritypolicies"]
+ verbs: ["use"]
+ resourceNames:
+ - {{ template "elasticsearch-curator.fullname" . }}-psp
+{{- end -}}
+{{- end -}}
diff --git a/stable/elasticsearch-curator/templates/rolebinding.yaml b/stable/elasticsearch-curator/templates/rolebinding.yaml
new file mode 100644
index 0000000..d25d2e1
--- /dev/null
+++ b/stable/elasticsearch-curator/templates/rolebinding.yaml
@@ -0,0 +1,21 @@
+{{- if .Values.rbac.enabled -}}
+kind: RoleBinding
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+ labels:
+ app: {{ template "elasticsearch-curator.name" . }}
+ chart: {{ template "elasticsearch-curator.chart" . }}
+ heritage: {{ .Release.Service }}
+ release: {{ .Release.Name }}
+ component: elasticsearch-curator-configmap
+ name: {{ template "elasticsearch-curator.name" . }}-rolebinding
+roleRef:
+ kind: Role
+ name: {{ template "elasticsearch-curator.name" . }}-role
+ apiGroup: rbac.authorization.k8s.io
+subjects:
+ - kind: ServiceAccount
+ name: {{ template "elasticsearch-curator.serviceAccountName" . }}
+ namespace: {{ .Release.Namespace }}
+{{- end -}}
+
diff --git a/stable/elasticsearch-curator/templates/serviceaccount.yaml b/stable/elasticsearch-curator/templates/serviceaccount.yaml
new file mode 100644
index 0000000..ad9c5c9
--- /dev/null
+++ b/stable/elasticsearch-curator/templates/serviceaccount.yaml
@@ -0,0 +1,12 @@
+{{- if and .Values.serviceAccount.create .Values.rbac.enabled }}
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+ name: {{ template "elasticsearch-curator.serviceAccountName" .}}
+ labels:
+ app: {{ template "elasticsearch-curator.fullname" . }}
+ chart: {{ template "elasticsearch-curator.chart" . }}
+ release: "{{ .Release.Name }}"
+ heritage: "{{ .Release.Service }}"
+{{- end }}
+
diff --git a/stable/elasticsearch-curator/values.yaml b/stable/elasticsearch-curator/values.yaml
index 3779be1..460f2a4 100644
--- a/stable/elasticsearch-curator/values.yaml
+++ b/stable/elasticsearch-curator/values.yaml
@@ -13,9 +13,25 @@ cronjob:
pod:
annotations: {}
+rbac:
+ # Specifies whether RBAC should be enabled
+ enabled: false
+
+serviceAccount:
+ # Specifies whether a ServiceAccount should be created
+ create: true
+ # The name of the ServiceAccount to use.
+ # If not set and create is true, a name is generated using the fullname template
+ name:
+
+
+psp:
+ # Specifies whether a podsecuritypolicy should be created
+ create: false
+
image:
- repository: quay.io/pires/docker-elasticsearch-curator
- tag: 5.5.4
+ repository: untergeek/curator
+ tag: 5.7.6
pullPolicy: IfNotPresent
hooks:
@@ -25,7 +41,7 @@ hooks:
# run curator in dry-run mode
dryrun: false
-command: ["curator"]
+command: ["/curator/curator"]
env: {}
configMaps:
@@ -101,5 +117,40 @@ priorityClassName: ""
# mountPath: /certs
# readOnly: true
+# Add your own init container or uncomment and modify the given example.
+extraInitContainers: {}
+ ## Don't configure S3 repository till Elasticsearch is reachable.
+ ## Ensure that it is available at http://elasticsearch:9200
+ ##
+ # elasticsearch-s3-repository:
+ # image: jwilder/dockerize:latest
+ # imagePullPolicy: "IfNotPresent"
+ # command:
+ # - "/bin/sh"
+ # - "-c"
+ # args:
+ # - |
+ # ES_HOST=elasticsearch
+ # ES_PORT=9200
+ # ES_REPOSITORY=backup
+ # S3_REGION=us-east-1
+ # S3_BUCKET=bucket
+ # S3_BASE_PATH=backup
+ # S3_COMPRESS=true
+ # S3_STORAGE_CLASS=standard
+ # apk add curl --no-cache && \
+ # dockerize -wait http://${ES_HOST}:${ES_PORT} --timeout 120s && \
+ # cat <<EOF | curl -sS -XPUT -H "Content-Type: application/json" -d @- http://${ES_HOST}:${ES_PORT}/_snapshot/${ES_REPOSITORY} \
+ # {
+ # "type": "s3",
+ # "settings": {
+ # "bucket": "${S3_BUCKET}",
+ # "base_path": "${S3_BASE_PATH}",
+ # "region": "${S3_REGION}",
+ # "compress": "${S3_COMPRESS}",
+ # "storage_class": "${S3_STORAGE_CLASS}"
+ # }
+ # }
+
securityContext:
runAsUser: 16 # run as cron user instead of root
--
1.8.3.1

View File

@ -208,6 +208,112 @@ metadata:
schema: metadata/Document/v1
---
schema: armada/Chart/v1
data:
chart_name: elasticsearch-curator
dependencies: []
install:
no_hooks: false
namespace: monitor
release: elasticsearch-curator
source:
location: http://172.17.0.1:8080/helm_charts/starlingx/elasticsearch-curator-2.0.2.tgz
reference: master
subpath: elasticsearch-curator
type: tar
test:
enabled: false
upgrade:
no_hooks: false
pre:
delete:
- labels:
release: mon-elasticsearch-curator
type: job
- labels:
component: test
release: mon-elasticsearch-curator
type: pod
values:
image:
repository: untergeek/curator
tag: 5.7.6
env:
FILEBEAT_INDEX_LIMIT_GB: 48
METRICBEAT_INDEX_LIMIT_GB: 38
COLLECTD_INDEX_LIMIT_GB: 9
configMaps:
action_file_yml: |-
---
actions:
1:
action: delete_indices
description: "Clean up ES filebeat indices"
options:
timeout_override:
continue_if_exception: False
disable_action: False
ignore_empty_list: True
filters:
- filtertype: pattern
kind: prefix
value: filebeat-
- filtertype: space
disk_space: ${FILEBEAT_INDEX_LIMIT_GB}
use_age: True
source: creation_date
2:
action: delete_indices
description: "Clean up ES metricbeat indices"
options:
timeout_override:
continue_if_exception: False
disable_action: False
ignore_empty_list: True
filters:
- filtertype: pattern
kind: prefix
value: metricbeat-
- filtertype: space
disk_space: ${METRICBEAT_INDEX_LIMIT_GB}
use_age: True
source: creation_date
3:
action: delete_indices
description: "Clean up ES collectd indices"
options:
timeout_override:
continue_if_exception: False
disable_action: False
ignore_empty_list: True
filters:
- filtertype: pattern
kind: prefix
value: collectd-
- filtertype: space
disk_space: ${COLLECTD_INDEX_LIMIT_GB}
use_age: True
source: creation_date
config_yml: |-
---
client:
hosts:
- mon-elasticsearch-client
port: 9200
logging:
loglevel: DEBUG
cronjob:
schedule: "0 1 * * *"
failedJobsHistoryLimit: "10"
successfulJobsHistoryLimit: "10"
nodeSelector:
elastic-controller: enabled
wait:
resources: []
metadata:
name: elasticsearch-curator
schema: metadata/Document/v1
---
schema: armada/Chart/v1
metadata:
schema: metadata/Document/v1
name: filebeat
@ -668,6 +774,16 @@ data:
chart_group:
- kibana
---
schema: armada/ChartGroup/v1
metadata:
schema: metadata/Document/v1
name: elasticsearch-curator
data:
description: "Deploy elasticsearch-curator"
sequenced: true
chart_group:
- elasticsearch-curator
---
schema: armada/Manifest/v1
metadata:
schema: metadata/Document/v1
@ -678,6 +794,7 @@ data:
- nginx-ingress
- kibana
- elasticsearch
- elasticsearch-curator
- logstash
- filebeat
- metricbeat