Enabling liveness probe for MariaDB

In a DX scenario, after lock-unlock a controller the remaining MariaDB
instance (lets say maria-server0) goes to a Non-Primary + Initializing
State (non-operational). After that it remains searching for the now
deleted pod (maria-server1) but using the old IP, the one before the
restart. maria-server0 flags the old IP as delayed and suspect for
eviction, however being a Non-Primary member it cannot in fact evict
the old node and start looking for new members. Setting a LivenessProbe
that detects nonoperational members and restart them fixes this, as the
recreated pod starts looking for a cluster to join.

Closes-Bug: #1938346
Signed-off-by: Thiago Brito <thiago.brito@windriver.com>
Change-Id: I38d788f720cbd6bd13b6b6147db6f3d2a2ff9c92
This commit is contained in:
Thiago Brito 2021-07-31 00:26:49 -03:00
parent 52b3185a19
commit e11888c1df
2 changed files with 15 additions and 5 deletions

View File

@ -17,21 +17,22 @@ parameters and override the related probe pareameters.
Signed-off-by: Hu, Yong <yong.hu@intel.com> Signed-off-by: Hu, Yong <yong.hu@intel.com>
Co-Authored-By: Zhipeng, Liu <zhipengs.liu@intel.com> Co-Authored-By: Zhipeng, Liu <zhipengs.liu@intel.com>
--- ---
mariadb/templates/statefulset.yaml | 3 ++- mariadb/templates/statefulset.yaml | 3 +++-
mariadb/values.yaml | 7 +++++++ mariadb/values.yaml | 20 +++++++++++++++
2 files changed, 9 insertions(+), 1 deletion(-) 2 files changed, 23 insertions(+), 1 deletion(-)
diff --git a/mariadb/templates/statefulset.yaml b/mariadb/templates/statefulset.yaml diff --git a/mariadb/templates/statefulset.yaml b/mariadb/templates/statefulset.yaml
index e31d4ac..83e135f 100644 index e31d4ac..83e135f 100644
--- a/mariadb/templates/statefulset.yaml --- a/mariadb/templates/statefulset.yaml
+++ b/mariadb/templates/statefulset.yaml +++ b/mariadb/templates/statefulset.yaml
@@ -187,7 +187,8 @@ spec: @@ -187,7 +187,9 @@ spec:
exec: exec:
command: command:
- /tmp/stop.sh - /tmp/stop.sh
-{{ dict "envAll" . "component" "server" "container" "mariadb" "type" "readiness" "probeTemplate" (include "mariadbReadinessProbe" . | fromYaml) | include "helm-toolkit.snippets.kubernetes_probe" | indent 10 -}} -{{ dict "envAll" . "component" "server" "container" "mariadb" "type" "readiness" "probeTemplate" (include "mariadbReadinessProbe" . | fromYaml) | include "helm-toolkit.snippets.kubernetes_probe" | indent 10 -}}
+{{ dict "envAll" . "component" "server" "container" "mariadb" "type" "readiness" "probeTemplate" (include "mariadbReadinessProbe" . | fromYaml) | include "helm-toolkit.snippets.kubernetes_probe" | indent 10 }} +{{ dict "envAll" . "component" "server" "container" "mariadb" "type" "readiness" "probeTemplate" (include "mariadbReadinessProbe" . | fromYaml) | include "helm-toolkit.snippets.kubernetes_probe" | indent 10 }}
+{{ dict "envAll" . "component" "server" "container" "mariadb" "type" "startup" "probeTemplate" (include "mariadbReadinessProbe" . | fromYaml) | include "helm-toolkit.snippets.kubernetes_probe" | indent 10 }} +{{ dict "envAll" . "component" "server" "container" "mariadb" "type" "startup" "probeTemplate" (include "mariadbReadinessProbe" . | fromYaml) | include "helm-toolkit.snippets.kubernetes_probe" | indent 10 }}
+{{ dict "envAll" . "component" "server" "container" "mariadb" "type" "liveness" "probeTemplate" (include "mariadbReadinessProbe" . | fromYaml) | include "helm-toolkit.snippets.kubernetes_probe" | indent 10 }}
volumeMounts: volumeMounts:
- name: pod-tmp - name: pod-tmp
mountPath: /tmp mountPath: /tmp
@ -39,7 +40,7 @@ diff --git a/mariadb/values.yaml b/mariadb/values.yaml
index 2d75f39..444bba3 100644 index 2d75f39..444bba3 100644
--- a/mariadb/values.yaml --- a/mariadb/values.yaml
+++ b/mariadb/values.yaml +++ b/mariadb/values.yaml
@@ -69,6 +69,13 @@ pod: @@ -69,6 +69,20 @@ pod:
initialDelaySeconds: 30 initialDelaySeconds: 30
periodSeconds: 30 periodSeconds: 30
timeoutSeconds: 15 timeoutSeconds: 15
@ -50,6 +51,13 @@ index 2d75f39..444bba3 100644
+ initialDelaySeconds: 60 + initialDelaySeconds: 60
+ periodSeconds: 60 + periodSeconds: 60
+ failureThreshold: 10 + failureThreshold: 10
+ liveness:
+ enabled: false
+ params:
+ initialDelaySeconds: 60
+ periodSeconds: 30
+ failureThreshold: 5
+ timeoutSeconds: 15
security_context: security_context:
server: server:
pod: pod:

View File

@ -179,6 +179,8 @@ data:
enabled: true enabled: true
startup: startup:
enabled: true enabled: true
liveness:
enabled: true
affinity: affinity:
anti: anti:
type: type: