e11888c1df
In a DX scenario, after lock-unlock a controller the remaining MariaDB instance (lets say maria-server0) goes to a Non-Primary + Initializing State (non-operational). After that it remains searching for the now deleted pod (maria-server1) but using the old IP, the one before the restart. maria-server0 flags the old IP as delayed and suspect for eviction, however being a Non-Primary member it cannot in fact evict the old node and start looking for new members. Setting a LivenessProbe that detects nonoperational members and restart them fixes this, as the recreated pod starts looking for a cluster to join. Closes-Bug: #1938346 Signed-off-by: Thiago Brito <thiago.brito@windriver.com> Change-Id: I38d788f720cbd6bd13b6b6147db6f3d2a2ff9c92
67 lines
3.0 KiB
Diff
67 lines
3.0 KiB
Diff
From a8ef0407811373b98a91be7787fcdcca18c76e23 Mon Sep 17 00:00:00 2001
|
|
From: "Hu, Yong" <yong.hu@intel.com>
|
|
Date: Tue, 17 Dec 2019 15:11:13 +0000
|
|
Subject: [PATCH] Enable override of mariadb-server probe parameters
|
|
|
|
Parameters are exported for startupProbe and readinessProbe.
|
|
They are set with the default values, and might be overridden in
|
|
some special cases. For example, some cases require more times
|
|
of "readinessProbe" failure before the peer mariadb-server node is
|
|
recovered. Other cases, in which the regular readiness probe cannot
|
|
be performed because only one signle mariadb-server exists, have to
|
|
take use of "startupProbe" to ensure the WSREP status are ready.
|
|
For all these exceptional cases above, which are different from the
|
|
default mariadb cluster with 3 nodes, we have to export the probe
|
|
parameters and override the related probe pareameters.
|
|
|
|
Signed-off-by: Hu, Yong <yong.hu@intel.com>
|
|
Co-Authored-By: Zhipeng, Liu <zhipengs.liu@intel.com>
|
|
---
|
|
mariadb/templates/statefulset.yaml | 3 +++-
|
|
mariadb/values.yaml | 20 +++++++++++++++
|
|
2 files changed, 23 insertions(+), 1 deletion(-)
|
|
|
|
diff --git a/mariadb/templates/statefulset.yaml b/mariadb/templates/statefulset.yaml
|
|
index e31d4ac..83e135f 100644
|
|
--- a/mariadb/templates/statefulset.yaml
|
|
+++ b/mariadb/templates/statefulset.yaml
|
|
@@ -187,7 +187,9 @@ spec:
|
|
exec:
|
|
command:
|
|
- /tmp/stop.sh
|
|
-{{ dict "envAll" . "component" "server" "container" "mariadb" "type" "readiness" "probeTemplate" (include "mariadbReadinessProbe" . | fromYaml) | include "helm-toolkit.snippets.kubernetes_probe" | indent 10 -}}
|
|
+{{ dict "envAll" . "component" "server" "container" "mariadb" "type" "readiness" "probeTemplate" (include "mariadbReadinessProbe" . | fromYaml) | include "helm-toolkit.snippets.kubernetes_probe" | indent 10 }}
|
|
+{{ dict "envAll" . "component" "server" "container" "mariadb" "type" "startup" "probeTemplate" (include "mariadbReadinessProbe" . | fromYaml) | include "helm-toolkit.snippets.kubernetes_probe" | indent 10 }}
|
|
+{{ dict "envAll" . "component" "server" "container" "mariadb" "type" "liveness" "probeTemplate" (include "mariadbReadinessProbe" . | fromYaml) | include "helm-toolkit.snippets.kubernetes_probe" | indent 10 }}
|
|
volumeMounts:
|
|
- name: pod-tmp
|
|
mountPath: /tmp
|
|
diff --git a/mariadb/values.yaml b/mariadb/values.yaml
|
|
index 2d75f39..444bba3 100644
|
|
--- a/mariadb/values.yaml
|
|
+++ b/mariadb/values.yaml
|
|
@@ -69,6 +69,20 @@ pod:
|
|
initialDelaySeconds: 30
|
|
periodSeconds: 30
|
|
timeoutSeconds: 15
|
|
+ failureThreshold: 3
|
|
+ startup:
|
|
+ enabled: false
|
|
+ params:
|
|
+ initialDelaySeconds: 60
|
|
+ periodSeconds: 60
|
|
+ failureThreshold: 10
|
|
+ liveness:
|
|
+ enabled: false
|
|
+ params:
|
|
+ initialDelaySeconds: 60
|
|
+ periodSeconds: 30
|
|
+ failureThreshold: 5
|
|
+ timeoutSeconds: 15
|
|
security_context:
|
|
server:
|
|
pod:
|
|
--
|
|
1.8.3.1
|
|
|