2462f3015a
The keystone-api pod gets stuck in CrashLoopBackOff on AIO-SX lock/unlock attempt. When Kubernetes decides to kill the keystone-api pod due to readiness probe failure or other reasons, it calls the preStop hook immediately before the container is terminated. This hook starts a graceful shutdown process which includes removing pid, shared memory segment and wsgi sock files. If the container is not terminated within the grace period, a SIGKILL is sent, and the container is forced to shut down. When the container was forced to terminate without clean up, the stale files were left behind. On the restart, the application detected the file existed, and treated it as configuration failure, hence the exit. As a result, the pod went into a crash loop. This update removes any stale files when the pod starts. Story: 2004520 Task: 28392 Change-Id: I613a0db674de9578b3f9d1fa781a1612d9caf214 Signed-off-by: Tao Liu <tao.liu@windriver.com>
96 lines
3.0 KiB
Diff
96 lines
3.0 KiB
Diff
From 44b9b086f22a561cec91519d03750c3e501d5739 Mon Sep 17 00:00:00 2001
|
|
From: Chris Friesen <chris.friesen@windriver.com>
|
|
Date: Wed, 28 Nov 2018 01:33:39 -0500
|
|
Subject: [PATCH] Remove stale Apache2 service pids when a POD starts.
|
|
|
|
Stale Apache2 pids will prevent Apache2 from starting and will leave
|
|
the POD in a crashed state.
|
|
|
|
Note: the pid file is somewhat confusingly called
|
|
/var/run/httpd/httpd.pid and /var/run/apache2 is just a symlink to
|
|
/var/run/httpd.
|
|
|
|
This is loosely based off the in-review upstream commit at
|
|
https://review.openstack.org/#/c/619747
|
|
---
|
|
aodh/templates/bin/_aodh-api.sh.tpl | 3 +++
|
|
ceilometer/templates/bin/_ceilometer-api.sh.tpl | 3 +++
|
|
keystone/templates/bin/_keystone-api.sh.tpl | 3 +++
|
|
nova/templates/bin/_nova-placement-api.sh.tpl | 3 +++
|
|
panko/templates/bin/_panko-api.sh.tpl | 3 +++
|
|
5 files changed, 15 insertions(+)
|
|
|
|
diff --git a/aodh/templates/bin/_aodh-api.sh.tpl b/aodh/templates/bin/_aodh-api.sh.tpl
|
|
index 708b327..dfc7abc 100644
|
|
--- a/aodh/templates/bin/_aodh-api.sh.tpl
|
|
+++ b/aodh/templates/bin/_aodh-api.sh.tpl
|
|
@@ -28,6 +28,9 @@ function start () {
|
|
source /etc/apache2/envvars
|
|
fi
|
|
|
|
+ # Get rid of stale pid file if present.
|
|
+ rm -f /var/run/apache2/*.pid
|
|
+
|
|
# Start Apache2
|
|
exec apache2 -DFOREGROUND
|
|
}
|
|
diff --git a/ceilometer/templates/bin/_ceilometer-api.sh.tpl b/ceilometer/templates/bin/_ceilometer-api.sh.tpl
|
|
index cdb02f7..0950f03 100644
|
|
--- a/ceilometer/templates/bin/_ceilometer-api.sh.tpl
|
|
+++ b/ceilometer/templates/bin/_ceilometer-api.sh.tpl
|
|
@@ -25,6 +25,9 @@ function start () {
|
|
source /etc/apache2/envvars
|
|
fi
|
|
|
|
+ # Get rid of stale pid file if present.
|
|
+ rm -f /var/run/apache2/*.pid
|
|
+
|
|
# Start Apache2
|
|
exec apache2 -DFOREGROUND
|
|
}
|
|
diff --git a/keystone/templates/bin/_keystone-api.sh.tpl b/keystone/templates/bin/_keystone-api.sh.tpl
|
|
index 217d942..a5950a4 100644
|
|
--- a/keystone/templates/bin/_keystone-api.sh.tpl
|
|
+++ b/keystone/templates/bin/_keystone-api.sh.tpl
|
|
@@ -31,6 +31,9 @@ function start () {
|
|
source /etc/apache2/envvars
|
|
fi
|
|
|
|
+ # Get rid of stale pid, shared memory segment and wsgi sock files if present.
|
|
+ rm -f /var/run/apache2/*
|
|
+
|
|
# Start Apache2
|
|
exec apache2 -DFOREGROUND
|
|
}
|
|
diff --git a/nova/templates/bin/_nova-placement-api.sh.tpl b/nova/templates/bin/_nova-placement-api.sh.tpl
|
|
index f9c8d7c..a7d753e 100644
|
|
--- a/nova/templates/bin/_nova-placement-api.sh.tpl
|
|
+++ b/nova/templates/bin/_nova-placement-api.sh.tpl
|
|
@@ -28,6 +28,9 @@ function start () {
|
|
source /etc/apache2/envvars
|
|
fi
|
|
|
|
+ # Get rid of stale pid file if present.
|
|
+ rm -f /var/run/apache2/*.pid
|
|
+
|
|
# Start Apache2
|
|
exec apache2 -DFOREGROUND
|
|
}
|
|
diff --git a/panko/templates/bin/_panko-api.sh.tpl b/panko/templates/bin/_panko-api.sh.tpl
|
|
index bd08a43..c4ffc3f 100755
|
|
--- a/panko/templates/bin/_panko-api.sh.tpl
|
|
+++ b/panko/templates/bin/_panko-api.sh.tpl
|
|
@@ -28,6 +28,9 @@ function start () {
|
|
source /etc/apache2/envvars
|
|
fi
|
|
|
|
+ # Get rid of stale pid file if present.
|
|
+ rm -f /var/run/apache2/*.pid
|
|
+
|
|
# Start Apache2
|
|
exec apache2 -DFOREGROUND
|
|
}
|
|
--
|
|
1.8.3.1
|
|
|