starlingx
/
ha
Code Issues Proposed changes

Reduce SM timeouts for sysinv-conductor 

SM runs ocf script to disable sysinv-conductor. Currently,
the timeout for sysinv-conductor ocf script to forcibly
terminate sysinv-conductor is 55s and the timeout for SM
to kill sysinv-conductor ocf script is 60s. In the case
that host-swact happens during application and k8s upgrade
operations, it will take a long time to complete swact
as these operations may spawn long-running processes or
greenthreads which prevent the sysinv-conductor from
shutting down before reaching the timeout to forcibly shutdown.
In a controller swact scenario, this results in the system
having almost no services running for almost a minute,
while SM is waiting for sysinv-conductor to shut down.

This commit updates the timeout for sysinv-conductor ocf
script to the default 15s and reduces the timeout for SM
to kill the script is 20s.

Tests conducted:
  - platform sanity
  - perform host-swact during stx-openstack upload/apply/remove
  - perform host-swact during kube upgrade operations
  - perform host-swact during platform-integ-apps apply
  - verified sysinv-conductor can be forcely shutdown after 15s
    when long-running processes are running

Change-Id: I337154a140f6cec3d6ab953003bf355b4396249e
Story: 2006781
Task: 38480
Signed-off-by: Angie Wang <angie.wang@windriver.com>
This commit is contained in:
Angie Wang 2020-02-05 15:36:18 -05:00
parent 477b500099
commit 26cc9d6cc0
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
service-mgmt/sm-db/database/create_sm_db.sql
View File

@ -393,7 +393,7 @@ INSERT INTO "SERVICE_ACTIONS" VALUES('sysinv-inv','disable','ocf-script','platfo
INSERT INTO "SERVICE_ACTIONS" VALUES('sysinv-inv','audit-enabled','ocf-script','platform','sysinv-api','monitor','',2,2,2,40,90);
INSERT INTO "SERVICE_ACTIONS" VALUES('sysinv-inv','audit-disabled','ocf-script','platform','sysinv-api','monitor','',0,0,0,40,90);
INSERT INTO "SERVICE_ACTIONS" VALUES('sysinv-conductor','enable','ocf-script','platform','sysinv-conductor','start','',2,2,2,30,'');
INSERT INTO "SERVICE_ACTIONS" VALUES('sysinv-conductor','disable','ocf-script','platform','sysinv-conductor','stop','timeout=60000',1,1,1,60,'');
INSERT INTO "SERVICE_ACTIONS" VALUES('sysinv-conductor','disable','ocf-script','platform','sysinv-conductor','stop','',1,1,1,20,'');
INSERT INTO "SERVICE_ACTIONS" VALUES('sysinv-conductor','audit-enabled','ocf-script','platform','sysinv-conductor','monitor','',2,2,2,15,40);
INSERT INTO "SERVICE_ACTIONS" VALUES('sysinv-conductor','audit-disabled','ocf-script','platform','sysinv-conductor','monitor','',0,0,0,15,40);
INSERT INTO "SERVICE_ACTIONS" VALUES('mtc-agent','enable','ocf-script','platform','mtcAgent','start','',2,2,2,30,'');