4 Commits
Author | SHA1 | Message | Date | |
---|---|---|---|---|
Tao Liu
|
9661e49411 |
Change compute node to worker node personality
This update replaces compute references to worker in mtce, kickstarts, installer and bsp files. Tests Performed: Non-containerized deployment AIO-SX: Sanity and Nightly automated test suite AIO-DX: Sanity and Nightly automated test suite 2+2 System: Sanity and Nightly automated test suite 2+2 System: Horizon Patch Orchestration Kubernetes deployment: AIO-SX: Create, delete, reboot and rebuild instances 2+2+2 System: worker nodes are unlock enable and no alarms Story: 2004022 Task: 27013 Depends-On: https://review.openstack.org/#/c/624452/ Change-Id: I225f7d7143d841f80459603b27b95ac3f846c46f Signed-off-by: Tao Liu <tao.liu@windriver.com> |
||
Eric MacDonald
|
0b922227ac |
Implement Active-Active Heartbeat as HA Improvement
This update introduces mtce changes to support Active-Active Heartbeating. The purpose of Active-Active Heartbeating is help avoid Split-Brain. Active-Active heartbeating has each controller maintain a 5 second heartbeat response history cache of each network for all monitored hosts as well as the on-going health of storage-0 if provisioned and enabled. This is referred to as the 'heartbeat cluster history' Each controller then includes its cluster history in each heartbeat pulse request message. The hbsClient, now modified to handle heartbeat from both controllers, saves each controllers' heartbeat cluster history in a local cache and criss-crosses the data in its pulse responses. So when the hbsClient receives a pulse request from controller-0 it saves its reported history and then replaces that history information in its response to controller-0 with what it saved from controller-1's last pulse request ; i.e. its view of the system. Controller-0, receiving a host's pulse response, saves its peers heartbeat cluster history so that it has summary of heartbeat cluster history for the last 5 seconds for each monitored network of every monitored host in the system from both controllers' perspectives. Same for controller-1 with controller-0's history. The hbsAgent is then further enhanced to support a query request for this information. So now SM, when it needs to make a decision to avoid Split-Brain or otherwise, can query either controller for its heartbeat cluster history and get the last 5 second summary view of heartbeat (network) responsivness from both controllers perspectives to help decide which controller to make active. This involved removing the hbsAgent process from SM control and monitor and adding a new hbsAgent LSB init script for process launch, service file to run the init script and pmon config file for hbsAgent process monitoring. With hbsAgent now running on both controllers, changes to maintenance were required to send inventory to hbsAgent on both controllers, listen for hbsAgent event messages over the management interface and inform both hbsAgents which controller is active. The hbsAgent running on the inactive controller does not - does not send heartbeat events to maintenance - does not send raise or clear alarms or produce customer logs Test Plan: Feature: PASS: Verify hbsAgent runs on both controllers PASS: Verify hbsAgent as pmon monitored process (not SM) PASS: Verify system install and cluster collection in all system types (10+) PASS: Verify active controller hbsAgent detects and handles heartbeat loss PASS: Verify inactive controller hbsAgent detects and logs heartbeat loss PASS: Verify heartbeat cluster history collection functions properly. PASS: Verify storage-0 state tracking in cluster into. PASS: Verify storage-0 not responding handling PASS: Verify heartbeat response is sent back to only the requesting controller. PASS: Verify heartbeat history is correct from each controller PASS: Verify MNFA from active controller after install to controller-0 PASS: Verify MNFA from active controller after swact to controller-1 PASS: Verify MNFA for 80%+ of the hosts in the storage system PASS: Verify SM cluster query operation and content from both controllers PASS: Verify restart of inactive hbsAgent doesn't clear existing heartbeat alarms Logging: PASS: Verify cluster info logs. PASS: Verify feature design logging. PASS: Verify hbsAgent and hbsClient design logs on all hosts add value PASS: Verify design logging from both controllers in heartbeat loss case PASS: Verify design logging from both controllers in MNFA case PASS: Verify clog logs cluster info vault status and updates for controllers PASS: Verify clog1 logs full cluster state change for all hosts PASS: Verify clog2 logs cluster info save/append logs for controllers PASS: Verify clog3 memory dumps a cluster history PASS: Verify USR2 forces heartbeat and cluster info log dump PASS: Verify hourly heartbeat and cluster info log dump PASS: Verify loss events force heartbeat and cluster info log dump Regression: PASS: Verify Large System DOR PASS: Verify pmond regression test that now includes hbsAgent PASS: Verify Lock/Unlock of inactive controller (x3) PASS: Verify Swact behavior (x10) PASS: Verify compute Lock/Unlock PASS: Verify storage-0 Lock/Unlock PASS: Verify compute Host Failure and Graceful Recovery PASS: Verify Graceful Recovery Retry to Max:3 then Full Enable PASS: Verify Delete Host PASS: Verify Patching hbsAgent and hbsClient PASS: Verify event driven cluster push Story: 2003576 Task: 24907 Change-Id: I5baf5bcca23601a99473d039356d58250ffb01b5 Signed-off-by: Eric MacDonald <eric.macdonald@windriver.com> |
||
Eric MacDonald
|
8a223f395d |
Mtce: Add heartbeat cluster information for SM query
This part one of a two part HA Improvements feature that introduces the collection of heartbeat health at the system level. The full feature is intended to provide service management (SM) with the last 2 seconds of maintenace's heartbeat health view that is reflective of each controller's connectivity to each host including its peer controller. The heartbeat cluster summary information is additional information for SM to draw on when needing to make a choice of which controller is healthier, if/when to switch over and to ultimately avoid split brain scenarios in a two controller system. Feature Behavior: A common heartbeat cluster data structure is introduced and published to the sysroot for SM. The heartbeat service populates and maintains a local copy of this structure with data that reflects the responsivness for each monitored network of all the monitored hosts for the last 20 heartbeat periods. Mtce sends the current cluster summary to SM upon request. General flow of cluster feature wrt hbsAgent: hbs_cluster_init: general data init hbs_cluster_nums: set controller and network numbers forever: select: hbs_cluster_add / hbs_cluster_del: - add/del hosts from mtcAgent hbs_sm_handler -> hbs_cluster_send: - send cluster to SM heartbeating: hbs_cluster_append: add controller cluster to pulse request hbs_cluster_update: get controller cluster data from pulse responses hbs_cluster_save: save other controller cluster view in cluster vault hbs_cluster_log: log cluster state changes (clog) Test Plan: PASS: Verify compute system install PASS: Verify storage system install PASS: Verify cluster data ; all members of structure PASS: Verify storage-0 state management PASS: Verify add of second controller PASS: Verify add of storage-0 node PASS: Verify behavior over Swact PASS: Verify lock/unlock of second controller ; overall behavior PASS: Verify lock/unlock of storage-0 ; overall behavior PASS: Verify lock/unlock of storage-1 ; overall behavior PASS: Verify lock/unlock of compute nodes ; overall behavior PASS: Verify heartbeat failure and recovery of compute node PASS: Verify heartbeat failure and recovery of storage-0 PASS: Verify heartbeat failure and recovery of controller PASS: Verify delete of controller node PASS: Verify delete of storage-0 PASS: Verify delete of compute node PASS: Verify cluster when controller-1 active / controller-0 disabled PASS: Verify MNFA and recovery handling PASS: Verify handling in presence of multiple failure conditions PASS: Verify hbsAgent memory leak soak test with continuous SM query. PASS: Verify active controller-1 infra network failure behavior. PASS: Verify inactive controller-1 infra network failure behavior. Change-Id: I4154287f6dcf5249be5ab3180f2752ab47c5da3c Story: 2003576 Task: 24907 Signed-off-by: Eric MacDonald <eric.macdonald@windriver.com> |
||
Jim Gauld
|
6a5e10492c |
Decouple Guest-server/agent from stx-metal
This decouples the build and packaging of guest-server, guest-agent from mtce, by splitting guest component into stx-nfv repo. This leaves existing C++ code, scripts, and resource files untouched, so there is no functional change. Code refactoring is beyond the scope of this update. Makefiles were modified to include devel headers directories /usr/include/mtce-common and /usr/include/mtce-daemon. This ensures there is no contamination with other system headers. The cgts-mtce-common package is renamed and split into: - repo stx-metal: mtce-common, mtce-common-dev - repo stx-metal: mtce - repo stx-nfv: mtce-guest - repo stx-ha: updates package dependencies to mtce-pmon for service-mgmt, sm, and sm-api mtce-common: - contains common and daemon shared source utility code mtce-common-dev: - based on mtce-common, contains devel package required to build mtce-guest and mtce - contains common library archives and headers mtce: - contains components: alarm, fsmon, fsync, heartbeat, hostw, hwmon, maintenance, mtclog, pmon, public, rmon mtce-guest: - contains guest component guest-server, guest-agent Story: 2002829 Task: 22748 Change-Id: I9c7a9b846fd69fd566b31aa3f12a043c08f19f1f Signed-off-by: Jim Gauld <james.gauld@windriver.com> |