The maintenance system implements a high availability (HA) feature
designed to detect the simultaneous heartbeat failure of a group
of hosts and avoid failing all those hosts until heartbeat resumes
or after a set period of time.
This feature is called Multi-Node Failure Avoidance, aka MNFA, and
currently has the hosts threshold set to 3 and timeout set to 100 secs.
This update implements enhancements to that existing feature by
making the 'number-of-hosts threshold' and 'timeout period'
customer configurable service parameters.
The new service parameters are listed under platform:maintenance which
display with the following command
> system service-parameter-list
mnfa_threshold: This new label and value is added to the puppet
managed /etc/mtc.ini and represents the number of hosts that are
required to fail heartbeat as a group; within the heartbeat
failure window (heartbeat_failure_threshold) after which maintenance
activates MNFA Mode.
This update changes the default number of failing hosts from
3 to 2 while allowing a configurable range from 2 to 100.
mnfa_timeout: This new label and value is added to the puppet
managed /etc/mtc.ini. While MNFA mode is active, it will remain active
until the number of failing hosts drop below the mnfa_threshold or this
timer expires. The MNFA mode deactivates on the first occurance of
either case. Upon deactivation the remaining failed hosts are no
longer treated as a failure group but instead are all Gracefully
Recovered individually. A value of zero imposes no timeout making the
deactivation criteria solely host based.
This update changes the default 100 second timer to 0; no-timeout
while permitting valid a times range from 100 to 86400 secs or 1 day.
Test Plan:
PASS - Verify duplex and 4 compute DOR
PASS - Verify default MNFA - 1 inactive controller and 4 computes
PASS - Verify default MNFA - 4 computes
PASS - Verify default MNFA - 1 active controller and 3 computes and failed host
PASS - Verify Single host heartbeat failure handling - fail host
PASS - Verify Multi Node failure below mnfa_threshold - fail hosts
PASS - Verify MNFA handling with timeout of zero and threshold of 3
PASS - Verify MNFA timeout handling with timeout set at 100 sec
PASS - Verify MNFA service parameter lising, default value and mtc.ini
PASS - Verify MNFA service parameter change and inservice apply
PASS - Verify MNFA timeout service parameter change from value to 0
PASS - Verify MNFA timeout service parameter change from to inrange value
PASS - Verify MNFA service parametrer out of range change handling
PASS - Verify MNFA timeout change from No-Timeout to 100 sec (while active)
DocImpact
Story: 2003576
Task: 24903
Change-Id: Ib56dd79b38c3726e042cf34aae361f229c89940b
Signed-off-by: Eric MacDonald <eric.macdonald@windriver.com>
All compute hosts seen to self reboot by hostw during patching due to
stuck pmond process
Current method to kill the running process leads to a race condition
that results in a user space futex dead lock that hangs pmond and
results in a watchdog self-reset due to quorum master 'pmond' failure.
The dead lock was traced to the ordering of the kill process.
Current steps to kill:
- kill process
- remove pidfile
- unregister pid with kernel
Deadlock is avoided by reversing the kill steps to what
is more logical.
- unregister pid with kernel
- remove pidfile
- kill process
Also introduced audit that registers manually restarted processes
with the kernel.
Failure Rate Before Fix: 1 every 25 process restarts.
Mostly fails before 5.
Failure Rate After Fix: No failures after 15000 process restarts
across 8 hosts including all host types between 2 different labs 2
different loads 18.07 and 18.08.
Test Method: Pmon restart regression test restarts all processes on
a host. Total soak restart of 25 monitored processes for 50 loops
over 12 hosts = 15000 restarts.
Also regressed process kill / recovery handling.
(5000 process recoveries)
Change-Id: Icac64df52df9d8074fcd886567dda6e53641572d
Signed-off-by: David Sullivan <david.sullivan@windriver.com>
Story: 2002993
Task: 23007
Two Node System: VMs did not switch to ERROR state after host reboot
A logically failed (rebooted) active controller is not being
administratively failed by maintenance. As a result the host's
offline availability state is not reported to the VIM and the
VMs on that (rebooted) All-in-one host are not evacuated.
This issue only applies to two node systems because of how the heartbeat
enable of an All-in-one host needs to be held off until its compute
manifests apply in the DOR case so as to avoid maintenance failing the
peer controller over a DOR.
The challange in maintenance is to distinguish between this spontaneous
failure and a DOR. For All-in-one hosts, DOR mode is active for a
whopping 600 seconds ; long enough to account for both sets of manifests
to apply.
It's that long delay that is making this silent fault stand out so
obviously.
This update uses 'active DOR mode' to decide whether or not to enable a
host's heartbeat in the add handler.
To better handle early active controller failure the qualifier for DOR
mode was reduced from 20 to 15 minutes. Meaning that maintenance DOR
mode is activated if its host up time is less than 15 minutes ; rather
than 20 as it was before this update. Note that normally the active
controller starts maintenance with an uptime of 5-7 minutes.
Story: 2002995
Task: 23009
Change-Id: I749aefef45b9db6e86a2c6b81d131ebeccc68926
Signed-off-by: David Sullivan <david.sullivan@windriver.com>
The mtcAgent process has been seen to segfault and coredump on process
exit.
The exit code is iterating over a c++ list that can change due to http
interrupt response handling.
The dump code is commented out with a note indicating why and when it
could be re-enabled.
Change-Id: Ie4ef684a65ded533c347ae07fdfa47f332412f7d
Signed-off-by: David Sullivan <david.sullivan@windriver.com>
Story: 2002994
Task: 23008
When the Hardware Monitor starts up it reads existing alarms and sensor
state from the sysinv database. It then uses this pre-existing state to
align its internal structure accordingly moving forward.
The hardware monitor manage_startup_states utility is incorrectly
requesting degrade clear rather than degrade set in response to finding
a pre-existing critical sensor assertion on process startup.
This update fixes this issue by calling the set_degraded_state rather
than clear_degraded_state against this sensor in this case.
Change-Id: Ic1ecc1f11d7a729c16da63c6d43b7d758bb9e467
Signed-off-by: David Sullivan <david.sullivan@windriver.com>
Story: 2002882
Task: 22845
As a part of changes to make sm-api independent, calling sm-api
requires keystone authentication.
This change is to enable mtce to call sm rest api with keystone
authentication.
Story: 2002827
Task: 22744
Change-Id: If3b58d3e36b9bd7fd88829d61e9c1daa00ab5048
Signed-off-by: Bin Qian <bin.qian@windriver.com>
Introduction of PTP service requires NTP service to be disabled.
Process monitoring of NTP daemon must be turned off as well.
There is no way to start/stop process monitoring from MTCE.
Puppet can check NTP status at startup and enable/disable monitoring.
So, it is needed to move NTP-related PMON script from MTCE to Puppet.
This is first step: removing NTP references from MTCE.
Change-Id: I1ca6045af8c5169220b7332d45b843fdb4960f01
Story: 2002935
Task: 24520
Signed-off-by: Alex Kozyrev <alex.kozyrev@windriver.com>
If the first mtcAlive message from a host that was supposed to be
rebooted reports uptime in excess of 40 minutes then that means it did
not reboot as expected.
This was seen to happen during an extended offline case where the host
failed heartbeat, then was reported offline during Graceful Recovery
which forced a full enable. When the host eventually came back online
its reported uptime made it clear that it never rebooted but mtce
allowed it to come into service anyway.
This is a security issue that can lead to a host disappearing, being
security hacked and brought back into the system without reboot.
To fix that, this update requires that a host's uptime, reported in its
first mtcAlive message, indicate that it has been up for less twice the
configured mtcAlive timeout or the enable will fail until it is proven
to reset.
Story: 2002882
Task: 22845
Change-Id: I9b3ff0bc1ba5af2ca5b07a58db9da9f288b59576
Signed-off-by: Jack Ding <jack.ding@windriver.com>
For the event of Heartbeat Failure with a host, the Mtce Heartbeat Agent
will declare heartbeat recovery upon the first successful heartbeat
reply after the loss is declared ; basically edge level trigger
recovery.
In cases where a networking issue causes heartbeat loss of a group of
hosts, Maintenance tracks the group of hosts that experienced heartbeta
loss and puts the system into 'Multi Node Failure Avoidance' mode.
maintenance then simply waits up to a configured timeout period for
hosts to regain heartbeat.
As heartbeat is regained for each host that host is attempted to be
'Gracefully Recovered'.
However, if the networking issue persists in a way that the occasional
transient heartbeat pulse gets through then the maintenance system can
prematurely take hosts and then 'the system' out of MNFA mode only to
find that heartbeat is actually not properly recovered/working only to
then fail and force reboot/reset each node that is still experiencing
heartbeat loss.
This update changes the heartbeat service from an 'edge' to 'level'
sensitive recovery by requiring a number of back-2-back heartbeat pulses
following a failure before that host is delared as recovered and pulled
out of the MMNFA pool.
Basically, This update makes the system's MNFA recovery algorithm more
robust in the face of transient heartbeat loss for a group of hosts.
Story: 2002882
Task: 22845
Change-Id: Ie36b73a14cfad317d900e3a3a9ddb434326737a1
Signed-off-by: Jack Ding <jack.ding@windriver.com>
Adds support to the mtcAgent for detecting the absence of the 'host
services execution enhancement feature' in the mtcClient and implements
the pre-upgrade implementation in that case. When mtcAgent tries to lock
a storage node running pre-upgrade verison it will implement a 90s
lock wait before proceeding to declare that storage host as
locked-disabled.
Story: 2002886
Task: 22847
Change-Id: I99fb5576e027621019adb5eff553d52773f608db
Signed-off-by: Jack Ding <jack.ding@windriver.com>
This update adds Maintenance support for receiving host degrade assert
and clear messages from collectd.
This update also disables platform memory, cpu and file system resource
monitoring in the maintenance resource monitor process rmon.
These disabled resources are now monitored by collectd and therefore
should not be monitored by rmond any longer.
Change-Id: I13fd033bb1d14f299dcb97fa80296641c958d0a9
Signed-off-by: Jack Ding <jack.ding@windriver.com>
Many new services being added to our system are no longer accompanied
with an init script ; only a service file. With the migration from sysvinit
to systemd pmond still requires process conf files to provide a script label.
This update removes that dependency. Instead, pmond will use the service
or script label to find the most appropriate process failure recover method
while handling the omition of either but not both of the service and script
labels.
The change is to first search for a service file that corresponds with the
service label in the conf file.
If the service label does not exist then the script label is looked at.
If the basename of the script has a corresponding service file then use it.
If no service file is found then the full pathed script is searched for.
If no script file is found then the process monitor errors out.
This update also makes an improvement to how pmond deals with the absence
of the hostw process. Current code base blocks startup if it cannot connect
to the hostw process.
This update implements host watchdog socket failure auto recovery while
continuing to monitor processes. With this update, if the host watchdog
process is restarted or is not running then pmond will continue to monitor
processes while periodically trying to recover connection to the host
watchdog once it does recover.
Change-Id: Icf27090d4d00954195b0ac931474587c67341207
Signed-off-by: Jack Ding <jack.ding@windriver.com>
While code reviewing, heartbeat typo was seeing.
Change-Id: I6fed9f82e1d3b3e78a7d91d2cfc9091f4bf83f70
Signed-off-by: Abraham Arce <abraham.arce.moreno@intel.com>
Add full support for Active/Active redudancy model
1. services could have enable dependency to services in other service
groups
(standby group)
2. An active/active service failure will degraded the service group it
is in
3. A failure of active/active service would not prevent a swact
4. Locking a controller that is sole active/active service
provider will be rejected. But lock with force option will still proceed
to lock the node.
5. sm-api bind to port 7777 on mgmt interface. (was localhost:7777)
Change-Id: I0da78a51a50fd60ec128edc91c2eeec31af4a956
Signed-off-by: Jack Ding <jack.ding@windriver.com>
Decouple NSLCD from the open-ldap SM service and manage it by PMOND
instead. This is needed because in the Shared LDAP case, we deprovision
the open-ldap service on the Secondary Region which renders NSLCD
unmanaged.
Additionally, we allow the Secondary Region or Sub Clouds to bind
anonymously, but still need to support LDAP read operations in these
regions such as ldapfinger or lsldap. For this purpose, the ldapscripts
runtime library has been modified to allow anonymous binds during LDAP
search operations.
Change-Id: Ic01a8097e8124348d493c9e0c82fda94700e28e2
Signed-off-by: Jack Ding <jack.ding@windriver.com>
The AIO-DX system type was enhanced to allow the customer to
provision additional compute hosts.
However, the maintenance system made a global assumption about the
'system type' and would run the enable subfunction FSMs against
these compute only hosts. In doing so the unlock of these added
hosts would timeout and heartbeat would not enable.
This update ensures that the subfunction FSM is only run against
hosts that have both controller and compute functions.
Change-Id: If7711519d3435ef19faa13e7905afae2ce9084bc
Signed-off-by: Eric MacDonald <eric.macdonald@windriver.com>
Fetching a keystone token is seen to take a long time When the system
is overloaded. Such delay, due to overload, is most often seen over a
Swact in All-In-One Duplex (AIO-DX) configuration.
Any blocking call that takes a long time to complete can cause a process
stall. If that stall is sufficiently long then the command will either
finish on or before the timeout. The timeout is currently set to 15
seconds which is comparable to the SM monitored audit timeout.
Although rare, then race condition does exist and if there are 2
timeouts within 2 minutes then SM fails the process and triggers a
swact.
Rather than tune the timeouts, this update implements a more robust
fix by making all token fetches ; both 'initial token get' and
'runtime token refresh' as non-blocking.
The change applies to both mtcAgent and hwmond ; both of which need
to get and manage their own authentication token.
Change-Id: I2730b76ae78daec4b9edeaff5c1ca614b75ab52c
Signed-off-by: Eric MacDonald <eric.macdonald@windriver.com>
The SM API event handler is calling event_base_loopbreak for all requests.
This is sometimes causing segfault when this is done in the http handler
for non-blocking requests.
This update prevents this call for non-blocking requests.
Change-Id: Ib083100ccd74aa984bd86921c7521bfec925e779
Signed-off-by: Eric MacDonald <eric.macdonald@windriver.com>
The maintenance resource monitor is not asserting a 'no reachable
NTP servers' alarm while the only reachable server is the peer
controller.
Change-Id: I8df7bda01676bbaaeca2d97cb6893265107700aa
Signed-off-by: Eric MacDonald <eric.macdonald@windriver.com>
The AIO (All-In-One) inactive controller is failed by maintenance in a
DOR (Dead-Office-Recovery) situation ; power off then power on of the
system.
This update scales the AIO DOR timeout to accomodate for the extra time
needed for the compute function manifest to apply.
Change-Id: I3006060fe04285881f95d2084cada40ec1002d1c
Signed-off-by: Eric MacDonald <eric.macdonald@windriver.com>
The mtcAgent's message inbox batch handling was needlessly zeroing a
buffer that it would never use after it reached the last received
message.
This update refactors the mtc_service_inbox entry code so that the
message buffer zeroing operation only occurs on the unused part of
fully received messages. To facilitate this change, a new utility for
zeroing a bounded part of a mtce message buf was introduced to common
utils.
Two additional enhancements were also made to the same procedure:
- variable scoping change.
- hostaddr and hostname lookup scoping change.
Change-Id: Ia2ef97dad611507b824927ed1652c8df8b54eee5
Signed-off-by: Eric MacDonald <eric.macdonald@windriver.com>
Due to race conditions, multiple messages might be received from a
single read by guestServer. guestServer in this case would only handle
the first message and discard the remaining ones.
In this particular issue, guestServer received a heartbeat challenge
response message and a vote notification response (reject) message from
a single read, and the latter message was discarded.
This fix rewrites message handler for virtio serial channel to handle
segmented and multiple messages. It uses newline character to deliminate
messages so it assumes any newline characters in client log message are
removed.
Change-Id: Ic6f0509c98fcedf3631f4d210f753c32c37aa442
Signed-off-by: Jack Ding <jack.ding@windriver.com>
Fixed a few day one bugs:
- Heartbeat status changes are not properly passed to VIM from
GuestAgent.
- Heartbeat reporting state is not properly updated in guestServer when
guest heartbeat is enabled.
These bugs could cause heartbeat status/states mismatch among
VIM-guestAgent-guestServer and result in intermittent issues.
Change-Id: I2198760345821fa4af0437af252e3ec6a39978d8
Signed-off-by: Jack Ding <jack.ding@windriver.com>
pop_front() only delete the internal copy of jobj_msg in the
message_list. The original object still need to be released.
Change-Id: I869bdfac9de59d512a50d10a073682e5dcd9bbcf
Signed-off-by: Jack Ding <jack.ding@windriver.com>
