The current offline handler assumes the node is offline after
'offline_search_count' reaches 'offline_threshold' count
regardless of whether mtcAlive messages were received during
the search window.
The offline algorithm requires that no mtcAlive messages
be seen for the full offline_threshold count.
During a slow shutdown the mtcClient runs for longer than
it should and as a result can lead to maintenance seeing
the node as recovered before it should.
This update manages the offline search counter to ensure that
it only reached the count threshold after seeing no mtcAlive
messages for the full search count. Any mtcAlive message seen
during the count triggers a count reset.
This update also
1. Adjusts the reset retry cadence from 7 to 12 secs
to prevent unnecessary reboot thrash during
the current shutdown.
2. Clears the hbsClient ready event at the start of the
subfunction handler so the heartbeat soak is only
started after seeing heartbeat client ready events
that follow the main config.
Test Plan:
PASS: Debian and CentOS Build and DX install
PASS: Verify search count management
PASS: Verify issue does not occur over lock/unlock soak (100+)
- where the same test without update did show issue.
PASS: Monitor alive logs for behavioral correctness
PASS: Verify recovery reset occurs after expected extended time.
Closes-Bug: 1993656
Signed-off-by: Eric MacDonald <eric.macdonald@windriver.com>
Change-Id: If10bb75a1fb01d0ecd3f88524d74c232658ca29e
da398e0c5f