CFmSocket::connect() to the FM Manager uses a default-blocking socket,
so when the cluster's floating IP is briefly unreachable, say during a
SWACT where the floating IP is temporarily dropped, the kernel walks
tcp_syn_retries (default 6 - verified) before returning ETIMEDOUT.
The kernel doubles the wait between SYN retransmits, a TCP exponential
backoff (1+2+4+8+16+32+64 = 127 s). Cumulative connect() ceiling at
each retry count progresses 1 -> 3 -> 7 -> 15 -> 31 -> 63 -> 127 s.
That stalls the caller for 127 s.
When pmond hits this stall during host startup it nearly exhausts its
180 s pmond/hostwd quorum budget: 30 s startup wait + 127 s blocked
connect() = 157 s, leaving only a 23 s margin. Any additional startup
latency tips the controller past the host watchdog quorum failure
ceiling, causing crashdump and reboot. The post-reboot pmond re-arms
the same race if the same SYN loss conditions resurface, producing a
perpetual crashdump loop.
The fix is to set TCP_SYNCNT=2 in CFmSocket::create_socket() so the
active-connect ceiling collapses to 7 s (1+2+4=7), in line with the
existing SOCKET_TIMEOUT_DEFAULT (5 s) already enforced via SO_RCVTIMEO
on read. Total worst-case pmond silence becomes 30 + 7 = 37 s, well
inside the 180 s host watchdog budget. Failure to set the option is
logged as a warning and treated as non-fatal.
The change applies to every process linked against libfmcommon
(pmond, mtcAgent, hbsAgent, hwmond, collectd extensions, ...) so all
FM client paths inherit the bounded connect time automatically.
Note:
A TODO comment was added in CFmSocket::read_packet() to flag the
existing dead retry loop (for-with-unconditional-break that is
equivalent to a single read() call). The comment is text-only and
does not change runtime behaviour. Cleanup of the dead loop is left
to a follow-on commit so this fix stays minimal for backport.
Test Plan:
PASS: Verify build, ISO create and multinode system install.
PASS: Verify pmond.log "Connected to FM Manager." appears within
seconds of "Starting to monitor processes" and no hostwd
quorum-miss warnings during boot.
PASS: Healthy controller, fm-manager up and accepting:
- Verify pmond's first query_alarms() at startup returns
FM_ERR_OK in <100 ms; "Connected to FM Manager." present.
PASS: fm-manager not running locally (kernel returns ECONNREFUSED):
verify connect() fails immediately (~ms) with
"Failed to connect to FM Manager." - matches pre-change
behaviour for this case (no regression).
PASS: Reproduce the unreachable active controller scenario by
silently dropping outbound SYNs to FM Manager:
# iptables -I OUTPUT -p tcp --syn --dport 8001 -j DROP
# pmon-restart pmond
# tail -F /var/log/pmond.log /var/log/hostwd.log
With this fix, pmond logs "Failed to connect to FM Manager."
within 7 s instead of 127 s. The hostwd does not warn of
missed Process Quorum Health updates and the controller does
not self-reboot. Without this fix, the pre-existing reboot
loop reproduces at least 2 consecutive quorum misses.
# iptables -D OUTPUT -p tcp --syn --dport 8001 -j DROP
PASS: Repeat fresh-install ISO:
PASS: Verify pmond logging when the hostwd process is not running
- process startup
- during runtime
PASS: Verify handling on
- worker node type
- standard controller node type
- all in one node type
Closes-Bug: 2157655
Assisted-By: Kiro - Claude Opus 4.7
Change-Id: Ibbb44f5c8ba47528436565c318b8e383ad67d91e
Signed-off-by: Eric Macdonald <eric.macdonald@windriver.com>
Eric Macdonald
47b84645af