Eric Macdonald
fc9db35265
The sm-api process resolves its WSGI bind address by calling
socket.getaddrinfo() on the configured bind host, which is typically
a hostname such as "controller" resolved via dnsmasq.
During early boot, unlock, swact, or DOR recovery, node configuration
can take up to ~5 minutes to complete and dnsmasq may not yet be
answering. In that window socket.getaddrinfo() raises socket.gaierror
([Errno -2] Name or service not known). The exception was unhandled, so
sm-api exited with a traceback only to be found as 'not running' only
to be restarted by pmon.
This change wraps the bind-address resolution in a new
resolve_bind_address() helper that:
- Retries socket.getaddrinfo() on socket.gaierror on bounded schedule
(ADDR_RESOLVE_MAX_RETRIES x ADDR_RESOLVE_RETRY_INTERVAL = 60 x 5s =
~5 min) to cover the time needed for configuration to complete, then
re-raises if resolution never succeeds.
- Uses socket.getaddrinfo() so the check matches the resolver path the
bind uses; a literal IP or 0.0.0.0 resolves immediately (no wait).
- Emits a throttled single-line log per attempt, tagged with the sm-api
process name, e.g.:
sm-api: Unable to bind to 'controller-0.internal' (attempt 1/60):
(errno:-2). dns name resolution may not be ready - check dnsmasq
running state - retry in 5s
The resolution/wait occurs only once, during process startup in main(),
before make_server()/serve_forever(); it is never entered from the WSGI
serving loop and adds no per-request DNS lookups.
Test Plan:
PASS: Verify sm-api build and systems install ; AIO SX/DX and Standard
PASS: Normal startup (resolvable bind host)
PASS: Verify sm-api kill and restart soaks on healthy controllers of
AIO-SX, AIO-DX and Standard systems.
PASS: Verify with both IPV4 and IPV6 addressing
Retry Handling:
PASS: Verify sm-api binds and serves without delay or retry log.
PASS: Verify retry handling behaviour when host name is unresolvable
PASS: - Verify basic retry handling ; 1 failure and retry log
PASS: - Verify throttled logging rate and log content
PASS: - Verify mid and late recovery handling
PASS: - Verify bounded give-up behaviour
* Leave name resolution broken for the full window.
- Verify: final attempt (60/60) is logged without retry suffix
* sm-api then exits and is restarted by pmon, which re-enters the
same bounded wait (no tight crash loop within the window).
PASS: - Verify sm-api process functions once recovered
ss -ltnp | grep 7777
curl -sS http://controller-0.internal:7777/ | head
curl -sS http://controller-0.internal:7777/v1/
Regression:
PASS: Verify mtcAgent communiucation with SM over sm-api
- node state change notifications
Assisted-By: Kiro (Claud Opus 4.7)
Closes-Bug: 2163360
Change-Id: Ib4f8a806b1b2e609cea2784d9a8c2181877c3499
Signed-off-by: Eric Macdonald <eric.macdonald@windriver.com>
ha
The starlingx/ha repository handles High Availability services1.
Its key component is the StarlingX Service Management (SM), which coordinates the StarlingX services.
This repository is not intended to be developed standalone, but rather as part of the StarlingX Source System, which is defined by the StarlingX manifest2.