Stress testing of repeated service provision and deprovision exposed
three latent concurrency issues in SM. Each can affect stability
when service-management activity overlaps with audit cycles or
external CLI access to the SM database. The fixes are independent
but share a common theme: SM's shared tables and database
connections needed proper serialization and retry handling.
Bug 1 - service / dependency tables had no mutex
------------------------------------------------
The in-memory service and service-dependency lists were accessed
by both the FSM thread and the API dispatch thread without
synchronization. This is fixed by adding a recursive mutex to
each table, with internal *_unlocked variants for code paths that
already hold the lock. Lock order is service -> dependency.
Recursive mutexes are required because the engine and dependency
callbacks legitimately re-enter the service table from inside a
dependency walk.
Bug 2 - SQLite connections had no busy handler
----------------------------------------------
Database connections opened without a busy handler returned
SQLITE_BUSY immediately on writer-writer contention. This is
fixed by registering sqlite3_busy_handler in sm_db_connect for
every SM connection, with a bounded retry budget and throttled
logging.
Bug 3 - process-failure deregister used a freed pointer
-------------------------------------------------------
sm_deprovision_service called the process-failure deregister
after sm_service_deprovision had already freed the service
struct. The deregister is reordered to run before the free so
the pointer is still valid. Deregister remains best-effort.
Files
-----
service-mgmt/sm-db/src/sm_db.c - busy handler
service-mgmt/sm/src/sm_service_table.c - mutex + locking
service-mgmt/sm/src/sm_service_dependency_table.c - mutex + locking
service-mgmt/sm/src/sm_configure.cpp - reorder dereg
Note: The provisioning / deprovisioning of SM services in the following
test plan was restricted to ceph services where on
AIO: services include
storage-services :ceph-osd
storage-services :mgr-restful-plugin
storage-services :storage-networking
controller-services :ceph-mon
controller-services :drbd-cephmon
storage-monitoring-services:ceph-manager
Standard Storage System: services include only
storage-services :mgr-restful-plugin
storage-services :storage-networking
storage-monitoring-services:ceph-manager
Note: Some tests were verified with additional logging added on top of
this update.
Test Plan
---------
PASS: Verify install AIO DX and 2+2+2 Storage system (6 times 3 each)
PASS: Repeated provision / deprovision soak across multiple ceph
services completes cleanly with SM stable across the run.
PASS: Perform the above test as a 1000 loop overnight soak in both
All-In-One and 2+2+2 Standard Storage System while verifying
in each case:
- no SM traps: sm-trap.log remains empty and nore sm coredumps
- no memory leak over soak
- no file descriptor leaks
- no unexpected logging
PASS: Verify swact back and forth following above reprovisioning soak
PASS: Verify new mutex lock handling confirming the locking is
exercised without deadlock. Verified with debug logging.
PASS: Inject sm_service_dependency_table_load concurrent with a
dependency walk; load blocks until the walk completes.
PASS: Verify the Busy retry handling is exercized and works properly
during the above reprovisioning soaks.
- debug log: "sm_db.c(58):
Database 'Busy': retry 1/200 (1000 us/retry) at 27016.528
Above log was hit twice in the Standard storage system soak.
Above log was hit 12 times, ~45 (30-60) mins apart in AIO soak.
PASS: Hold the database write lock long enough to force the
give-up branch. Tested by FIT during a reprovisioning cycle
The following 4 debug logs
show the max retry handling
- sm_db.c(58):
Database 'Busy': retry 200/200 (1000 us/retry) at 65910.756
- sm_db.c(51): Database has been 'Busy' for 200 msecs :
gave up after 200 retries (1000 us/retry, 200 ms max)
- sm_db_nodes.c(409): Failed to update, sql=UPDATE NODES SET
READY_STATE = 'disabled' WHERE NAME = 'controller-0';,
rc=5, error=database is locked.
- debug Shutting down.
The busy condition is detected, SM exits as before but now only
after 200 retries 1 msec retries.
PASS: Verify no log corruption with Bug3 fix.
Observed debug log showing
"Process Failure Deregistration BEFORE Freeing the service"
PASS: AIO-DX swact loop, 50 swacts per side.
PASS: After above reprovisioning soaks Lock/Unlock compute, controller,
and storage hosts; Verify proper handling ; no SM-side regression.
Closes-Bug: 2154600
Assisted-By: Amazon Q (Auto)
Change-Id: I8ebeae48d00ba50fd4952ef54ca0d4d1f4019b7d
Signed-off-by: Eric Macdonald <eric.macdonald@windriver.com>
Eric Macdonald
e7e987ec2c