This commit converts the boot_failure variable to be a counter
instead of a boolean variable, adds a logic to set the default
grub menu entry as the other available in case of three
consecutive boot failures, and adds the system_mode variable,
that will be assigned with the actual system_mode value by the
change in [1].
Note: The 'Depends-On' commit adds a service to reset the
boot_failure counter value to 0 once the system boots
successfully
[1] https://review.opendev.org/c/starlingx/stx-puppet/+/952869
Test Plan
PASS: simulate a boot failure (e.g. reset the host while
it is being booted) and verify the counter increments
PASS: simulate a boot failure until the counter reaches the
maximum retry value, then verify grub sets the default
entry as the rollback
PASS: integrated test for automatic boot recovery (AIO-SX)
1. upgrade host from stx-10 to stx-11 with LVM snapshot
feature enabled
2. force failure during the boot sequence (3x)
3. system boots in the rollback deployment
4. LVM snapshots are restored during the boot sequence
and system is rebooted again
5. system boots in from-release ready to 'deploy delete'
Story: 2011357
Task: 52402
Depends-On: https://review.opendev.org/c/starlingx/update/+/952865
Change-Id: I2a03acf2b589373d4eaab08a71db6a58c2672200
Signed-off-by: Heitor Matsui <heitorvieira.matsui@windriver.com>