This fix service enabling timeout timer incorrect left active after service is
enabled.
Also ensure properly transitioning to service enabling state from enabling-throttle
state (transition logging).
Closes-Bug: 1931617
Signed-off-by: Bin Qian <bin.qian@windriver.com>
Change-Id: I0be01c335ef9e39d1f14df12cccdf97ce09fbb5c
sm-schedule.log contains a recurring log message "no scheduling
changes required." The amount of those log entries is huge and
in a large environment, it adds a significant burden for the
monitoring elements, while providing no apparent value. This
commit fixes the issue.
Change-Id: Ia03bc7161d5b6a8059fd8daba5e40450711f2830
Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>
Closes-Bug: 1929618
Currently when a controller goes active, a chain operations of setting
desired state of each service domain assignment (group) to
"enabled-active", and each service within the group to "enabled-active",
which triggers a service going enabled, where service enabled dependency
is being checked. However some groups still have desired state
"enabled-standby", which allows a dependency being met when the service
is still in "disabled" state.
Ensure setting all services group desired state to target state before
enabling services.
Closes-Bug: 1928707
Signed-off-by: Bin Qian <bin.qian@windriver.com>
Change-Id: Id38540bc0a5292602ae49c524b0dceb77e303a8b
This fix is to ensure keeping service failure counting over successful
audit.
When service enabled audit successfully completes, SM reset the service
failure state. However it should not reset the service fail-count.
The fail-count should only be reset after the grace period.
Closes-Bug: 1893669
Change-Id: I6996fe3f1c08c38da6f26243aee2b95b083069f0
Signed-off-by: Bin Qian <bin.qian@windriver.com>
This fix is to fix AIO unexpected failover behaviors.
1. active controller reboots itself when standby controller
reboot/lost power
2. standby controller becomes degraded after active controller
reboot/lost power
Closes-bug: 1927133
Change-Id: If3c9f6251f689a89cd206c672092ba296f00bd6b
Signed-off-by: Bin Qian <bin.qian@windriver.com>
A forced reboot of the active controller in an AIO DC system
puts SM into a failover failure recovery loop that prevents
maintenance from detecting the heartbeat failure of the just-
rebooted controller.
The SM's failover failure recovery handling algorithm includes
a self (sm process) restart preceded by a restart of the
hbsAgent, both added by the following update last year.
update: Add unhealthy state recovery audit to service management (sm)
review: https://review.opendev.org/c/starlingx/ha/+/735219
The self restart of SM was and is required in this case. However,
the restart of the hbsAgent was only included as a safety measure,
at the time, to ensure SM received updated cluster state info. The
hbsAgent restart was only added at that time with the longer term
intention to have it removed once the hbsAgent cluster state change
notification improvement was implemented. That change is now
implemented and merged by the following update.
update: Mtce heartbeat cluster state change notification improvement
review: https://review.opendev.org/c/starlingx/metal/+/769936
Testing of the fix for the following issue in an AIO DC system
resulted in the takeover controller not detecting a heartbeat loss
of the just rebooted standby controller.
title: Force active controller reboot results in a second reboot
issue: https://bugs.launchpad.net/starlingx/+bug/1922584
The hbsAgent is not able to detect the heartbeat loss of the just-
booted controller because SM keeps restarting it before it reaches
the heartbeat loss state.
With the cluster notification improvement update now implemented
and merged it's time to remove the hbsAgent restart from SM's
failover failure recovery algorithm.
Test Plan:
PASS: Active controller force reboot handling in AIO DC, DX and
standard systems.
PASS: Standby controller force reboot handling in AIO DC, DX and
standard systems
Partial-Bug: 1922584
Signed-off-by: Eric MacDonald <eric.macdonald@windriver.com>
Change-Id: I26aa5ed9e0faec7294816269dbaa49cbb4696f66
Pin hacking to < 4.0.1 to fix zuul gate issues.
Test:
Ran tox -e pep8 command to validate the pep8 job and result.
Related-Bug: 1926172
Signed-off-by: Charles Short <charles.short@windriver.com>
Change-Id: I2a3709342a2150e57461561ca1299adabc993667
SQLite3 does not support concurrent access with multiple connections
that have writeable access. Currently SM opens database connections
with full access, which causes concurrent issue.
This fix includes:
1. open readonly connection whenever the write permission is not needed
2. remove code that open connections that are not being used
3. remove reattempt and loggings from previous partial fix
Now all writable connections are opened and used in main thread, this
can ensure no more concurrent issue.
Closes-Bug: 1915894
Signed-off-by: Bin Qian <bin.qian@windriver.com>
Change-Id: I200647a3733ac899b0b7498abd52992c7a87bd32
Stop the troubleshooting log once the execution passes the
checkpoint.
Change-Id: I4e1d7710d5216f7b5a908f56e72d5f95c35a6586
Partial-bug: 1915894
Signed-off-by: Bin Qian <bin.qian@windriver.com>
Previous commit:
f39ca95924
added an incorrect #include which cause build failure.
Closes-bug: 1917527
Change-Id: I5d93d77fb0b14446e21a1ba160ffd0848533e970
Signed-off-by: Bin Qian <bin.qian@windriver.com>
Multiple report to AIO-SX that SM failed its intialization due to
a SQL failure. The issue had not been reproduced in DEV environment.
This change adds logging, reattempt and collect SM troubleshooting
data when SM fails in such situation.
For potential recovery before pmon start actively monitoring SM,
setting systemd restart=on-failure. Also set RestartSec=10 seconds
in order to give pmon enough time to catch the failure and restart
SM.
Partial-bug: 1915894
Change-Id: I5899e401742510158cd9c59a664b1dc329bb1075
Signed-off-by: Bin Qian <bin.qian@windriver.com>
We're moving the bulk of the dcmanager subcloud audits to separate
worker processes, so we need to add a service for the main worker
processes (which will then spawn additional workers).
In order to ensure that audits can be processed as soon as
dcmanager-audit starts up, we make enabling it dependent on
dcmanager-audit-worker being already running.
Story: 2007267
Task: 41870
Signed-off-by: Chris Friesen <chris.friesen@windriver.com>
Depends-On: https://review.opendev.org/c/starlingx/distcloud/+/769216
Change-Id: I162c00a3e8dba07f1912171e9371c29e5fd9a689
Added a device-image-fs SM service to manage the device image
repository filesystem.
Tests performed on the following systems:
AIO-DX, AIO-DX plus compute, Standard 2+1
DC with AIO-DX plus subcloud
DC with Standard subcloud
Story: 2007875
Task: 41880
Depends-On: https://review.opendev.org/c/starlingx/ansible-playbooks/+/776488
Change-Id: I068c26c524357176e4b526c405785768044c379c
Signed-off-by: Teresa Ho <teresa.ho@windriver.com>
Skip logging state changes of interfaces that are not being
monitored by SM. This is to reduce the noise in the sm.log.
Closes-Bug: 1910770
Change-Id: I6e3d78255dc41c03f10af2fd5d778e2398ea8816
Signed-off-by: Bin Qian <bin.qian@windriver.com>
For duplex, when make host swact, sm should firstly remove ceph-mon
and ceph-osd pod, which open the /var/lib/ceph/mon-a folder.
Remove these pods on active controller and make drbd set to
secondary to swact to the other controller
Story: 2005527
Task: 41328
Change-Id: I7cb7af3b3a56afcff71087d7f3b4f09a384c8dc2
Signed-off-by: Martin, Chen <haochuan.z.chen@intel.com>
This change is to detect SM failure/stall.
1. SM sends alive pulse to hbsAgent, (hbsAgent sends SM failed state
along with hbs cluster info)
2. When SM lost heartbeat from peer, SM detects if peer has failed
from hbs cluster info.
3. On standby controller, if peer SM is stalled, it will take over
to become active after signaling mtce to fail peer node.
TCs passed:
When SM detects peer failure from hbs cluster info, and signals
mtce to fail peer node.
Depends-on: https://review.opendev.org/#/c/751558
Partial-bug: 1895350
Change-Id: Id51e9adb4ef30bf806159366e6fdf115e743fe97
Signed-off-by: Bin Qian <bin.qian@windriver.com>
A DRBD service needs dependency on disable to its related fs
service, for example: drbd-rabbit -> rabbit-fs, so that
SM disables drbd-rabbit after rabbit-fs is disabled.
This is important especially in AIO-SX in which drbd services
are disabled when host is being unlocked.
The following disable dependency are added in this change:
drbd-pg -> pg-fs
drbd-rabbit -> rabbit-fs
drbd-platform -> platform-fs
drbd-extension -> extension-fs
drbd-cinder -> cinder-lvm
drbd-dc-vault -> dc-vault-fs
drbd-dockerdistribution -> dockerdistribution-fs
drbd-cephmon -> cephmon-fs
Test host lock/unlock in AIO-SX and standard DX environments.
Closes-bug: 1907490
Change-Id: I631f718add2c1d2756a36f3770a4d48f02904f1a
Signed-off-by: Bin Qian <bin.qian@windriver.com>
In AIO-SX, the peer IP address is not configured (blank),
which is translated into '::'. When the '::' is used as
dest address, it is translated to loopback '::1'.
SM should skip sending packets to destination '::'.
Closes-Bug: 1909769
Change-Id: Id8a9a00adce6573bcccd60b1b2112b6ee8b2f8a3
Signed-off-by: Bin Qian <bin.qian@windriver.com>
The install_command for docs, newnote and api-ref
needed to be overridden to not use upper constraints.
The bandit requirement needed to be made python3 only.
The bandit scan was failing, so it is now updated to
allow individual bandit failures to be suppressed in tox.ini
Need to include a py file change in order for bandit to be
triggered by zuul.
Partial-Bug: #1907678
Signed-off-by: albailey <Al.Bailey@windriver.com>
Change-Id: Ic73d0ea590ab1b7857f7275fa9c71828b0d343ee
Update remaining StarlingX packages with hardcoded TIS_PATCH_VER to
use PKG_GITREVCOUNT where possible, with offsets as needed to ensure
the version is incremented above the hardcoded version.
Story: 2008455
Task: 41447
Signed-off-by: Don Penney <don.penney@windriver.com>
Change-Id: Idf5ef476192cdf4923d6c903f1a15e03cfe9d03f
When a domain interface state is changed, hardware information is
verified to ensure the interface is OK to enter into the new state.
However, when an interface is entering into Not-In-Use state, it should
be always OK no matter what the h/w interface state is. Especially when
the interface is back on lo, in which case getting hardware information
will fail. This prevents moving interface to Not-In-Use state.
This change skip verifying h/w state if state of an interface is changed
to Not-In-Use.
This fix will also skip checking h/w information for lo interface and
always returns enabled = true.
Closes-Bug: 1898629
Signed-off-by: Bin Qian <bin.qian@windriver.com>
Change-Id: I709708bce622f52bf84fc3fec749f204cfeee533
flake8 2.5.5 fails on ubuntu-focal zuul machines running python3.8
with the following error:
AttributeError: 'FlakesChecker' object has no attribute 'CONSTANT'
The update removes the version constraint to use newer flake8.
The linters can be run in python3.
Pylint cannot be run in python3 because mysql-python is not
compatable, so a new zuul job for pylint is now added.
The flake8 errors that the newer version raises are all suppressed,
and some of them should be addressed by someone with familiarity in
this repo.
Change-Id: Ida6447728d4175173c02130cb04a6013e4f966f9
Partial-Bug: 1895054
Signed-off-by: albailey <Al.Bailey@windriver.com>
The DC manager orchestration is being removed from the
dcmanager-manager process and it is running in
dcmanager-orchestrator process.
This update adds associated sm config for the new process.
Change-Id: I7cc0869a123713d85b8167bd1f8a4481b8da0902
Story: 2007267
Task: 40715
Depends-On: https://review.opendev.org/#/c/748452/
Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>
Move the cert-mon service to controller-services service group, so
to make it more generic for other platform certificate monitoring
features.
Story: 2007347
Task: 40119
Change-Id: Ib82a579dd2f1d0dcf97e90eed44fb095ee9ab6ca
Signed-off-by: Bin Qian <bin.qian@windriver.com>
Add new critical service cert-mon to under SM manage,
in controller-services group.
The new service will monitor admin endpoint service renewal in
cert-manager and apply new certificates to controller nodes in a
DC setup.
Tests:
Provision DC system controllers. Swact. all successful.
Added and provisioned dummy service process, swact, all successful.
Change-Id: Ic545fafc88be4acb4e5e0ea3e4449ade57dcef8c
Story: 2007347
Task: 40119
Signed-off-by: Bin Qian <bin.qian@windriver.com>
Service Management (SM) monitors connectivity and health of
its peer controller over the OAM, Mgmt and (if provisioned)
Cluster-Host networks.
If SM sees all the links to its peer go 'carrier down' virtually
simultaneously, it is possible that both controllers might
simultaneously declare themselves unhealthy and both go
disabled; i.e. shutdown all services with no automatic recovery.
This update adds an 'Unhealthy State Recovery Audit' to SM which
forces a self restart when all of its monitored links recover
for cases where both controllers go unhealthy-shutdown or both
controllers remain active in split-brain.
Test Plan:
PASS: Verify AIO SX install
PASS: Verify Standard system install and unhealthy state recovery
PASS: Verify single link failure end to end behavior
PASS: Verify 2 of 3 link failure end to end behavior
PASS: Verify all link failure end to end behavior
PASS: Verify SM and Mtce heartbeat recovery over unhealthy state recovery
PASS: Verify swact back and forth following a recovery
PASS: Verify process restart as part of unhealthy state recovery
PASS: Verify AIO DX install and unhealthy state recovery
Change-Id: Ie906eaf04bec607328b7e0af09b37fa0558e3bbe
Closes-Bug: 1883004
Signed-off-by: Eric MacDonald <eric.macdonald@windriver.com>
Setting up the bandit tool for the scanning of HIGH severity issues
in the python codes under Starlingx/ha folder.
Expecting this merge will enable zuul job for CI/CD of bandit scan.
Configuration files:
1. tox.ini for adding bandit environment and command.
2. test-requirements.txt for adding bandit version.
3. .zuul.yaml file for adding bandit job and configuring under
check job to run code scan every time before code commit.
Test:
Run tox -e bandit command inside the fault folder to validate the
bandit scan and result.
Please note:
Changes will be implemented in batches and this is Batch3 change.
Story: 2007541
Task: 39621
Depends-On: https://review.opendev.org/#/c/721294/
Change-Id: I01f81d7c52c12432965106f9603e4db600381971
Signed-off-by: Sharath Kumar K <sharath.kumar@intel.com>
Switch to openstackdocstheme 2.2.1 and reno 3.1.0 versions. Using
these versions will allow especially:
* Linking from HTML to PDF document
* Allow parallel building of documents
* Fix some rendering problems
Update Sphinx version as well.
Disable openstackdocs_auto_name to use 'project' variable as name.
Change pygments_style to 'native' since old theme version always used
'native' and the theme now respects the setting and using 'sphinx' can
lead to some strange rendering.
openstackdocstheme renames some variables, so follow the renames
before the next release removes them. A couple of variables are also
not needed anymore, remove them.
See also
http://lists.openstack.org/pipermail/openstack-discuss/2020-May/014971.html
Change-Id: Iab15b05918b73ce9ba2ff0b479fdb8a0631fad42
"env:UPPER_CONSTRAINTS_FILE" doesn't work with latest Zuul.
Remove it to make sure upper-constraints working.
Closes-Bug: 1880097
Change-Id: Ibc6ab625eda1b5695798d044ef367f375086c87c
Signed-off-by: Shuicheng Lin <shuicheng.lin@intel.com>
Subcloud audit is being removed from the dcmanager-manager
process and it is running in dcmanager-audit process.
This update adds associated sm config for the new process.
Story: 2007267
Task: 39639
Depends-On: https://review.opendev.org/#/c/725627/
Change-Id: Icf7dff67756c6bc168cc768e1e5cb42e1254c77f
Signed-off-by: Tao Liu <tao.liu@windriver.com>
The filesystem /opt/patch-vault is renamed to /opt/dc-vault so that
it can be re-used to store FPGA images and software loads. Thus,
the SM services have been renamed from drbd-patch-vault and
patch-vault-fs to drbd-dc-vault and dc-vault-fs.
Change-Id: Ifeed62ad86f043b5b2d9602cbfffa0ea95f2c685
Story: 2006740
Task: 39550
Depends-On: https://review.opendev.org/#/c/723007/
Signed-off-by: Jessica Castelino <jessica.castelino@windriver.com>
This update adds dcmanager-manager as dcorch-patch-api-proxy enable
dependency.
Test cases
1. Swact few times and ensure the dcorch-patch-api-proxy is enabled
after the dcmanager-manager is enabled.
2. Power off/on both AIO controllers and ensure that
dcorch-patch-api-proxy is enabled after the
dcmanager-manager is enabled.
Change-Id: Ief61bcbd973398acce4473c7cd429f03d34b5a98
Partial-Bug: 1868604
Signed-off-by: Tao Liu <tao.liu@windriver.com>
SM runs ocf script to disable sysinv-conductor. Currently,
the timeout for sysinv-conductor ocf script to forcibly
terminate sysinv-conductor is 55s and the timeout for SM
to kill sysinv-conductor ocf script is 60s. In the case
that host-swact happens during application and k8s upgrade
operations, it will take a long time to complete swact
as these operations may spawn long-running processes or
greenthreads which prevent the sysinv-conductor from
shutting down before reaching the timeout to forcibly shutdown.
In a controller swact scenario, this results in the system
having almost no services running for almost a minute,
while SM is waiting for sysinv-conductor to shut down.
This commit updates the timeout for sysinv-conductor ocf
script to the default 15s and reduces the timeout for SM
to kill the script is 20s.
Tests conducted:
- platform sanity
- perform host-swact during stx-openstack upload/apply/remove
- perform host-swact during kube upgrade operations
- perform host-swact during platform-integ-apps apply
- verified sysinv-conductor can be forcely shutdown after 15s
when long-running processes are running
Change-Id: I337154a140f6cec3d6ab953003bf355b4396249e
Story: 2006781
Task: 38480
Signed-off-by: Angie Wang <angie.wang@windriver.com>
Zuul
Bin Qian
Jessica Castelino
Eric MacDonald
Charles Short
Chris Friesen
Teresa Ho
Martin, Chen
Takamasa Takenaka
albailey
Don Penney
Sharath Kumar K
Andreas Jaeger
Shuicheng Lin
Tao Liu
Gerry Kopec
Angie Wang