Retry get alarms/events for SNMP if it fails

Root cause:
SNMP sub-agent uses fm-common library to connect database
to obtain alarm/event list. But after swact, it does not
detect disconnection and as the result, the functions to
obtain alarm/event returns false and does not return proper
list.

Analysis:
fm-common library to connect to the database. It checks
connection by PQstatus() and it will reconnect if it find
connection error. But PQstatus responses CONNECTION_OK
even after swact. The other option to check connection is
using PQPing(). But this method also responses PQPING_OK
even after awact, too.
Other option is sending real SQL and it is same as to
retry. So this solution is to retry to obtain alarm/
event list if it fails.

Solution:
When SNMP sub-agent calls to obtain alarm/event list
and it returns false, retry it. Internally, it reconnects
automatically and it returns the proper list.

Test Plan:
Test the following steps for snmpget, snmpgetnext, snmpbulkget
and snmpwalk in IPv4 and IPv6:
1. Execute "system host-swact controller-n"
2. Send snmp command and record response
3. Send snmp command and record response
4. Compare 2. and 3. response and confirm they are matched
=> all cases PASS

Closes-bug: 1949083

Signed-off-by: Takamasa Takenaka <takamasa.takenaka@windriver.com>
Change-Id: Ib58f9f657bb5ac8c3b505eee5443a24bedb2463e

stx-snmp-helm/centos/docker/stx-fm-subagent/src/snmpActiveAlarms.c

@@ -1,5 +1,5 @@
 /*
-* Copyright (c) 2020 Wind River Systems, Inc.
+* Copyright (c) 2020-2021 Wind River Systems, Inc.
 *
 * SPDX-License-Identifier: Apache-2.0
 *
@@ -26,6 +26,7 @@ static long Active_Alarm_Count = 0;
 static struct activealarm *alarm_list;
 static struct activealarm *alarmaddr, savealarm, *savealarmaddr;
 static int saveIndex = 0;
+static int retryCount = 5;
 static char saveUuid[36];
 static long LastLoad = 0;     /* ET in secs at last table load */
 extern long long_return;
@@ -64,7 +65,19 @@ Alarm_Scan_Init()
     /*
      * query active alarm list from DB
      */
-    if (fm_snmp_util_get_all_alarms(getAlarmSession(), &aquery) != true){
+    bool isAlarmObtained = false;
+    for (i = 0; i < retryCount; i++){
+        if (fm_snmp_util_get_all_alarms(getAlarmSession(), &aquery)){
+            DEBUGMSG(("cgtsAgentPlugin", "get_all_alarms done\n"));
+            isAlarmObtained = true;
+            break;
+        } else {
+            DEBUGMSG(("cgtsAgentPlugin", 
+            "get_all_alarms returns false (%zu/%d). Try again\n",
+             (i+1), retryCount));
+        }
+    }
+    if (!isAlarmObtained) {
         DEBUGMSG(("cgtsAgentPlugin", "get_all_alarms from db failed\n"));
         return;
     }

stx-snmp-helm/centos/docker/stx-fm-subagent/src/snmpEventLogs.c

@@ -1,5 +1,5 @@
 /*
-* Copyright (c) 2020 Wind River Systems, Inc.
+* Copyright (c) 2020-2021 Wind River Systems, Inc.
 *
 * SPDX-License-Identifier: Apache-2.0
 *
@@ -26,6 +26,7 @@ static long Event_Log_Count = 0;
 static struct activealarm *alarm_list;
 static struct activealarm *alarmaddr, savealarm, *savealarmaddr;
 static int saveIndex = 0;
+static int retryCount = 5;
 static char saveUuid[36];
 static long LastLoad = 0;     /* ET in secs at last table load */
 extern long long_return;
@@ -64,7 +65,19 @@ Event_Log_Scan_Init()
     /*
      * query event log list from DB
      */
-    if (fm_snmp_util_get_all_event_logs(getAlarmSession(), &aquery) != true){
+    bool isEventObtained = false;
+    for (i = 0; i < retryCount; i++){
+        if (fm_snmp_util_get_all_event_logs(getAlarmSession(), &aquery)){
+            DEBUGMSG(("cgtsAgentPlugin", "get_all_event_logs done\n"));
+            isEventObtained = true;
+            break;
+        } else {
+            DEBUGMSG(("cgtsAgentPlugin", 
+            "get_all_event_logs returns false (%zu/%d). Try again\n",
+             (i+1), retryCount));
+        }
+    }
+    if (!isEventObtained) {
         DEBUGMSG(("cgtsAgentPlugin", "get_all_event_logs from db failed\n"));
         return;
     }