Add pod health status to kube rootca check

As part of the kube rootca certificate update, it's recommended to
have all pods in ready state to avoid problems during it. This
commit adds an optional flag to 'system health-query-kube-upgrade'
command to check for pod health and returns a list of pods that are
not ready or completed.

Usage: system health-query-kube-upgrade --rootca

Test Cases:
1) PASS: Run the following commands and verify their output remains
         unchanged.
         - system health-query
         - system health-query-upgrade
         - system health-query-kube-upgrade (without --rootca)
2) PASS: Run "system health-query-kube-upgrade --rootca" without any
         pod in failure state and verify that the correct success
         message was included in the command output.
3) PASS: Repeat test 2 but adding pods in unhealthy state (Error,
         Evicted and CrashLoopBackOff) and verify that the output
         contains the correct error message and a list of the
         unhealthy pods.
4) PASS: Repeat test 2 but adding pods with completed and pending
         status and verify the completed pod was not added to the
         failed pod list and the correct success message was showned.
5) PASS: Repeat test 3 but adding pods with completed and pending
         status and verify this pods weren't added to the failed pod
         list and the correct failure message was showned.
6) PASS: Run 'system kube-rootca-update-start' with pods in unhealthy
         state and verify the update did not start and the correct
         error message was displayed.
7) PASS: Run 'system kube-rootca-update-start' with all pods in
         healthy state and verify the update process started
         successfully.
8) PASS: Create and apply a new sw-manager kube-rootca-update-strategy
         with pods in unhealthy state and verify the apply was aborted
         and the correct error message was displayed.
9) PASS: Create and apply a new sw-manager kube-rootca-update-strategy
         with all pods in healthy state and verify the update was
         applied successfully.

Story: 2010852
Task: 49085

Change-Id: I463ecc8a1107375e4e0997e07581b10ec8d129e2
Signed-off-by: Victor Romano <victor.gluzromano@windriver.com>

sysinv/cgts-client/cgts-client/cgtsclient/v1/health.py

@@ -1,6 +1,6 @@
 # -*- encoding: utf-8 -*-
 #
-# Copyright (c) 2015-2016 Wind River Systems, Inc.
+# Copyright (c) 2015-2023 Wind River Systems, Inc.
 #
 # SPDX-License-Identifier: Apache-2.0
 #
@@ -23,9 +23,12 @@ class HealthManager(base.Manager):
         resp, body = self.api.json_request('GET', path)
         return body
 
-    def get_kube_upgrade(self, relaxed=None):
+    def get_kube_upgrade(self, args: dict, relaxed=None):
         path = '/v1/health/kube-upgrade'
         if relaxed:
             path += '/relaxed'
+        rootca = args.get('rootca')
+        if rootca:
+            path += f'?rootca={rootca}'
         resp, body = self.api.json_request('GET', path)
         return body

sysinv/cgts-client/cgts-client/cgtsclient/v1/health_shell.py

@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2016 Wind River Systems, Inc.
+# Copyright (c) 2016-2023 Wind River Systems, Inc.
 #
 # SPDX-License-Identifier: Apache-2.0
 #
@@ -8,6 +8,8 @@
 # All Rights Reserved.
 #
 
+from cgtsclient.common import utils
+
 
 def do_health_query(cc, args):
     """Run the Health Check."""
@@ -19,6 +21,13 @@ def do_health_query_upgrade(cc, args):
     print(cc.health.get_upgrade())
 
 
+@utils.arg('--rootca',
+           action='store_true',
+           default=False,
+           help='Whether additional RootCA verifications should be executed')
 def do_health_query_kube_upgrade(cc, args):
     """Run the Health Check for a Kubernetes Upgrade."""
-    print(cc.health.get_kube_upgrade())
+    attributes = {}
+    if args.rootca is not None:
+        attributes.update({'rootca': args.rootca})
+    print(cc.health.get_kube_upgrade(attributes))