starlingx/config
Code Issues Proposed changes

Enhance app updates during Kubernetes upgrades

Browse Source 
Add the folowing enhancements to application updates during
Kubernetes upgrades:

1) Move the pre application update logic from kube-upgrade-start step
   to a specific separated step called via a new command line option
   named kube-pre-application-update, which can be triggered after the
   download images step and before upgrading networking.
2) Move the post application update logic from kube-upgrade-complete
   step to a specific separated step called via a new command line
   option named kube-post-application-update, which can be triggered
   after the kube-upgrade-complete stage and before the upgrade is
   deleted.
3) Introduce validation logic to kube-upgrade-start step to check if
   all applied apps have available versions compatible with intermediate
   and target Kubernetes versions. Upgrades are blocked if apps marked
   to be pre updated are incompatible with current and target Kubernetes
   versions. Upgrades are also blocked if apps marked to be post updated
   are incompatible with the target Kubernetes version.
4) Delete uploaded applications incompatible with the target Kubernetes
   version and upload one that is compatible if available.
5) Restore kube-upgrade-start and kube-upgrade-complete to their
   original logic before application updates during Kubernetes upgrades
   was implemented on task 49416. The kube-upgrade-start step is
   synchronous as it used to be before that change.
6) Update sysinv and cgts-client unit tests to account for the new
   Kubernetes upgrade steps.
7) Create a helper function called "patch_kube_upgrade" to improve code
   reuse when creating patch requests for new shell commands related to
   Kubernetes upgrades.

Test Plan:
AIO-SX Test Cases:
PASS: Fresh install.
PASS: Successful Kubernetes single version upgrade with no apps that
      need to be updated.
PASS: Successful Kubernetes multi-version upgrade with no apps that need
      to be updated.
PASS: Successful Kubernetes upgrade with apps that need to be updated
      before and after the new version is deployed.
PASS: Check if the upgrade is blocked if an app is incompatible with a
      Kubernetes intermediate version during a multi-version
      upgrade.
PASS: Check if the upgrade is blocked if an app marked to be pre updated
      is incompatible with the Kubernetes target version.
PASS: Check if the upgrade is blocked if an app marked to be post
      updated is incompatible with the Kubernetes target version.
PASS: Check if uploaded apps have been replaced by compatible versions.
PASS: Check if uploaded apps that do not have compatible versions were
      removed.
PASS: Failure to run kube-pre-application-update and successful
      retry.
PASS: Failure to run kube-post-application-update and successful
      retry.
PASS: Abort during kube-pre-application-update and start over.
PASS: Reject aborting Kubernetes upgrade after post-updated-apps state.
AIO-DX Test Cases:
PASS: Fresh install.
PASS  Successful Kubernetes upgrade with no apps that need to be
      updated.
PASS: Successful Kubernetes upgrade with apps that need to be updated
      before and after the new version is deployed.
PASS: Check if the upgrade is blocked if an app marked to be pre updated
      is incompatible with the Kubernetes target version.
PASS: Check if the upgrade is blocked if an app marked to be post
      updated is incompatible with the Kubernetes target version.

Story: 2010929
Task: 49595

Change-Id: I9b48567c39c9a12b7563d56ab90fbfe9dd7082aa
Signed-off-by: Igor Soares <Igor.PiresSoares@windriver.com>
This commit is contained in:
Igor Soares
2024-02-15 19:39:07 -03:00
parent 8b82a82372
commit 7bd361b69b
10 changed files with 874 additions and 309 deletions
Download Patch File Download Diff File Expand all files Collapse all files

54
sysinv/cgts-client/cgts-client/cgtsclient/tests/v1/test_kube_upgrade_shell.py
View File

@@ -85,6 +85,33 @@ class KubeUpgradeTest(test_shell.ShellTest):
self.assertIn(fake_kube_upgrade['created_at'], results)
self.assertIn(fake_kube_upgrade['updated_at'], results)
@mock.patch('cgtsclient.v1.kube_upgrade.KubeUpgradeManager.update')
def test_kube_pre_application_update(self, mock_update):
fake_kube_upgrade = {'from_version': 'v1.42.1',
'to_version': 'v1.42.2',
'state': 'pre-updating-apps',
'uuid': 'cb737aba-1820-4184-b0dc-9b073822af48',
'created_at': 'fake-created-time',
'updated_at': 'fake-updated-time',
}
mock_update.return_value = KubeUpgrade(None, fake_kube_upgrade, True)
self.make_env()
results = self.shell("kube-pre-application-update")
patch = {'op': 'replace',
'path': '/state',
'value': 'pre-updating-apps'
}
mock_update.assert_called_once_with([patch])
self.assertIn(fake_kube_upgrade['from_version'], results)
self.assertIn(fake_kube_upgrade['to_version'], results)
self.assertIn(fake_kube_upgrade['state'], results)
self.assertIn(fake_kube_upgrade['uuid'], results)
self.assertIn(fake_kube_upgrade['created_at'], results)
self.assertIn(fake_kube_upgrade['updated_at'], results)
@mock.patch('cgtsclient.v1.kube_upgrade.KubeUpgradeManager.update')
def test_kube_upgrade_download_images(self, mock_update):
fake_kube_upgrade = {'from_version': 'v1.42.1',
@@ -125,6 +152,33 @@ class KubeUpgradeTest(test_shell.ShellTest):
self.assertIn(fake_kube_upgrade['created_at'], results)
self.assertIn(fake_kube_upgrade['updated_at'], results)
@mock.patch('cgtsclient.v1.kube_upgrade.KubeUpgradeManager.update')
def test_kube_post_application_update(self, mock_update):
fake_kube_upgrade = {'from_version': 'v1.42.1',
'to_version': 'v1.42.2',
'state': 'post-updating-apps',
'uuid': 'cb737aba-1820-4184-b0dc-9b073822af48',
'created_at': 'fake-created-time',
'updated_at': 'fake-updated-time',
}
mock_update.return_value = KubeUpgrade(None, fake_kube_upgrade, True)
self.make_env()
results = self.shell("kube-post-application-update")
patch = {'op': 'replace',
'path': '/state',
'value': 'post-updating-apps'
}
mock_update.assert_called_once_with([patch])
self.assertIn(fake_kube_upgrade['from_version'], results)
self.assertIn(fake_kube_upgrade['to_version'], results)
self.assertIn(fake_kube_upgrade['state'], results)
self.assertIn(fake_kube_upgrade['uuid'], results)
self.assertIn(fake_kube_upgrade['created_at'], results)
self.assertIn(fake_kube_upgrade['updated_at'], results)
@mock.patch('cgtsclient.v1.kube_upgrade.KubeUpgradeManager.update')
def test_kube_upgrade_complete(self, mock_update):
fake_kube_upgrade = {'from_version': 'v1.42.1',

94
sysinv/cgts-client/cgts-client/cgtsclient/v1/kube_upgrade_shell.py
View File

@@ -9,6 +9,7 @@ from cgtsclient import exc
from cgtsclient.v1 import ihost as ihost_utils
# Kubernetes constants
KUBE_UPGRADE_STATE_PRE_UPDATING_APPS = 'pre-updating-apps'
KUBE_UPGRADE_STATE_DOWNLOADING_IMAGES = 'downloading-images'
KUBE_UPGRADE_STATE_UPGRADING_NETWORKING = 'upgrading-networking'
KUBE_UPGRADE_STATE_UPGRADING_STORAGE = 'upgrading-storage'
@@ -18,6 +19,7 @@ KUBE_UPGRADE_STATE_UPGRADING_SECOND_MASTER = 'upgrading-second-master'
KUBE_UPGRADE_STATE_ABORTING = 'upgrade-aborting'
KUBE_UPGRADE_STATE_CORDON = 'cordon-started'
KUBE_UPGRADE_STATE_UNCORDON = 'uncordon-started'
KUBE_UPGRADE_STATE_POST_UPDATING_APPS = 'post-updating-apps'
def _print_kube_upgrade_show(obj):
@@ -56,11 +58,8 @@ def do_kube_upgrade_start(cc, args):
_print_kube_upgrade_show(kube_upgrade)
def do_kube_upgrade_download_images(cc, args):
"""Download kubernetes images."""
data = dict()
data['state'] = KUBE_UPGRADE_STATE_DOWNLOADING_IMAGES
def patch_kube_upgrade(cc, data):
"""" Call patch HTTP method for kube upgrades"""
patch = []
for (k, v) in data.items():
@@ -73,6 +72,24 @@ def do_kube_upgrade_download_images(cc, args):
_print_kube_upgrade_show(kube_upgrade)
def do_kube_upgrade_download_images(cc, args):
"""Download kubernetes images."""
data = dict()
data['state'] = KUBE_UPGRADE_STATE_DOWNLOADING_IMAGES
patch_kube_upgrade(cc, data)
def do_kube_pre_application_update(cc, args):
"""Update applications before Kubernetes is upgraded."""
data = dict()
data['state'] = KUBE_UPGRADE_STATE_PRE_UPDATING_APPS
patch_kube_upgrade(cc, data)
@utils.arg('hostid', metavar='<hostname or id>',
help="Name or ID of host")
def do_kube_host_cordon(cc, args):
@@ -83,15 +100,7 @@ def do_kube_host_cordon(cc, args):
data['hostname'] = ihost.hostname
data['state'] = KUBE_UPGRADE_STATE_CORDON
patch = []
for (k, v) in data.items():
patch.append({'op': 'replace', 'path': '/' + k, 'value': v})
try:
kube_upgrade = cc.kube_upgrade.update(patch)
except exc.HTTPNotFound:
raise exc.CommandError('Kubernetes upgrade UUID not found')
_print_kube_upgrade_show(kube_upgrade)
patch_kube_upgrade(cc, data)
@utils.arg('hostid', metavar='<hostname or id>',
@@ -104,15 +113,7 @@ def do_kube_host_uncordon(cc, args):
data['hostname'] = ihost.hostname
data['state'] = KUBE_UPGRADE_STATE_UNCORDON
patch = []
for (k, v) in data.items():
patch.append({'op': 'replace', 'path': '/' + k, 'value': v})
try:
kube_upgrade = cc.kube_upgrade.update(patch)
except exc.HTTPNotFound:
raise exc.CommandError('Kubernetes upgrade UUID not found')
_print_kube_upgrade_show(kube_upgrade)
patch_kube_upgrade(cc, data)
def do_kube_upgrade_networking(cc, args):
@@ -121,15 +122,7 @@ def do_kube_upgrade_networking(cc, args):
data = dict()
data['state'] = KUBE_UPGRADE_STATE_UPGRADING_NETWORKING
patch = []
for (k, v) in data.items():
patch.append({'op': 'replace', 'path': '/' + k, 'value': v})
try:
kube_upgrade = cc.kube_upgrade.update(patch)
except exc.HTTPNotFound:
raise exc.CommandError('Kubernetes upgrade UUID not found')
_print_kube_upgrade_show(kube_upgrade)
patch_kube_upgrade(cc, data)
def do_kube_upgrade_storage(cc, args):
@@ -138,15 +131,16 @@ def do_kube_upgrade_storage(cc, args):
data = dict()
data['state'] = KUBE_UPGRADE_STATE_UPGRADING_STORAGE
patch = []
for (k, v) in data.items():
patch.append({'op': 'replace', 'path': '/' + k, 'value': v})
try:
kube_upgrade = cc.kube_upgrade.update(patch)
except exc.HTTPNotFound:
raise exc.CommandError('Kubernetes upgrade UUID not found')
patch_kube_upgrade(cc, data)
_print_kube_upgrade_show(kube_upgrade)
def do_kube_post_application_update(cc, args):
"""Update applications after Kubernetes is upgraded."""
data = dict()
data['state'] = KUBE_UPGRADE_STATE_POST_UPDATING_APPS
patch_kube_upgrade(cc, data)
def do_kube_upgrade_abort(cc, args):
@@ -155,15 +149,7 @@ def do_kube_upgrade_abort(cc, args):
data = dict()
data['state'] = KUBE_UPGRADE_STATE_ABORTING
patch = []
for (k, v) in data.items():
patch.append({'op': 'replace', 'path': '/' + k, 'value': v})
try:
kube_upgrade = cc.kube_upgrade.update(patch)
except exc.HTTPNotFound:
raise exc.CommandError('Kubernetes upgrade not found')
_print_kube_upgrade_show(kube_upgrade)
patch_kube_upgrade(cc, data)
def do_kube_upgrade_complete(cc, args):
@@ -172,15 +158,7 @@ def do_kube_upgrade_complete(cc, args):
data = dict()
data['state'] = KUBE_UPGRADE_STATE_COMPLETE
patch = []
for (k, v) in data.items():
patch.append({'op': 'replace', 'path': '/' + k, 'value': v})
try:
kube_upgrade = cc.kube_upgrade.update(patch)
except exc.HTTPNotFound:
raise exc.CommandError('Kubernetes upgrade UUID not found')
_print_kube_upgrade_show(kube_upgrade)
patch_kube_upgrade(cc, data)
def do_kube_upgrade_delete(cc, args):