starlingx/ansible-playbooks
Code Issues Proposed changes

Fix ansible download_images handle invalid registry

Browse Source 
Exit download_images on exception or when number of failed downloads
threshold is exceeded.  The case for invalid registries or other error,
such as invalid DNS, is measured about 20 minutes vs 4 hours in the bug
reported initial error case.

The playbook will retry and pull the remaining required images; as
docker will not pull images for the already downloaded images.

Tests Performed:
o initial installation and sanity which performs download_images
o modify registries to point to invalid.docker.registry and
  measure timeouts
o perform ansible-playbook replay with invalid registries
  Either APIError with Forbidden or 'no auth host' may occur
 (replay with invalid registry or invalid DNS)
o perform ansible-playbook replay with good registries

Closes-Bug: 1933863
Signed-off-by: John Kung <john.kung@windriver.com>
Change-Id: Id2dec495ae6f956af7c66f0bcf88931991ece0c4
This commit is contained in:
John Kung 2021-06-28 17:06:47 -05:00
parent 95e3828e75
commit c0bc0d61a4
1 changed files with 39 additions and 42 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

23
playbookconfig/src/playbooks/roles/common/push-docker-images/files/download_images.py
View File

@ -14,8 +14,10 @@ import json
import keyring
import subprocess
MAX_DOWNLOAD_ATTEMPTS = 3
MAX_DOWNLOAD_THREAD = 5
MAX_FAILED_DOWNLOADS = 5
num_failed_downloads = 0
DEFAULT_REGISTRIES = {
'docker.io': 'docker.io',
@ -73,7 +75,11 @@ def download_an_image(img):
except docker.errors.APIError as e:
print(str(e))
print("Image {} not found on local registry, attempt to download...".format(target_img))
for i in range(MAX_DOWNLOAD_ATTEMPTS):
# pull if not already too many failed downloads
if num_failed_downloads > MAX_FAILED_DOWNLOADS:
print("Too many failed downloads {}, skipping {}".format(
num_failed_downloads, target_img))
return target_img, False
try:
client.pull(target_img)
print("Image download succeeded: %s" % target_img)
@ -102,19 +108,8 @@ def download_an_image(img):
print(delete_warn % local_img)
return target_img, True
except docker.errors.NotFound as e:
print(err_msg + str(e))
return target_img, False
except docker.errors.APIError as e:
print(err_msg + str(e))
if "no basic auth credentials" in str(e):
return target_img, False
except Exception as e:
print(err_msg + str(e))
print("Sleep 20s before retry downloading image %s ..." % target_img)
time.sleep(20)
return target_img, False
@ -132,6 +127,7 @@ def get_img_tag_with_registry(pub_img):
def download_images(images):
global num_failed_downloads
failed_images = []
threads = min(MAX_DOWNLOAD_THREAD, len(images))
@ -139,6 +135,7 @@ def download_images(images):
for img, success in pool.imap(download_an_image, images):
if not success:
failed_images.append(img)
num_failed_downloads += 1
return failed_images