861ecd49cf
When we build application docker images if "docker build" fails, we sleep a bit and retry. But some dockerfiles contain commands that falsely succeed, causing failures later. That falsely successful command gets cached by docker build cache, so the retry is ineffective. HYPOTHETICAL EXAMPLE ==================== FROM ubuntu:latest # At this point apt cache already exists (inherited from base image) # Lets say this fails because some of the repo URLs aren't responding # apt-get will print a warning, but "succeed" and keep using the old # cache RUN apt-get update # This fails, because package index cache is too old. But when we # retry the build, it resumes from here because the previous # "apt-get update" was seemingly successful and got cached by docker # build. RUN apt-get install SOME_PACKAGE SOLUTION ======== Always disable the cache after the 1st build attempt fails. TESTS ===== Build the base image, a "loci" and a "docker"-type containers with an artificial error in them. Make sure the second build attempt does not use the cache, even when "--cache" is present on the command line. Story: 2010055 Task: 52260 Signed-off-by: Davlet Panech <davlet.panech@windriver.com> Change-Id: Ie251faaac8c2537bd2361cabe09800ddd1204af6
50 lines
1.3 KiB
Bash
50 lines
1.3 KiB
Bash
# bash
|
|
|
|
# Usage: MAX_ATTEMPTS=1 RETRY_DELAY=2 USE_DOCKER_CACHE=yes \
|
|
# docker_build_with_retries DOCKER_ARGS...
|
|
function docker_build_with_retries {
|
|
|
|
if [[ -z "$MAX_ATTEMPTS" ]] ; then
|
|
echo "ERROR: docker_build_with_retries(): MAX_ATTEMPTS must be defined!" >&2
|
|
return 1
|
|
fi
|
|
|
|
if [[ -z "$RETRY_DELAY" ]] ; then
|
|
echo "ERROR: docker_build_with_retries(): RETRY_DELAY must be defined!" >&2
|
|
return 1
|
|
fi
|
|
|
|
local max_attempts=$MAX_ATTEMPTS
|
|
local delay=$RETRY_DELAY
|
|
|
|
local -a cache_args
|
|
if [[ "$USE_DOCKER_CACHE" != "yes" ]] ; then
|
|
cache_args=("--no-cache")
|
|
fi
|
|
|
|
local -i attempt=0
|
|
while true ; do
|
|
let attempt++
|
|
if [[ attempt -gt 1 ]] ; then
|
|
cache_args=("--no-cache")
|
|
fi
|
|
echo "Running: docker build ${cache_args[@]} $@" >&2
|
|
docker build "${cache_args[@]}" "$@"
|
|
local rc=$?
|
|
if [ $rc -eq 0 ]; then
|
|
return 0
|
|
fi
|
|
|
|
echo "Command [docker build] failed [rc=$rc], attempt ${attempt} of ${max_attempts}." >&2
|
|
if [ ${attempt} -lt ${max_attempts} ]; then
|
|
echo "Waiting ${delay} seconds before retrying..." >&2
|
|
sleep ${delay}
|
|
continue
|
|
else
|
|
echo "Max command attempts reached. Aborting..." >&2
|
|
return 1
|
|
fi
|
|
done
|
|
|
|
}
|