Build Avoidance
Purpose: Reduce build times after a repo sync by pulling in pre-generated srpms and rpms and other build products created by a local reference build. Usage: repo sync generate-cgcs-centos-repo.sh ... populate_downloads.sh ... build-pkgs --build-avoidance [--build-avoidance-user <user> \ --build-avoidance-host <addr> --build-avoidance-dir <dir>] Reference builds: - A server performs a regular (daily?), automated builds using existing methods. Call these the reference builds. - The builds are timestamped, and preserved for some time. (weeks?) The MY_WORKSPACE directory for the build shall have a common root directory, and a leaf directory that is a UTC time stamp of format YYYYMMDDThhmmssZ. e.g. MY_WORKSPACE=/localdisk/loadbuild/jenkins/StarlingX/20180719T113021Z Alternative formats are possible by setting values in ... "$MY_REPO/local-build-data/build_avoidance_source" e.g. BUILD_AVOIDANCE_DATE_FORMAT="%Y-%m-%d" BUILD_AVOIDANCE_TIME_FORMAT="%H-%M-%S" BUILD_AVOIDANCE_DATE_TIME_DELIM="_" BUILD_AVOIDANCE_DATE_TIME_POSTFIX="" BUILD_AVOIDANCE_DATE_UTC=0 Which results in YYYY-MM-DD_hh-mm-ss format using local time. The one property that the timestamp must have is that they are sortable, and that the reference build and the consumer of the reference builds agree on the format. - A build CONTEXT is captured, consisting of the SHA of each and every git that contributed to the build. - For each package built, a file shall capture he md5sums of all the source code inputs to the build of that package. - All these build products are accessible locally (e.g. a regional office) via rsync (other protocols can be added later). ssh is also required to run remote query commands on the reference build. Initial ground work to support a selection variable .... BUILD_AVOIDANCE_FILE_TRANSFER="my-transfer-protocol" in $MY_REPO/local-build-data/build_avoidance_source" has been created, but "rsync" is the only valid value at this time. - Location of the reference build can be specified via command line, or defaults can be put in $MY_REPO/local-build-data/build_avoidance_source. The local-build-data directory is gitignored by stx-root and so can be customized for local needs. e.g. cat $MY_REPO/local-build-data/build_avoidance_source BUILD_AVOIDANCE_USR="jenkins" BUILD_AVOIDANCE_HOST="stx-build-server.myco.com" BUILD_AVOIDANCE_DIR="/localdisk/loadbuild/jenkins/StarlingX" Notes: - Build avoidance is only used if requested. - Build avoidance does not necessarily use the latest reference build. It compares the git context of all available reference builds vs your own git context, and chooses the most recent for which you gits have all the conent. i.e. all your gits will be same or newer than that used by the reference build. This also meens that some packages might still need to be rebuilt after the download step. - Normally build avoidance remembers the last download context and will only consider reference builds newer than the last download. You can reset using 'build-pkgs --build-avoidance --clear' to erase the download history. When might this matter to me? If you change to an old branch that hasn't been synced recently and want to build in that context. - The primary assumtion of Build Avoidance is that it is faster to download packages than to build them. This is typically true of a good LAN, but likely not true of a WAN. This is why we emphasize the local nature of your reference build server. Also in this update: - reworked context generation to be relative to 'dirname $MY_REPO' - Moved md5sum calculation to a common file, and fixed case where symlinks where canonacalized to paths outside of $MY_REPO. We'll make an exception to canonacalization to keep paths relative to $MY_REPO. - In future other functions could be moved to the common file. Story: 2002835 Task: 22754 Change-Id: I757780190cc6063d0a2d3ad9d0a6020ab5169e99 Signed-off-by: Scott Little <scott.little@windriver.com>
This commit is contained in:
parent
2f9d9a5672
commit
b20ac0164d
1
.gitignore
vendored
1
.gitignore
vendored
@ -2,3 +2,4 @@
|
||||
/cgcs-centos-repo
|
||||
/cgcs-tis-repo
|
||||
/cgcs-3rd-party-repo
|
||||
/local-build-data
|
||||
|
23
build-data/build_avoidance_source
Normal file
23
build-data/build_avoidance_source
Normal file
@ -0,0 +1,23 @@
|
||||
#
|
||||
# Copyright (c) 2018 Wind River Systems, Inc.
|
||||
#
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
#
|
||||
|
||||
|
||||
#
|
||||
# What files and directories need to be copied
|
||||
#
|
||||
BUILD_AVOIDANCE_SRPM_DIRECTORIES="inputs srpm_assemble rpmbuild/SRPMS rpmbuild/SOURCES"
|
||||
BUILD_AVOIDANCE_SRPM_FILES=""
|
||||
BUILD_AVOIDANCE_RPM_DIRECTORIES="results rpmbuild/RPMS rpmbuild/SPECS"
|
||||
BUILD_AVOIDANCE_RPM_FILES=".platform_release"
|
||||
|
||||
|
||||
#
|
||||
# Copy the lines below to $MY_REPO/local-build-data/build_avoidance_source,
|
||||
# then uncomment and fill in the values giving the location of your local reference build.
|
||||
#
|
||||
# BUILD_AVOIDANCE_USR="jenkins"
|
||||
# BUILD_AVOIDANCE_HOST="machine.corp.com"
|
||||
# BUILD_AVOIDANCE_DIR="/localdisk/loadbuild/jenkins/StarlingX_Build"
|
842
build-tools/build-avoidance-utils.sh
Normal file
842
build-tools/build-avoidance-utils.sh
Normal file
@ -0,0 +1,842 @@
|
||||
#
|
||||
# Copyright (c) 2018 Wind River Systems, Inc.
|
||||
#
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
#
|
||||
|
||||
#
|
||||
# Functions related to build avoidance.
|
||||
#
|
||||
# Do not call directly. Used by build-pkgs.
|
||||
#
|
||||
# Build avoidance downloads rpm, src.rpm and other artifacts of
|
||||
# build-pkgs for a local reference build. The reference would
|
||||
# typically be an automated build run atleast daily.
|
||||
# The MY_WORKSPACE directory for the reference build shall have
|
||||
# a common root directory, and a leaf directory that is a time stamp
|
||||
# in a sortable parsable format. Default YYYYMMDDThhmmssZ.
|
||||
# e.g. /localdisk/loadbuild/jenkins/StarlingX/20180719T113021Z
|
||||
#
|
||||
# Other formats can be used by setting the following variables
|
||||
# in $MY_REPO/local-build-data/build_avoidance_source.
|
||||
# e.g. to allow format YYYY-MM-DD_hh-mm-ss
|
||||
# BUILD_AVOIDANCE_DATE_FORMAT="%Y-%m-%d"
|
||||
# BUILD_AVOIDANCE_TIME_FORMAT="%H-%M-%S"
|
||||
# BUILD_AVOIDANCE_DATE_TIME_DELIM="_"
|
||||
# BUILD_AVOIDANCE_DATE_TIME_POSTFIX=""
|
||||
#
|
||||
# Note: Must be able to rsync and ssh to the machine that holds the
|
||||
# reference builds.
|
||||
#
|
||||
# In future alternative transfer protocols may be supported.
|
||||
# Select the alternate protocol by setting the following variables
|
||||
# in $MY_REPO/local-build-data/build_avoidance_source.
|
||||
# e.g.
|
||||
# BUILD_AVOIDANCE_FILE_TRANSFER="my-supported-prototcol"
|
||||
#
|
||||
|
||||
BUILD_AVOIDANCE_USR=""
|
||||
BUILD_AVOIDANCE_HOST=""
|
||||
BUILD_AVOIDANCE_DIR=""
|
||||
BUILD_AVOIDANCE_URL=""
|
||||
|
||||
# Default date/time format, iso-8601 compact, 20180912T143913Z
|
||||
# Syntax is a subset of that use by the unix 'date' command.
|
||||
BUILD_AVOIDANCE_DATE_FORMAT="%Y%m%d"
|
||||
BUILD_AVOIDANCE_TIME_FORMAT="%H%M%S"
|
||||
BUILD_AVOIDANCE_DATE_TIME_DELIM="T"
|
||||
BUILD_AVOIDANCE_DATE_TIME_POSTFIX="Z"
|
||||
|
||||
# Default file transfer method
|
||||
BUILD_AVOIDANCE_FILE_TRANSFER="rsync"
|
||||
|
||||
# Default is to use timestamps and days in UTC
|
||||
#
|
||||
# If you prefer local time, then set 'BUILD_AVOIDANCE_DATE_UTC=0'
|
||||
# in '$MY_REPO/local-build-data/build_avoidance_source'
|
||||
BUILD_AVOIDANCE_DATE_UTC=1
|
||||
|
||||
BUILD_AVOIDANCE_DATA_DIR="$MY_WORKSPACE/build_avoidance_data"
|
||||
BUILD_AVOIDANCE_SOURCE="$MY_REPO/build-data/build_avoidance_source"
|
||||
BUILD_AVOIDANCE_LOCAL_SOURCE="$MY_REPO/local-build-data/build_avoidance_source"
|
||||
BUILD_AVOIDANCE_TEST_CONTEXT="$BUILD_AVOIDANCE_DATA_DIR/test_context"
|
||||
BUILD_AVOIDANCE_LAST_SYNC_FILE="$BUILD_AVOIDANCE_DATA_DIR/last_sync_context"
|
||||
|
||||
if [ ! -f $BUILD_AVOIDANCE_SOURCE ]; then
|
||||
echo "Couldn't read $BUILD_AVOIDANCE_SOURCE"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "Reading: $BUILD_AVOIDANCE_SOURCE"
|
||||
source $BUILD_AVOIDANCE_SOURCE
|
||||
|
||||
if [ -f $BUILD_AVOIDANCE_LOCAL_SOURCE ]; then
|
||||
echo "Reading: $BUILD_AVOIDANCE_LOCAL_SOURCE"
|
||||
source $BUILD_AVOIDANCE_LOCAL_SOURCE
|
||||
fi
|
||||
|
||||
UTC=""
|
||||
|
||||
if [ $BUILD_AVOIDANCE_DATE_UTC -eq 1 ]; then
|
||||
UTC="--utc"
|
||||
fi
|
||||
|
||||
|
||||
if [ "x$BUILD_AVOIDANCE_OVERRIDE_DIR" != "x" ]; then
|
||||
BUILD_AVOIDANCE_DIR="$BUILD_AVOIDANCE_OVERRIDE_DIR"
|
||||
fi
|
||||
|
||||
if [ "x$BUILD_AVOIDANCE_OVERRIDE_HOST" != "x" ]; then
|
||||
BUILD_AVOIDANCE_HOST="$BUILD_AVOIDANCE_OVERRIDE_HOST"
|
||||
fi
|
||||
|
||||
if [ "x$BUILD_AVOIDANCE_OVERRIDE_USR" != "x" ]; then
|
||||
BUILD_AVOIDANCE_USR="$BUILD_AVOIDANCE_OVERRIDE_USR"
|
||||
fi
|
||||
|
||||
echo "BUILD_AVOIDANCE_DIR=$BUILD_AVOIDANCE_DIR"
|
||||
echo "BUILD_AVOIDANCE_HOST=$BUILD_AVOIDANCE_HOST"
|
||||
echo "BUILD_AVOIDANCE_USR=$BUILD_AVOIDANCE_USR"
|
||||
|
||||
build_avoidance_clean () {
|
||||
if [ -f $BUILD_AVOIDANCE_LAST_SYNC_FILE ]; then
|
||||
\rm -f -v "$BUILD_AVOIDANCE_LAST_SYNC_FILE"
|
||||
fi
|
||||
}
|
||||
|
||||
|
||||
date_to_iso_8601 () {
|
||||
local DATE="$1"
|
||||
local CENTURY=""
|
||||
local YEAR_IN_CENTURY="00"
|
||||
local MONTH="01"
|
||||
local DAY="01"
|
||||
local DAY_OF_YEAR=""
|
||||
|
||||
CENTURY="$(date '+%C')"
|
||||
|
||||
for x in $(echo "${BUILD_AVOIDANCE_DATE_FORMAT}" | tr ' ' '#' | sed 's/%%/#/g' | tr '%' ' ' ); do
|
||||
# Consume format case options
|
||||
case ${x:0:1} in
|
||||
^) x=${x:1};;
|
||||
\#) x=${x:1};;
|
||||
*) ;;
|
||||
esac
|
||||
|
||||
# Process format
|
||||
case $x in
|
||||
Y*) CENTURY=${DATE:0:2}; YEAR_IN_CENTURY=${DATE:2:2}; DATE=${DATE:4}; x=${x:1};;
|
||||
0Y*) CENTURY=${DATE:0:2}; YEAR_IN_CENTURY=${DATE:2:2}; DATE=${DATE:4}; x=${x:2};;
|
||||
_Y*) CENTURY=$(echo "${DATE:0:2}" | tr ' ' '0'); YEAR_IN_CENTURY=${DATE:2:2}; DATE=${DATE:4}; x=${x:2};;
|
||||
|
||||
y*) YEAR_IN_CENTURY=${DATE:0:2}; DATE=${DATE:2}; x=${x:1};;
|
||||
0y*) YEAR_IN_CENTURY=${DATE:0:2}; DATE=${DATE:2}; x=${x:2};;
|
||||
_y*) YEAR_IN_CENTURY=$(echo "${DATE:0:2}" | tr ' ' '0'); DATE=${DATE:2}; x=${x:2};;
|
||||
|
||||
C*) CENTURY=${DATE:0:2}; DATE=${DATE:2}; x=${x:1};;
|
||||
0C*) CENTURY=${DATE:0:2}; DATE=${DATE:2}; x=${x:2};;
|
||||
_C*) CENTURY=$(echo "${DATE:0:2}" | tr ' ' '0'); DATE=${DATE:2}; x=${x:2};;
|
||||
|
||||
m*) MONTH=${DATE:0:2}; DATE=${DATE:2}; x=${x:1};;
|
||||
0m*) MONTH=${DATE:0:2}; DATE=${DATE:2}; x=${x:2};;
|
||||
_m*) MONTH=$(echo "${DATE:0:2}" | tr ' ' '0'); DATE=${DATE:2}; x=${x:2};;
|
||||
e*) MONTH=$(echo "${DATE:0:2}" | tr ' ' '0'); DATE=${DATE:2}; x=${x:1};;
|
||||
0e*) MONTH=${DATE:0:2}; DATE=${DATE:2}; x=${x:2};;
|
||||
_e*) MONTH=$(echo "${DATE:0:2}" | tr ' ' '0'); DATE=${DATE:2}; x=${x:2};;
|
||||
b*) MONTH="$(date -d "${DATE:0:3} 1 2000" '+%m')"; DATE=${DATE:3}; x=${x:1};;
|
||||
h*) MONTH="$(date -d "${DATE:0:3} 1 2000" '+%m')"; DATE=${DATE:3}; x=${x:1};;
|
||||
|
||||
d*) DAY=${DATE:0:2}; DATE=${DATE:2}; x=${x:1};;
|
||||
0d*) DAY=${DATE:0:2}; DATE=${DATE:2}; x=${x:2};;
|
||||
_d*) DAY=$(echo "${DATE:0:2}" | tr ' ' '0'); DATE=${DATE:2}; x=${x:2};;
|
||||
|
||||
j*) DAY_OF_YEAR=${DATE:0:3}; DATE=${DATE:3}; x=${x:1};;
|
||||
0j*) DAY_OF_YEAR=${DATE:0:3}; DATE=${DATE:3}; x=${x:2};;
|
||||
_j*) DAY_OF_YEAR=$(echo "${DATE:0:3}" | tr ' ' '0'); DATE=${DATE:3}; x=${x:2};;
|
||||
|
||||
D*) MONTH=${DATE:0:2}; DAY=${DATE:3:2}; YEAR_IN_CENTURY=${DATE:6:2}; DATE=${DATE:8}; x=${x:1};;
|
||||
F*) CENTURY=${DATE:0:2}; YEAR_IN_CENTURY=${DATE:2:2}; MONTH=${DATE:5:2}; DAY=${DATE:8:2}; DATE=${DATE:10}; x=${x:1};;
|
||||
*) >&2 echo "$FUNCNAME (${LINENO}): Unsupported date format: ${BUILD_AVOIDANCE_DATE_FORMAT}"; return 1;;
|
||||
esac
|
||||
|
||||
# consume remaing non-interpreted content
|
||||
if [ "$(echo "${DATE:0:${#x}}" | tr ' ' '#')" != "${x}" ]; then
|
||||
>&2 echo "$FUNCNAME (${LINENO}): Unexpected content '${DATE:0:${#x}}' does not match expected '${x}': '$1' being parsed vs '${BUILD_AVOIDANCE_DATE_FORMAT}'"
|
||||
return 1
|
||||
fi
|
||||
DATE=${DATE:${#x}}
|
||||
done
|
||||
|
||||
if [ "${DAY_OF_YEAR}" != "" ]; then
|
||||
local YEAR_SEC
|
||||
local DOY_SEC
|
||||
YEAR_SEC="$(date -d "${CENTURY}${YEAR_IN_CENTURY}-01-01" '+%s')"
|
||||
DOY_SEC=$((YEAR_SEC+(DAY_OF_YEAR-1)*24*60*60))
|
||||
MONTH="$(date "@$DOY_SEC" "+%m")"
|
||||
DAY="$(date "@$DOY_SEC" "+%d")"
|
||||
fi
|
||||
|
||||
echo "${CENTURY}${YEAR_IN_CENTURY}-${MONTH}-${DAY}"
|
||||
return 0
|
||||
}
|
||||
|
||||
time_to_iso_8601 () {
|
||||
TIME="$1"
|
||||
local HOUR="00"
|
||||
local H12=""
|
||||
local AMPM=""
|
||||
local MINUTE="00"
|
||||
local SECOND="00"
|
||||
|
||||
CENTURY="$(date '+%C')"
|
||||
|
||||
for x in $(echo "${BUILD_AVOIDANCE_TIME_FORMAT}" | tr ' ' '#' | sed 's/%%/#/g' | tr '%' ' ' ); do
|
||||
# Consume format case options
|
||||
case ${x:0:1} in
|
||||
^) x=${x:1};;
|
||||
\#) x=${x:1};;
|
||||
*) ;;
|
||||
esac
|
||||
|
||||
# Process format
|
||||
case $x in
|
||||
H*) HOUR=${TIME:0:2}; TIME=${TIME:2}; x=${x:1};;
|
||||
0H*) HOUR=${TIME:0:2}; TIME=${TIME:2}; x=${x:2};;
|
||||
_H*) HOUR="$(echo "${TIME:0:2}" | tr ' ' '0')"; TIME=${TIME:2}; x=${x:2};;
|
||||
k*) HOUR="$(echo "${TIME:0:2}" | tr ' ' '0')"; TIME=${TIME:2}; x=${x:1};;
|
||||
0k*) HOUR=${TIME:0:2}; TIME=${TIME:2}; x=${x:2};;
|
||||
_k*) HOUR="$(echo "${TIME:0:2}" | tr ' ' '0')"; TIME=${TIME:2}; x=${x:2};;
|
||||
|
||||
I*) H12=${TIME:0:2}; TIME=${TIME:2}; x=${x:1};;
|
||||
0I*) H12=${TIME:0:2}; TIME=${TIME:2}; x=${x:2};;
|
||||
_I*) H12="$(echo "${TIME:0:2}" | tr ' ' '0')"; TIME=${TIME:2}; x=${x:2};;
|
||||
l*) H12="$(echo "${TIME:0:2}" | tr ' ' '0')"; TIME=${TIME:2}; x=${x:1};;
|
||||
0l*) H12=${TIME:0:2}; TIME=${TIME:2}; x=${x:2};;
|
||||
_l*) H12="$(echo "${TIME:0:2}" | tr ' ' '0')"; TIME=${TIME:2}; x=${x:2};;
|
||||
p*) AMPM=${TIME:0:2}; TIME=${TIME:2}; x=${x:1};;
|
||||
|
||||
M*) MINUTE=${TIME:0:2}; TIME=${TIME:2}; x=${x:1};;
|
||||
0M*) MINUTE=${TIME:0:2}; TIME=${TIME:2}; x=${x:2};;
|
||||
_M*) MINUTE="$(echo "${TIME:0:2}" | tr ' ' '0')"; TIME=${TIME:2}; x=${x:2};;
|
||||
|
||||
S*) SECOND=${TIME:0:2}; TIME=${TIME:2}; x=${x:1};;
|
||||
0S*) SECOND=${TIME:0:2}; TIME=${TIME:2}; x=${x:2};;
|
||||
_S*) SECOND="$(echo "${TIME:0:2}" | tr ' ' '0')"; TIME=${TIME:2}; x=${x:2};;
|
||||
|
||||
R*) HOUR=${TIME:0:2}; MINUTE=${TIME:3:2} TIME=${TIME:5}; x=${x:1};;
|
||||
r*) H12=${TIME:0:2}; MINUTE=${TIME:3:2}; SECOND=${TIME:6:2}; AMPM=${TIME:9:2}; TIME=${TIME:11}; x=${x:1};;
|
||||
T*) HOUR=${TIME:0:2}; MINUTE=${TIME:3:2}; SECOND=${TIME:6:2}; TIME=${TIME:8}; x=${x:1};;
|
||||
|
||||
*) >&2 echo "$FUNCNAME (${LINENO}): Unsupported time format: ${BUILD_AVOIDANCE_TIME_FORMAT}"; return 1;;
|
||||
esac
|
||||
|
||||
# consume remaing non-interpreted content
|
||||
if [ "$(echo "${TIME:0:${#x}}" | tr ' ' '#')" != "${x}" ]; then
|
||||
>&2 echo "$FUNCNAME (${LINENO}): Unexpected content '${TIME:0:${#x}}' does not match expected '${x}': '$1' being parsed vs '${BUILD_AVOIDANCE_TIME_FORMAT}'"
|
||||
return 1
|
||||
fi
|
||||
TIME=${TIME:${#x}}
|
||||
done
|
||||
|
||||
if [ "$H12" != "" ] && [ "$AMPM" != "" ]; then
|
||||
HOUR="$(date "$H12:01:01 $AMPM" '+%H')"
|
||||
else
|
||||
if [ "$H12" != "" ] && [ "$AMPM" != "" ]; then
|
||||
>&2 echo "$FUNCNAME (${LINENO}): Unsupported time format: ${BUILD_AVOIDANCE_TIME_FORMAT}"
|
||||
return 1
|
||||
fi
|
||||
fi
|
||||
|
||||
echo "${HOUR}:${MINUTE}:${SECOND}"
|
||||
return 0
|
||||
}
|
||||
|
||||
date_time_to_iso_8601 () {
|
||||
local DATE_TIME="$1"
|
||||
local DATE
|
||||
local TIME
|
||||
local DECODED_DATE
|
||||
local DECODED_TIME
|
||||
DATE=$(echo "${DATE_TIME}" | cut -d ${BUILD_AVOIDANCE_DATE_TIME_DELIM} -f 1)
|
||||
TIME=$(echo "${DATE_TIME}" | cut -d ${BUILD_AVOIDANCE_DATE_TIME_DELIM} -f 2 | sed "s#${BUILD_AVOIDANCE_DATE_TIME_POSTFIX}\$##")
|
||||
DECODED_DATE=$(date_to_iso_8601 "${DATE}")
|
||||
DECODED_TIME=$(time_to_iso_8601 "${TIME}")
|
||||
echo "${DECODED_DATE}T${DECODED_TIME}$(date $UTC '+%:z')"
|
||||
}
|
||||
|
||||
#
|
||||
# test_build_avoidance_context <path-to-context-file>
|
||||
#
|
||||
# Is the provided context file compatible with the current
|
||||
# state of all of our gits? A compatible context is one
|
||||
# where every commit in the context file is visible in our
|
||||
# current git history.
|
||||
#
|
||||
# Returns: Timestamp of context tested.
|
||||
# Exit code: 0 = Compatible
|
||||
# 1 = This context is older than the last applied
|
||||
# build avoidance context. If you are searching
|
||||
# newest to oldest, you might as well stop.
|
||||
# 2 = Not compatible
|
||||
#
|
||||
test_build_avoidance_context () {
|
||||
local context="$1"
|
||||
local BA_LAST_SYNC_CONTEXT="$2"
|
||||
local BA_CONTEXT=""
|
||||
|
||||
BA_CONTEXT=$(basename $context | cut -d '.' -f 1)
|
||||
>&2 echo "test: $BA_CONTEXT"
|
||||
|
||||
if [ "$BA_CONTEXT" == "$BA_LAST_SYNC_CONTEXT" ]; then
|
||||
# Stop the search. We've reached the last sync point
|
||||
BA_CONTEXT=""
|
||||
echo "$BA_CONTEXT"
|
||||
return 1
|
||||
fi
|
||||
|
||||
git_test_context "$context"
|
||||
result=$?
|
||||
if [ $result -eq 0 ]; then
|
||||
# found a new context !!!
|
||||
echo "$BA_CONTEXT"
|
||||
return 0
|
||||
fi
|
||||
|
||||
# Continue the search
|
||||
BA_CONTEXT=""
|
||||
echo "$BA_CONTEXT"
|
||||
return 2
|
||||
}
|
||||
|
||||
|
||||
#
|
||||
# get_build_avoidance_context
|
||||
#
|
||||
# Return URL of the most recent jenkins build that is compatable with
|
||||
# the current software context under $MY_REPO.
|
||||
#
|
||||
get_build_avoidance_context () {
|
||||
(
|
||||
local context
|
||||
local BA_CONTEXT=""
|
||||
local BA_LAST_SYNC_CONTEXT=""
|
||||
|
||||
# Load last synced context
|
||||
if [ -f $BUILD_AVOIDANCE_LAST_SYNC_FILE ]; then
|
||||
BA_LAST_SYNC_CONTEXT=$(head -n 1 $BUILD_AVOIDANCE_LAST_SYNC_FILE)
|
||||
fi
|
||||
|
||||
mkdir -p $BUILD_AVOIDANCE_DATA_DIR
|
||||
if [ $? -ne 0 ]; then
|
||||
>&2 echo "Error: $FUNCNAME (${LINENO}): mkdir -p $BUILD_AVOIDANCE_DATA_DIR"
|
||||
return 1
|
||||
fi
|
||||
|
||||
local REMOTE_CTX_DIR="context"
|
||||
local LOCAL_CTX_DIR="$BUILD_AVOIDANCE_DATA_DIR/context"
|
||||
|
||||
# First copy the directory containing all the context files for
|
||||
# the reference builds.
|
||||
>&2 echo "Download latest reference build contexts"
|
||||
|
||||
# Must set this prior to build_avoidance_copy_dir.
|
||||
# The setting is not exported outside of the subshell.
|
||||
BUILD_AVOIDANCE_URL="$BUILD_AVOIDANCE_HOST:$BUILD_AVOIDANCE_DIR"
|
||||
|
||||
build_avoidance_copy_dir "$REMOTE_CTX_DIR" "$LOCAL_CTX_DIR"
|
||||
if [ $? -ne 0 ]; then
|
||||
>&2 echo "Error: $FUNCNAME (${LINENO}): build_avoidance_copy_dir '$REMOTE_CTX_DIR' '$LOCAL_CTX_DIR'"
|
||||
return 1
|
||||
fi
|
||||
|
||||
# Search for a new context to sync
|
||||
cd $MY_REPO
|
||||
|
||||
if [ "$BUILD_AVOIDANCE_DAY" == "" ]; then
|
||||
# Normal case:
|
||||
# Search all contexts, newest to oldest, for a good context.
|
||||
for context in $(ls -1rd $LOCAL_CTX_DIR/*context); do
|
||||
>&2 echo "context=$context"
|
||||
BA_CONTEXT=$(test_build_avoidance_context $context $BA_LAST_SYNC_CONTEXT)
|
||||
if [ $? -le 1 ]; then
|
||||
# Stop search. Might or might not have found a good context.
|
||||
break;
|
||||
fi
|
||||
done
|
||||
else
|
||||
# Special case when a target day is specified. Why would we do this?
|
||||
# Reason is we might want the reference build to itself use build
|
||||
# avoidance referencing prior builds of itself, except for one build
|
||||
# a week when we use a full build rather than a build avoidance build.
|
||||
# e.g. Sunday - full build
|
||||
# Mon-Sat - avoidance builds that refernce Sunday build.
|
||||
#
|
||||
# Starting from last <TARG_DAY> (e.g. "Sunday"), search newest to
|
||||
# oldest for a good context. If none found, increment the target
|
||||
# day (e.g. Monday) and search again. Keep incrementing until a
|
||||
# good build is found, or target day + offset days would be a date
|
||||
# in the furure.
|
||||
#
|
||||
local TARG_DAY=$BUILD_AVOIDANCE_DAY
|
||||
local TODAY_DATE
|
||||
local TODAY_DAY
|
||||
local TARG_DATE=""
|
||||
local TARG_TS
|
||||
local TODAY_TS
|
||||
|
||||
TODAY_DATE=$(date $UTC +%Y-%m-%d)
|
||||
TODAY_DAY=$(date $UTC "+%A")
|
||||
|
||||
for OFFSET_DAYS in 0 1 2 3 4 5 6; do
|
||||
if [ "$TARG_DAY" != "" ]; then
|
||||
# Convert TARG_DAY+OFFSET_DAYS to TARG_DATE
|
||||
|
||||
if [ "$TODAY_DAY" == "$TARG_DAY" ]; then
|
||||
TARG_DATE=$(date $UTC -d"$TARG_DAY+$OFFSET_DAYS days" +%Y-%m-%d)
|
||||
else
|
||||
TARG_DATE=$(date $UTC -d"last-$TARG_DAY+$OFFSET_DAYS days" +%Y-%m-%d)
|
||||
fi
|
||||
>&2 echo "TARG_DATE=$TARG_DATE"
|
||||
|
||||
TARG_TS=$(date $UTC -d "$TARG_DATE" +%s)
|
||||
TODAY_TS=$(date $UTC -d "$TODAY_DATE" +%s)
|
||||
if [ $TARG_TS -gt $TODAY_TS ]; then
|
||||
# Skip if offset has pushed us into future dates
|
||||
continue;
|
||||
fi
|
||||
|
||||
if [ "$TARG_DATE" == "$TODAY_DATE" ]; then
|
||||
TARG_DATE=""
|
||||
fi
|
||||
fi
|
||||
|
||||
# Search build, newest to oldest, satisfying TARG_DATE
|
||||
for f in $(ls -1rd $LOCAL_CTX_DIR/*context); do
|
||||
DATE=$(date_to_iso_8601 $(basename "$f"))
|
||||
if [ $? -ne 0 ]; then
|
||||
>&2 echo "Failed to extract date from filename '$(basename "$f")', ignoring file"
|
||||
continue
|
||||
fi
|
||||
|
||||
>&2 echo " DATE=$DATE, TARG_DATE=$TARG_DATE"
|
||||
|
||||
if [ "$DATE" == "$TARG_DATE" ] || [ "$TARG_DATE" == "" ] ; then
|
||||
context=$f;
|
||||
else
|
||||
continue
|
||||
fi
|
||||
|
||||
>&2 echo "context=$context"
|
||||
|
||||
BA_CONTEXT=$(test_build_avoidance_context $context $BA_LAST_SYNC_CONTEXT)
|
||||
|
||||
if [ $? -le 1 ]; then
|
||||
# Stop search. Might or might not have found a good context.
|
||||
break;
|
||||
fi
|
||||
done
|
||||
|
||||
if [ "$BA_CONTEXT" != "" ]; then
|
||||
# Found a good context.
|
||||
break
|
||||
fi
|
||||
done
|
||||
fi
|
||||
|
||||
if [ "$BA_CONTEXT" == "" ]; then
|
||||
# No new context found
|
||||
return 1
|
||||
fi
|
||||
|
||||
# test that the reference build context hasn't been deleted
|
||||
local BA_CONTEXT_DIR="$BUILD_AVOIDANCE_DIR/$BA_CONTEXT"
|
||||
>&2 echo "ssh $BUILD_AVOIDANCE_HOST '[ -d $BA_CONTEXT_DIR ]'"
|
||||
if ! ssh $BUILD_AVOIDANCE_HOST '[ -d $BA_CONTEXT_DIR ]' ; then
|
||||
return 1
|
||||
fi
|
||||
|
||||
# Save the latest context
|
||||
>&2 echo "BA_CONTEXT=$BA_CONTEXT"
|
||||
>&2 echo "BUILD_AVOIDANCE_LAST_SYNC_FILE=$BUILD_AVOIDANCE_LAST_SYNC_FILE"
|
||||
echo $BA_CONTEXT > $BUILD_AVOIDANCE_LAST_SYNC_FILE
|
||||
|
||||
# The location of the load with the most compatable new context
|
||||
URL=$BUILD_AVOIDANCE_HOST:$BA_CONTEXT_DIR
|
||||
|
||||
# return URL to caller.
|
||||
echo $URL
|
||||
return 0
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
#
|
||||
# build_avoidance_pre_clean <build-type>
|
||||
#
|
||||
# A place for any cleanup actions that must preceed a build avoidance build.
|
||||
#
|
||||
build_avoidance_pre_clean () {
|
||||
local BUILD_TYPE="$1"
|
||||
|
||||
if [ "$BUILD_TYPE" == "" ]; then
|
||||
>&2 echo "Error: $FUNCNAME (${LINENO}): BUILD_TYPE required"
|
||||
return 1
|
||||
fi
|
||||
|
||||
# clean prior builds
|
||||
if [ -d $MY_WORKSPACE/$BUILD_TYPE ]; then
|
||||
build-pkgs --clean --$BUILD_TYPE --no-build-avoidance
|
||||
if [ $? -ne 0 ]; then
|
||||
return 1
|
||||
fi
|
||||
fi
|
||||
|
||||
for f in $BUILD_AVOIDANCE_SRPM_FILES $BUILD_AVOIDANCE_RPM_FILES; do
|
||||
if [ -f $MY_WORKSPACE/$BUILD_TYPE/$f ]; then
|
||||
\rm -f $MY_WORKSPACE/$BUILD_TYPE/$f
|
||||
if [ $? -ne 0 ]; then
|
||||
>&2 echo "Error: $FUNCNAME (${LINENO}): rm -f $MY_WORKSPACE/$BUILD_TYPE/$f"
|
||||
return 1
|
||||
fi
|
||||
fi
|
||||
done
|
||||
|
||||
for d in $BUILD_AVOIDANCE_SRPM_DIRECTORIES $BUILD_AVOIDANCE_RPM_DIRECTORIES; do
|
||||
|
||||
if [ -d $MY_WORKSPACE/$BUILD_TYPE/$d ]; then
|
||||
\rm -rf $MY_WORKSPACE/$BUILD_TYPE/$d
|
||||
if [ $? -ne 0 ]; then
|
||||
>&2 echo "Error: $FUNCNAME (${LINENO}): rm -rf $MY_WORKSPACE/$BUILD_TYPE/$d"
|
||||
return 1
|
||||
fi
|
||||
fi
|
||||
done
|
||||
|
||||
return 0
|
||||
}
|
||||
|
||||
|
||||
#
|
||||
# build_avoidance_copy_dir_rsync <remote-dir-path-rel> <local-dir-path> ['verbose']
|
||||
#
|
||||
# Copy a file from $BUILD_AVOIDANCE_URL/<remote-dir-path-rel>
|
||||
# to <local-dir-path> using rsync.
|
||||
#
|
||||
build_avoidance_copy_dir_rsync () {
|
||||
local FROM="$1"
|
||||
local TO="$2"
|
||||
local VERBOSE="$3"
|
||||
local FLAGS="-a -u"
|
||||
|
||||
if [ "$BUILD_AVOIDANCE_URL" == "" ]; then
|
||||
>&2 echo "Error: $FUNCNAME (${LINENO}): BUILD_AVOIDANCE_URL no set"
|
||||
return 1
|
||||
fi
|
||||
if [ "$VERBOSE" != "" ]; then
|
||||
FLAGS="$FLAGS -v"
|
||||
echo "rsync $FLAGS '$BUILD_AVOIDANCE_URL/$FROM/*' '$TO/'"
|
||||
fi
|
||||
rsync $FLAGS "$BUILD_AVOIDANCE_URL/$FROM/*" "$TO/"
|
||||
return $?
|
||||
}
|
||||
|
||||
#
|
||||
# build_avoidance_copy_file_rsync <remote-file-path-rel> <local-file-path> ['verbose']
|
||||
#
|
||||
# Copy a file from $BUILD_AVOIDANCE_URL/<remote-file-path-rel>
|
||||
# to <local-file-path> using rsync.
|
||||
#
|
||||
build_avoidance_copy_file_rsync () {
|
||||
local FROM="$1"
|
||||
local TO="$2"
|
||||
local VERBOSE="$3"
|
||||
local FLAGS="-a -u"
|
||||
|
||||
if [ "$BUILD_AVOIDANCE_URL" == "" ]; then
|
||||
>&2 echo "Error: $FUNCNAME (${LINENO}): BUILD_AVOIDANCE_URL no set"
|
||||
return 1
|
||||
fi
|
||||
if [ "$VERBOSE" != "" ]; then
|
||||
FLAGS="$FLAGS -v"
|
||||
echo "rsync $FLAGS '$BUILD_AVOIDANCE_URL/$FROM' '$TO'"
|
||||
fi
|
||||
rsync $FLAGS "$BUILD_AVOIDANCE_URL/$FROM" "$TO"
|
||||
return $?
|
||||
}
|
||||
|
||||
#
|
||||
# build_avoidance_copy_dir <remote-dir-path-rel> <local-dir-path> ['verbose']
|
||||
#
|
||||
# Copy a file from $BUILD_AVOIDANCE_URL/<remote-dir-path-rel>
|
||||
# to <local-dir-path>. The copy method will be determined by
|
||||
# BUILD_AVOIDANCE_FILE_TRANSFER. Only 'rsync' is supported at present.
|
||||
#
|
||||
# <local-dir-path> should be a directory,
|
||||
# mkdir -p will be called on <local-file-path>.
|
||||
#
|
||||
build_avoidance_copy_dir () {
|
||||
local FROM="$1"
|
||||
local TO="$2"
|
||||
local VERBOSE="$3"
|
||||
|
||||
if [ "$VERBOSE" != "" ]; then
|
||||
echo "mkdir -p '$TO'"
|
||||
fi
|
||||
mkdir -p "$TO"
|
||||
if [ $? -ne 0 ]; then
|
||||
>&2 echo "Error: $FUNCNAME (${LINENO}): mkdir -p $TO"
|
||||
return 1
|
||||
fi
|
||||
|
||||
case ${BUILD_AVOIDANCE_FILE_TRANSFER} in
|
||||
rsync)
|
||||
build_avoidance_copy_dir_rsync "$FROM" "$TO" "$VERBOSE"
|
||||
return $?
|
||||
;;
|
||||
*)
|
||||
>&2 echo "Error: $FUNCNAME (${LINENO}): Unknown BUILD_AVOIDANCE_FILE_TRANSFER '${BUILD_AVOIDANCE_FILE_TRANSFER}'"
|
||||
return 1
|
||||
;;
|
||||
esac
|
||||
return 1
|
||||
}
|
||||
|
||||
#
|
||||
# build_avoidance_copy_file <remote-file-path-rel> <local-file-path> ['verbose']
|
||||
#
|
||||
# Copy a file from $BUILD_AVOIDANCE_URL/<remote-file-path-rel>
|
||||
# to <local-file-path>. The copy method will be determined by
|
||||
# BUILD_AVOIDANCE_FILE_TRANSFER. Only 'rsync' is supported at present.
|
||||
#
|
||||
# <local-file-path> should be a file, not a directory,
|
||||
# mkdir -p will be called on $(dirname <local-file-path>)
|
||||
#
|
||||
build_avoidance_copy_file () {
|
||||
local FROM="$1"
|
||||
local TO="$2"
|
||||
local VERBOSE="$3"
|
||||
|
||||
if [ "$VERBOSE" != "" ]; then
|
||||
echo "mkdir -p $(dirname '$TO')"
|
||||
fi
|
||||
mkdir -p "$(dirname "$TO")"
|
||||
if [ $? -ne 0 ]; then
|
||||
>&2 echo "Error: $FUNCNAME (${LINENO}): mkdir -p $(dirname "$TO")"
|
||||
return 1
|
||||
fi
|
||||
|
||||
case ${BUILD_AVOIDANCE_FILE_TRANSFER} in
|
||||
rsync)
|
||||
build_avoidance_copy_file_rsync "$FROM" "$TO" "$VERBOSE"
|
||||
return $?
|
||||
;;
|
||||
*)
|
||||
>&2 echo "Error: $FUNCNAME (${LINENO}): Unknown BUILD_AVOIDANCE_FILE_TRANSFER '${BUILD_AVOIDANCE_FILE_TRANSFER}'"
|
||||
return 1
|
||||
;;
|
||||
esac
|
||||
return 1
|
||||
}
|
||||
|
||||
#
|
||||
# build_avoidance_copy <build-type> ['verbose']
|
||||
#
|
||||
# Copy the needed build artifacts for <build-type> from $BUILD_AVOIDANCE_URL.
|
||||
#
|
||||
build_avoidance_copy () {
|
||||
local BUILD_TYPE="$1"
|
||||
local VERBOSE="$2"
|
||||
|
||||
if [ "$BUILD_TYPE" == "" ]; then
|
||||
>&2 echo "Error: $FUNCNAME (${LINENO}): BUILD_TYPE required"
|
||||
return 1
|
||||
fi
|
||||
|
||||
# Iterate through list of directories to copy
|
||||
for d in $BUILD_AVOIDANCE_SRPM_DIRECTORIES $BUILD_AVOIDANCE_RPM_DIRECTORIES; do
|
||||
build_avoidance_copy_dir "$BUILD_TYPE/$d" "$MY_WORKSPACE/$BUILD_TYPE/$d" "$VERBOSE"
|
||||
if [ $? -ne 0 ]; then
|
||||
>&2 echo "Error: $FUNCNAME (${LINENO}): build_avoidance_copy_dir '$BUILD_TYPE/$d' '$MY_WORKSPACE/$BUILD_TYPE/$d'"
|
||||
return 1
|
||||
fi
|
||||
done
|
||||
|
||||
# Iterate through list of files to copy
|
||||
for f in $BUILD_AVOIDANCE_SRPM_FILES $BUILD_AVOIDANCE_RPM_FILES; do
|
||||
build_avoidance_copy_file "$BUILD_TYPE/$f" "$MY_WORKSPACE/$BUILD_TYPE/$f" "$VERBOSE"
|
||||
if [ $? -ne 0 ]; then
|
||||
>&2 echo "Error: $FUNCNAME (${LINENO}): build_avoidance_copy_file '$BUILD_TYPE/$f' '$MY_WORKSPACE/$BUILD_TYPE/$f'"
|
||||
return 1
|
||||
fi
|
||||
done
|
||||
|
||||
return 0
|
||||
}
|
||||
|
||||
#
|
||||
# build_avoidance_fixups <build-type>
|
||||
#
|
||||
# Fix paths in the build artifacts that we coppied that contain
|
||||
# the user name.
|
||||
#
|
||||
# Also, our credentials may differ from the reference build,
|
||||
# so substitute unsigned packages in place of signed packages.
|
||||
#
|
||||
build_avoidance_fixups () {
|
||||
local BUILD_TYPE="$1"
|
||||
|
||||
local BA_SOURCE_BUILD_ENVIRONMENT
|
||||
BA_SOURCE_BUILD_ENVIRONMENT="${BUILD_AVOIDANCE_USR}-$(basename $(dirname $BUILD_AVOIDANCE_URL))-$(basename $BUILD_AVOIDANCE_URL)-${SRC_BUILD_ENVIRONMENT}"
|
||||
local RESULT_DIR=""
|
||||
local FROM_DIR=""
|
||||
local TO_DIR=""
|
||||
local rpm_path_post_signing
|
||||
local rpm_path_pre_signing
|
||||
local rpm_name
|
||||
local md5sum_post_signing
|
||||
local md5sum_pre_signing
|
||||
|
||||
if [ "$BUILD_TYPE" == "" ]; then
|
||||
>&2 echo "Error: $FUNCNAME (${LINENO}): BUILD_TYPE required"
|
||||
return 1
|
||||
fi
|
||||
|
||||
RESULT_DIR="$MY_WORKSPACE/$BUILD_TYPE/results"
|
||||
FROM_DIR="${RESULT_DIR}/${BA_SOURCE_BUILD_ENVIRONMENT}-${BUILD_TYPE}"
|
||||
TO_DIR="${RESULT_DIR}/${MY_BUILD_ENVIRONMENT}-${BUILD_TYPE}"
|
||||
echo "$FUNCNAME: FROM_DIR=$FROM_DIR"
|
||||
echo "$FUNCNAME: TO_DIR=$TO_DIR"
|
||||
echo "$FUNCNAME: MY_BUILD_ENVIRONMENT=$MY_BUILD_ENVIRONMENT"
|
||||
|
||||
# Fix patchs the use MY_BUILD_ENVIRONMENT
|
||||
if [ ! -d "$FROM_DIR" ]; then
|
||||
>&2 echo "Error: $FUNCNAME (${LINENO}): Expected directory '$FROM_DIR' is missing."
|
||||
return 1
|
||||
fi
|
||||
|
||||
echo "$FUNCNAME: mv '$FROM_DIR' '$TO_DIR'"
|
||||
\mv "$FROM_DIR" "$TO_DIR"
|
||||
if [ $? -ne 0 ]; then
|
||||
>&2 echo "Error: $FUNCNAME (${LINENO}): mv '$FROM_DIR' '$TO_DIR'"
|
||||
return 1
|
||||
fi
|
||||
|
||||
local MY_WS_BT="$MY_WORKSPACE/$BUILD_TYPE"
|
||||
|
||||
# Replace signed rpms with non-signed copies .... we aren't a formal build
|
||||
for rpm_path_post_signing in $(find $MY_WS_BT/rpmbuild/RPMS -type f -name '*.rpm' | grep -v src.rpm); do
|
||||
|
||||
rpm_name=$(basename $rpm_path_post_signing)
|
||||
rpm_path_pre_signing=$(find $MY_WS_BT/results -name $b | head -n1)
|
||||
if [ "$rpm_path_pre_signing" != "" ]; then
|
||||
md5sum_post_signing=$(md5sum ${rpm_path_post_signing} | cut -d ' ' -f 1)
|
||||
md5sum_pre_signing=$(md5sum ${rpm_path_pre_signing} | cut -d ' ' -f 1)
|
||||
if [ "${md5sum_post_signing}" != "${md5sum_pre_signing}" ]; then
|
||||
echo "$FUNCNAME: fixing $rpm_name"
|
||||
\rm -f ${rpm_path_post_signing}
|
||||
if [ $? -ne 0 ]; then
|
||||
>&2 echo "Error: $FUNCNAME (${LINENO}): rm -f ${rpm_path_post_signing}"
|
||||
return 1
|
||||
fi
|
||||
|
||||
\cp ${rpm_path_pre_signing} ${rpm_path_post_signing}
|
||||
if [ $? -ne 0 ]; then
|
||||
>&2 echo "Error: $FUNCNAME (${LINENO}): cp ${rpm_path_pre_signing} ${rpm_path_post_signing}"
|
||||
return 1
|
||||
fi
|
||||
fi
|
||||
fi;
|
||||
done
|
||||
|
||||
return 0
|
||||
}
|
||||
|
||||
|
||||
#
|
||||
# build_avoidance <build-type>
|
||||
#
|
||||
# Look for a reference build that is applicable to our current git context.
|
||||
# and copy it to our local workspace, if we haven't already done so.
|
||||
#
|
||||
build_avoidance () {
|
||||
local BUILD_TYPE="$1"
|
||||
|
||||
echo "==== Build Avoidance Start ===="
|
||||
|
||||
if [ "$BUILD_TYPE" == "" ]; then
|
||||
>&2 echo "Error: $FUNCNAME (${LINENO}): BUILD_TYPE required"
|
||||
return 1
|
||||
fi
|
||||
|
||||
if [ "$BUILD_TYPE" == "installer" ]; then
|
||||
>&2 echo "build_avoidance: BUILD_TYPE==installer not supported"
|
||||
return 1
|
||||
fi
|
||||
|
||||
build_avoidance_pre_clean $BUILD_TYPE
|
||||
if [ $? -ne 0 ]; then
|
||||
>&2 echo "Error: $FUNCNAME (${LINENO}): build_avoidance_pre_clean $BUILD_TYPE"
|
||||
return 1
|
||||
fi
|
||||
|
||||
build_avoidance_copy $BUILD_TYPE 'verbose'
|
||||
if [ $? -ne 0 ]; then
|
||||
>&2 echo "Error: $FUNCNAME (${LINENO}): build_avoidance_copy $BUILD_TYPE"
|
||||
return 1
|
||||
fi
|
||||
|
||||
build_avoidance_fixups $BUILD_TYPE
|
||||
if [ $? -ne 0 ]; then
|
||||
>&2 echo "Error: $FUNCNAME (${LINENO}): build_avoidance_fixups $BUILD_TYPE"
|
||||
return 1
|
||||
fi
|
||||
|
||||
echo "==== Build Avoidance Complete ===="
|
||||
return 0
|
||||
}
|
||||
|
||||
#
|
||||
# build_avoidance_save_reference_context
|
||||
#
|
||||
# For use by a reference build. Copy the 'CONTEXT' file
|
||||
# from the build into a central directory where we save
|
||||
# the context of old builds.
|
||||
#
|
||||
# Individual reference builds use:
|
||||
# MY_WORKSPACE=<common-dir>/<timestamp>
|
||||
# and context files are collected in dir:
|
||||
# DEST_CTX_DIR=<common-dir>/context
|
||||
# using name:
|
||||
# DEST_CTX=<timestamp>.context
|
||||
|
||||
build_avoidance_save_reference_context () {
|
||||
local DIR
|
||||
DIR=$(dirname "${MY_WORKSPACE}")
|
||||
|
||||
# Note: SUB_DIR should be a timestamp
|
||||
local SUB_DIR
|
||||
SUB_DIR=$(basename "${MY_WORKSPACE}")
|
||||
|
||||
local SRC_CTX="${MY_WORKSPACE}/CONTEXT"
|
||||
local DEST_CTX_DIR="${DIR}/context"
|
||||
local DEST_CTX="${DEST_CTX_DIR}/${SUB_DIR}.context"
|
||||
|
||||
if [ ! -f "${SRC_CTX}" ]; then
|
||||
echo "Context file not found at '${SRC_CTX}'"
|
||||
return 1
|
||||
fi
|
||||
|
||||
mkdir -p "${DEST_CTX_DIR}"
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "Error: $FUNCNAME (${LINENO}): Failed to create directory '${DEST_CTX_DIR}'"
|
||||
return 1
|
||||
fi
|
||||
|
||||
cp "${SRC_CTX}" "${DEST_CTX}"
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "Error: $FUNCNAME (${LINENO}): Failed to copy ${SRC_CTX} -> ${DEST_CTX}"
|
||||
return 1
|
||||
fi
|
||||
|
||||
return 0
|
||||
}
|
@ -6,6 +6,9 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
#
|
||||
|
||||
#
|
||||
# Build first src.rpms, then rpms, from source, or from a downloaded tarball
|
||||
# or src.rpm plus our additional patches.
|
||||
#
|
||||
# This program is a wrapper around build-pkgs-parallel and build-pkgs-serial
|
||||
#
|
||||
|
@ -1,10 +1,22 @@
|
||||
#!/bin/bash
|
||||
|
||||
#
|
||||
# Copyright (c) 2018 Wind River Systems, Inc.
|
||||
#
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
#
|
||||
|
||||
#
|
||||
# Build first src.rpms, then rpms, from source, or from a downloaded tarball
|
||||
# or src.rpm plus our additional patches.
|
||||
#
|
||||
# This program is a wrapper around build-srpms-parallel and build-rpms-parallel
|
||||
#
|
||||
|
||||
BUILD_PKGS_PARALLEL_DIR="$(dirname "$(readlink -f "${BASH_SOURCE[0]}" )" )"
|
||||
|
||||
source "${BUILD_PKGS_PARALLEL_DIR}/git-utils.sh"
|
||||
source "${BUILD_PKGS_PARALLEL_DIR}/build-avoidance-utils.sh"
|
||||
source "${BUILD_PKGS_PARALLEL_DIR}/spec-utils"
|
||||
|
||||
|
||||
@ -13,14 +25,15 @@ usage () {
|
||||
echo "Usage: "
|
||||
echo " Create source and Binary rpms:"
|
||||
echo " Build optimizations (--no-descendants, --no-required, --no-build-info,"
|
||||
echo " --no-autoclean) are not recommended for the first build after a clone/pull,"
|
||||
echo " nor the final build prior to creating an iso or patch, but can be used"
|
||||
echo " for intermediate builds. i.e. while debugging compilation failures."
|
||||
echo " build-pkgs-parallel [--no-descendants] [--no-required] [--no-build-info] [--no-autoclean] [--careful] [--formal] [ list of package names ]"
|
||||
echo " --no-autoclean, --no-build-avoidance) are not recommended for the first build"
|
||||
echo " after a clone/pull, nor the final build prior to creating an iso or patch,"
|
||||
echo " but can be used for intermediate builds."
|
||||
echo " i.e. while debugging compilation failures."
|
||||
echo " build-pkgs-parallel [--build-avoidance | --no-build-avoidance] [--no-descendants] [--no-required] [--no-build-info] [--no-autoclean] [--careful] [--formal] [ list of package names ]"
|
||||
echo ""
|
||||
echo " Delete source rpms, and the directories associated with it's creation:"
|
||||
echo " Note: does not clean an edit environment"
|
||||
echo " build-pkgs-parallel --clean [ list of package names ]"
|
||||
echo " build-pkgs-parallel --clean [--build-avoidance | --no-build-avoidance] [ list of package names ]"
|
||||
echo ""
|
||||
echo " Extract an src.rpm into a pair of git trees to aid in editing it's contents,"
|
||||
echo " one for source code and one for metadata such as the spec file."
|
||||
@ -39,15 +52,20 @@ usage () {
|
||||
HELP=0
|
||||
CLEAN_FLAG=0
|
||||
EDIT_FLAG=0
|
||||
APPEND_LOG_FLAG=0
|
||||
BUILD_AVOIDANCE_FLAG=0
|
||||
STD_BUILD=1
|
||||
RT_BUILD=1
|
||||
INSTALLER_BUILD=0
|
||||
|
||||
export BUILD_AVOIDANCE_URL=""
|
||||
|
||||
|
||||
# read the options
|
||||
TEMP=$(getopt -o h --long parallel,rt,std,installer,edit,no-meta-patch,no-descendants,no-required,no-build-info,no-autoclean,formal,careful,help,clean -n 'build-pkgs-parallel' -- "$@")
|
||||
TEMP=$(getopt -o h --long parallel,rt,std,installer,edit,build-avoidance,no-build-avoidance,build-avoidance-dir:,build-avoidance-host:,build-avoidance-user:,build-avoidance-day:,no-meta-patch,no-descendants,no-required,no-build-info,no-autoclean,formal,careful,help,clean,append-log -n 'build-pkgs-parallel' -- "$@")
|
||||
if [ $? -ne 0 ]; then
|
||||
usage
|
||||
exit 0
|
||||
exit 1
|
||||
fi
|
||||
eval set -- "$TEMP"
|
||||
|
||||
@ -55,8 +73,21 @@ eval set -- "$TEMP"
|
||||
EXTRA_ARGS_COMMON=""
|
||||
EXTRA_ARGS_SRPM=""
|
||||
EXTRA_ARGS_RPM=""
|
||||
|
||||
export BUILD_AVOIDANCE_OVERRIDE_DIR=""
|
||||
export BUILD_AVOIDANCE_OVERRIDE_HOST=""
|
||||
export BUILD_AVOIDANCE_OVERRIDE_USR=""
|
||||
export BUILD_AVOIDANCE_DAY=""
|
||||
|
||||
while true ; do
|
||||
case "$1" in
|
||||
--append-log) APPEND_LOG_FLAG=1 ; shift ;;
|
||||
--build-avoidance) BUILD_AVOIDANCE_FLAG=1 ; shift ;;
|
||||
--no-build-avoidance) BUILD_AVOIDANCE_FLAG=0 ; shift ;;
|
||||
--build-avoidance-dir) BUILD_AVOIDANCE_OVERRIDE_DIR=$2; shift 2 ;;
|
||||
--build-avoidance-host) BUILD_AVOIDANCE_OVERRIDE_HOST=$2; shift 2 ;;
|
||||
--build-avoidance-user) BUILD_AVOIDANCE_OVERRIDE_USR=$2; shift 2 ;;
|
||||
--build-avoidance-day) BUILD_AVOIDANCE_DAY=$2; shift 2 ;;
|
||||
--no-descendants) EXTRA_ARGS_COMMON+=" --no-descendants" ; shift ;;
|
||||
--formal) EXTRA_ARGS_COMMON+=" --formal" ; shift ;;
|
||||
--careful) EXTRA_ARGS_RPM+=" --careful" ; shift ;;
|
||||
@ -125,11 +156,15 @@ trap my_sighup HUP
|
||||
trap my_sigabrt ABRT
|
||||
trap my_sigterm TERM
|
||||
|
||||
# Note: For ease of parsing, a TARGETS list always begins and ends
|
||||
# with a space. An empty target list consistes of two spaces.
|
||||
TARGETS=" $@ "
|
||||
TARGETS_STD=" "
|
||||
TARGETS_RT=" "
|
||||
TARGETS_INSTALLER=" "
|
||||
TARGETS_MISC=" "
|
||||
EMPTY_TARGETS=" "
|
||||
|
||||
TARGETS_STD="$EMPTY_TARGETS"
|
||||
TARGETS_RT="$EMPTY_TARGETS"
|
||||
TARGETS_INSTALLER="$EMPTY_TARGETS"
|
||||
TARGETS_MISC="$EMPTY_TARGETS"
|
||||
|
||||
find_targets () {
|
||||
local centos_pkg_dirs=$1
|
||||
@ -199,14 +234,28 @@ find_targets () {
|
||||
fi
|
||||
done
|
||||
|
||||
echo "$RESULT"
|
||||
if [ "$RESULT" == " " ]; then
|
||||
echo "$EMPTY_LIST"
|
||||
else
|
||||
echo "$RESULT"
|
||||
fi
|
||||
return 0
|
||||
}
|
||||
|
||||
if [ $EDIT_FLAG -eq 1 ] || [ "$TARGETS" != "$EMPTY_TARGETS" ]; then
|
||||
BUILD_AVOIDANCE_FLAG=0
|
||||
fi
|
||||
|
||||
echo "BUILD_AVOIDANCE_FLAG=$BUILD_AVOIDANCE_FLAG"
|
||||
echo "CLEAN_FLAG=$CLEAN_FLAG"
|
||||
echo "EDIT_FLAG=$EDIT_FLAG"
|
||||
|
||||
if [ "x$TARGETS" != "x " ]; then
|
||||
if [ $BUILD_AVOIDANCE_FLAG -eq 1 ]; then
|
||||
# Build Avoidance requested. Get URL of a usable context, if any.
|
||||
export BUILD_AVOIDANCE_URL=$(get_build_avoidance_context)
|
||||
fi
|
||||
|
||||
if [ "$TARGETS" != "$EMPTY_TARGETS" ]; then
|
||||
TARGETS_STD="$(find_targets centos_pkg_dirs)"
|
||||
|
||||
BUILD_TYPE_SAVE="$BUILD_TYPE"
|
||||
@ -247,44 +296,50 @@ echo "EXTRA_ARGS_RPM='$EXTRA_ARGS_RPM'"
|
||||
echo "TARGETS='$TARGETS'"
|
||||
echo "TARGETS_STD='$TARGETS_STD'"
|
||||
echo "TARGETS_RT='$TARGETS_RT'"
|
||||
echo "TARGETS_INSTALLER='$TARGETS_INSTALLER'"
|
||||
echo "TARGETS_MISC='$TARGETS_MISC'"
|
||||
|
||||
if [ $CLEAN_FLAG -eq 1 ]; then
|
||||
|
||||
if [ "$TARGETS" == "$EMPTY_TARGETS" ] && [ $BUILD_AVOIDANCE_FLAG -eq 1 ] ; then
|
||||
build_avoidance_clean
|
||||
fi
|
||||
|
||||
if [ $STD_BUILD -eq 1 ]; then
|
||||
if [ "x$TARGETS" == "x " ] || [ "$TARGETS_STD" != " " ] || [ "$TARGETS_MISC" != " " ]; then
|
||||
if [ "$TARGETS" == "$EMPTY_TARGETS" ] || [ "$TARGETS_STD" != "$EMPTY_TARGETS" ] || [ "$TARGETS_MISC" != "$EMPTY_TARGETS" ]; then
|
||||
if [ $EDIT_FLAG -ne 1 ]; then
|
||||
echo "${BUILD_PKGS_PARALLEL_DIR}/build-rpms-parallel --std --clean $EXTRA_ARGS_COMMON $EXTRA_ARGS_RPM $TARGETS_STD $TARGETS_MISC"
|
||||
${BUILD_PKGS_PARALLEL_DIR}/build-rpms-parallel --std --clean $EXTRA_ARGS_COMMON $EXTRA_ARGS_RPM $TARGETS_STD $TARGETS_MISC || exit 1
|
||||
|
||||
fi
|
||||
fi
|
||||
if [ "x$TARGETS" == "x " ] || [ "$TARGETS_STD" != " " ] || [ "$TARGETS_MISC" != " " ]; then
|
||||
if [ "$TARGETS" == "$EMPTY_TARGETS" ] || [ "$TARGETS_STD" != "$EMPTY_TARGETS" ] || [ "$TARGETS_MISC" != "$EMPTY_TARGETS" ]; then
|
||||
echo "${BUILD_PKGS_PARALLEL_DIR}/build-srpms-parallel --std --clean $EXTRA_ARGS_COMMON $EXTRA_ARGS_SRPM $TARGETS_STD $TARGETS_MISC"
|
||||
${BUILD_PKGS_PARALLEL_DIR}/build-srpms-parallel --std --clean $EXTRA_ARGS_COMMON $EXTRA_ARGS_SRPM $TARGETS_STD $TARGETS_MISC || exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
if [ $RT_BUILD -eq 1 ]; then
|
||||
if [ "x$TARGETS" == "x " ] || [ "$TARGETS_RT" != " " ] || [ "$TARGETS_MISC" != " " ]; then
|
||||
if [ "$TARGETS" == "$EMPTY_TARGETS" ] || [ "$TARGETS_RT" != "$EMPTY_TARGETS" ] || [ "$TARGETS_MISC" != "$EMPTY_TARGETS" ]; then
|
||||
if [ $EDIT_FLAG -ne 1 ]; then
|
||||
echo "${BUILD_PKGS_PARALLEL_DIR}/build-rpms-parallel --rt --clean $EXTRA_ARGS_COMMON $EXTRA_ARGS_RPM $TARGETS_RT $TARGETS_MISC"
|
||||
${BUILD_PKGS_PARALLEL_DIR}/build-rpms-parallel --rt --clean $EXTRA_ARGS_COMMON $EXTRA_ARGS_RPM $TARGETS_RT $TARGETS_MISC || exit 1
|
||||
fi
|
||||
fi
|
||||
if [ "x$TARGETS" == "x " ] || [ "$TARGETS_RT" != " " ] || [ "$TARGETS_MISC" != " " ]; then
|
||||
if [ "$TARGETS" == "$EMPTY_TARGETS" ] || [ "$TARGETS_RT" != "$EMPTY_TARGETS" ] || [ "$TARGETS_MISC" != "$EMPTY_TARGETS" ]; then
|
||||
echo "${BUILD_PKGS_PARALLEL_DIR}/build-srpms-parallel --rt --clean $EXTRA_ARGS_COMMON $EXTRA_ARGS_SRPM $TARGETS_RT $TARGETS_MISC"
|
||||
${BUILD_PKGS_PARALLEL_DIR}/build-srpms-parallel --rt --clean $EXTRA_ARGS_COMMON $EXTRA_ARGS_SRPM $TARGETS_RT $TARGETS_MISC || exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
if [ $INSTALLER_BUILD -eq 1 ]; then
|
||||
if [ "x$TARGETS" == "x" ] || [ "$TARGETS_INSTALLER" != " " ]; then
|
||||
if [ "$TARGETS" == "$EMPTY_TARGETS" ] || [ "$TARGETS_INSTALLER" != "$EMPTY_TARGETS" ]; then
|
||||
if [ $EDIT_FLAG -ne 1 ]; then
|
||||
echo "${BUILD_PKGS_PARALLEL_DIR}/build-rpms-serial --installer --clean $EXTRA_ARGS_COMMON $EXTRA_ARGS_RPM $TARGETS_INSTALLER"
|
||||
${BUILD_PKGS_PARALLEL_DIR}/build-rpms-serial --installer --clean $EXTRA_ARGS_COMMON $EXTRA_ARGS_RPM $TARGETS_INSTALLER || exit 1
|
||||
fi
|
||||
fi
|
||||
if [ "x$TARGETS" == "x " ] || [ "$TARGETS_INSTALLER" != " " ]; then
|
||||
if [ "$TARGETS" == "$EMPTY_TARGETS" ] || [ "$TARGETS_INSTALLER" != "$EMPTY_TARGETS" ]; then
|
||||
echo "${BUILD_PKGS_PARALLEL_DIR}/build-srpms-serial --installer --clean $EXTRA_ARGS_COMMON $EXTRA_ARGS_SRPM $TARGETS_INSTALLER"
|
||||
${BUILD_PKGS_PARALLEL_DIR}/build-srpms-serial --installer --clean $EXTRA_ARGS_COMMON $EXTRA_ARGS_SRPM $TARGETS_INSTALLER || exit 1
|
||||
fi
|
||||
@ -317,13 +372,22 @@ function launch_build()
|
||||
fi
|
||||
|
||||
echo "Launching $build_type build, logging to $logfile"
|
||||
\rm $logfile
|
||||
if [ $APPEND_LOG_FLAG -eq 0 ] && [ -f $logfile ]; then
|
||||
\rm $logfile
|
||||
fi
|
||||
|
||||
|
||||
echo -e "\n######## $(date): Launching build-srpms-parallel --$build_type $EXTRA_ARGS $@\n" | tee --append $logfile
|
||||
|
||||
echo "BUILD_AVOIDANCE_URL=$BUILD_AVOIDANCE_URL" | tee --append $logfile
|
||||
if [ "x$BUILD_AVOIDANCE_URL" != "x" ]; then
|
||||
echo "build_avoidance $build_type" | tee --append $logfile
|
||||
build_avoidance $build_type 2>&1 | tee --append $logfile
|
||||
fi
|
||||
|
||||
# No clean flag, call build-srpms-parallel followed by build-rpms-parallel
|
||||
echo "${BUILD_PKGS_PARALLEL_DIR}/build-srpms-parallel --$build_type $EXTRA_ARGS_COMMON $EXTRA_ARGS_SRPM $targets" | tee --append $logfile
|
||||
${BUILD_PKGS_PARALLEL_DIR}/build-srpms-parallel --$build_type $EXTRA_ARGS_COMMON $EXTRA_ARGS_SRPM $targets 2>&1 | tee --append $logfile
|
||||
|
||||
rc=${PIPESTATUS[0]}
|
||||
if [ $rc -eq 0 ]; then
|
||||
echo -e "\n######## $(date): build-srpm-parallel --$build_type was successful" | tee --append $logfile
|
||||
@ -371,19 +435,18 @@ fi
|
||||
echo "Capture build context"
|
||||
git_context > "${MY_WORKSPACE}/CONTEXT"
|
||||
|
||||
|
||||
if [ $STD_BUILD -eq 1 ]; then
|
||||
if [ "x$TARGETS" == "x " ] || [ "$TARGETS_STD" != " " ] || [ "$TARGETS_MISC" != " " ]; then
|
||||
if [ "$TARGETS" == "$EMPTY_TARGETS" ] || [ "$TARGETS_STD" != "$EMPTY_TARGETS" ] || [ "$TARGETS_MISC" != "$EMPTY_TARGETS" ]; then
|
||||
launch_build std
|
||||
else
|
||||
echo "Skipping 'std' build, no valid targets in list: $TARGETS"
|
||||
echo "Skipping 'std' build, no valid targets in list: '$TARGETS'"
|
||||
fi
|
||||
else
|
||||
echo "Skipping 'std' build"
|
||||
fi
|
||||
|
||||
if [ $RT_BUILD -eq 1 ]; then
|
||||
if [ "x$TARGETS" == "x " ] || [ "$TARGETS_RT" != " " ] || [ "$TARGETS_MISC" != " " ]; then
|
||||
if [ "$TARGETS" == "$EMPTY_TARGETS" ] || [ "$TARGETS_RT" != "$EMPTY_TARGETS" ] || [ "$TARGETS_MISC" != "$EMPTY_TARGETS" ]; then
|
||||
launch_build rt
|
||||
else
|
||||
echo "Skipping 'rt' build, no valid targets in list: $TARGETS"
|
||||
@ -393,7 +456,7 @@ else
|
||||
fi
|
||||
|
||||
if [ $INSTALLER_BUILD -eq 1 ]; then
|
||||
if [ "x$TARGETS" == "x " ] || [ "$TARGETS_INSTALLER" != " " ] || [ "$TARGETS_MISC" != " " ]; then
|
||||
if [ "$TARGETS" == "$EMPTY_TARGETS" ] || [ "$TARGETS_INSTALLER" != "$EMPTY_TARGETS" ] || [ "$TARGETS_MISC" != "$EMPTY_TARGETS" ]; then
|
||||
launch_build installer
|
||||
else
|
||||
echo "Skipping 'installer' build, no valid targets in list: $TARGETS"
|
||||
@ -402,6 +465,15 @@ else
|
||||
echo "Skipping 'installer' build"
|
||||
fi
|
||||
|
||||
# Make sure REFERENCE_BUILD is set to something
|
||||
if [ -z $REFERENCE_BUILD ]; then
|
||||
REFERENCE_BUILD=0
|
||||
fi
|
||||
|
||||
if [ $REFERENCE_BUILD -eq 1 ]; then
|
||||
echo "Saving reference context"
|
||||
build_avoidance_save_reference_context
|
||||
fi
|
||||
|
||||
echo "All builds were successful"
|
||||
|
||||
|
@ -1,10 +1,22 @@
|
||||
#!/bin/bash
|
||||
|
||||
# This program is a wrapper around build-srpms and build-rpms
|
||||
#
|
||||
# Copyright (c) 2018 Wind River Systems, Inc.
|
||||
#
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
#
|
||||
|
||||
#
|
||||
# Build first src.rpms, then rpms, from source, or from a downloaded tarball
|
||||
# or src.rpm plus our additional patches.
|
||||
#
|
||||
# This program is a wrapper around build-srpms-serial and build-rpms-serial
|
||||
#
|
||||
|
||||
BUILD_PKGS_SERIAL_DIR="$(dirname "$(readlink -f "${BASH_SOURCE[0]}" )" )"
|
||||
|
||||
source "${BUILD_PKGS_SERIAL_DIR}/git-utils.sh"
|
||||
source "${BUILD_PKGS_SERIAL_DIR}/build-avoidance-utils.sh"
|
||||
source "${BUILD_PKGS_SERIAL_DIR}/spec-utils"
|
||||
|
||||
|
||||
@ -13,14 +25,15 @@ usage () {
|
||||
echo "Usage: "
|
||||
echo " Create source and Binary rpms:"
|
||||
echo " Build optimizations (--no-descendants, --no-required, --no-build-info,"
|
||||
echo " --no-autoclean) are not recommended for the first build after a clone/pull,"
|
||||
echo " nor the final build prior to creating an iso or patch, but can be used"
|
||||
echo " for intermediate builds. i.e. while debugging compilation failures."
|
||||
echo " build-pkgs-serial [--no-descendants] [--no-required] [--no-build-info] [--no-autoclean] [--careful] [--formal] [ list of package names ]"
|
||||
echo " --no-autoclean, --no-build-avoidance) are not recommended for the first build"
|
||||
echo " after a clone/pull, nor the final build prior to creating an iso or patch,"
|
||||
echo " but can be used for intermediate builds."
|
||||
echo " i.e. while debugging compilation failures."
|
||||
echo " build-pkgs-serial [--build-avoidance | --no-build-avoidance] [--no-descendants] [--no-required] [--no-build-info] [--no-autoclean] [--careful] [--formal] [ list of package names ]"
|
||||
echo ""
|
||||
echo " Delete source rpms, and the directories associated with it's creation:"
|
||||
echo " Note: does not clean an edit environment"
|
||||
echo " build-pkgs-serial --clean [ list of package names ]"
|
||||
echo " build-pkgs-serial --clean [--build-avoidance | --no-build-avoidance] [ list of package names ]"
|
||||
echo ""
|
||||
echo " Extract an src.rpm into a pair of git trees to aid in editing it's contents,"
|
||||
echo " one for source code and one for metadata such as the spec file."
|
||||
@ -39,12 +52,17 @@ usage () {
|
||||
HELP=0
|
||||
CLEAN_FLAG=0
|
||||
EDIT_FLAG=0
|
||||
APPEND_LOG_FLAG=0
|
||||
BUILD_AVOIDANCE_FLAG=0
|
||||
STD_BUILD=1
|
||||
RT_BUILD=1
|
||||
INSTALLER_BUILD=0
|
||||
|
||||
export BUILD_AVOIDANCE_URL=""
|
||||
|
||||
|
||||
# read the options
|
||||
TEMP=$(getopt -o h --long serial,rt,std,edit,no-meta-patch,no-descendants,no-required,no-build-info,no-autoclean,formal,careful,help,clean -n 'build-pkgs' -- "$@")
|
||||
TEMP=$(getopt -o h --long serial,rt,std,installer,edit,build-avoidance,no-build-avoidance,build-avoidance-dir:,build-avoidance-host:,build-avoidance-user:,build-avoidance-day:,no-meta-patch,no-descendants,no-required,no-build-info,no-autoclean,formal,careful,help,clean,append-log -n 'build-pkgs-serial' -- "$@")
|
||||
if [ $? -ne 0 ]; then
|
||||
usage
|
||||
exit 1
|
||||
@ -55,8 +73,21 @@ eval set -- "$TEMP"
|
||||
EXTRA_ARGS_COMMON=""
|
||||
EXTRA_ARGS_SRPM=""
|
||||
EXTRA_ARGS_RPM=""
|
||||
|
||||
export BUILD_AVOIDANCE_OVERRIDE_DIR=""
|
||||
export BUILD_AVOIDANCE_OVERRIDE_HOST=""
|
||||
export BUILD_AVOIDANCE_OVERRIDE_USR=""
|
||||
export BUILD_AVOIDANCE_DAY=""
|
||||
|
||||
while true ; do
|
||||
case "$1" in
|
||||
--append-log) APPEND_LOG_FLAG=1 ; shift ;;
|
||||
--build-avoidance) BUILD_AVOIDANCE_FLAG=1 ; shift ;;
|
||||
--no-build-avoidance) BUILD_AVOIDANCE_FLAG=0 ; shift ;;
|
||||
--build-avoidance-dir) BUILD_AVOIDANCE_OVERRIDE_DIR=$2; shift 2 ;;
|
||||
--build-avoidance-host) BUILD_AVOIDANCE_OVERRIDE_HOST=$2; shift 2 ;;
|
||||
--build-avoidance-user) BUILD_AVOIDANCE_OVERRIDE_USR=$2; shift 2 ;;
|
||||
--build-avoidance-day) BUILD_AVOIDANCE_DAY=$2; shift 2 ;;
|
||||
--no-descendants) EXTRA_ARGS_COMMON+=" --no-descendants" ; shift ;;
|
||||
--formal) EXTRA_ARGS_COMMON+=" --formal" ; shift ;;
|
||||
--careful) EXTRA_ARGS_RPM+=" --careful" ; shift ;;
|
||||
@ -81,11 +112,59 @@ if [ $HELP -eq 1 ]; then
|
||||
exit 0
|
||||
fi
|
||||
|
||||
function my_exit() {
|
||||
build-rpms-parallel --std --tmpfs-clean
|
||||
build-rpms-parallel --rt --tmpfs-clean
|
||||
}
|
||||
|
||||
function my_sigint() {
|
||||
echo "build-pkgs-parallel sigint"
|
||||
pkill -SIGABRT -P $BASHPID &> /dev/null
|
||||
echo "build-pkgs-parallel waiting"
|
||||
wait
|
||||
echo "build-pkgs-parallel wait complete"
|
||||
|
||||
}
|
||||
|
||||
function my_sighup() {
|
||||
echo "build-pkgs-parallel sighup"
|
||||
pkill -SIGABRT -P $BASHPID &> /dev/null
|
||||
echo "build-pkgs-parallel waiting"
|
||||
wait
|
||||
echo "build-pkgs-parallel wait complete"
|
||||
}
|
||||
|
||||
function my_sigabrt() {
|
||||
echo "build-pkgs-parallel sigabrt"
|
||||
pkill -SIGABRT -P $BASHPID &> /dev/null
|
||||
echo "build-pkgs-parallel waiting"
|
||||
wait
|
||||
echo "build-pkgs-parallel wait complete"
|
||||
}
|
||||
|
||||
function my_sigterm() {
|
||||
echo "build-pkgs-parallel sigterm"
|
||||
pkill -SIGABRT -P $BASHPID &> /dev/null
|
||||
echo "build-pkgs-parallel waiting"
|
||||
wait
|
||||
echo "build-pkgs-parallel wait complete"
|
||||
}
|
||||
|
||||
trap my_exit EXIT
|
||||
trap my_sigint INT
|
||||
trap my_sighup HUP
|
||||
trap my_sigabrt ABRT
|
||||
trap my_sigterm TERM
|
||||
|
||||
# Note: For ease of parsing, a TARGETS list always begins and ends
|
||||
# with a space. An empty target list consistes of two spaces.
|
||||
TARGETS=" $@ "
|
||||
TARGETS_STD=" "
|
||||
TARGETS_RT=" "
|
||||
TARGETS_INSTALLER=" "
|
||||
TARGETS_MISC=" "
|
||||
EMPTY_TARGETS=" "
|
||||
|
||||
TARGETS_STD="$EMPTY_TARGETS"
|
||||
TARGETS_RT="$EMPTY_TARGETS"
|
||||
TARGETS_INSTALLER="$EMPTY_TARGETS"
|
||||
TARGETS_MISC="$EMPTY_TARGETS"
|
||||
|
||||
find_targets () {
|
||||
local centos_pkg_dirs=$1
|
||||
@ -155,14 +234,28 @@ find_targets () {
|
||||
fi
|
||||
done
|
||||
|
||||
echo "$RESULT"
|
||||
if [ "$RESULT" == " " ]; then
|
||||
echo "$EMPTY_LIST"
|
||||
else
|
||||
echo "$RESULT"
|
||||
fi
|
||||
return 0
|
||||
}
|
||||
|
||||
if [ $EDIT_FLAG -eq 1 ] || [ "$TARGETS" != "$EMPTY_TARGETS" ]; then
|
||||
BUILD_AVOIDANCE_FLAG=0
|
||||
fi
|
||||
|
||||
echo "BUILD_AVOIDANCE_FLAG=$BUILD_AVOIDANCE_FLAG"
|
||||
echo "CLEAN_FLAG=$CLEAN_FLAG"
|
||||
echo "EDIT_FLAG=$EDIT_FLAG"
|
||||
|
||||
if [ "x$TARGETS" != "x " ]; then
|
||||
if [ $BUILD_AVOIDANCE_FLAG -eq 1 ]; then
|
||||
# Build Avoidance requested. Get URL of a usable context, if any.
|
||||
export BUILD_AVOIDANCE_URL=$(get_build_avoidance_context)
|
||||
fi
|
||||
|
||||
if [ "$TARGETS" != "$EMPTY_TARGETS" ]; then
|
||||
TARGETS_STD="$(find_targets centos_pkg_dirs)"
|
||||
|
||||
BUILD_TYPE_SAVE="$BUILD_TYPE"
|
||||
@ -207,45 +300,51 @@ echo "TARGETS_INSTALLER='$TARGETS_INSTALLER'"
|
||||
echo "TARGETS_MISC='$TARGETS_MISC'"
|
||||
|
||||
if [ $CLEAN_FLAG -eq 1 ]; then
|
||||
|
||||
if [ "$TARGETS" == "$EMPTY_TARGETS" ] && [ $BUILD_AVOIDANCE_FLAG -eq 1 ] ; then
|
||||
build_avoidance_clean
|
||||
fi
|
||||
|
||||
if [ $STD_BUILD -eq 1 ]; then
|
||||
if [ "x$TARGETS" == "x " ] || [ "$TARGETS_STD" != " " ] || [ "$TARGETS_MISC" != " " ]; then
|
||||
if [ "$TARGETS" == "$EMPTY_TARGETS" ] || [ "$TARGETS_STD" != "$EMPTY_TARGETS" ] || [ "$TARGETS_MISC" != "$EMPTY_TARGETS" ]; then
|
||||
if [ $EDIT_FLAG -ne 1 ]; then
|
||||
echo "${BUILD_PKGS_SERIAL_DIR}/build-rpms-serial --std --clean $EXTRA_ARGS_COMMON $EXTRA_ARGS_RPM $TARGETS_STD $TARGETS_MISC"
|
||||
${BUILD_PKGS_SERIAL_DIR}/build-rpms-serial --std --clean $EXTRA_ARGS_COMMON $EXTRA_ARGS_RPM $TARGETS_STD $TARGETS_MISC || exit 1
|
||||
|
||||
fi
|
||||
fi
|
||||
if [ "x$TARGETS" == "x " ] || [ "$TARGETS_STD" != " " ] || [ "$TARGETS_MISC" != " " ]; then
|
||||
if [ "$TARGETS" == "$EMPTY_TARGETS" ] || [ "$TARGETS_STD" != "$EMPTY_TARGETS" ] || [ "$TARGETS_MISC" != "$EMPTY_TARGETS" ]; then
|
||||
echo "${BUILD_PKGS_SERIAL_DIR}/build-srpms-serial --std --clean $EXTRA_ARGS_COMMON $EXTRA_ARGS_SRPM $TARGETS_STD $TARGETS_MISC"
|
||||
${BUILD_PKGS_SERIAL_DIR}/build-srpms-serial --std --clean $EXTRA_ARGS_COMMON $EXTRA_ARGS_SRPM $TARGETS_STD $TARGETS_MISC || exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
if [ $RT_BUILD -eq 1 ]; then
|
||||
if [ "x$TARGETS" == "x" ] || [ "$TARGETS_RT" != " " ] || [ "$TARGETS_MISC" != " " ]; then
|
||||
if [ "$TARGETS" == "$EMPTY_TARGETS" ] || [ "$TARGETS_RT" != "$EMPTY_TARGETS" ] || [ "$TARGETS_MISC" != "$EMPTY_TARGETS" ]; then
|
||||
if [ $EDIT_FLAG -ne 1 ]; then
|
||||
echo "${BUILD_PKGS_SERIAL_DIR}/build-rpms-serial --rt --clean $EXTRA_ARGS_COMMON $EXTRA_ARGS_RPM $TARGETS_RT $TARGETS_MISC"
|
||||
${BUILD_PKGS_SERIAL_DIR}/build-rpms-serial --rt --clean $EXTRA_ARGS_COMMON $EXTRA_ARGS_RPM $TARGETS_RT $TARGETS_MISC || exit 1
|
||||
fi
|
||||
fi
|
||||
if [ "x$TARGETS" == "x " ] || [ "$TARGETS_RT" != " " ] || [ "$TARGETS_MISC" != " " ]; then
|
||||
if [ "$TARGETS" == "$EMPTY_TARGETS" ] || [ "$TARGETS_RT" != "$EMPTY_TARGETS" ] || [ "$TARGETS_MISC" != "$EMPTY_TARGETS" ]; then
|
||||
echo "${BUILD_PKGS_SERIAL_DIR}/build-srpms-serial --rt --clean $EXTRA_ARGS_COMMON $EXTRA_ARGS_SRPM $TARGETS_RT $TARGETS_MISC"
|
||||
${BUILD_PKGS_SERIAL_DIR}/build-srpms-serial --rt --clean $EXTRA_ARGS_COMMON $EXTRA_ARGS_SRPM $TARGETS_RT $TARGETS_MISC || exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
if [ $INSTALLER_BUILD -eq 1 ]; then
|
||||
if [ "x$TARGETS" == "x" ] || [ "$TARGETS_INSTALLER" != " " ]; then
|
||||
if [ "$TARGETS" == "$EMPTY_TARGETS" ] || [ "$TARGETS_INSTALLER" != "$EMPTY_TARGETS" ]; then
|
||||
if [ $EDIT_FLAG -ne 1 ]; then
|
||||
echo "${BUILD_PKGS_SERIAL_DIR}/build-rpms-serial --installer --clean $EXTRA_ARGS_COMMON $EXTRA_ARGS_RPM $TARGETS_INSTALLER"
|
||||
${BUILD_PKGS_SERIAL_DIR}/build-rpms-serial --installer --clean $EXTRA_ARGS_COMMON $EXTRA_ARGS_RPM $TARGETS_INSTALLER || exit 1
|
||||
fi
|
||||
fi
|
||||
if [ "x$TARGETS" == "x " ] || [ "$TARGETS_INSTALLER" != " " ]; then
|
||||
if [ "$TARGETS" == "$EMPTY_TARGETS" ] || [ "$TARGETS_INSTALLER" != "$EMPTY_TARGETS" ]; then
|
||||
echo "${BUILD_PKGS_SERIAL_DIR}/build-srpms-serial --installer --clean $EXTRA_ARGS_COMMON $EXTRA_ARGS_SRPM $TARGETS_INSTALLER"
|
||||
${BUILD_PKGS_SERIAL_DIR}/build-srpms-serial --installer --clean $EXTRA_ARGS_COMMON $EXTRA_ARGS_SRPM $TARGETS_INSTALLER || exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
exit $?
|
||||
fi
|
||||
|
||||
@ -273,8 +372,18 @@ function launch_build()
|
||||
fi
|
||||
|
||||
echo "Launching $build_type build, logging to $logfile"
|
||||
if [ $APPEND_LOG_FLAG -eq 0 ] && [ -f $logfile ]; then
|
||||
\rm $logfile
|
||||
fi
|
||||
|
||||
echo -e "\n######## $(date): Launching build-srpms-serial --$build_type $EXTRA_ARGS $@\n" | tee --append $logfile
|
||||
|
||||
echo "BUILD_AVOIDANCE_URL=$BUILD_AVOIDANCE_URL" | tee --append $logfile
|
||||
if [ "x$BUILD_AVOIDANCE_URL" != "x" ]; then
|
||||
echo "build_avoidance $build_type" | tee --append $logfile
|
||||
build_avoidance $build_type 2>&1 | tee --append $logfile
|
||||
fi
|
||||
|
||||
# No clean flag, call build-srpms-serial followed by build-rpms-serial
|
||||
echo "${BUILD_PKGS_SERIAL_DIR}/build-srpms-serial --$build_type $EXTRA_ARGS_COMMON $EXTRA_ARGS_SRPM $targets" | tee --append $logfile
|
||||
${BUILD_PKGS_SERIAL_DIR}/build-srpms-serial --$build_type $EXTRA_ARGS_COMMON $EXTRA_ARGS_SRPM $targets 2>&1 | tee --append $logfile
|
||||
@ -326,7 +435,7 @@ echo "Capture build context"
|
||||
git_context > "${MY_WORKSPACE}/CONTEXT"
|
||||
|
||||
if [ $STD_BUILD -eq 1 ]; then
|
||||
if [ "x$TARGETS" == "x " ] || [ "$TARGETS_STD" != " " ] || [ "$TARGETS_MISC" != " " ]; then
|
||||
if [ "$TARGETS" == "$EMPTY_TARGETS" ] || [ "$TARGETS_STD" != "$EMPTY_TARGETS" ] || [ "$TARGETS_MISC" != "$EMPTY_TARGETS" ]; then
|
||||
launch_build std
|
||||
else
|
||||
echo "Skipping 'std' build, no valid targets in list: $TARGETS"
|
||||
@ -336,7 +445,7 @@ else
|
||||
fi
|
||||
|
||||
if [ $RT_BUILD -eq 1 ]; then
|
||||
if [ "x$TARGETS" == "x " ] || [ "$TARGETS_RT" != " " ] || [ "$TARGETS_MISC" != " " ]; then
|
||||
if [ "$TARGETS" == "$EMPTY_TARGETS" ] || [ "$TARGETS_RT" != "$EMPTY_TARGETS" ] || [ "$TARGETS_MISC" != "$EMPTY_TARGETS" ]; then
|
||||
launch_build rt
|
||||
else
|
||||
echo "Skipping 'rt' build, no valid targets in list: $TARGETS"
|
||||
@ -346,7 +455,7 @@ else
|
||||
fi
|
||||
|
||||
if [ $INSTALLER_BUILD -eq 1 ]; then
|
||||
if [ "x$TARGETS" == "x " ] || [ "$TARGETS_INSTALLER" != " " ] || [ "$TARGETS_MISC" != " " ]; then
|
||||
if [ "$TARGETS" == "$EMPTY_TARGETS" ] || [ "$TARGETS_INSTALLER" != "$EMPTY_TARGETS" ] || [ "$TARGETS_MISC" != "$EMPTY_TARGETS" ]; then
|
||||
launch_build installer
|
||||
else
|
||||
echo "Skipping 'installer' build, no valid targets in list: $TARGETS"
|
||||
@ -355,6 +464,16 @@ else
|
||||
echo "Skipping 'installer' build"
|
||||
fi
|
||||
|
||||
# Make sure REFERENCE_BUILD is set to something
|
||||
if [ -z $REFERENCE_BUILD ]; then
|
||||
REFERENCE_BUILD=0
|
||||
fi
|
||||
|
||||
if [ $REFERENCE_BUILD -eq 1 ]; then
|
||||
echo "Saving reference context"
|
||||
build_avoidance_save_reference_context
|
||||
fi
|
||||
|
||||
echo "All builds were successful"
|
||||
|
||||
exit 0
|
||||
|
@ -1,6 +1,16 @@
|
||||
#!/bin/bash
|
||||
|
||||
#
|
||||
# Copyright (c) 2018 Wind River Systems, Inc.
|
||||
#
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
#
|
||||
|
||||
#
|
||||
# Builds rpm files from src.rpm files.
|
||||
#
|
||||
# This program is a wrapper around build-rpms-parallel and build-rpms-serial
|
||||
#
|
||||
|
||||
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
|
||||
|
||||
|
@ -6,6 +6,24 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
#
|
||||
|
||||
#
|
||||
# Builds rpm files from src.rpm files.
|
||||
#
|
||||
# This version compiles packages in parrallel if sufficient resources
|
||||
# (cpu's and memory) are available.
|
||||
#
|
||||
# The location of packages to be built is
|
||||
# $MY_WORKSPACE/<build-type>/rpmbuild/SRPMS.
|
||||
#
|
||||
# The build order is a derived from the BuildRequires in the
|
||||
# spec files in the src.rpms. Note that the BuildRequires sometimes
|
||||
# create dependency loops, so no correct order can be computed. In these
|
||||
# cases we add a retry loop. As long as one new package builds, we
|
||||
# keep retrying the loop, until all are built, or no progress is made.
|
||||
# So please don't panic and CTRL-C just because you see a few error
|
||||
# messages go by!
|
||||
#
|
||||
|
||||
|
||||
export ME=$(basename "$0")
|
||||
CMDLINE="$ME $@"
|
||||
@ -122,12 +140,10 @@ create-no-clean-list () {
|
||||
if [ ! -f $NO_CLEAN_LIST_FILE ]; then
|
||||
NEED_REBUILD=1
|
||||
else
|
||||
if [ -f MY_BUILD_CFG ]; then
|
||||
if [ -f MY_BUILD_CFG ]; then
|
||||
find "$MY_BUILD_CFG" -not -newermm "$NO_CLEAN_LIST_FILE" | grep -q $(basename $MY_BUILD_CFG)
|
||||
if [ $? -eq 0 ]; then
|
||||
NEED_REBUILD=1
|
||||
fi
|
||||
if [ -f $MY_BUILD_CFG ]; then
|
||||
find "$MY_BUILD_CFG" -not -newermm "$NO_CLEAN_LIST_FILE" | grep -q $(basename $MY_BUILD_CFG)
|
||||
if [ $? -eq 0 ]; then
|
||||
NEED_REBUILD=1
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|