109 lines
2.6 KiB
Plaintext
Raw Normal View History

#!/bin/bash
#
# Copyright (c) 2018 Wind River Systems, Inc.
#
# SPDX-License-Identifier: Apache-2.0
#
Build Avoidance Purpose: Reduce build times after a repo sync by pulling in pre-generated srpms and rpms and other build products created by a local reference build. Usage: repo sync generate-cgcs-centos-repo.sh ... populate_downloads.sh ... build-pkgs --build-avoidance [--build-avoidance-user <user> \ --build-avoidance-host <addr> --build-avoidance-dir <dir>] Reference builds: - A server performs a regular (daily?), automated builds using existing methods. Call these the reference builds. - The builds are timestamped, and preserved for some time. (weeks?) The MY_WORKSPACE directory for the build shall have a common root directory, and a leaf directory that is a UTC time stamp of format YYYYMMDDThhmmssZ. e.g. MY_WORKSPACE=/localdisk/loadbuild/jenkins/StarlingX/20180719T113021Z Alternative formats are possible by setting values in ... "$MY_REPO/local-build-data/build_avoidance_source" e.g. BUILD_AVOIDANCE_DATE_FORMAT="%Y-%m-%d" BUILD_AVOIDANCE_TIME_FORMAT="%H-%M-%S" BUILD_AVOIDANCE_DATE_TIME_DELIM="_" BUILD_AVOIDANCE_DATE_TIME_POSTFIX="" BUILD_AVOIDANCE_DATE_UTC=0 Which results in YYYY-MM-DD_hh-mm-ss format using local time. The one property that the timestamp must have is that they are sortable, and that the reference build and the consumer of the reference builds agree on the format. - A build CONTEXT is captured, consisting of the SHA of each and every git that contributed to the build. - For each package built, a file shall capture he md5sums of all the source code inputs to the build of that package. - All these build products are accessible locally (e.g. a regional office) via rsync (other protocols can be added later). ssh is also required to run remote query commands on the reference build. Initial ground work to support a selection variable .... BUILD_AVOIDANCE_FILE_TRANSFER="my-transfer-protocol" in $MY_REPO/local-build-data/build_avoidance_source" has been created, but "rsync" is the only valid value at this time. - Location of the reference build can be specified via command line, or defaults can be put in $MY_REPO/local-build-data/build_avoidance_source. The local-build-data directory is gitignored by stx-root and so can be customized for local needs. e.g. cat $MY_REPO/local-build-data/build_avoidance_source BUILD_AVOIDANCE_USR="jenkins" BUILD_AVOIDANCE_HOST="stx-build-server.myco.com" BUILD_AVOIDANCE_DIR="/localdisk/loadbuild/jenkins/StarlingX" Notes: - Build avoidance is only used if requested. - Build avoidance does not necessarily use the latest reference build. It compares the git context of all available reference builds vs your own git context, and chooses the most recent for which you gits have all the conent. i.e. all your gits will be same or newer than that used by the reference build. This also meens that some packages might still need to be rebuilt after the download step. - Normally build avoidance remembers the last download context and will only consider reference builds newer than the last download. You can reset using 'build-pkgs --build-avoidance --clear' to erase the download history. When might this matter to me? If you change to an old branch that hasn't been synced recently and want to build in that context. - The primary assumtion of Build Avoidance is that it is faster to download packages than to build them. This is typically true of a good LAN, but likely not true of a WAN. This is why we emphasize the local nature of your reference build server. Also in this update: - reworked context generation to be relative to 'dirname $MY_REPO' - Moved md5sum calculation to a common file, and fixed case where symlinks where canonacalized to paths outside of $MY_REPO. We'll make an exception to canonacalization to keep paths relative to $MY_REPO. - In future other functions could be moved to the common file. Story: 2002835 Task: 22754 Change-Id: I757780190cc6063d0a2d3ad9d0a6020ab5169e99 Signed-off-by: Scott Little <scott.little@windriver.com>
2018-09-05 16:56:37 -04:00
#
# Build first src.rpms, then rpms, from source, or from a downloaded tarball
# or src.rpm plus our additional patches.
#
# This program is a wrapper around build-pkgs-parallel and build-pkgs-serial
#
BUILD_PKGS_DIR="$(dirname "$(readlink -f "${BASH_SOURCE[0]}" )" )"
usage () {
echo ""
echo "Usage: "
echo " Create source and binary rpms:"
echo " build-pkgs [--serial] [args]"
}
SERIAL_FLAG=0
RC=0
for arg in "$@"; do
case "$1" in
--serial) SERIAL_FLAG=1 ;;
esac
done
which mock_tmpfs_umount >> /dev/null
if [ $? -ne 0 ]; then
SERIAL_FLAG=1
fi
export TMPDIR=$MY_WORKSPACE/tmp
mkdir -p $TMPDIR
# Make sure we have a dependency cache
DEP_CACHE="$MY_REPO/cgcs-tis-repo/dependancy-cache"
BUILD_TYPES=(" std rt installer containers")
DEP_RPM_TYPE=(" RPMS SRPMS ")
DEP_DELTAS="$DEP_CACHE/deltas-rpms-srpms"
make_cache_current_rpms () {
FILE=${1}
if [ -z "${FILE}" ]; then
echo "File not specified"
return;
fi
if [ -f ${FILE} ]; then
rm ${FILE}
fi
for build_type in $BUILD_TYPES; do
for rpm_type in $DEP_RPM_TYPE; do
if [ -d $MY_WORKSPACE/$build_type/rpmbuild/$rpm_type/repodata ]; then
current=$MY_WORKSPACE/$build_type/rpmbuild/$rpm_type/
repoquery \
--repofrompath=$build_type-$rpm_type,$current \
--repoid=$build_type-$rpm_type --arch=noarch,src,x86_64 -a \
--qf "%-10{repoid} %-40{name} %-10{version} %-10{release}" \
>> ${FILE}
fi
done;
done;
}
if [ ! -d $DEP_CACHE ]; then
echo "Dependency cache is missing. Creating it now."
$BUILD_PKGS_DIR/create_dependancy_cache.py > $MY_WORKSPACE/create_dependancy_cache.log
make_cache_current_rpms $DEP_DELTAS
echo "Dependency cache created."
else
DEP_TMP=$(mktemp)
make_cache_current_rpms $DEP_TMP
if diff $DEP_DELTAS $DEP_TMP > /dev/null; then
echo "No changes for stx projects"
rm $DEP_TMP
else
echo "Changes detected for stx projects"
echo "Recreating dependecy cache now."
mv $DEP_TMP $DEP_DELTAS
$BUILD_PKGS_DIR/create_dependancy_cache.py > $MY_WORKSPACE/create_dependancy_cache.log
echo "Dependency cache recreated."
fi
fi
if [ $SERIAL_FLAG -eq 1 ]; then
echo "build-pkgs-serial $@"
build-pkgs-serial "$@"
RC=$?
else
echo "build-pkgs-parallel $@"
build-pkgs-parallel "$@"
RC=$?
fi
exit $RC