Build Avoidance
Purpose:
Reduce build times after a repo sync by pulling in pre-generated
srpms and rpms and other build products created by a local reference build.
Usage:
repo sync
generate-cgcs-centos-repo.sh ...
populate_downloads.sh ...
build-pkgs --build-avoidance [--build-avoidance-user <user> \
--build-avoidance-host <addr> --build-avoidance-dir <dir>]
Reference builds:
- A server performs a regular (daily?), automated builds using
existing methods. Call these the reference builds.
- The builds are timestamped, and preserved for some time. (weeks?)
The MY_WORKSPACE directory for the build shall have a common root
directory, and a leaf directory that is a UTC time stamp of format
YYYYMMDDThhmmssZ.
e.g.
MY_WORKSPACE=/localdisk/loadbuild/jenkins/StarlingX/20180719T113021Z
Alternative formats are possible by setting values in ...
"$MY_REPO/local-build-data/build_avoidance_source"
e.g.
BUILD_AVOIDANCE_DATE_FORMAT="%Y-%m-%d"
BUILD_AVOIDANCE_TIME_FORMAT="%H-%M-%S"
BUILD_AVOIDANCE_DATE_TIME_DELIM="_"
BUILD_AVOIDANCE_DATE_TIME_POSTFIX=""
BUILD_AVOIDANCE_DATE_UTC=0
Which results in YYYY-MM-DD_hh-mm-ss format using local time.
The one property that the timestamp must have is that they
are sortable, and that the reference build and the consumer of
the reference builds agree on the format.
- A build CONTEXT is captured, consisting of the SHA of each and every
git that contributed to the build.
- For each package built, a file shall capture he md5sums of all the
source code inputs to the build of that package.
- All these build products are accessible locally (e.g. a regional
office) via rsync (other protocols can be added later). ssh
is also required to run remote query commands on the reference build.
Initial ground work to support a selection variable ....
BUILD_AVOIDANCE_FILE_TRANSFER="my-transfer-protocol"
in $MY_REPO/local-build-data/build_avoidance_source"
has been created, but "rsync" is the only valid value at this time.
- Location of the reference build can be specified via command line, or
defaults can be put in $MY_REPO/local-build-data/build_avoidance_source.
The local-build-data directory is gitignored by stx-root and so can be
customized for local needs.
e.g.
cat $MY_REPO/local-build-data/build_avoidance_source
BUILD_AVOIDANCE_USR="jenkins"
BUILD_AVOIDANCE_HOST="stx-build-server.myco.com"
BUILD_AVOIDANCE_DIR="/localdisk/loadbuild/jenkins/StarlingX"
Notes:
- Build avoidance is only used if requested.
- Build avoidance does not necessarily use the latest reference build.
It compares the git context of all available reference builds vs your
own git context, and chooses the most recent for which you gits have
all the conent. i.e. all your gits will be same or newer than that
used by the reference build. This also meens that some packages might
still need to be rebuilt after the download step.
- Normally build avoidance remembers the last download context and will only
consider reference builds newer than the last download. You can reset
using 'build-pkgs --build-avoidance --clear' to erase the download history.
When might this matter to me? If you change to an old branch that
hasn't been synced recently and want to build in that context.
- The primary assumtion of Build Avoidance is that it is faster to
download packages than to build them. This is typically true of a
good LAN, but likely not true of a WAN. This is why we emphasize the
local nature of your reference build server.
Also in this update:
- reworked context generation to be relative to 'dirname $MY_REPO'
- Moved md5sum calculation to a common file, and fixed case where
symlinks where canonacalized to paths outside of $MY_REPO.
We'll make an exception to canonacalization to keep paths
relative to $MY_REPO.
- In future other functions could be moved to the common file.
Story: 2002835
Task: 22754
Change-Id: I757780190cc6063d0a2d3ad9d0a6020ab5169e99
Signed-off-by: Scott Little <scott.little@windriver.com>
2018-09-05 16:56:37 -04:00
|
|
|
#
|
|
|
|
# Copyright (c) 2018 Wind River Systems, Inc.
|
|
|
|
#
|
|
|
|
# SPDX-License-Identifier: Apache-2.0
|
|
|
|
#
|
|
|
|
|
|
|
|
#
|
|
|
|
# Functions common to build-srpm-serial and build-srpm-parallel.
|
|
|
|
#
|
|
|
|
|
|
|
|
SRC_BUILD_TYPE_SRPM="srpm"
|
|
|
|
SRC_BUILD_TYPE_SPEC="spec"
|
|
|
|
SRC_BUILD_TYPES="$SRC_BUILD_TYPE_SRPM $SRC_BUILD_TYPE_SPEC"
|
|
|
|
|
Update to build tools to support Build layering.
1: Building rpms now produces an rpm.lst file. This file serves to
list all rpms produced by the build. A seperate rpm.lst file
is producede for each build type (std, rt, installer). The file is
co-resident with the matching repodata directory.
The rpm.lst files will need to be published by cengn for each layer build.
The download tools in stx-tools will need to find the rpm.lst files
of lower layer builds, and use it to direct the download of rpms
from the lower layer.
2: Building rpms now produces an image.inc file. This file serves
to list all rpms that the layer recommends be built into an iso.
The file is stored under the $MY_WORKSPACE/<build-type> subdirectory,
although it has identical content for any build-type.
The image.inc file will need to be published by cengn for each layer
build.
The download tools in stx-tools will need to download the per-layer
image.inc files to the $MY_REPO/cgcs-centos-repo/layer_image_inc/
sub-directory, renaming the file in some layer specific way.
The build-iso tool will process any *.inc files found under
$MY_REPO/cgcs-centos-repo/layer_image_inc/ ,
adding those rpms to the iso.
3) Add a mechanism for layer specific mock.cfg prototypes.
This will allow special handling of the real-time (aka 'rt'),
repositories for layer builds that need to do a 'rt' build.
4) Add support for a $MY_REPO/cgcs-centos-repo/rt subdirectory,
a place to build a repo of real-time rpms originating from
lower layer builds.
The download tools in stx-tools will need to populate the new
rt repos.
As of this writing, non-rt rpms remain in $MY_REPO/cgcs-centos-repo/.
i.e. there is not a new $MY_REPO/cgcs-centos-repo/std/ directory.
5) Some changes to make us more flexible about where we find
realease and bsp files.
6) Found that kernel mudules were not reliably building against
the hearnel-headers of our modified kernel. Found that adding '--update'
to our mock build command was not working. Does mock expect '--update'
to only be used independently of a build command? It does work when
used in that manner, so that's what we will do.
7) The build-pkgs, build-srpms, build-rpms family of commands
can take a layer argument and/or will read the LAYER environment
variable. Current the only use of this variable is to modify
the build-info. It does NOT limit the compile to packages
for a specific layer.
Story: 2006166
Task: 37094
Depends-On: https://review.opendev.org/698756
Depends-On: https://review.opendev.org/700819
Change-Id: I817e08a19cdabe08b3fcc47dee63a36b461c13c0
Co-Authored-by: Martin Chen <haochuan.z.chen@intel.com>
Signed-off-by: Scott Little <scott.little@windriver.com>
2019-12-31 15:46:45 -05:00
|
|
|
set_build_info () {
|
|
|
|
local info_file="$MY_WORKSPACE/BUILD_INFO"
|
|
|
|
local layer_prefix="${LAYER^^}_"
|
|
|
|
if [ "${LAYER}" == "" ]; then
|
|
|
|
layer_prefix=""
|
|
|
|
fi
|
|
|
|
mkdir -p "$(dirname ${info_file})"
|
|
|
|
echo "${layer_prefix}OS=\"centos\"" > "${info_file}"
|
|
|
|
echo "${layer_prefix}JOB=\"n/a\"" >> "${info_file}"
|
|
|
|
echo "${layer_prefix}BUILD_BY=\"${USER}\"" >> "${info_file}"
|
|
|
|
echo "${layer_prefix}BUILD_NUMBER=\"n/a\"" >> "${info_file}"
|
|
|
|
echo "${layer_prefix}BUILD_HOST=\"$(hostname)\"" >> "${info_file}"
|
|
|
|
echo "${layer_prefix}BUILD_DATE=\"$(date '+%Y-%m-%d %H:%M:%S %z')\"" >> "${info_file}"
|
|
|
|
}
|
|
|
|
|
Build Avoidance
Purpose:
Reduce build times after a repo sync by pulling in pre-generated
srpms and rpms and other build products created by a local reference build.
Usage:
repo sync
generate-cgcs-centos-repo.sh ...
populate_downloads.sh ...
build-pkgs --build-avoidance [--build-avoidance-user <user> \
--build-avoidance-host <addr> --build-avoidance-dir <dir>]
Reference builds:
- A server performs a regular (daily?), automated builds using
existing methods. Call these the reference builds.
- The builds are timestamped, and preserved for some time. (weeks?)
The MY_WORKSPACE directory for the build shall have a common root
directory, and a leaf directory that is a UTC time stamp of format
YYYYMMDDThhmmssZ.
e.g.
MY_WORKSPACE=/localdisk/loadbuild/jenkins/StarlingX/20180719T113021Z
Alternative formats are possible by setting values in ...
"$MY_REPO/local-build-data/build_avoidance_source"
e.g.
BUILD_AVOIDANCE_DATE_FORMAT="%Y-%m-%d"
BUILD_AVOIDANCE_TIME_FORMAT="%H-%M-%S"
BUILD_AVOIDANCE_DATE_TIME_DELIM="_"
BUILD_AVOIDANCE_DATE_TIME_POSTFIX=""
BUILD_AVOIDANCE_DATE_UTC=0
Which results in YYYY-MM-DD_hh-mm-ss format using local time.
The one property that the timestamp must have is that they
are sortable, and that the reference build and the consumer of
the reference builds agree on the format.
- A build CONTEXT is captured, consisting of the SHA of each and every
git that contributed to the build.
- For each package built, a file shall capture he md5sums of all the
source code inputs to the build of that package.
- All these build products are accessible locally (e.g. a regional
office) via rsync (other protocols can be added later). ssh
is also required to run remote query commands on the reference build.
Initial ground work to support a selection variable ....
BUILD_AVOIDANCE_FILE_TRANSFER="my-transfer-protocol"
in $MY_REPO/local-build-data/build_avoidance_source"
has been created, but "rsync" is the only valid value at this time.
- Location of the reference build can be specified via command line, or
defaults can be put in $MY_REPO/local-build-data/build_avoidance_source.
The local-build-data directory is gitignored by stx-root and so can be
customized for local needs.
e.g.
cat $MY_REPO/local-build-data/build_avoidance_source
BUILD_AVOIDANCE_USR="jenkins"
BUILD_AVOIDANCE_HOST="stx-build-server.myco.com"
BUILD_AVOIDANCE_DIR="/localdisk/loadbuild/jenkins/StarlingX"
Notes:
- Build avoidance is only used if requested.
- Build avoidance does not necessarily use the latest reference build.
It compares the git context of all available reference builds vs your
own git context, and chooses the most recent for which you gits have
all the conent. i.e. all your gits will be same or newer than that
used by the reference build. This also meens that some packages might
still need to be rebuilt after the download step.
- Normally build avoidance remembers the last download context and will only
consider reference builds newer than the last download. You can reset
using 'build-pkgs --build-avoidance --clear' to erase the download history.
When might this matter to me? If you change to an old branch that
hasn't been synced recently and want to build in that context.
- The primary assumtion of Build Avoidance is that it is faster to
download packages than to build them. This is typically true of a
good LAN, but likely not true of a WAN. This is why we emphasize the
local nature of your reference build server.
Also in this update:
- reworked context generation to be relative to 'dirname $MY_REPO'
- Moved md5sum calculation to a common file, and fixed case where
symlinks where canonacalized to paths outside of $MY_REPO.
We'll make an exception to canonacalization to keep paths
relative to $MY_REPO.
- In future other functions could be moved to the common file.
Story: 2002835
Task: 22754
Change-Id: I757780190cc6063d0a2d3ad9d0a6020ab5169e99
Signed-off-by: Scott Little <scott.little@windriver.com>
2018-09-05 16:56:37 -04:00
|
|
|
|
|
|
|
str_lst_contains() {
|
|
|
|
TARGET="$1"
|
|
|
|
LST="$2"
|
|
|
|
|
|
|
|
if [[ $LST =~ (^|[[:space:]])$TARGET($|[[:space:]]) ]] ; then
|
|
|
|
return 0
|
|
|
|
else
|
|
|
|
return 1
|
|
|
|
fi
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
#
|
Extend GITREVCOUNT to inputs other than SRC_DIR and PKG_BASE
Changes in brief:
1) TIS_PATCH_VER will now accept new automatic variables
'OTHER_GITREVCOUNT' and 'SRC_GITREVCOUNT'.
2) New BASE_SRCREV variables are supported, 'SRC_BASE_SRCREV' and
'BASE_SRCREV_FOR_PATH'
3) OPT_DEP_LIST_FOR_BUILD_TYPE[<build-type>] joins OPT_DEP_LIST as
a way to list files that affect the build, but are not included
in the assembled src.rpm.
4) Some code relating to interpreting build_srpm.data files,
and producing a list of files that affect the build, are relocated
to 'srpm-util' and restructured for greater utility.
The details:
TIS_PATCH_VER=<expression>
# An integer, or one of the supported
# variables listed below, or the sum of
# variables and integers.
# e.g.
# TIS_PATCH_VER=PKG_GITREVCOUNT+SRC_GITREVCOUNT+5
PKG_GITREVCOUNT # Count git revisions relative to PKG_BASE.
# Optionally only count from PKG_BASE_SRCREV
SRC_GITREVCOUNT # Count git revisions relative to SRC_DIR.
# Optionally only count from SRC_BASE_SRCREV
GITREVCOUNT # Deprecated, please use SRC_GITREVCOUNT instead.
# Count git revisions relative to SRC_DIR.
# Optionally only count from TIS_BASE_SRCREV
OTHER_GITREVCOUNT # count git revisions from all sources excluding
# PKG_BASE and SRC_DIR
# Optionally only count from
# BASE_SRCREV_FOR_PATH[<path>]
PKG_BASE_SRCREV=<sha> # Limit PKG_GITREVCOUNT revision count to
# commits since <sha>
SRC_BASE_SRCREV=<sha> # Limit SRC_GITREVCOUNT revision count to
# commits since <sha>
TIS_BASE_SRCREV=<sha> # Deprecated, please use SRC_BASE_SRCREV
# instead
# Limit GITREVCOUNT revision count to commits
# since <sha>
BASE_SRCREV_FOR_PATH[<path>]=[<sha>|OTHER_PKG_BASE_SRCREV]
# Limit OTHER_GITREVCOUNT revision count for
# commits under <path> to commits since <sha>.
# If <path> is the PKG_BASE of another package
# (not the current package) then the keyword
# 'OTHER_PKG_BASE_SRCREV' can be used to extract
# the 'PKG_BASE_SRCREV' value of the other
# package.
#
# The <path> can reference variables like
# $STX_BASE and $GIT_BASE.
OPT_DEP_LIST=<path-list> # Add a space separated list of paths that
# don't contribute to the content of a src.rpm
# but do contribute to triggering a rebuild,
# and possibly modifying the TIS_PATCH_VER via
# use of OTHER_GITREVCOUNT.
OPT_DEP_LIST_FOR_BUILD_TYPE[<build-type>]=<path-list>
# For a specific build type only, add a space
# separated list of paths that don't
# contribute to the content of src.rpm,
# but do contribute to triggering a
# rebuild, and possibly modifying the
# TIS_PATCH_VER via use of OTHER_GITREVCOUNT.
Story: 2006166
Task: 39765
Change-Id: I9e7d409d4eefdb41a7083db1b801d531c443b678
Signed-off-by: Scott Little <scott.little@windriver.com>
2020-06-04 23:58:11 -04:00
|
|
|
# md5sums_from_input_vars <src-build-type> <srpm-or-spec-path> <work-dir>
|
Build Avoidance
Purpose:
Reduce build times after a repo sync by pulling in pre-generated
srpms and rpms and other build products created by a local reference build.
Usage:
repo sync
generate-cgcs-centos-repo.sh ...
populate_downloads.sh ...
build-pkgs --build-avoidance [--build-avoidance-user <user> \
--build-avoidance-host <addr> --build-avoidance-dir <dir>]
Reference builds:
- A server performs a regular (daily?), automated builds using
existing methods. Call these the reference builds.
- The builds are timestamped, and preserved for some time. (weeks?)
The MY_WORKSPACE directory for the build shall have a common root
directory, and a leaf directory that is a UTC time stamp of format
YYYYMMDDThhmmssZ.
e.g.
MY_WORKSPACE=/localdisk/loadbuild/jenkins/StarlingX/20180719T113021Z
Alternative formats are possible by setting values in ...
"$MY_REPO/local-build-data/build_avoidance_source"
e.g.
BUILD_AVOIDANCE_DATE_FORMAT="%Y-%m-%d"
BUILD_AVOIDANCE_TIME_FORMAT="%H-%M-%S"
BUILD_AVOIDANCE_DATE_TIME_DELIM="_"
BUILD_AVOIDANCE_DATE_TIME_POSTFIX=""
BUILD_AVOIDANCE_DATE_UTC=0
Which results in YYYY-MM-DD_hh-mm-ss format using local time.
The one property that the timestamp must have is that they
are sortable, and that the reference build and the consumer of
the reference builds agree on the format.
- A build CONTEXT is captured, consisting of the SHA of each and every
git that contributed to the build.
- For each package built, a file shall capture he md5sums of all the
source code inputs to the build of that package.
- All these build products are accessible locally (e.g. a regional
office) via rsync (other protocols can be added later). ssh
is also required to run remote query commands on the reference build.
Initial ground work to support a selection variable ....
BUILD_AVOIDANCE_FILE_TRANSFER="my-transfer-protocol"
in $MY_REPO/local-build-data/build_avoidance_source"
has been created, but "rsync" is the only valid value at this time.
- Location of the reference build can be specified via command line, or
defaults can be put in $MY_REPO/local-build-data/build_avoidance_source.
The local-build-data directory is gitignored by stx-root and so can be
customized for local needs.
e.g.
cat $MY_REPO/local-build-data/build_avoidance_source
BUILD_AVOIDANCE_USR="jenkins"
BUILD_AVOIDANCE_HOST="stx-build-server.myco.com"
BUILD_AVOIDANCE_DIR="/localdisk/loadbuild/jenkins/StarlingX"
Notes:
- Build avoidance is only used if requested.
- Build avoidance does not necessarily use the latest reference build.
It compares the git context of all available reference builds vs your
own git context, and chooses the most recent for which you gits have
all the conent. i.e. all your gits will be same or newer than that
used by the reference build. This also meens that some packages might
still need to be rebuilt after the download step.
- Normally build avoidance remembers the last download context and will only
consider reference builds newer than the last download. You can reset
using 'build-pkgs --build-avoidance --clear' to erase the download history.
When might this matter to me? If you change to an old branch that
hasn't been synced recently and want to build in that context.
- The primary assumtion of Build Avoidance is that it is faster to
download packages than to build them. This is typically true of a
good LAN, but likely not true of a WAN. This is why we emphasize the
local nature of your reference build server.
Also in this update:
- reworked context generation to be relative to 'dirname $MY_REPO'
- Moved md5sum calculation to a common file, and fixed case where
symlinks where canonacalized to paths outside of $MY_REPO.
We'll make an exception to canonacalization to keep paths
relative to $MY_REPO.
- In future other functions could be moved to the common file.
Story: 2002835
Task: 22754
Change-Id: I757780190cc6063d0a2d3ad9d0a6020ab5169e99
Signed-off-by: Scott Little <scott.little@windriver.com>
2018-09-05 16:56:37 -04:00
|
|
|
#
|
|
|
|
# Returns md5 data for all input files of a src.rpm.
|
|
|
|
# Assumes PKG_BASE, ORIG_SRPM_PATH have been defined and the
|
|
|
|
# build_srpm.data file has already been sourced.
|
|
|
|
#
|
|
|
|
# Arguments:
|
|
|
|
# src-build-type: Any single value from $SRC_BUILD_TYPES.
|
|
|
|
# e.g. 'srpm' or 'spec'
|
|
|
|
# srpm-or-spec-path: Absolute path to an src.rpm, or to a
|
|
|
|
# spec file.
|
|
|
|
# work-dir: Optional working directory. If a path is
|
|
|
|
# specified but does not exist, it will be created.
|
|
|
|
#
|
|
|
|
# Returns: output of md5sum command with canonical path names
|
|
|
|
#
|
|
|
|
md5sums_from_input_vars () {
|
|
|
|
local SRC_BUILD_TYPE="$1"
|
|
|
|
local SRPM_OR_SPEC_PATH="$2"
|
|
|
|
local WORK_DIR="$3"
|
|
|
|
|
|
|
|
local TMP_FLAG=0
|
|
|
|
local LINK_FILTER='[/]stx[/]downloads[/]'
|
|
|
|
|
|
|
|
if ! str_lst_contains "$SRC_BUILD_TYPE" "$SRC_BUILD_TYPES" ; then
|
|
|
|
>&2 echo "ERROR: $FUNCNAME (${LINENO}): invalid arg: SRC_BUILD_TYPE='$SRC_BUILD_TYPE'"
|
|
|
|
return 1
|
|
|
|
fi
|
|
|
|
|
|
|
|
if [ -z $WORK_DIR ]; then
|
|
|
|
WORK_DIR=$(mktemp -d /tmp/${FUNCNAME}_XXXXXX)
|
|
|
|
if [ $? -ne 0 ]; then
|
|
|
|
>&2 echo "ERROR: $FUNCNAME (${LINENO}): mktemp -d /tmp/${FUNCNAME}_XXXXXX"
|
|
|
|
return 1
|
|
|
|
fi
|
|
|
|
TMP_FLAG=1
|
|
|
|
else
|
|
|
|
mkdir -p "$WORK_DIR"
|
|
|
|
if [ $? -ne 0 ]; then
|
|
|
|
>&2 echo "ERROR: $FUNCNAME (${LINENO}): mkdir -p '$WORK_DIR'"
|
|
|
|
return 1
|
|
|
|
fi
|
|
|
|
fi
|
|
|
|
|
|
|
|
local INPUT_FILES_SORTED="$WORK_DIR/srpm_sorted_input.files"
|
|
|
|
|
|
|
|
# Create lists of input files (INPUT_FILES) and symlinks (INPUT_LINKS).
|
Extend GITREVCOUNT to inputs other than SRC_DIR and PKG_BASE
Changes in brief:
1) TIS_PATCH_VER will now accept new automatic variables
'OTHER_GITREVCOUNT' and 'SRC_GITREVCOUNT'.
2) New BASE_SRCREV variables are supported, 'SRC_BASE_SRCREV' and
'BASE_SRCREV_FOR_PATH'
3) OPT_DEP_LIST_FOR_BUILD_TYPE[<build-type>] joins OPT_DEP_LIST as
a way to list files that affect the build, but are not included
in the assembled src.rpm.
4) Some code relating to interpreting build_srpm.data files,
and producing a list of files that affect the build, are relocated
to 'srpm-util' and restructured for greater utility.
The details:
TIS_PATCH_VER=<expression>
# An integer, or one of the supported
# variables listed below, or the sum of
# variables and integers.
# e.g.
# TIS_PATCH_VER=PKG_GITREVCOUNT+SRC_GITREVCOUNT+5
PKG_GITREVCOUNT # Count git revisions relative to PKG_BASE.
# Optionally only count from PKG_BASE_SRCREV
SRC_GITREVCOUNT # Count git revisions relative to SRC_DIR.
# Optionally only count from SRC_BASE_SRCREV
GITREVCOUNT # Deprecated, please use SRC_GITREVCOUNT instead.
# Count git revisions relative to SRC_DIR.
# Optionally only count from TIS_BASE_SRCREV
OTHER_GITREVCOUNT # count git revisions from all sources excluding
# PKG_BASE and SRC_DIR
# Optionally only count from
# BASE_SRCREV_FOR_PATH[<path>]
PKG_BASE_SRCREV=<sha> # Limit PKG_GITREVCOUNT revision count to
# commits since <sha>
SRC_BASE_SRCREV=<sha> # Limit SRC_GITREVCOUNT revision count to
# commits since <sha>
TIS_BASE_SRCREV=<sha> # Deprecated, please use SRC_BASE_SRCREV
# instead
# Limit GITREVCOUNT revision count to commits
# since <sha>
BASE_SRCREV_FOR_PATH[<path>]=[<sha>|OTHER_PKG_BASE_SRCREV]
# Limit OTHER_GITREVCOUNT revision count for
# commits under <path> to commits since <sha>.
# If <path> is the PKG_BASE of another package
# (not the current package) then the keyword
# 'OTHER_PKG_BASE_SRCREV' can be used to extract
# the 'PKG_BASE_SRCREV' value of the other
# package.
#
# The <path> can reference variables like
# $STX_BASE and $GIT_BASE.
OPT_DEP_LIST=<path-list> # Add a space separated list of paths that
# don't contribute to the content of a src.rpm
# but do contribute to triggering a rebuild,
# and possibly modifying the TIS_PATCH_VER via
# use of OTHER_GITREVCOUNT.
OPT_DEP_LIST_FOR_BUILD_TYPE[<build-type>]=<path-list>
# For a specific build type only, add a space
# separated list of paths that don't
# contribute to the content of src.rpm,
# but do contribute to triggering a
# rebuild, and possibly modifying the
# TIS_PATCH_VER via use of OTHER_GITREVCOUNT.
Story: 2006166
Task: 39765
Change-Id: I9e7d409d4eefdb41a7083db1b801d531c443b678
Signed-off-by: Scott Little <scott.little@windriver.com>
2020-06-04 23:58:11 -04:00
|
|
|
srpm_source_file_list "$SRC_BUILD_TYPE" "$SRPM_OR_SPEC_PATH" "$INPUT_FILES_SORTED"
|
Build Avoidance
Purpose:
Reduce build times after a repo sync by pulling in pre-generated
srpms and rpms and other build products created by a local reference build.
Usage:
repo sync
generate-cgcs-centos-repo.sh ...
populate_downloads.sh ...
build-pkgs --build-avoidance [--build-avoidance-user <user> \
--build-avoidance-host <addr> --build-avoidance-dir <dir>]
Reference builds:
- A server performs a regular (daily?), automated builds using
existing methods. Call these the reference builds.
- The builds are timestamped, and preserved for some time. (weeks?)
The MY_WORKSPACE directory for the build shall have a common root
directory, and a leaf directory that is a UTC time stamp of format
YYYYMMDDThhmmssZ.
e.g.
MY_WORKSPACE=/localdisk/loadbuild/jenkins/StarlingX/20180719T113021Z
Alternative formats are possible by setting values in ...
"$MY_REPO/local-build-data/build_avoidance_source"
e.g.
BUILD_AVOIDANCE_DATE_FORMAT="%Y-%m-%d"
BUILD_AVOIDANCE_TIME_FORMAT="%H-%M-%S"
BUILD_AVOIDANCE_DATE_TIME_DELIM="_"
BUILD_AVOIDANCE_DATE_TIME_POSTFIX=""
BUILD_AVOIDANCE_DATE_UTC=0
Which results in YYYY-MM-DD_hh-mm-ss format using local time.
The one property that the timestamp must have is that they
are sortable, and that the reference build and the consumer of
the reference builds agree on the format.
- A build CONTEXT is captured, consisting of the SHA of each and every
git that contributed to the build.
- For each package built, a file shall capture he md5sums of all the
source code inputs to the build of that package.
- All these build products are accessible locally (e.g. a regional
office) via rsync (other protocols can be added later). ssh
is also required to run remote query commands on the reference build.
Initial ground work to support a selection variable ....
BUILD_AVOIDANCE_FILE_TRANSFER="my-transfer-protocol"
in $MY_REPO/local-build-data/build_avoidance_source"
has been created, but "rsync" is the only valid value at this time.
- Location of the reference build can be specified via command line, or
defaults can be put in $MY_REPO/local-build-data/build_avoidance_source.
The local-build-data directory is gitignored by stx-root and so can be
customized for local needs.
e.g.
cat $MY_REPO/local-build-data/build_avoidance_source
BUILD_AVOIDANCE_USR="jenkins"
BUILD_AVOIDANCE_HOST="stx-build-server.myco.com"
BUILD_AVOIDANCE_DIR="/localdisk/loadbuild/jenkins/StarlingX"
Notes:
- Build avoidance is only used if requested.
- Build avoidance does not necessarily use the latest reference build.
It compares the git context of all available reference builds vs your
own git context, and chooses the most recent for which you gits have
all the conent. i.e. all your gits will be same or newer than that
used by the reference build. This also meens that some packages might
still need to be rebuilt after the download step.
- Normally build avoidance remembers the last download context and will only
consider reference builds newer than the last download. You can reset
using 'build-pkgs --build-avoidance --clear' to erase the download history.
When might this matter to me? If you change to an old branch that
hasn't been synced recently and want to build in that context.
- The primary assumtion of Build Avoidance is that it is faster to
download packages than to build them. This is typically true of a
good LAN, but likely not true of a WAN. This is why we emphasize the
local nature of your reference build server.
Also in this update:
- reworked context generation to be relative to 'dirname $MY_REPO'
- Moved md5sum calculation to a common file, and fixed case where
symlinks where canonacalized to paths outside of $MY_REPO.
We'll make an exception to canonacalization to keep paths
relative to $MY_REPO.
- In future other functions could be moved to the common file.
Story: 2002835
Task: 22754
Change-Id: I757780190cc6063d0a2d3ad9d0a6020ab5169e99
Signed-off-by: Scott Little <scott.little@windriver.com>
2018-09-05 16:56:37 -04:00
|
|
|
if [ $? -eq 1 ]; then
|
2018-10-15 14:49:25 -04:00
|
|
|
return 1
|
Build Avoidance
Purpose:
Reduce build times after a repo sync by pulling in pre-generated
srpms and rpms and other build products created by a local reference build.
Usage:
repo sync
generate-cgcs-centos-repo.sh ...
populate_downloads.sh ...
build-pkgs --build-avoidance [--build-avoidance-user <user> \
--build-avoidance-host <addr> --build-avoidance-dir <dir>]
Reference builds:
- A server performs a regular (daily?), automated builds using
existing methods. Call these the reference builds.
- The builds are timestamped, and preserved for some time. (weeks?)
The MY_WORKSPACE directory for the build shall have a common root
directory, and a leaf directory that is a UTC time stamp of format
YYYYMMDDThhmmssZ.
e.g.
MY_WORKSPACE=/localdisk/loadbuild/jenkins/StarlingX/20180719T113021Z
Alternative formats are possible by setting values in ...
"$MY_REPO/local-build-data/build_avoidance_source"
e.g.
BUILD_AVOIDANCE_DATE_FORMAT="%Y-%m-%d"
BUILD_AVOIDANCE_TIME_FORMAT="%H-%M-%S"
BUILD_AVOIDANCE_DATE_TIME_DELIM="_"
BUILD_AVOIDANCE_DATE_TIME_POSTFIX=""
BUILD_AVOIDANCE_DATE_UTC=0
Which results in YYYY-MM-DD_hh-mm-ss format using local time.
The one property that the timestamp must have is that they
are sortable, and that the reference build and the consumer of
the reference builds agree on the format.
- A build CONTEXT is captured, consisting of the SHA of each and every
git that contributed to the build.
- For each package built, a file shall capture he md5sums of all the
source code inputs to the build of that package.
- All these build products are accessible locally (e.g. a regional
office) via rsync (other protocols can be added later). ssh
is also required to run remote query commands on the reference build.
Initial ground work to support a selection variable ....
BUILD_AVOIDANCE_FILE_TRANSFER="my-transfer-protocol"
in $MY_REPO/local-build-data/build_avoidance_source"
has been created, but "rsync" is the only valid value at this time.
- Location of the reference build can be specified via command line, or
defaults can be put in $MY_REPO/local-build-data/build_avoidance_source.
The local-build-data directory is gitignored by stx-root and so can be
customized for local needs.
e.g.
cat $MY_REPO/local-build-data/build_avoidance_source
BUILD_AVOIDANCE_USR="jenkins"
BUILD_AVOIDANCE_HOST="stx-build-server.myco.com"
BUILD_AVOIDANCE_DIR="/localdisk/loadbuild/jenkins/StarlingX"
Notes:
- Build avoidance is only used if requested.
- Build avoidance does not necessarily use the latest reference build.
It compares the git context of all available reference builds vs your
own git context, and chooses the most recent for which you gits have
all the conent. i.e. all your gits will be same or newer than that
used by the reference build. This also meens that some packages might
still need to be rebuilt after the download step.
- Normally build avoidance remembers the last download context and will only
consider reference builds newer than the last download. You can reset
using 'build-pkgs --build-avoidance --clear' to erase the download history.
When might this matter to me? If you change to an old branch that
hasn't been synced recently and want to build in that context.
- The primary assumtion of Build Avoidance is that it is faster to
download packages than to build them. This is typically true of a
good LAN, but likely not true of a WAN. This is why we emphasize the
local nature of your reference build server.
Also in this update:
- reworked context generation to be relative to 'dirname $MY_REPO'
- Moved md5sum calculation to a common file, and fixed case where
symlinks where canonacalized to paths outside of $MY_REPO.
We'll make an exception to canonacalization to keep paths
relative to $MY_REPO.
- In future other functions could be moved to the common file.
Story: 2002835
Task: 22754
Change-Id: I757780190cc6063d0a2d3ad9d0a6020ab5169e99
Signed-off-by: Scott Little <scott.little@windriver.com>
2018-09-05 16:56:37 -04:00
|
|
|
fi
|
|
|
|
|
|
|
|
# Remove $MY_REPO prefix from paths
|
|
|
|
cat $INPUT_FILES_SORTED | xargs md5sum | sed "s# $(readlink -f $MY_REPO)/# #"
|
|
|
|
|
|
|
|
if [ $TMP_FLAG -eq 0 ]; then
|
|
|
|
\rm -f $INPUT_FILES_SORTED
|
|
|
|
else
|
|
|
|
\rm -rf $WORK_DIR
|
|
|
|
fi
|
|
|
|
|
|
|
|
return 0
|
|
|
|
}
|