d51f8050cd
Script changes to download content by layer. Valid options are 'all', 'compiler', 'distro', 'flock'. Current .lst and yum files under directory centos-mirror-tools are relocated. Generic package dependencies are relocated to centos-mirror-tools/config/<os>/<layer>/ . Lst entries for compilable content have been relocated to other git repos by prior updates. i.e. those that list tarballs or srpms to be compiled within that repo. The original .lst files are deleted to make it easier to identify new content during development. Layer 'all' builds all layers in a single workspace. The lst files are identical to current content, minus the src.rpm and tarball entries. Other layers get only a subset of packages download. The minimum required to build the layer. The 'flock' layer will have additional content to satisfy the run time requirements as well as the build time requirements. An upper layer does not need to list rpms known to be provided by a lower layer. Instead the config file 'required_layer_pkgs.cfg' lists urls for lst files for lower layer build outputs. These build outputs are generated and published by cengn for each layer. A second layer config file, 'required_layer_iso_inc.cfg' lists image.inc files for lower layer builds. These build outputs are generated and published by cengn for each layer, summarizing the image.inc files found in individual git repos. Image.inc files inform the build-iso process, listing rpms that that provide services and commands that need to be included in the iso. The transitive list of required rpms need not be listed. Finally the layer config should include a yum.repos.d directory in which supplementary yum repos are defined to pick up cengn built content from lower layers. To allow a designer to do cross-layer building using local sources rather than those provided by CENGN, there are several options. The designer can modify the urls for lower layer build outputs, as found in the .cfg and .repo files within the config directory 'stx-tools/centos-mirror-tools/config/<distro>/<layer-to-build>' directly within the git. Substitute urls can use the file:/// syntax. Just be sure to remove these changes before submitting. Alternatively new args have been added to download_mirror.sh, generate-cgcs-centos-repo.sh and commands that override the normal config. The easiest to use is a command argurement that substitutes a new config directory, replacing stx-tools/centos-mirror-tools/config. The intent is for the designer to do a recursive copy of that directory into a side location. make his changes there, outside of git, and provide the path to that directory as an extra arguement to download_mirror.sh and generate-cgcs-centos-repo.sh. e.g. For simplicity I'll only list the 'extra' arguements download_mirror.sh -C <my-config-dir> \ -l <layer> \ ... generate-cgcs-centos-repo.sh --config-dir=<my-config-dir> \ --layer=<layer> \ ... populate_downloads.sh --config-dir=<my-config-dir> \ --layer=<layer> \ ... These arguements can also be suplied via the environment. For the purpose of containerized builds, these arguements should be defined in your localrc. e.g. export STX_CONFIG_DIR=<my-config-dir> export LAYER=<layer> The final alternative is to override things at a more granular level, replacing a single lst file of image.inc file. Here you can replace a single line found in a required_layer_pkgs.cfg or required_layer_iso_inc.cfg file. e.g. We are doing a flock build and want to modify the content picked up from the distro layer's rt build, and that content delivers a service we want in the iso. For simplicity I'll only list the 'extra' arguments ./download_mirror.sh -l flock \ -L distro,rt,file:///<my-distro-workspace>/rt/rpmbuild/RPMS/rpm.lst \ -I distro,std,file:///<my-distro-workspace>/rt/image.inc \ ... generate-cgcs-centos-repo.sh --layer=flock \ --layer-pkg-url=distro,rt,file:///<my-distro-workspace>/rt/rpmbuild/RPMS/rpm.lst \ --layer-inc-url=distro,std,file:////<my-distro-workspace>/rt/image.inc \ ... NOTE: The triplet syntax for a package list url is <lower-layer>,<build-type>,<url-to-rpm.lst> lower-layer: 'compiler', 'distro' build-type: 'std', 'rt', 'installer' Also if 'file:///' syntax is used, a matching change is made to the yum *.repo file. This assumes that the rpm.lst is co-resident with repodata directory, as is the norm for our build outputs. NOTE: The triplet syntax for a image inc url is <lower-layer>,<include-type>,<url-to-image.inc> lower-layer: 'compiler', 'distro' build-type: 'std', 'dev' A typical user is likely only working in the flock layer on the master branch. He should be content to use the compiler and distro layer outputs from cengn. His workflow looks like ... 1, sync code for flock layer $ repo init -u https://opendev.org/starlingx/manifest.git -b master -m flock.xml $ repo sync 2, download rpms for flock layer, and populate a local mirror $ LOCAL_MIRROR=/import/mirrors/starlingx $ cd stx-tools/centos-mirror-tools $ ./download_mirror.sh -n -g -c yum.conf.sample -S -l flock $ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/ 3, Prepare a virtual repo and downloads directory for building $ cd ../toCOPY $ ./generate-cgcs-centos-repo.sh --layer=flock $LOCAL_MIRROR $ ./populate_downloads.sh --layer=flock $LOCAL_MIRROR 4, rpm package and iso building $ build-pkgs && build-iso && build-helm-charts.sh Building all layers in a single workspace is still supported, and looks identical to the previous workflow. 1, sync code $ repo init -u https://opendev.org/starlingx/manifest.git -b master $ repo sync 2, download rpms for flock layer, and populate a local mirror $ LOCAL_MIRROR=/import/mirrors/starlingx $ cd stx-tools/centos-mirror-tools $ ./download_mirror.sh -n -g -c yum.conf.sample -S $ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/ 3, create repo named "StxCentos7Distro" for building $ cd ../toCOPY $ generate-cgcs-centos-repo.sh $LOCAL_MIRROR $ populate_downloads.sh $LOCAL_MIRROR 4, rpm package and iso building $ build-pkgs && build-iso && build-helm-charts.sh Only a cross-layer developer should setup two or three copies of the building environment, one per layer. We suggest you use seperate shells for each layer, as the various paths (MY_REPO, MY_WORKSPACE ...) need to be unique, Shell 1, compiler layer $ LOCAL_MIRROR=/import/mirrors/starlingx $ LOCAL_CONFIG=<some-dir>/config $ MY_REPO_ROOT_DIR=<some-dir>/layer-compiler $ MY_REPO=$MY_REPO_ROOT_DIR/cgcs-root $ MY_WORKSPACE=$MY_REPO_ROOT_DIR/workspace ... $ mkdir -p $MY_REPO_ROOT_DIR $ cd $MY_REPO_ROOT_DIR $ repo init -u https://opendev.org/starlingx/manifest.git -b master -m compiler.xml $ cd stx-tools/centos-mirror-tools $ cp -r config/* $LOCAL_CONFIG ... edit urls in *.cfg and *.repo files under $LOCAL_CONFIG ... $ ./download_mirror.sh -n -g -c yum.conf.sample -S -C $LOCAL_CONFIG -l compiler $ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/ $ cd ../toCOPY $ ./generate-cgcs-centos-repo.sh --config-dir=$LOCAL_CONFIG --layer=compiler $LOCAL_MIRROR $ ./populate_downloads.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR $ build-pkgs Shell 2, distro layer $ LOCAL_MIRROR=/import/mirrors/starlingx $ LOCAL_CONFIG=<some-dir>/config $ MY_REPO_ROOT_DIR=<some-dir>/layer-distro $ MY_REPO=$MY_REPO_ROOT_DIR/cgcs-root $ MY_WORKSPACE=$MY_REPO_ROOT_DIR/workspace ... $ mkdir -p $MY_REPO_ROOT_DIR $ cd $MY_REPO_ROOT_DIR $ repo init -u https://opendev.org/starlingx/manifest.git -b master -m distro.xml $ repo sync $ cd stx-tools/centos-mirror-tools $ ./download_mirror.sh -n -g -c yum.conf.sample -S -C $LOCAL_CONFIG -l distro $ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/ $ cd ../toCOPY $ ./generate-cgcs-centos-repo.sh --config-dir=$LOCAL_CONFIG --layer=distro $LOCAL_MIRROR $ ./populate_downloads.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR $ build-pkgs Shell 3, flock layer $ LOCAL_MIRROR=/import/mirrors/starlingx $ LOCAL_CONFIG=<some-dir>/config $ MY_REPO_ROOT_DIR=<some-dir>/layer-flock $ MY_REPO=$MY_REPO_ROOT_DIR/cgcs-root $ MY_WORKSPACE=$MY_REPO_ROOT_DIR/workspace ... $ mkdir -p $MY_REPO_ROOT_DIR $ cd $MY_REPO_ROOT_DIR $ repo init -u https://opendev.org/starlingx/manifest.git -b master -m flock.xml $ repo sync $ cd stx-tools/centos-mirror-tools $ ./download_mirror.sh -n -g -c yum.conf.sample -S -C $LOCAL_CONFIG -l flock $ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/ $ cd ../toCOPY $ ./generate-cgcs-centos-repo.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR $ ./populate_downloads.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR $ build-pkgs && build-iso && build-helm-charts.sh Story: 2006166 Task: 37103 Depends-On: https://review.opendev.org/698756 Depends-On: https://review.opendev.org/700819 Depends-On: https://review.opendev.org/700821 Change-Id: I088020b81f08656e50aa29b5584bbc1dd1378f12 Signed-off-by: Scott Little <scott.little@windriver.com>
219 lines
5.3 KiB
Bash
219 lines
5.3 KiB
Bash
#
|
|
# SPDX-License-Identifier: Apache-2.0
|
|
#
|
|
# Copyright (C) 2019 Intel Corporation
|
|
#
|
|
|
|
UTILS_DIR="$(dirname "$(readlink -f "${BASH_SOURCE[0]}" )" )"
|
|
|
|
source $UTILS_DIR/url_utils.sh
|
|
|
|
get_yum_command() {
|
|
local _file=$1
|
|
local _level=$2
|
|
local rpm_name=""
|
|
local arr=( $(split_filename $_file) )
|
|
local arch=${arr[3]}
|
|
local yumdownloader_extra_opts=""
|
|
rpm_name="$(get_rpm_level_name $_file $_level)"
|
|
|
|
if [ "$arch" == "src" ]; then
|
|
yumdownloader_extra_opts="--source"
|
|
else
|
|
yumdownloader_extra_opts="--exclude='*.i686' --archlist=noarch,x86_64"
|
|
fi
|
|
|
|
echo "yumdownloader -q -C ${YUMCONFOPT} ${RELEASEVER} $yumdownloader_extra_opts $rpm_name"
|
|
}
|
|
|
|
get_wget_command() {
|
|
local _name="$1"
|
|
local _ret=""
|
|
if [[ "$_name" == http?(s)://* ]]; then
|
|
_ret="wget -q $_name"
|
|
else
|
|
_ret="wget -q $(koji_url $_name)"
|
|
fi
|
|
echo "$_ret"
|
|
}
|
|
|
|
get_rpm_level_name() {
|
|
local _rpm_name=$1
|
|
local _level=$2
|
|
if [ $_level == "L1" ]; then
|
|
SFILE=`echo $_rpm_name | rev | cut -d'.' -f3- | rev`
|
|
elif [ $_level == "$dl_from_stx_mirror" ];then
|
|
# stx mirror uses L1 matches
|
|
SFILE=`echo $_rpm_name | rev | cut -d'.' -f3- | rev`
|
|
elif [ $_level == "L2" ];then
|
|
SFILE=`echo $_rpm_name | rev | cut -d'-' -f2- | rev`
|
|
else
|
|
SFILE=`echo $_rpm_name | rev | cut -d'-' -f3- | rev`
|
|
fi
|
|
echo "$SFILE"
|
|
}
|
|
|
|
get_url() {
|
|
local _name="$1"
|
|
local _level="$2"
|
|
local _ret=""
|
|
|
|
if [ "$_level" == "K1" ]; then
|
|
_ret="$(koji_url $_name)"
|
|
elif [[ "$_name" == *"#"* ]]; then
|
|
_ret="$(echo $_name | cut -d'#' -f2-2)"
|
|
if [ $_level == "stx_mirror" ]; then
|
|
_ret="$(url_to_stx_mirror_url $_ret $distro)"
|
|
fi
|
|
else
|
|
_url_cmd="$(get_yum_command $_name $_level)"
|
|
|
|
# When we add --url to the yum download command,
|
|
# --archlist is no longer enforced. Multiple
|
|
# url's might be returned. So use grep to
|
|
# filter urls for the desitered arch.
|
|
local arr=( $(split_filename $_name) )
|
|
local arch=${arr[3]}
|
|
_ret="$($_url_cmd --url | grep "[.]$arch[.]rpm$")"
|
|
fi
|
|
echo "$_ret"
|
|
}
|
|
|
|
# Function to split an rpm filename into parts.
|
|
#
|
|
# Returns a space seperated list containing:
|
|
# <NAME> <VERSION> <RELEASE> <ARCH> <EPOCH>
|
|
#
|
|
split_filename () {
|
|
local rpm_filename=$1
|
|
|
|
local RPM=""
|
|
local SFILE=""
|
|
local ARCH=""
|
|
local RELEASE=""
|
|
local VERSION=""
|
|
local NAME=""
|
|
local EPOCH=""
|
|
|
|
RPM=$(echo $rpm_filename | rev | cut -d'.' -f-1 | rev)
|
|
SFILE=$(echo $rpm_filename | rev | cut -d'.' -f2- | rev)
|
|
ARCH=$(echo $SFILE | rev | cut -d'.' -f-1 | rev)
|
|
SFILE=$(echo $SFILE | rev | cut -d'.' -f2- | rev)
|
|
RELEASE=$(echo $SFILE | rev | cut -d'-' -f-1 | rev)
|
|
SFILE=$(echo $SFILE | rev | cut -d'-' -f2- | rev)
|
|
VERSION=$(echo $SFILE | rev | cut -d'-' -f-1 | rev)
|
|
NAME=$(echo $SFILE | rev | cut -d'-' -f2- | rev)
|
|
|
|
if [[ $NAME = *":"* ]]; then
|
|
EPOCH=$(echo $NAME | cut -d':' -f-1)
|
|
NAME=$(echo $NAME | cut -d':' -f2-)
|
|
fi
|
|
|
|
echo "$NAME" "$VERSION" "$RELEASE" "$ARCH" "$EPOCH"
|
|
}
|
|
|
|
# Function to predict the URL where a rpm might be found.
|
|
# Assumes the rpm was compile for EPEL by fedora's koji.
|
|
koji_url () {
|
|
local rpm_filename=$1
|
|
|
|
local arr=( $(split_filename $rpm_filename) )
|
|
|
|
local n=${arr[0]}
|
|
local v=${arr[1]}
|
|
local r=${arr[2]}
|
|
local a=${arr[3]}
|
|
|
|
echo "https://kojipkgs.fedoraproject.org/packages/$n/$v/$r/$a/$n-$v-$r.$a.rpm"
|
|
}
|
|
|
|
get_dest_directory() {
|
|
local _type=$1
|
|
local _dest=""
|
|
if [ "$_type" == "src" ]; then
|
|
_dest="$MDIR_SRC"
|
|
else
|
|
_dest="$MDIR_BIN/$_type"
|
|
fi
|
|
echo "$_dest"
|
|
}
|
|
|
|
process_result() {
|
|
local _type="$1"
|
|
local dest_dir="$2"
|
|
local url="$3"
|
|
local sfile="$4"
|
|
|
|
if [ "$_type" != "src" ] && [ ! -d $dest_dir ]; then
|
|
mkdir -p $dest_dir
|
|
fi
|
|
|
|
echo "url_srpm:$url"
|
|
|
|
if ! mv -f $sfile* $dest_dir ; then
|
|
echo "FAILED to move $rpm_name"
|
|
echo "fail_move_srpm:$rpm_name" >> $LOG
|
|
return 1
|
|
fi
|
|
|
|
echo "found_srpm:$rpm_name"
|
|
echo $rpm_name >> $FOUND_SRPMS
|
|
return 0
|
|
}
|
|
|
|
|
|
get_download_cmd() {
|
|
local ff="$1"
|
|
local _level="$2"
|
|
|
|
# Decide if the list will be downloaded using yumdownloader or wget
|
|
if [[ $ff != *"#"* ]]; then
|
|
rpm_name=$ff
|
|
if [ $_level == "K1" ]; then
|
|
download_cmd="$(get_wget_command $rpm_name)"
|
|
else
|
|
# yumdownloader with the appropriate flag for src, noarch or x86_64
|
|
download_cmd="${SUDOCMD} $(get_yum_command $rpm_name $_level)"
|
|
fi
|
|
else
|
|
# Build wget command
|
|
rpm_url=$(get_url "$ff" "$_level")
|
|
download_cmd="$(get_wget_command $rpm_url)"
|
|
fi
|
|
|
|
echo "$download_cmd"
|
|
}
|
|
|
|
get_rpm_name() {
|
|
local ret=""
|
|
|
|
if [[ "$1" != *"#"* ]]; then
|
|
ret="$1"
|
|
else
|
|
ret="$(echo $1 | cut -d"#" -f1-1)"
|
|
fi
|
|
echo "$ret"
|
|
}
|
|
|
|
get_arch_from_rpm() {
|
|
local _file=$1
|
|
local _split=()
|
|
local _arch=""
|
|
if [[ "$1" == *"#"* ]]; then
|
|
_file=$(echo $_file | cut -d"#" -f1-1)
|
|
fi
|
|
|
|
_split=( $(split_filename $_file) )
|
|
_arch=${_split[3]}
|
|
|
|
echo "$_arch"
|
|
}
|
|
|
|
get_from() {
|
|
list=$1
|
|
base=$(basename $list .lst) # removing lst extension
|
|
base=$(basename $base .log) # removing log extension
|
|
from=$(echo $base | rev | cut -d'_' -f1-1 | rev)
|
|
echo $from
|
|
}
|