tools/centos-mirror-tools/utils.sh
Scott Little d51f8050cd Build layering, script and lst update
Script changes to download content by layer.
Valid options are 'all', 'compiler', 'distro', 'flock'.

Current .lst and yum files under directory centos-mirror-tools
are relocated.  Generic package dependencies are relocated to
centos-mirror-tools/config/<os>/<layer>/ .

Lst entries for compilable content have been relocated to other
git repos by prior updates.  i.e.  those that list tarballs or
srpms to be compiled within that repo.

The original .lst files are deleted to make it easier
to identify new content during development.

Layer 'all' builds all layers in a single workspace.  The
lst files are identical to current content, minus the src.rpm and
tarball entries.

Other layers get only a subset of packages download. The minimum
required to build the layer.  The 'flock' layer will have additional
content to satisfy the run time requirements as well as the build
time requirements.

An upper layer does not need to list rpms known to be provided by
a lower layer.  Instead the config file 'required_layer_pkgs.cfg'
lists urls for lst files for lower layer build outputs.
These build outputs are generated and published by cengn for
each layer.

A second layer config file, 'required_layer_iso_inc.cfg' lists
image.inc files for lower layer builds.  These build outputs are
generated and published by cengn for each layer, summarizing
the image.inc files found in individual git repos.

Image.inc files inform the build-iso process, listing rpms that
that provide services and commands that need to be included in
the iso.  The transitive list of required rpms need not be listed.

Finally the layer config should include a yum.repos.d
directory in which supplementary yum repos are defined to
pick up cengn built content from lower layers.

To allow a designer to do cross-layer building using local sources
rather than those provided by CENGN, there are several options.

The designer can modify the urls for lower layer build outputs,
as found in the .cfg and .repo files within the config directory
'stx-tools/centos-mirror-tools/config/<distro>/<layer-to-build>'
directly within the git.  Substitute urls can use the file:///
syntax.  Just be sure to remove these changes before submitting.

Alternatively new args have been added to download_mirror.sh,
generate-cgcs-centos-repo.sh and commands that override the
normal config.

The easiest to use is a command argurement that substitutes a new
config directory, replacing stx-tools/centos-mirror-tools/config.
The intent is for the designer to do a recursive copy of that
directory into a side location. make his changes there, outside of
git, and provide the path to that directory as an extra arguement
to download_mirror.sh and generate-cgcs-centos-repo.sh.

e.g. For simplicity I'll only list the 'extra' arguements

download_mirror.sh -C <my-config-dir> \
                   -l <layer> \
                   ...

generate-cgcs-centos-repo.sh --config-dir=<my-config-dir> \
                             --layer=<layer> \
                             ...

populate_downloads.sh --config-dir=<my-config-dir> \
                      --layer=<layer> \
                      ...

These arguements can also be suplied via the environment.
For the purpose of containerized builds, these arguements
should be defined in your localrc.

e.g.
export STX_CONFIG_DIR=<my-config-dir>
export LAYER=<layer>

The final alternative is to override things at a more granular level,
replacing a single lst file of image.inc file.  Here you can replace
a single line found in a required_layer_pkgs.cfg or
required_layer_iso_inc.cfg file.

e.g. We are doing a flock build and want to modify the content picked up
from the distro layer's rt build, and that content delivers a service
we want in the iso.  For simplicity I'll only list the 'extra' arguments

./download_mirror.sh -l flock \
    -L distro,rt,file:///<my-distro-workspace>/rt/rpmbuild/RPMS/rpm.lst \
    -I distro,std,file:///<my-distro-workspace>/rt/image.inc \
    ...

generate-cgcs-centos-repo.sh --layer=flock \
    --layer-pkg-url=distro,rt,file:///<my-distro-workspace>/rt/rpmbuild/RPMS/rpm.lst \
    --layer-inc-url=distro,std,file:////<my-distro-workspace>/rt/image.inc \
    ...

NOTE: The triplet syntax for a package list url is
    <lower-layer>,<build-type>,<url-to-rpm.lst>

    lower-layer: 'compiler', 'distro'
    build-type: 'std', 'rt', 'installer'

    Also if 'file:///' syntax is used, a matching change is made to
    the yum *.repo file.  This assumes that the rpm.lst is co-resident with
    repodata directory, as is the norm for our build outputs.

NOTE: The triplet syntax for a image inc url is
    <lower-layer>,<include-type>,<url-to-image.inc>

    lower-layer: 'compiler', 'distro'
    build-type: 'std', 'dev'

A typical user is likely only working in the flock layer on the master
branch.  He should be content to use the compiler and distro layer
outputs from cengn.

His workflow looks like ...

1, sync code for flock layer
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m flock.xml
$ repo sync

2, download rpms for flock layer, and populate a local mirror
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S -l flock
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/

3, Prepare a virtual repo and downloads directory for building
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --layer=flock $LOCAL_MIRROR
$ ./populate_downloads.sh --layer=flock $LOCAL_MIRROR

4, rpm package and iso building
$ build-pkgs && build-iso && build-helm-charts.sh

Building all layers in a single workspace is still supported, and
looks identical to the previous workflow.

1, sync code
$ repo init -u https://opendev.org/starlingx/manifest.git -b master
$ repo sync

2, download rpms for flock layer, and populate a local mirror
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/

3, create repo named "StxCentos7Distro" for building
$ cd ../toCOPY
$ generate-cgcs-centos-repo.sh $LOCAL_MIRROR
$ populate_downloads.sh $LOCAL_MIRROR

4, rpm package and iso building
$ build-pkgs && build-iso && build-helm-charts.sh

Only a cross-layer developer should setup two or three copies of the
building environment, one per layer.  We suggest you use seperate shells
for each layer, as the various paths (MY_REPO, MY_WORKSPACE ...) need to
be unique,

Shell 1, compiler layer
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ LOCAL_CONFIG=<some-dir>/config
$ MY_REPO_ROOT_DIR=<some-dir>/layer-compiler
$ MY_REPO=$MY_REPO_ROOT_DIR/cgcs-root
$ MY_WORKSPACE=$MY_REPO_ROOT_DIR/workspace
  ...

$ mkdir -p $MY_REPO_ROOT_DIR
$ cd $MY_REPO_ROOT_DIR
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m compiler.xml
$ cd stx-tools/centos-mirror-tools
$ cp -r config/* $LOCAL_CONFIG
 ... edit urls in *.cfg and *.repo files under $LOCAL_CONFIG ...

$ ./download_mirror.sh -n -g -c yum.conf.sample -S -C $LOCAL_CONFIG -l compiler
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --config-dir=$LOCAL_CONFIG --layer=compiler $LOCAL_MIRROR
$ ./populate_downloads.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ build-pkgs

Shell 2, distro layer
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ LOCAL_CONFIG=<some-dir>/config
$ MY_REPO_ROOT_DIR=<some-dir>/layer-distro
$ MY_REPO=$MY_REPO_ROOT_DIR/cgcs-root
$ MY_WORKSPACE=$MY_REPO_ROOT_DIR/workspace
  ...
$ mkdir -p $MY_REPO_ROOT_DIR
$ cd $MY_REPO_ROOT_DIR
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m distro.xml
$ repo sync
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S -C $LOCAL_CONFIG -l distro
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --config-dir=$LOCAL_CONFIG --layer=distro $LOCAL_MIRROR
$ ./populate_downloads.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ build-pkgs

Shell 3, flock layer
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ LOCAL_CONFIG=<some-dir>/config
$ MY_REPO_ROOT_DIR=<some-dir>/layer-flock
$ MY_REPO=$MY_REPO_ROOT_DIR/cgcs-root
$ MY_WORKSPACE=$MY_REPO_ROOT_DIR/workspace
  ...
$ mkdir -p $MY_REPO_ROOT_DIR
$ cd $MY_REPO_ROOT_DIR
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m flock.xml
$ repo sync
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S -C $LOCAL_CONFIG -l flock
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ ./populate_downloads.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ build-pkgs && build-iso && build-helm-charts.sh

Story: 2006166
Task: 37103

Depends-On: https://review.opendev.org/698756
Depends-On: https://review.opendev.org/700819
Depends-On: https://review.opendev.org/700821
Change-Id: I088020b81f08656e50aa29b5584bbc1dd1378f12
Signed-off-by: Scott Little <scott.little@windriver.com>
2020-02-10 10:45:40 -05:00

219 lines
5.3 KiB
Bash

#
# SPDX-License-Identifier: Apache-2.0
#
# Copyright (C) 2019 Intel Corporation
#
UTILS_DIR="$(dirname "$(readlink -f "${BASH_SOURCE[0]}" )" )"
source $UTILS_DIR/url_utils.sh
get_yum_command() {
local _file=$1
local _level=$2
local rpm_name=""
local arr=( $(split_filename $_file) )
local arch=${arr[3]}
local yumdownloader_extra_opts=""
rpm_name="$(get_rpm_level_name $_file $_level)"
if [ "$arch" == "src" ]; then
yumdownloader_extra_opts="--source"
else
yumdownloader_extra_opts="--exclude='*.i686' --archlist=noarch,x86_64"
fi
echo "yumdownloader -q -C ${YUMCONFOPT} ${RELEASEVER} $yumdownloader_extra_opts $rpm_name"
}
get_wget_command() {
local _name="$1"
local _ret=""
if [[ "$_name" == http?(s)://* ]]; then
_ret="wget -q $_name"
else
_ret="wget -q $(koji_url $_name)"
fi
echo "$_ret"
}
get_rpm_level_name() {
local _rpm_name=$1
local _level=$2
if [ $_level == "L1" ]; then
SFILE=`echo $_rpm_name | rev | cut -d'.' -f3- | rev`
elif [ $_level == "$dl_from_stx_mirror" ];then
# stx mirror uses L1 matches
SFILE=`echo $_rpm_name | rev | cut -d'.' -f3- | rev`
elif [ $_level == "L2" ];then
SFILE=`echo $_rpm_name | rev | cut -d'-' -f2- | rev`
else
SFILE=`echo $_rpm_name | rev | cut -d'-' -f3- | rev`
fi
echo "$SFILE"
}
get_url() {
local _name="$1"
local _level="$2"
local _ret=""
if [ "$_level" == "K1" ]; then
_ret="$(koji_url $_name)"
elif [[ "$_name" == *"#"* ]]; then
_ret="$(echo $_name | cut -d'#' -f2-2)"
if [ $_level == "stx_mirror" ]; then
_ret="$(url_to_stx_mirror_url $_ret $distro)"
fi
else
_url_cmd="$(get_yum_command $_name $_level)"
# When we add --url to the yum download command,
# --archlist is no longer enforced. Multiple
# url's might be returned. So use grep to
# filter urls for the desitered arch.
local arr=( $(split_filename $_name) )
local arch=${arr[3]}
_ret="$($_url_cmd --url | grep "[.]$arch[.]rpm$")"
fi
echo "$_ret"
}
# Function to split an rpm filename into parts.
#
# Returns a space seperated list containing:
# <NAME> <VERSION> <RELEASE> <ARCH> <EPOCH>
#
split_filename () {
local rpm_filename=$1
local RPM=""
local SFILE=""
local ARCH=""
local RELEASE=""
local VERSION=""
local NAME=""
local EPOCH=""
RPM=$(echo $rpm_filename | rev | cut -d'.' -f-1 | rev)
SFILE=$(echo $rpm_filename | rev | cut -d'.' -f2- | rev)
ARCH=$(echo $SFILE | rev | cut -d'.' -f-1 | rev)
SFILE=$(echo $SFILE | rev | cut -d'.' -f2- | rev)
RELEASE=$(echo $SFILE | rev | cut -d'-' -f-1 | rev)
SFILE=$(echo $SFILE | rev | cut -d'-' -f2- | rev)
VERSION=$(echo $SFILE | rev | cut -d'-' -f-1 | rev)
NAME=$(echo $SFILE | rev | cut -d'-' -f2- | rev)
if [[ $NAME = *":"* ]]; then
EPOCH=$(echo $NAME | cut -d':' -f-1)
NAME=$(echo $NAME | cut -d':' -f2-)
fi
echo "$NAME" "$VERSION" "$RELEASE" "$ARCH" "$EPOCH"
}
# Function to predict the URL where a rpm might be found.
# Assumes the rpm was compile for EPEL by fedora's koji.
koji_url () {
local rpm_filename=$1
local arr=( $(split_filename $rpm_filename) )
local n=${arr[0]}
local v=${arr[1]}
local r=${arr[2]}
local a=${arr[3]}
echo "https://kojipkgs.fedoraproject.org/packages/$n/$v/$r/$a/$n-$v-$r.$a.rpm"
}
get_dest_directory() {
local _type=$1
local _dest=""
if [ "$_type" == "src" ]; then
_dest="$MDIR_SRC"
else
_dest="$MDIR_BIN/$_type"
fi
echo "$_dest"
}
process_result() {
local _type="$1"
local dest_dir="$2"
local url="$3"
local sfile="$4"
if [ "$_type" != "src" ] && [ ! -d $dest_dir ]; then
mkdir -p $dest_dir
fi
echo "url_srpm:$url"
if ! mv -f $sfile* $dest_dir ; then
echo "FAILED to move $rpm_name"
echo "fail_move_srpm:$rpm_name" >> $LOG
return 1
fi
echo "found_srpm:$rpm_name"
echo $rpm_name >> $FOUND_SRPMS
return 0
}
get_download_cmd() {
local ff="$1"
local _level="$2"
# Decide if the list will be downloaded using yumdownloader or wget
if [[ $ff != *"#"* ]]; then
rpm_name=$ff
if [ $_level == "K1" ]; then
download_cmd="$(get_wget_command $rpm_name)"
else
# yumdownloader with the appropriate flag for src, noarch or x86_64
download_cmd="${SUDOCMD} $(get_yum_command $rpm_name $_level)"
fi
else
# Build wget command
rpm_url=$(get_url "$ff" "$_level")
download_cmd="$(get_wget_command $rpm_url)"
fi
echo "$download_cmd"
}
get_rpm_name() {
local ret=""
if [[ "$1" != *"#"* ]]; then
ret="$1"
else
ret="$(echo $1 | cut -d"#" -f1-1)"
fi
echo "$ret"
}
get_arch_from_rpm() {
local _file=$1
local _split=()
local _arch=""
if [[ "$1" == *"#"* ]]; then
_file=$(echo $_file | cut -d"#" -f1-1)
fi
_split=( $(split_filename $_file) )
_arch=${_split[3]}
echo "$_arch"
}
get_from() {
list=$1
base=$(basename $list .lst) # removing lst extension
base=$(basename $base .log) # removing log extension
from=$(echo $base | rev | cut -d'_' -f1-1 | rev)
echo $from
}