starlingx/tools
Code Issues Proposed changes

Build layering, script and lst update

Browse Source 
Script changes to download content by layer.
Valid options are 'all', 'compiler', 'distro', 'flock'.

Current .lst and yum files under directory centos-mirror-tools
are relocated.  Generic package dependencies are relocated to
centos-mirror-tools/config/<os>/<layer>/ .

Lst entries for compilable content have been relocated to other
git repos by prior updates.  i.e.  those that list tarballs or
srpms to be compiled within that repo.

The original .lst files are deleted to make it easier
to identify new content during development.

Layer 'all' builds all layers in a single workspace.  The
lst files are identical to current content, minus the src.rpm and
tarball entries.

Other layers get only a subset of packages download. The minimum
required to build the layer.  The 'flock' layer will have additional
content to satisfy the run time requirements as well as the build
time requirements.

An upper layer does not need to list rpms known to be provided by
a lower layer.  Instead the config file 'required_layer_pkgs.cfg'
lists urls for lst files for lower layer build outputs.
These build outputs are generated and published by cengn for
each layer.

A second layer config file, 'required_layer_iso_inc.cfg' lists
image.inc files for lower layer builds.  These build outputs are
generated and published by cengn for each layer, summarizing
the image.inc files found in individual git repos.

Image.inc files inform the build-iso process, listing rpms that
that provide services and commands that need to be included in
the iso.  The transitive list of required rpms need not be listed.

Finally the layer config should include a yum.repos.d
directory in which supplementary yum repos are defined to
pick up cengn built content from lower layers.

To allow a designer to do cross-layer building using local sources
rather than those provided by CENGN, there are several options.

The designer can modify the urls for lower layer build outputs,
as found in the .cfg and .repo files within the config directory
'stx-tools/centos-mirror-tools/config/<distro>/<layer-to-build>'
directly within the git.  Substitute urls can use the file:///
syntax.  Just be sure to remove these changes before submitting.

Alternatively new args have been added to download_mirror.sh,
generate-cgcs-centos-repo.sh and commands that override the
normal config.

The easiest to use is a command argurement that substitutes a new
config directory, replacing stx-tools/centos-mirror-tools/config.
The intent is for the designer to do a recursive copy of that
directory into a side location. make his changes there, outside of
git, and provide the path to that directory as an extra arguement
to download_mirror.sh and generate-cgcs-centos-repo.sh.

e.g. For simplicity I'll only list the 'extra' arguements

download_mirror.sh -C <my-config-dir> \
                   -l <layer> \
                   ...

generate-cgcs-centos-repo.sh --config-dir=<my-config-dir> \
                             --layer=<layer> \
                             ...

populate_downloads.sh --config-dir=<my-config-dir> \
                      --layer=<layer> \
                      ...

These arguements can also be suplied via the environment.
For the purpose of containerized builds, these arguements
should be defined in your localrc.

e.g.
export STX_CONFIG_DIR=<my-config-dir>
export LAYER=<layer>

The final alternative is to override things at a more granular level,
replacing a single lst file of image.inc file.  Here you can replace
a single line found in a required_layer_pkgs.cfg or
required_layer_iso_inc.cfg file.

e.g. We are doing a flock build and want to modify the content picked up
from the distro layer's rt build, and that content delivers a service
we want in the iso.  For simplicity I'll only list the 'extra' arguments

./download_mirror.sh -l flock \
    -L distro,rt,file:///<my-distro-workspace>/rt/rpmbuild/RPMS/rpm.lst \
    -I distro,std,file:///<my-distro-workspace>/rt/image.inc \
    ...

generate-cgcs-centos-repo.sh --layer=flock \
    --layer-pkg-url=distro,rt,file:///<my-distro-workspace>/rt/rpmbuild/RPMS/rpm.lst \
    --layer-inc-url=distro,std,file:////<my-distro-workspace>/rt/image.inc \
    ...

NOTE: The triplet syntax for a package list url is
    <lower-layer>,<build-type>,<url-to-rpm.lst>

    lower-layer: 'compiler', 'distro'
    build-type: 'std', 'rt', 'installer'

    Also if 'file:///' syntax is used, a matching change is made to
    the yum *.repo file.  This assumes that the rpm.lst is co-resident with
    repodata directory, as is the norm for our build outputs.

NOTE: The triplet syntax for a image inc url is
    <lower-layer>,<include-type>,<url-to-image.inc>

    lower-layer: 'compiler', 'distro'
    build-type: 'std', 'dev'

A typical user is likely only working in the flock layer on the master
branch.  He should be content to use the compiler and distro layer
outputs from cengn.

His workflow looks like ...

1, sync code for flock layer
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m flock.xml
$ repo sync

2, download rpms for flock layer, and populate a local mirror
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S -l flock
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/

3, Prepare a virtual repo and downloads directory for building
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --layer=flock $LOCAL_MIRROR
$ ./populate_downloads.sh --layer=flock $LOCAL_MIRROR

4, rpm package and iso building
$ build-pkgs && build-iso && build-helm-charts.sh

Building all layers in a single workspace is still supported, and
looks identical to the previous workflow.

1, sync code
$ repo init -u https://opendev.org/starlingx/manifest.git -b master
$ repo sync

2, download rpms for flock layer, and populate a local mirror
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/

3, create repo named "StxCentos7Distro" for building
$ cd ../toCOPY
$ generate-cgcs-centos-repo.sh $LOCAL_MIRROR
$ populate_downloads.sh $LOCAL_MIRROR

4, rpm package and iso building
$ build-pkgs && build-iso && build-helm-charts.sh

Only a cross-layer developer should setup two or three copies of the
building environment, one per layer.  We suggest you use seperate shells
for each layer, as the various paths (MY_REPO, MY_WORKSPACE ...) need to
be unique,

Shell 1, compiler layer
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ LOCAL_CONFIG=<some-dir>/config
$ MY_REPO_ROOT_DIR=<some-dir>/layer-compiler
$ MY_REPO=$MY_REPO_ROOT_DIR/cgcs-root
$ MY_WORKSPACE=$MY_REPO_ROOT_DIR/workspace
  ...

$ mkdir -p $MY_REPO_ROOT_DIR
$ cd $MY_REPO_ROOT_DIR
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m compiler.xml
$ cd stx-tools/centos-mirror-tools
$ cp -r config/* $LOCAL_CONFIG
 ... edit urls in *.cfg and *.repo files under $LOCAL_CONFIG ...

$ ./download_mirror.sh -n -g -c yum.conf.sample -S -C $LOCAL_CONFIG -l compiler
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --config-dir=$LOCAL_CONFIG --layer=compiler $LOCAL_MIRROR
$ ./populate_downloads.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ build-pkgs

Shell 2, distro layer
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ LOCAL_CONFIG=<some-dir>/config
$ MY_REPO_ROOT_DIR=<some-dir>/layer-distro
$ MY_REPO=$MY_REPO_ROOT_DIR/cgcs-root
$ MY_WORKSPACE=$MY_REPO_ROOT_DIR/workspace
  ...
$ mkdir -p $MY_REPO_ROOT_DIR
$ cd $MY_REPO_ROOT_DIR
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m distro.xml
$ repo sync
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S -C $LOCAL_CONFIG -l distro
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --config-dir=$LOCAL_CONFIG --layer=distro $LOCAL_MIRROR
$ ./populate_downloads.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ build-pkgs

Shell 3, flock layer
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ LOCAL_CONFIG=<some-dir>/config
$ MY_REPO_ROOT_DIR=<some-dir>/layer-flock
$ MY_REPO=$MY_REPO_ROOT_DIR/cgcs-root
$ MY_WORKSPACE=$MY_REPO_ROOT_DIR/workspace
  ...
$ mkdir -p $MY_REPO_ROOT_DIR
$ cd $MY_REPO_ROOT_DIR
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m flock.xml
$ repo sync
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S -C $LOCAL_CONFIG -l flock
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ ./populate_downloads.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ build-pkgs && build-iso && build-helm-charts.sh

Story: 2006166
Task: 37103

Depends-On: https://review.opendev.org/698756
Depends-On: https://review.opendev.org/700819
Depends-On: https://review.opendev.org/700821
Change-Id: I088020b81f08656e50aa29b5584bbc1dd1378f12
Signed-off-by: Scott Little <scott.little@windriver.com>
This commit is contained in:
Scott Little 2019-09-12 11:57:25 -04:00
parent 9d3df76891
commit d51f8050cd
33 changed files with 3337 additions and 1194 deletions

3
Dockerfile

@ -108,6 +108,7 @@ COPY toCOPY/finishSetup.sh /usr/local/bin
COPY toCOPY/populate_downloads.sh /usr/local/bin
COPY toCOPY/generate-cgcs-tis-repo /usr/local/bin
COPY toCOPY/generate-cgcs-centos-repo.sh /usr/local/bin
COPY toCOPY/lst_utils.sh /usr/local/bin
COPY toCOPY/.inputrc /home/$MYUNAME/
COPY toCOPY/builder-constraints.txt /home/$MYUNAME/
@ -158,7 +159,7 @@ RUN echo "# Load stx-builder configuration" >> /etc/profile.d/TC.sh && \
echo "if [[ -r \${HOME}/buildrc ]]; then" >> /etc/profile.d/TC.sh && \
echo " source \${HOME}/buildrc" >> /etc/profile.d/TC.sh && \
echo " export PROJECT SRC_BUILD_ENVIRONMENT MYPROJECTNAME MYUNAME" >> /etc/profile.d/TC.sh && \
echo " export MY_BUILD_CFG MY_BUILD_CFG_RT MY_BUILD_CFG_STD MY_BUILD_DIR MY_BUILD_ENVIRONMENT MY_BUILD_ENVIRONMENT_FILE MY_BUILD_ENVIRONMENT_FILE_RT MY_BUILD_ENVIRONMENT_FILE_STD MY_DEBUG_BUILD_CFG_RT MY_DEBUG_BUILD_CFG_STD MY_LOCAL_DISK MY_MOCK_ROOT MY_REPO MY_REPO_ROOT_DIR MY_SRC_RPM_BUILD_DIR MY_TC_RELEASE MY_WORKSPACE" >> /etc/profile.d/TC.sh && \
echo " export MY_BUILD_CFG MY_BUILD_CFG_RT MY_BUILD_CFG_STD MY_BUILD_DIR MY_BUILD_ENVIRONMENT MY_BUILD_ENVIRONMENT_FILE MY_BUILD_ENVIRONMENT_FILE_RT MY_BUILD_ENVIRONMENT_FILE_STD MY_DEBUG_BUILD_CFG_RT MY_DEBUG_BUILD_CFG_STD MY_LOCAL_DISK MY_MOCK_ROOT MY_REPO MY_REPO_ROOT_DIR MY_SRC_RPM_BUILD_DIR MY_TC_RELEASE MY_WORKSPACE LAYER" >> /etc/profile.d/TC.sh && \
echo "fi" >> /etc/profile.d/TC.sh && \
echo "export FORMAL_BUILD=0" >> /etc/profile.d/TC.sh && \
echo "export PATH=\$MY_REPO/build-tools:\$PATH" >> /etc/profile.d/TC.sh

2
buildrc

@ -82,3 +82,5 @@ export MY_BUILD_CFG_RT=${MY_WORKSPACE}/rt/${MY_BUILD_ENVIRONMENT_FILE_RT}
export MY_MOCK_ROOT=$MY_WORKSPACE/mock/root
export MY_DEBUG_BUILD_CFG_STD=${MY_WORKSPACE}/std/configs/${MY_BUILD_ENVIRONMENT}-std/${MY_BUILD_ENVIRONMENT}-std.b0.cfg
export MY_DEBUG_BUILD_CFG_RT=${MY_WORKSPACE}/std/configs/${MY_BUILD_ENVIRONMENT}-std/${MY_BUILD_ENVIRONMENT}-rt.b0.cfg
export LAYER=$LAYER
export STX_CONFIG_DIR="$STX_CONFIG_DIR"

287
centos-mirror-tools/config/centos/compiler/rpms_centos.lst Normal file

@ -0,0 +1,287 @@
apr-1.4.8-3.el7_4.1.x86_64.rpm
apr-util-1.5.2-6.el7.x86_64.rpm
atk-devel-2.28.1-1.el7.x86_64.rpm
at-spi2-atk-devel-2.26.2-1.el7.x86_64.rpm
autoconf-2.69-11.el7.noarch.rpm
automake-1.13.4-3.el7.noarch.rpm
# basesystem-10.0-7.el7.centos.noarch.rpm provided by mock
bash-4.2.46-31.el7.x86_64.rpm
# binutils-2.27-34.base.el7.x86_64.rpm provided by mock
binutils-devel-2.27-34.base.el7.x86_64.rpm
bison-3.0.4-2.el7.x86_64.rpm
bluez-libs-5.44-4.el7_4.x86_64.rpm
bluez-libs-devel-5.44-4.el7_4.x86_64.rpm
# bzip2-1.0.6-13.el7.x86_64.rpm provided by mock
bzip2-devel-1.0.6-13.el7.x86_64.rpm
# bzip2-libs-1.0.6-13.el7.x86_64.rpm provided by mock
# ca-certificates-2018.2.22-70.0.el7_5.noarch.rpm provided by mock
# centos-release-7-6.1810.2.el7.centos.x86_64.rpm provided by mock
# chkconfig-1.7.4-1.el7.x86_64.rpm provided by mock
# coreutils-8.22-23.el7.x86_64.rpm provided by mock
# cpio-2.11-27.el7.x86_64.rpm provided by mock
# cpp-4.8.5-36.el7.x86_64.rpm provided by mock
# cracklib-2.9.0-11.el7.x86_64.rpm provided by mock
# cracklib-dicts-2.9.0-11.el7.x86_64.rpm provided by mock
# cryptsetup-libs-2.0.3-3.el7.x86_64.rpm provided by mock
# curl-7.29.0-51.el7_6.3.x86_64.rpm provided by mock
# cyrus-sasl-lib-2.1.26-23.el7.x86_64.rpm provided by mock
# dbus-1.10.24-12.el7.x86_64.rpm provided by mock
dbus-devel-1.10.24-12.el7.x86_64.rpm
# dbus-libs-1.10.24-12.el7.x86_64.rpm provided by mock
dejavu-fonts-common-2.33-6.el7.noarch.rpm
dejavu-sans-fonts-2.33-6.el7.noarch.rpm
# device-mapper-1.02.146-4.el7.x86_64.rpm provided by mock
device-mapper-devel-1.02.146-4.el7.x86_64.rpm
device-mapper-event-1.02.146-4.el7.x86_64.rpm
device-mapper-event-libs-1.02.146-4.el7.x86_64.rpm
# device-mapper-libs-1.02.146-4.el7.x86_64.rpm provided by mock
device-mapper-multipath-0.4.9-119.el7.x86_64.rpm
device-mapper-multipath-libs-0.4.9-119.el7.x86_64.rpm
device-mapper-persistent-data-0.7.3-3.el7.x86_64.rpm
# diffutils-3.3-4.el7.x86_64.rpm provided by mock
# dracut-033-554.el7.x86_64.rpm provided by mock
# expat-2.1.0-10.el7_3.x86_64.rpm provided by mock
expat-devel-2.1.0-10.el7_3.x86_64.rpm
file-devel-5.11-35.el7.x86_64.rpm
# filesystem-3.2-25.el7.x86_64.rpm provided by mock
# findutils-4.5.11-6.el7.x86_64.rpm provided by mock
fipscheck-1.4.1-6.el7.x86_64.rpm
fipscheck-lib-1.4.1-6.el7.x86_64.rpm
fontconfig-2.13.0-4.3.el7.x86_64.rpm
fontconfig-devel-2.13.0-4.3.el7.x86_64.rpm
fontpackages-filesystem-1.44-8.el7.noarch.rpm
freetype-2.8-12.el7.x86_64.rpm
freetype-devel-2.8-12.el7.x86_64.rpm
# gawk-4.0.2-4.el7_3.1.x86_64.rpm provided by mock
# gcc-4.8.5-36.el7.x86_64.rpm provided by mock
# gcc-c++-4.8.5-36.el7.x86_64.rpm provided by mock
# gdbm-1.10-8.el7.x86_64.rpm provided by mock
gdbm-devel-1.10-8.el7.x86_64.rpm
gettext-0.19.8.1-2.el7.x86_64.rpm
gettext-common-devel-0.19.8.1-2.el7.noarch.rpm
gettext-devel-0.19.8.1-2.el7.x86_64.rpm
gettext-libs-0.19.8.1-2.el7.x86_64.rpm
git-1.8.3.1-20.el7.x86_64.rpm
# glibc-2.17-260.el7.x86_64.rpm provided by mock
# glibc-common-2.17-260.el7.x86_64.rpm provided by mock
# glibc-devel-2.17-260.el7.x86_64.rpm provided by mock
# glibc-headers-2.17-260.el7.x86_64.rpm provided by mock
glibc-static-2.17-260.el7.x86_64.rpm
gl-manpages-1.1-7.20130122.el7.noarch.rpm
# gmp-6.0.0-15.el7.x86_64.rpm provided by mock
gmp-devel-6.0.0-15.el7.x86_64.rpm
gnutls-3.3.29-8.el7.x86_64.rpm
golang-1.11.2-3.el7.x86_64.rpm
golang-bin-1.11.2-3.el7.x86_64.rpm
golang-github-cpuguy83-go-md2man-1.0.4-4.el7.x86_64.rpm
golang-src-1.11.2-3.el7.noarch.rpm
# grep-2.20-3.el7.x86_64.rpm provided by mock
# groff-base-1.22.2-8.el7.x86_64.rpm provided by mock
# gzip-1.5-10.el7.x86_64.rpm provided by mock
# hardlink-1.0-19.el7.x86_64.rpm provided by mock
hwdata-0.252-9.1.el7.x86_64.rpm
# info-5.1-5.el7.x86_64.rpm provided by mock
# json-c-0.11-4.el7_0.x86_64.rpm provided by mock
json-c-devel-0.11-4.el7_0.x86_64.rpm
kernel-headers-3.10.0-1062.1.2.el7.x86_64.rpm
# keyutils-libs-1.5.8-3.el7.x86_64.rpm provided by mock
keyutils-libs-devel-1.5.8-3.el7.x86_64.rpm
# kmod-20-23.el7.x86_64.rpm provided by mock
# kmod-libs-20-23.el7.x86_64.rpm provided by mock
# kpartx-0.4.9-119.el7.x86_64.rpm provided by mock
krb5-devel-1.15.1-34.el7.x86_64.rpm
# krb5-libs-1.15.1-34.el7.x86_64.rpm provided by mock
less-458-9.el7.x86_64.rpm
# libacl-2.2.51-14.el7.x86_64.rpm provided by mock
libacl-devel-2.2.51-14.el7.x86_64.rpm
libarchive-devel-3.1.2-10.el7_2.x86_64.rpm
# libattr-2.4.46-13.el7.x86_64.rpm provided by mock
libattr-devel-2.4.46-13.el7.x86_64.rpm
# libcap-2.22-9.el7.x86_64.rpm provided by mock
libcap-devel-2.22-9.el7.x86_64.rpm
# libcap-ng-0.7.5-4.el7.x86_64.rpm provided by mock
libcap-ng-devel-0.7.5-4.el7.x86_64.rpm
libcroco-0.6.12-4.el7.x86_64.rpm
# libcurl-7.29.0-51.el7_6.3.x86_64.rpm provided by mock
# libcurl-devel-7.29.0-51.el7_6.3.x86_64.rpm provided by mock
# libdb-5.3.21-24.el7.x86_64.rpm provided by mock
libdb-devel-5.3.21-24.el7.x86_64.rpm
# libdb-utils-5.3.21-24.el7.x86_64.rpm provided by mock
libedit-3.0-12.20121213cvs.el7.x86_64.rpm
# libffi-3.0.13-18.el7.x86_64.rpm provided by mock
libffi-devel-3.0.13-18.el7.x86_64.rpm
# libgcc-4.8.5-36.el7.x86_64.rpm provided by mock
# libgcrypt-1.5.3-14.el7.x86_64.rpm provided by mock
libglvnd-1.0.1-0.8.git5baa1e5.el7.x86_64.rpm
libglvnd-core-devel-1.0.1-0.8.git5baa1e5.el7.x86_64.rpm
libglvnd-devel-1.0.1-0.8.git5baa1e5.el7.x86_64.rpm
libglvnd-egl-1.0.1-0.8.git5baa1e5.el7.x86_64.rpm
libglvnd-gles-1.0.1-0.8.git5baa1e5.el7.x86_64.rpm
libglvnd-glx-1.0.1-0.8.git5baa1e5.el7.x86_64.rpm
libglvnd-opengl-1.0.1-0.8.git5baa1e5.el7.x86_64.rpm
# libgomp-4.8.5-36.el7.x86_64.rpm provided by mock
# libgpg-error-1.12-3.el7.x86_64.rpm provided by mock
libgpg-error-devel-1.12-3.el7.x86_64.rpm
# libidn-1.28-4.el7.x86_64.rpm provided by mock
libkadm5-1.15.1-34.el7.x86_64.rpm
libmodman-2.0.1-8.el7.x86_64.rpm
# libmount-2.23.2-59.el7.x86_64.rpm provided by mock
# libmpc-1.0.1-3.el7.x86_64.rpm provided by mock
libpciaccess-0.14-1.el7.x86_64.rpm
libpng-1.5.13-7.el7_2.x86_64.rpm
libpng-devel-1.5.13-7.el7_2.x86_64.rpm
libproxy-0.4.11-11.el7.x86_64.rpm
# libpwquality-1.2.3-5.el7.x86_64.rpm provided by mock
# libselinux-2.5-14.1.el7.x86_64.rpm provided by mock
libselinux-devel-2.5-14.1.el7.x86_64.rpm
# libsepol-2.5-10.el7.x86_64.rpm provided by mock
libsepol-devel-2.5-10.el7.x86_64.rpm
# libsmartcols-2.23.2-59.el7.x86_64.rpm provided by mock
# libssh2-1.4.3-12.el7.x86_64.rpm provided by mock
# libstdc++-4.8.5-36.el7.x86_64.rpm provided by mock
# libstdc++-devel-4.8.5-36.el7.x86_64.rpm provided by mock
# libtasn1-4.10-1.el7.x86_64.rpm provided by mock
libtool-2.4.2-22.el7_3.x86_64.rpm
libunistring-0.9.3-9.el7.x86_64.rpm
# libuser-0.60-9.el7.x86_64.rpm provided by mock
# libutempter-1.1.6-4.el7.x86_64.rpm provided by mock
# libuuid-2.23.2-59.el7.x86_64.rpm provided by mock
libuuid-devel-2.23.2-59.el7.x86_64.rpm
# libverto-0.2.5-4.el7.x86_64.rpm provided by mock
libverto-devel-0.2.5-4.el7.x86_64.rpm
libwayland-client-1.15.0-1.el7.x86_64.rpm
libwayland-server-1.15.0-1.el7.x86_64.rpm
libX11-1.6.7-2.el7.x86_64.rpm
libX11-common-1.6.7-2.el7.noarch.rpm
libX11-devel-1.6.7-2.el7.x86_64.rpm
libXau-1.0.8-2.1.el7.x86_64.rpm
libXau-devel-1.0.8-2.1.el7.x86_64.rpm
libxcb-1.13-1.el7.x86_64.rpm
libxcb-devel-1.13-1.el7.x86_64.rpm
libXdamage-1.1.4-4.1.el7.x86_64.rpm
libXdamage-devel-1.1.4-4.1.el7.x86_64.rpm
libXext-1.3.3-3.el7.x86_64.rpm
libXext-devel-1.3.3-3.el7.x86_64.rpm
libXfixes-5.0.3-1.el7.x86_64.rpm
libXfixes-devel-5.0.3-1.el7.x86_64.rpm
libXft-2.3.2-2.el7.x86_64.rpm
libXft-devel-2.3.2-2.el7.x86_64.rpm
# libxml2-2.9.1-6.el7_2.3.x86_64.rpm provided by mock
libXrender-0.9.10-1.el7.x86_64.rpm
libXrender-devel-0.9.10-1.el7.x86_64.rpm
libxshmfence-1.2-1.el7.x86_64.rpm
libXxf86vm-1.1.4-1.el7.x86_64.rpm
libXxf86vm-devel-1.1.4-1.el7.x86_64.rpm
# lua-5.1.4-15.el7.x86_64.rpm provided by mock
lua-devel-5.1.4-15.el7.x86_64.rpm
# lz4-1.7.5-2.el7.x86_64.rpm provided by mock
m4-1.4.16-10.el7.x86_64.rpm
mercurial-2.6.2-8.el7_4.x86_64.rpm
mesa-libGLU-9.0.0-4.el7.x86_64.rpm
mesa-libGLU-devel-9.0.0-4.el7.x86_64.rpm
# mpfr-3.1.1-4.el7.x86_64.rpm provided by mock
# ncurses-5.9-14.20130511.el7_4.x86_64.rpm provided by mock
# ncurses-base-5.9-14.20130511.el7_4.noarch.rpm provided by mock
ncurses-devel-5.9-14.20130511.el7_4.x86_64.rpm
# ncurses-libs-5.9-14.20130511.el7_4.x86_64.rpm provided by mock
neon-0.30.0-3.el7.x86_64.rpm
nettle-2.7.1-8.el7.x86_64.rpm
# nspr-4.19.0-1.el7_5.x86_64.rpm provided by mock
nspr-devel-4.19.0-1.el7_5.x86_64.rpm
# nss-3.36.0-7.el7_5.x86_64.rpm provided by mock
nss-devel-3.36.0-7.el7_5.x86_64.rpm
# nss-pem-1.0.3-5.el7.x86_64.rpm provided by mock
# nss-sysinit-3.36.0-7.el7_5.x86_64.rpm provided by mock
# nss-tools-3.36.0-7.el7_5.x86_64.rpm provided by mock
# nss-util-3.36.0-1.el7_5.x86_64.rpm provided by mock
nss-util-devel-3.36.0-1.el7_5.x86_64.rpm
openssh-7.4p1-16.el7.x86_64.rpm
openssh-clients-7.4p1-16.el7.x86_64.rpm
openssl-devel-1.0.2k-16.el7.x86_64.rpm
# openssl-libs-1.0.2k-16.el7.x86_64.rpm provided by mock
# p11-kit-0.23.5-3.el7.x86_64.rpm provided by mock
p11-kit-devel-0.23.5-3.el7.x86_64.rpm
# p11-kit-trust-0.23.5-3.el7.x86_64.rpm provided by mock
pakchois-0.4-10.el7.x86_64.rpm
# pam-1.1.8-22.el7.x86_64.rpm provided by mock
# pcre-8.32-17.el7.x86_64.rpm provided by mock
pcre-devel-8.32-17.el7.x86_64.rpm
# perl-5.16.3-294.el7_6.x86_64.rpm provided by mock
# perl-Carp-1.26-244.el7.noarch.rpm provided by mock
# perl-constant-1.27-2.el7.noarch.rpm provided by mock
perl-Data-Dumper-2.145-3.el7.x86_64.rpm
# perl-Encode-2.51-7.el7.x86_64.rpm provided by mock
perl-Encode-Locale-1.03-5.el7.noarch.rpm
perl-Error-0.17020-2.el7.noarch.rpm
# perl-Exporter-5.68-3.el7.noarch.rpm provided by mock
# perl-File-Path-2.09-2.el7.noarch.rpm provided by mock
# perl-File-Temp-0.23.01-3.el7.noarch.rpm provided by mock
# perl-Filter-1.49-3.el7.x86_64.rpm provided by mock
# perl-Getopt-Long-2.40-3.el7.noarch.rpm provided by mock
perl-Git-1.8.3.1-20.el7.noarch.rpm
# perl-HTTP-Tiny-0.033-3.el7.noarch.rpm provided by mock
perl-libintl-1.20-12.el7.x86_64.rpm
# perl-libs-5.16.3-294.el7_6.x86_64.rpm provided by mock
# perl-macros-5.16.3-294.el7_6.x86_64.rpm provided by mock
# perl-parent-0.225-244.el7.noarch.rpm provided by mock
# perl-PathTools-3.40-5.el7.x86_64.rpm provided by mock
# perl-Pod-Escapes-1.04-294.el7_6.noarch.rpm provided by mock
# perl-podlators-2.5.1-3.el7.noarch.rpm provided by mock
# perl-Pod-Perldoc-3.20-4.el7.noarch.rpm provided by mock
# perl-Pod-Simple-3.28-4.el7.noarch.rpm provided by mock
# perl-Pod-Usage-1.63-3.el7.noarch.rpm provided by mock
# perl-Scalar-List-Utils-1.27-248.el7.x86_64.rpm provided by mock
# perl-Socket-2.010-4.el7.x86_64.rpm provided by mock
# perl-Storable-2.45-3.el7.x86_64.rpm provided by mock
perl-TermReadKey-2.30-20.el7.x86_64.rpm
perl-Test-Harness-3.28-3.el7.noarch.rpm
# perl-Text-ParseWords-3.29-4.el7.noarch.rpm provided by mock
perl-Text-Unidecode-0.04-20.el7.noarch.rpm
# perl-threads-1.87-4.el7.x86_64.rpm provided by mock
# perl-threads-shared-1.43-6.el7.x86_64.rpm provided by mock
# perl-Time-HiRes-1.9725-3.el7.x86_64.rpm provided by mock
# perl-Time-Local-1.2300-2.el7.noarch.rpm provided by mock
# pkgconfig-0.27.1-4.el7.x86_64.rpm provided by mock
# popt-1.13-16.el7.x86_64.rpm provided by mock
popt-devel-1.13-16.el7.x86_64.rpm
# procps-ng-3.3.10-23.el7.x86_64.rpm provided by mock
pyparsing-2.1.10-3.el7.noarch.rpm
python2-pyparsing-2.1.10-3.el7.noarch.rpm
python-2.7.5-76.el7.x86_64.rpm
python-devel-2.7.5-76.el7.x86_64.rpm
python-libs-2.7.5-76.el7.x86_64.rpm
# qrencode-libs-3.4.1-3.el7.x86_64.rpm provided by mock
# readline-6.2-10.el7.x86_64.rpm provided by mock
readline-devel-6.2-10.el7.x86_64.rpm
rpm-4.11.3-35.el7.x86_64.rpm
rpm-build-4.11.3-35.el7.x86_64.rpm
rpm-build-libs-4.11.3-35.el7.x86_64.rpm
rpm-libs-4.11.3-35.el7.x86_64.rpm
rpm-python-4.11.3-35.el7.x86_64.rpm
rsync-3.1.2-4.el7.x86_64.rpm
# sed-4.2.2-5.el7.x86_64.rpm provided by mock
# setup-2.8.71-10.el7.noarch.rpm provided by mock
# shadow-utils-4.1.5.1-25.el7.x86_64.rpm provided by mock
# shared-mime-info-1.8-4.el7.x86_64.rpm provided by mock
# sqlite-3.7.17-8.el7.x86_64.rpm provided by mock
sqlite-devel-3.7.17-8.el7.x86_64.rpm
subversion-1.7.14-14.el7.x86_64.rpm
subversion-libs-1.7.14-14.el7.x86_64.rpm
# tar-1.26-35.el7.x86_64.rpm provided by mock
tcl-8.5.13-8.el7.x86_64.rpm
tcl-devel-8.5.13-8.el7.x86_64.rpm
texinfo-5.1-5.el7.x86_64.rpm
tix-8.4.3-12.el7.x86_64.rpm
tix-devel-8.4.3-12.el7.x86_64.rpm
tk-8.5.13-6.el7.x86_64.rpm
tk-devel-8.5.13-6.el7.x86_64.rpm
trousers-0.3.14-2.el7.x86_64.rpm
# tzdata-2018g-1.el7.noarch.rpm provided by mock
# ustr-1.0.4-16.el7.x86_64.rpm provided by mock
ustr-devel-1.0.4-16.el7.x86_64.rpm
xorg-x11-proto-devel-2018.4-1.el7.noarch.rpm
# xz-5.2.2-1.el7.x86_64.rpm provided by mock
xz-devel-5.2.2-1.el7.x86_64.rpm
# xz-libs-5.2.2-1.el7.x86_64.rpm provided by mock
# zlib-1.2.7-18.el7.x86_64.rpm provided by mock
zlib-devel-1.2.7-18.el7.x86_64.rpm

38
centos-mirror-tools/config/centos/compiler/rpms_centos3rdparties.lst Normal file

@ -0,0 +1,38 @@
# audit-libs-2.8.4-4.el7.x86_64.rpm provided by mock
audit-libs-devel-2.8.4-4.el7.x86_64.rpm
audit-libs-python-2.8.4-4.el7.x86_64.rpm
devtoolset-7-gcc-c++-7.3.1-5.10.el7.x86_64.rpm
devtoolset-7-libstdc++-devel-7.3.1-5.10.el7.x86_64.rpm
# elfutils-default-yama-scope-0.176-2.el7.noarch.rpm provided by mock
# elfutils-devel-0.176-2.el7.x86_64.rpm provided by mock
# elfutils-libelf-0.176-2.el7.x86_64.rpm provided by mock
# elfutils-libelf-devel-0.176-2.el7.x86_64.rpm provided by mock
# elfutils-libs-0.176-2.el7.x86_64.rpm provided by mock
# glib2-2.56.1-2.el7.x86_64.rpm provided by mock
ima-evm-utils-devel-1.1-2.el7.x86_64.rpm
# libblkid-2.23.2-59.el7.x86_64.rpm provided by mock
# libcom_err-1.42.9-13.el7.x86_64.rpm provided by mock
libcom_err-devel-1.42.9-13.el7.x86_64.rpm
libdrm-2.4.91-3.el7.x86_64.rpm
libdrm-devel-2.4.91-3.el7.x86_64.rpm
# libsemanage-2.5-14.el7.x86_64.rpm provided by mock
libsemanage-devel-2.5-14.el7.x86_64.rpm
mesa-libEGL-18.0.5-3.el7.x86_64.rpm
mesa-libEGL-devel-18.0.5-3.el7.x86_64.rpm
mesa-libgbm-18.0.5-3.el7.x86_64.rpm
mesa-libGL-18.0.5-3.el7.x86_64.rpm
mesa-libglapi-18.0.5-3.el7.x86_64.rpm
mesa-libGL-devel-18.0.5-3.el7.x86_64.rpm
# nss-softokn-3.36.0-5.el7_5.x86_64.rpm provided by mock
nss-softokn-devel-3.36.0-5.el7_5.x86_64.rpm
# nss-softokn-freebl-3.36.0-5.el7_5.x86_64.rpm provided by mock
nss-softokn-freebl-devel-3.36.0-5.el7_5.x86_64.rpm
# openldap-2.4.44-20.el7.x86_64.rpm provided by mock
# systemd-219-67.el7.x86_64.rpm provided by mock
# systemd-devel-219-67.el7.x86_64.rpm provided by mock
# systemd-libs-219-67.el7.x86_64.rpm provided by mock
# systemd-sysv-219-67.el7.x86_64.rpm provided by mock
systemtap-sdt-devel-3.3-3.el7.x86_64.rpm
# util-linux-2.23.2-59.el7.x86_64.rpm provided by mock
valgrind-3.13.0-13.el7.x86_64.rpm
valgrind-devel-3.13.0-13.el7.x86_64.rpm

0
centos-mirror-tools/other_downloads.lst → centos-mirror-tools/config/centos/distro/other_downloads.lst

2
centos-mirror-tools/config/centos/distro/required_layer_iso_inc.cfg Normal file

@ -0,0 +1,2 @@
compiler,std,http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/compiler/latest_build/outputs/image.inc
compiler,dev,http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/compiler/latest_build/outputs/image-dev.inc

1
centos-mirror-tools/config/centos/distro/required_layer_pkgs.cfg Normal file

@ -0,0 +1 @@
compiler,std,http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/compiler/latest_build/outputs/RPMS/std/rpm.lst

7
centos-mirror-tools/config/centos/distro/rpms_3rdparties.lst Normal file

@ -0,0 +1,7 @@
python2-openstackdocstheme-1.23.2-1.el7.noarch.rpm#https://cbs.centos.org/kojifiles/packages/python-openstackdocstheme/1.23.2/1.el7/noarch/python2-openstackdocstheme-1.23.2-1.el7.noarch.rpm
python2-pip-9.0.3-6.fc29.noarch.rpm#https://kojipkgs.fedoraproject.org/packages/python-pip/9.0.3/6.fc29/noarch/python2-pip-9.0.3-6.fc29.noarch.rpm
python2-pytest-httpbin-0.2.3-6.el7.noarch.rpm#http://cbs.centos.org/kojifiles/packages/python-pytest-httpbin/0.2.3/6.el7/noarch/python2-pytest-httpbin-0.2.3-6.el7.noarch.rpm
python2-pytest-mock-1.6.0-2.el7.noarch.rpm#http://cbs.centos.org/kojifiles/packages/python-pytest-mock/1.6.0/2.el7/noarch/python2-pytest-mock-1.6.0-2.el7.noarch.rpm
python2-wheel-0.31.1-3.fc30.noarch.rpm#https://kojipkgs.fedoraproject.org/packages/python-wheel/0.31.1/3.fc30/noarch/python2-wheel-0.31.1-3.fc30.noarch.rpm
python3-pip-9.0.3-6.fc29.noarch.rpm#https://kojipkgs.fedoraproject.org/packages/python-pip/9.0.3/6.fc29/noarch/python3-pip-9.0.3-6.fc29.noarch.rpm
python3-wheel-0.31.1-3.fc30.noarch.rpm#https://kojipkgs.fedoraproject.org/packages/python-wheel/0.31.1/3.fc30/noarch/python3-wheel-0.31.1-3.fc30.noarch.rpm

1095
centos-mirror-tools/rpms_centos.lst → centos-mirror-tools/config/centos/distro/rpms_centos.lst

File diff suppressed because it is too large Load Diff

90
centos-mirror-tools/config/centos/distro/rpms_centos3rdparties.lst Normal file

@ -0,0 +1,90 @@
adwaita-cursor-theme-3.28.0-1.el7.noarch.rpm
adwaita-icon-theme-3.28.0-1.el7.noarch.rpm
alsa-lib-1.1.6-2.el7.x86_64.rpm
alsa-lib-devel-1.1.6-2.el7.x86_64.rpm
# audit-libs-2.8.4-4.el7.x86_64.rpm provided by mock
audit-libs-devel-2.8.4-4.el7.x86_64.rpm
audit-libs-python-2.8.4-4.el7.x86_64.rpm
copy-jdk-configs-3.3-10.el7_5.noarch.rpm
CUnit-2.1.3-8.el7.x86_64.rpm
CUnit-devel-2.1.3-8.el7.x86_64.rpm
devtoolset-7-binutils-2.28-11.el7.x86_64.rpm
devtoolset-7-gcc-7.3.1-5.10.el7.x86_64.rpm
devtoolset-7-gcc-c++-7.3.1-5.10.el7.x86_64.rpm
devtoolset-7-libstdc++-devel-7.3.1-5.10.el7.x86_64.rpm
devtoolset-7-runtime-7.1-4.el7.x86_64.rpm
# elfutils-0.176-2.el7.x86_64.rpm provided by mock
# elfutils-default-yama-scope-0.176-2.el7.noarch.rpm provided by mock
# elfutils-devel-0.176-2.el7.x86_64.rpm provided by mock
# elfutils-libelf-0.176-2.el7.x86_64.rpm provided by mock
# elfutils-libelf-devel-0.176-2.el7.x86_64.rpm provided by mock
# elfutils-libs-0.176-2.el7.x86_64.rpm provided by mock
# glib2-2.56.1-2.el7.x86_64.rpm provided by mock
glib2-devel-2.56.1-2.el7.x86_64.rpm
glib2-doc-2.56.1-2.el7.noarch.rpm
gsettings-desktop-schemas-3.28.0-2.el7.x86_64.rpm
gssproxy-0.7.0-21.el7.x86_64.rpm
gtk3-3.22.30-3.el7.x86_64.rpm
gtk3-devel-3.22.30-3.el7.x86_64.rpm
gtk3-devel-docs-3.22.30-3.el7.x86_64.rpm
gtk-update-icon-cache-3.22.30-3.el7.x86_64.rpm
java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64.rpm
java-1.8.0-openjdk-devel-1.8.0.191.b12-1.el7_6.x86_64.rpm
java-1.8.0-openjdk-headless-1.8.0.191.b12-1.el7_6.x86_64.rpm
# libblkid-2.23.2-59.el7.x86_64.rpm provided by mock
# libcom_err-1.42.9-13.el7.x86_64.rpm provided by mock
libcom_err-devel-1.42.9-13.el7.x86_64.rpm
libdrm-2.4.91-3.el7.x86_64.rpm
libdrm-devel-2.4.91-3.el7.x86_64.rpm
liboath-2.4.1-9.el7.x86_64.rpm
liboath-devel-2.4.1-9.el7.x86_64.rpm
# libsemanage-2.5-14.el7.x86_64.rpm provided by mock
libsemanage-python-2.5-14.el7.x86_64.rpm
libss-1.42.9-13.el7.x86_64.rpm
libtalloc-2.1.13-1.el7.x86_64.rpm
libtevent-0.9.36-1.el7.x86_64.rpm
lvm2-2.02.177-4.el7.x86_64.rpm
lvm2-libs-2.02.177-4.el7.x86_64.rpm
mesa-libEGL-18.0.5-3.el7.x86_64.rpm
mesa-libEGL-devel-18.0.5-3.el7.x86_64.rpm
mesa-libgbm-18.0.5-3.el7.x86_64.rpm
mesa-libGL-18.0.5-3.el7.x86_64.rpm
mesa-libglapi-18.0.5-3.el7.x86_64.rpm
mesa-libGL-devel-18.0.5-3.el7.x86_64.rpm
NetworkManager-glib-1.12.0-8.el7_6.x86_64.rpm
NetworkManager-glib-devel-1.12.0-8.el7_6.x86_64.rpm
# nss-softokn-3.36.0-5.el7_5.x86_64.rpm provided by mock
nss-softokn-devel-3.36.0-5.el7_5.x86_64.rpm
# nss-softokn-freebl-3.36.0-5.el7_5.x86_64.rpm provided by mock
nss-softokn-freebl-devel-3.36.0-5.el7_5.x86_64.rpm
# openldap-2.4.44-20.el7.x86_64.rpm provided by mock
policycoreutils-2.5-29.el7.x86_64.rpm
policycoreutils-devel-2.5-29.el7.x86_64.rpm
policycoreutils-python-2.5-29.el7.x86_64.rpm
pulseaudio-libs-10.0-5.el7.x86_64.rpm
pulseaudio-libs-devel-10.0-5.el7.x86_64.rpm
pulseaudio-libs-glib2-10.0-5.el7.x86_64.rpm
pykickstart-1.99.66.19-2.el7.noarch.rpm
python2-msgpack-0.6.1-2.el7.x86_64.rpm
python2-pysocks-1.6.8-6.el7.noarch.rpm
python2-scapy-2.4.0-3.el7.noarch.rpm
python2-urllib3-1.21.1-1.el7.noarch.rpm
python-requests-mock-1.3.0-1.el7.noarch.rpm
rest-0.8.1-2.el7.x86_64.rpm
selinux-policy-3.13.1-229.el7_6.6.noarch.rpm
selinux-policy-devel-3.13.1-229.el7_6.6.noarch.rpm
spice-protocol-0.12.14-1.el7.noarch.rpm
spice-server-0.14.0-6.el7.x86_64.rpm
spice-server-devel-0.14.0-6.el7.x86_64.rpm
# systemd-219-67.el7.x86_64.rpm provided by mock
# systemd-devel-219-67.el7.x86_64.rpm provided by mock
# systemd-libs-219-67.el7.x86_64.rpm provided by mock
# systemd-sysv-219-67.el7.x86_64.rpm provided by mock
systemtap-3.3-3.el7.x86_64.rpm
systemtap-client-3.3-3.el7.x86_64.rpm
systemtap-devel-3.3-3.el7.x86_64.rpm
systemtap-runtime-3.3-3.el7.x86_64.rpm
systemtap-sdt-devel-3.3-3.el7.x86_64.rpm
# util-linux-2.23.2-59.el7.x86_64.rpm provided by mock
valgrind-3.13.0-13.el7.x86_64.rpm
valgrind-devel-3.13.0-13.el7.x86_64.rpm

12
centos-mirror-tools/config/centos/distro/yum.repos.d/StarlingX_cengn_compiler_layer.repo Normal file

@ -0,0 +1,12 @@
#####
# StarlingX compiler layer at CENGN
[Starlingx-cengn_compiler_std_layer]
name=Starlingx-cengn_compiler_std_layer
baseurl=http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/compiler/latest_build/outputs/RPMS/std/
enabled=1
[Starlingx-cengn_compiler_installer_layer]
name=Starlingx-cengn_compiler_installer_layer
baseurl=http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/compiler/latest_build/outputs/RPMS/installer/
enabled=1

29
centos-mirror-tools/config/centos/flock/other_downloads.lst Normal file

@ -0,0 +1,29 @@
folder:EFI
folder:EFI/BOOT
#file:EFI/BOOT/mmx64.efi
#file:EFI/BOOT/grubia32.efi
file:EFI/BOOT/grub.cfg
file:EFI/BOOT/BOOTX64.EFI
file:EFI/BOOT/grubx64.efi
#file:EFI/BOOT/BOOTIA32.EFI
#file:EFI/BOOT/mmia32.efi
folder:EFI/BOOT/fonts
file:EFI/BOOT/fonts/unicode.pf2
folder:LiveOS
file:LiveOS/squashfs.img
folder:images
folder:images/pxeboot
file:images/pxeboot/initrd.img
file:images/pxeboot/vmlinuz
file:images/efiboot.img
#file:images/boot.iso
folder:isolinux
file:isolinux/memtest
file:isolinux/grub.conf
file:isolinux/initrd.img
file:isolinux/boot.msg
file:isolinux/vmlinuz
file:isolinux/isolinux.bin
file:isolinux/splash.png
file:isolinux/isolinux.cfg
file:isolinux/vesamenu.c32

4
centos-mirror-tools/config/centos/flock/required_layer_iso_inc.cfg Normal file

@ -0,0 +1,4 @@
compiler,std,http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/compiler/latest_build/outputs/image.inc
compiler,dev,http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/compiler/latest_build/outputs/image-dev.inc
distro,std,http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/distro/latest_build/outputs/image.inc
distro,dev,http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/distro/latest_build/outputs/image-dev.inc

4
centos-mirror-tools/config/centos/flock/required_layer_pkgs.cfg Normal file

@ -0,0 +1,4 @@
compiler,std,http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/compiler/latest_build/outputs/RPMS/std/rpm.lst
distro,std,http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/distro/latest_build/outputs/RPMS/std/rpm.lst
distro,rt,http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/distro/latest_build/outputs/RPMS/rt/rpm.lst
distro,installer,http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/distro/latest_build/outputs/RPMS/installer/rpm.lst

13
centos-mirror-tools/rpms_3rdparties.lst → centos-mirror-tools/config/centos/flock/rpms_3rdparties.lst

@ -1,15 +1,10 @@
ansible-2.7.5-1.el7.ans.noarch.rpm#https://releases.ansible.com/ansible/rpm/release/epel-7-x86_64/ansible-2.7.5-1.el7.ans.noarch.rpm
libvirt-python-4.7.0-1.fc28.src.rpm#https://libvirt.org/sources/python/libvirt-python-4.7.0-1.fc28.src.rpm
influxdb-0.9.5.1-1.x86_64.rpm#https://s3.amazonaws.com/influxdb/influxdb-0.9.5.1-1.x86_64.rpm
novnc-0.6.2-1.el7.noarch.rpm#http://cbs.centos.org/kojifiles/packages/novnc/0.6.2/1.el7/noarch/novnc-0.6.2-1.el7.noarch.rpm
libvirt-python-4.7.0-1.fc28.src.rpm#https://libvirt.org/sources/python/libvirt-python-4.7.0-1.fc28.src.rpm
python2-httpbin-0.5.0-6.el7.noarch.rpm#http://cbs.centos.org/kojifiles/packages/python-httpbin/0.5.0/6.el7/noarch/python2-httpbin-0.5.0-6.el7.noarch.rpm
python2-kubernetes-8.0.0-8.el7.noarch.rpm#https://cbs.centos.org/kojifiles/packages/python-kubernetes/8.0.0/8.el7/noarch/python2-kubernetes-8.0.0-8.el7.noarch.rpm
python2-pytest-httpbin-0.2.3-6.el7.noarch.rpm#http://cbs.centos.org/kojifiles/packages/python-pytest-httpbin/0.2.3/6.el7/noarch/python2-pytest-httpbin-0.2.3-6.el7.noarch.rpm
python2-pytest-mock-1.6.0-2.el7.noarch.rpm#http://cbs.centos.org/kojifiles/packages/python-pytest-mock/1.6.0/2.el7/noarch/python2-pytest-mock-1.6.0-2.el7.noarch.rpm
python2-storops-0.4.7-2.el7.noarch.rpm#http://cbs.centos.org/kojifiles/packages/python-storops/0.4.7/2.el7/noarch/python2-storops-0.4.7-2.el7.noarch.rpm
influxdb-0.9.5.1-1.x86_64.rpm#https://s3.amazonaws.com/influxdb/influxdb-0.9.5.1-1.x86_64.rpm
python2-wheel-0.31.1-3.fc30.noarch.rpm#https://kojipkgs.fedoraproject.org/packages/python-wheel/0.31.1/3.fc30/noarch/python2-wheel-0.31.1-3.fc30.noarch.rpm
python3-wheel-0.31.1-3.fc30.noarch.rpm#https://kojipkgs.fedoraproject.org/packages/python-wheel/0.31.1/3.fc30/noarch/python3-wheel-0.31.1-3.fc30.noarch.rpm
python2-pip-9.0.3-6.fc29.noarch.rpm#https://kojipkgs.fedoraproject.org/packages/python-pip/9.0.3/6.fc29/noarch/python2-pip-9.0.3-6.fc29.noarch.rpm
python3-pip-9.0.3-6.fc29.noarch.rpm#https://kojipkgs.fedoraproject.org/packages/python-pip/9.0.3/6.fc29/noarch/python3-pip-9.0.3-6.fc29.noarch.rpm
python2-openstackdocstheme-1.23.2-1.el7.noarch.rpm#https://cbs.centos.org/kojifiles/packages/python-openstackdocstheme/1.23.2/1.el7/noarch/python2-openstackdocstheme-1.23.2-1.el7.noarch.rpm
python2-storops-0.4.7-2.el7.noarch.rpm#http://cbs.centos.org/kojifiles/packages/python-storops/0.4.7/2.el7/noarch/python2-storops-0.4.7-2.el7.noarch.rpm
python2-wheel-0.31.1-3.fc30.noarch.rpm#https://kojipkgs.fedoraproject.org/packages/python-wheel/0.31.1/3.fc30/noarch/python2-wheel-0.31.1-3.fc30.noarch.rpm
python2-sphinxcontrib-apidoc-0.2.1-6.el7.noarch.rpm#https://cbs.centos.org/kojifiles/packages/python-sphinxcontrib-apidoc/0.2.1/6.el7/noarch/python2-sphinxcontrib-apidoc-0.2.1-6.el7.noarch.rpm

1311
centos-mirror-tools/config/centos/flock/rpms_centos.lst Normal file

File diff suppressed because it is too large Load Diff

85
centos-mirror-tools/rpms_centos3rdparties.lst → centos-mirror-tools/config/centos/flock/rpms_centos3rdparties.lst

@ -1,46 +1,25 @@
adwaita-cursor-theme-3.28.0-1.el7.noarch.rpm
adwaita-icon-theme-3.28.0-1.el7.noarch.rpm
alsa-lib-1.1.6-2.el7.x86_64.rpm
alsa-lib-devel-1.1.6-2.el7.x86_64.rpm
audit-2.8.4-4.el7.x86_64.rpm
audit-libs-2.8.4-4.el7.x86_64.rpm
# audit-libs-2.8.4-4.el7.x86_64.rpm provided by mock
audit-libs-devel-2.8.4-4.el7.x86_64.rpm
audit-libs-python-2.8.4-4.el7.x86_64.rpm
collectd-5.8.1-4.el7.x86_64.rpm
collectd-python-5.8.1-4.el7.x86_64.rpm
containernetworking-plugins-0.8.1-1.el7.centos.x86_64.rpm
copy-jdk-configs-3.3-10.el7_5.noarch.rpm
cppcheck-1.88-3.el7.x86_64.rpm
CUnit-2.1.3-8.el7.x86_64.rpm
CUnit-devel-2.1.3-8.el7.x86_64.rpm
devtoolset-7-binutils-2.28-11.el7.x86_64.rpm
devtoolset-7-gcc-7.3.1-5.10.el7.x86_64.rpm
devtoolset-7-gcc-c++-7.3.1-5.10.el7.x86_64.rpm
devtoolset-7-libstdc++-devel-7.3.1-5.10.el7.x86_64.rpm
devtoolset-7-runtime-7.1-4.el7.x86_64.rpm
docker-ce-18.09.6-3.el7.x86_64.rpm
docker-ce-cli-18.09.6-3.el7.x86_64.rpm
elfutils-0.176-2.el7.x86_64.rpm
elfutils-default-yama-scope-0.176-2.el7.noarch.rpm
elfutils-devel-0.176-2.el7.x86_64.rpm
elfutils-libelf-0.176-2.el7.x86_64.rpm
elfutils-libelf-devel-0.176-2.el7.x86_64.rpm
elfutils-libs-0.176-2.el7.x86_64.rpm
glib2-2.56.1-2.el7.x86_64.rpm
# elfutils-default-yama-scope-0.176-2.el7.noarch.rpm provided by mock
# elfutils-devel-0.176-2.el7.x86_64.rpm provided by mock
# elfutils-libelf-0.176-2.el7.x86_64.rpm provided by mock
# elfutils-libelf-devel-0.176-2.el7.x86_64.rpm provided by mock
# elfutils-libs-0.176-2.el7.x86_64.rpm provided by mock
# glib2-2.56.1-2.el7.x86_64.rpm provided by mock
glib2-devel-2.56.1-2.el7.x86_64.rpm
glib2-doc-2.56.1-2.el7.noarch.rpm
gsettings-desktop-schemas-3.28.0-2.el7.x86_64.rpm
gssproxy-0.7.0-21.el7.x86_64.rpm
gtk3-3.22.30-3.el7.x86_64.rpm
gtk3-devel-3.22.30-3.el7.x86_64.rpm
gtk3-devel-docs-3.22.30-3.el7.x86_64.rpm
gtk-update-icon-cache-3.22.30-3.el7.x86_64.rpm
ima-evm-utils-1.1-2.el7.x86_64.rpm
ima-evm-utils-devel-1.1-2.el7.x86_64.rpm
# ima-evm-utils-1.1-2.el7.x86_64.rpm provided by mock
iprutils-2.4.16.1-1.el7.x86_64.rpm
java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64.rpm
java-1.8.0-openjdk-devel-1.8.0.191.b12-1.el7_6.x86_64.rpm
java-1.8.0-openjdk-headless-1.8.0.191.b12-1.el7_6.x86_64.rpm
kata-containers-image-1.10.0-4.1.x86_64.rpm
kata-ksm-throttler-1.10.0-4.1.x86_64.rpm
kata-linux-container-4.19.86.60-4.1.x86_64.rpm
@ -50,18 +29,14 @@ kata-proxy-bin-1.10.0-4.1.x86_64.rpm
kata-runtime-1.10.0-4.1.x86_64.rpm
kata-shim-1.10.0-4.1.x86_64.rpm
kata-shim-bin-1.10.0-4.1.x86_64.rpm
kernel-rt-3.10.0-1062.1.2.rt56.1025.el7.src.rpm
kexec-tools-2.0.15-21.el7.x86_64.rpm
libblkid-2.23.2-59.el7.x86_64.rpm
libcom_err-1.42.9-13.el7.x86_64.rpm
# libblkid-2.23.2-59.el7.x86_64.rpm provided by mock
# libcom_err-1.42.9-13.el7.x86_64.rpm provided by mock
libcom_err-devel-1.42.9-13.el7.x86_64.rpm
libdrm-2.4.91-3.el7.x86_64.rpm
libdrm-devel-2.4.91-3.el7.x86_64.rpm
libldb-1.3.4-1.el7.x86_64.rpm
liboath-2.4.1-9.el7.x86_64.rpm
liboath-devel-2.4.1-9.el7.x86_64.rpm
libsemanage-2.5-14.el7.x86_64.rpm
libsemanage-devel-2.5-14.el7.x86_64.rpm
# libsemanage-2.5-14.el7.x86_64.rpm provided by mock
libsemanage-python-2.5-14.el7.x86_64.rpm
libss-1.42.9-13.el7.x86_64.rpm
libss-devel-1.42.9-13.el7.x86_64.rpm
@ -71,36 +46,22 @@ libtevent-0.9.36-1.el7.x86_64.rpm
libwbclient-4.8.3-4.el7.x86_64.rpm
lvm2-2.02.177-4.el7.x86_64.rpm
lvm2-libs-2.02.177-4.el7.x86_64.rpm
mesa-libEGL-18.0.5-3.el7.x86_64.rpm
mesa-libEGL-devel-18.0.5-3.el7.x86_64.rpm
mesa-libgbm-18.0.5-3.el7.x86_64.rpm
mesa-libGL-18.0.5-3.el7.x86_64.rpm
mesa-libglapi-18.0.5-3.el7.x86_64.rpm
mesa-libGL-devel-18.0.5-3.el7.x86_64.rpm
NetworkManager-glib-1.12.0-8.el7_6.x86_64.rpm
NetworkManager-glib-devel-1.12.0-8.el7_6.x86_64.rpm
nss-softokn-3.36.0-5.el7_5.x86_64.rpm
# nss-softokn-3.36.0-5.el7_5.x86_64.rpm provided by mock
nss-softokn-devel-3.36.0-5.el7_5.x86_64.rpm
nss-softokn-freebl-3.36.0-5.el7_5.x86_64.rpm
# nss-softokn-freebl-3.36.0-5.el7_5.x86_64.rpm provided by mock
nss-softokn-freebl-devel-3.36.0-5.el7_5.x86_64.rpm
ntfs-3g-2017.3.23-11.el7.x86_64.rpm
ntfs-3g-devel-2017.3.23-11.el7.x86_64.rpm
ntfsprogs-2017.3.23-11.el7.x86_64.rpm
openldap-2.4.44-20.el7.x86_64.rpm
openscap-1.2.17-2.el7.x86_64.rpm
openscap-scanner-1.2.17-2.el7.x86_64.rpm
openstack-tempest-17.2.0-4.el7.noarch.rpm
osinfo-db-20180531-1.el7.noarch.rpm
OVMF-20180508-6.gitee3198e672e2.el7.noarch.rpm
policycoreutils-2.5-29.el7.x86_64.rpm
policycoreutils-devel-2.5-29.el7.x86_64.rpm
policycoreutils-newrole-2.5-29.el7.x86_64.rpm
policycoreutils-python-2.5-29.el7.x86_64.rpm
pulseaudio-libs-10.0-5.el7.x86_64.rpm
pulseaudio-libs-devel-10.0-5.el7.x86_64.rpm
pulseaudio-libs-glib2-10.0-5.el7.x86_64.rpm
pykickstart-1.99.66.19-2.el7.noarch.rpm
python-2.7.5-76.el7.x86_64.rpm
python2-google-api-client-1.6.3-1.el7.noarch.rpm
python2-manilaclient-1.17.3-1.el7.noarch.rpm
python2-mistralclient-3.1.4-1.el7.noarch.rpm
@ -111,43 +72,25 @@ python2-scapy-2.4.0-3.el7.noarch.rpm
python2-sushy-1.1.1-1.el7.noarch.rpm
python2-tempestconf-1.1.3-1.el7.noarch.rpm
python2-urllib3-1.21.1-1.el7.noarch.rpm
python-devel-2.7.5-76.el7.x86_64.rpm
python-ironic-inspector-client-2.1.0-1.el7.noarch.rpm
python-ironic-lib-2.10.1-1.el7.noarch.rpm
python-libguestfs-1.38.2-12.el7_6.1.x86_64.rpm
python-libs-2.7.5-76.el7.x86_64.rpm
python-requests-mock-1.3.0-1.el7.noarch.rpm
python-setuptools-22.0.5-1.el7.src.rpm
python-tempest-17.2.0-4.el7.noarch.rpm
python-virtualenv-15.1.0-2.el7.noarch.rpm
qemu-vanilla-4.1.0+git.9e06029aea-4.1.x86_64.rpm
qemu-vanilla-bin-4.1.0+git.9e06029aea-4.1.x86_64.rpm
qemu-vanilla-data-4.1.0+git.9e06029aea-4.1.x86_64.rpm
rest-0.8.1-2.el7.x86_64.rpm
samba-client-libs-4.8.3-4.el7.x86_64.rpm
samba-common-4.8.3-4.el7.noarch.rpm
samba-common-libs-4.8.3-4.el7.x86_64.rpm
selinux-policy-3.13.1-229.el7_6.6.noarch.rpm
selinux-policy-devel-3.13.1-229.el7_6.6.noarch.rpm
selinux-policy-minimum-3.13.1-229.el7_6.6.noarch.rpm
selinux-policy-mls-3.13.1-229.el7_6.6.noarch.rpm
selinux-policy-targeted-3.13.1-229.el7_6.6.noarch.rpm
spice-protocol-0.12.14-1.el7.noarch.rpm
spice-server-0.14.0-6.el7.x86_64.rpm
spice-server-devel-0.14.0-6.el7.x86_64.rpm
systemd-219-67.el7.x86_64.rpm
systemd-devel-219-67.el7.x86_64.rpm
systemd-libs-219-67.el7.x86_64.rpm
systemd-sysv-219-67.el7.x86_64.rpm
systemtap-3.3-3.el7.x86_64.rpm
systemtap-client-3.3-3.el7.x86_64.rpm
systemtap-devel-3.3-3.el7.x86_64.rpm
systemtap-runtime-3.3-3.el7.x86_64.rpm
systemtap-sdt-devel-3.3-3.el7.x86_64.rpm
tcpdump-4.9.2-3.el7.x86_64.rpm
tkinter-2.7.5-76.el7.x86_64.rpm
tuned-2.8.0-5.el7.noarch.rpm
tuned-profiles-realtime-2.8.0-5.el7.noarch.rpm
util-linux-2.23.2-59.el7.x86_64.rpm
valgrind-3.13.0-13.el7.x86_64.rpm
valgrind-devel-3.13.0-13.el7.x86_64.rpm
# util-linux-2.23.2-59.el7.x86_64.rpm provided by mock

12
centos-mirror-tools/config/centos/flock/yum.repos.d/StarlingX_cengn_compiler_layer.repo Normal file

@ -0,0 +1,12 @@
#####
# StarlingX compiler layer at CENGN
[Starlingx-cengn_compiler_std_layer]
name=Starlingx-cengn_compiler_std_layer
baseurl=http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/compiler/latest_build/outputs/RPMS/std/
enabled=1
[Starlingx-cengn_compiler_installer_layer]
name=Starlingx-cengn_compiler_installer_layer
baseurl=http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/compiler/latest_build/outputs/RPMS/installer/
enabled=1

18
centos-mirror-tools/config/centos/flock/yum.repos.d/StarlingX_cengn_distro_layer.repo Normal file

@ -0,0 +1,18 @@
#####
# StarlingX distro layer at CENGN
[Starlingx-cengn_distro_std_layer]
name=Starlingx-cengn_distro_std_layer
baseurl=http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/distro/latest_build/outputs/RPMS/std/
enabled=1
[Starlingx-cengn_distro_rt_layer]
name=Starlingx-cengn_distro_rt_layer
baseurl=http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/distro/latest_build/outputs/RPMS/rt/
enabled=1
[Starlingx-cengn_distro_installer_layer]
name=Starlingx-cengn_distro_installer_layer
baseurl=http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/distro/latest_build/outputs/RPMS/installer/
enabled=1

180
centos-mirror-tools/config/centos/mock/rpms_centos.lst Normal file

@ -0,0 +1,180 @@
acl-2.2.51-14.el7.x86_64.rpm
basesystem-10.0-7.el7.centos.noarch.rpm
# bash-4.2.46-31.el7.x86_64.rpm
binutils-2.27-34.base.el7.x86_64.rpm
bzip2-1.0.6-13.el7.x86_64.rpm
bzip2-libs-1.0.6-13.el7.x86_64.rpm
ca-certificates-2018.2.22-70.0.el7_5.noarch.rpm
centos-release-7-6.1810.2.el7.centos.x86_64.rpm
chkconfig-1.7.4-1.el7.x86_64.rpm
coreutils-8.22-23.el7.x86_64.rpm
cpio-2.11-27.el7.x86_64.rpm
cpp-4.8.5-36.el7.x86_64.rpm
cracklib-2.9.0-11.el7.x86_64.rpm
cracklib-dicts-2.9.0-11.el7.x86_64.rpm
cryptsetup-libs-2.0.3-3.el7.x86_64.rpm
curl-7.29.0-51.el7_6.3.x86_64.rpm
cyrus-sasl-lib-2.1.26-23.el7.x86_64.rpm
dbus-1.10.24-12.el7.x86_64.rpm
dbus-libs-1.10.24-12.el7.x86_64.rpm
device-mapper-1.02.146-4.el7.x86_64.rpm
device-mapper-libs-1.02.146-4.el7.x86_64.rpm
diffutils-3.3-4.el7.x86_64.rpm
dracut-033-554.el7.x86_64.rpm
dwz-0.11-3.el7.x86_64.rpm
epel-release-7-11.noarch.rpm
epel-rpm-macros-7-21.noarch.rpm
expat-2.1.0-10.el7_3.x86_64.rpm
file-5.11-35.el7.x86_64.rpm
file-libs-5.11-35.el7.x86_64.rpm
filesystem-3.2-25.el7.x86_64.rpm
findutils-4.5.11-6.el7.x86_64.rpm
gawk-4.0.2-4.el7_3.1.x86_64.rpm
gcc-4.8.5-36.el7.x86_64.rpm
gcc-c++-4.8.5-36.el7.x86_64.rpm
gdb-7.6.1-114.el7.x86_64.rpm
gdbm-1.10-8.el7.x86_64.rpm
glibc-2.17-260.el7.x86_64.rpm
glibc-common-2.17-260.el7.x86_64.rpm
glibc-devel-2.17-260.el7.x86_64.rpm
glibc-headers-2.17-260.el7.x86_64.rpm
gmp-6.0.0-15.el7.x86_64.rpm
gnupg2-2.0.22-5.el7_5.x86_64.rpm
gpgme-1.3.2-5.el7.x86_64.rpm
grep-2.20-3.el7.x86_64.rpm
groff-base-1.22.2-8.el7.x86_64.rpm
gzip-1.5-10.el7.x86_64.rpm
hardlink-1.0-19.el7.x86_64.rpm
hostname-3.13-3.el7.x86_64.rpm
info-5.1-5.el7.x86_64.rpm
json-c-0.11-4.el7_0.x86_64.rpm
keyutils-libs-1.5.8-3.el7.x86_64.rpm
kmod-20-23.el7.x86_64.rpm
kmod-libs-20-23.el7.x86_64.rpm
kpartx-0.4.9-119.el7.x86_64.rpm
krb5-libs-1.15.1-34.el7.x86_64.rpm
lbzip2-2.5-1.el7.x86_64.rpm
libacl-2.2.51-14.el7.x86_64.rpm
libarchive-3.1.2-10.el7_2.x86_64.rpm
libassuan-2.1.0-3.el7.x86_64.rpm
libattr-2.4.46-13.el7.x86_64.rpm
libcap-2.22-9.el7.x86_64.rpm
libcap-ng-0.7.5-4.el7.x86_64.rpm
libcurl-7.29.0-51.el7_6.3.x86_64.rpm
libcurl-devel-7.29.0-51.el7_6.3.x86_64.rpm
libdb-5.3.21-24.el7.x86_64.rpm
libdb-utils-5.3.21-24.el7.x86_64.rpm
libffi-3.0.13-18.el7.x86_64.rpm
libgcc-4.8.5-36.el7.x86_64.rpm
libgcrypt-1.5.3-14.el7.x86_64.rpm
libgomp-4.8.5-36.el7.x86_64.rpm
libgpg-error-1.12-3.el7.x86_64.rpm
libidn-1.28-4.el7.x86_64.rpm
libmount-2.23.2-59.el7.x86_64.rpm
libmpc-1.0.1-3.el7.x86_64.rpm
libpwquality-1.2.3-5.el7.x86_64.rpm
libselinux-2.5-14.1.el7.x86_64.rpm
libsepol-2.5-10.el7.x86_64.rpm
libsmartcols-2.23.2-59.el7.x86_64.rpm
libssh2-1.4.3-12.el7.x86_64.rpm
libstdc++-4.8.5-36.el7.x86_64.rpm
libstdc++-devel-4.8.5-36.el7.x86_64.rpm
libtasn1-4.10-1.el7.x86_64.rpm
libuser-0.60-9.el7.x86_64.rpm
libutempter-1.1.6-4.el7.x86_64.rpm
libuuid-2.23.2-59.el7.x86_64.rpm
libverto-0.2.5-4.el7.x86_64.rpm
libxml2-2.9.1-6.el7_2.3.x86_64.rpm
lua-5.1.4-15.el7.x86_64.rpm
lz4-1.7.5-2.el7.x86_64.rpm
lzo-2.06-8.el7.x86_64.rpm
make-3.82-23.el7.x86_64.rpm
mpfr-3.1.1-4.el7.x86_64.rpm
ncurses-5.9-14.20130511.el7_4.x86_64.rpm
ncurses-base-5.9-14.20130511.el7_4.noarch.rpm
ncurses-libs-5.9-14.20130511.el7_4.x86_64.rpm
nspr-4.19.0-1.el7_5.x86_64.rpm
nss-3.36.0-7.el7_5.x86_64.rpm
nss-pem-1.0.3-5.el7.x86_64.rpm
nss-sysinit-3.36.0-7.el7_5.x86_64.rpm
nss-tools-3.36.0-7.el7_5.x86_64.rpm
nss-util-3.36.0-1.el7_5.x86_64.rpm
openldap-2.4.44-20.el7.x86_64.rpm
openssl-libs-1.0.2k-16.el7.x86_64.rpm
p11-kit-0.23.5-3.el7.x86_64.rpm
p11-kit-trust-0.23.5-3.el7.x86_64.rpm
pam-1.1.8-22.el7.x86_64.rpm
patch-2.7.1-10.el7_5.x86_64.rpm
pcre-8.32-17.el7.x86_64.rpm
perl-5.16.3-294.el7_6.x86_64.rpm
perl-Carp-1.26-244.el7.noarch.rpm
perl-constant-1.27-2.el7.noarch.rpm
perl-Encode-2.51-7.el7.x86_64.rpm
perl-Exporter-5.68-3.el7.noarch.rpm
perl-File-Path-2.09-2.el7.noarch.rpm
perl-File-Temp-0.23.01-3.el7.noarch.rpm
perl-Filter-1.49-3.el7.x86_64.rpm
perl-Getopt-Long-2.40-3.el7.noarch.rpm
perl-HTTP-Tiny-0.033-3.el7.noarch.rpm
perl-libs-5.16.3-294.el7_6.x86_64.rpm
perl-macros-5.16.3-294.el7_6.x86_64.rpm
perl-parent-0.225-244.el7.noarch.rpm
perl-PathTools-3.40-5.el7.x86_64.rpm
perl-Pod-Escapes-1.04-294.el7_6.noarch.rpm
perl-podlators-2.5.1-3.el7.noarch.rpm
perl-Pod-Perldoc-3.20-4.el7.noarch.rpm
perl-Pod-Simple-3.28-4.el7.noarch.rpm
perl-Pod-Usage-1.63-3.el7.noarch.rpm
perl-Scalar-List-Utils-1.27-248.el7.x86_64.rpm
perl-Socket-2.010-4.el7.x86_64.rpm
perl-srpm-macros-1-8.el7.noarch.rpm
perl-Storable-2.45-3.el7.x86_64.rpm
perl-Text-ParseWords-3.29-4.el7.noarch.rpm
perl-Thread-Queue-3.02-2.el7.noarch.rpm
perl-threads-1.87-4.el7.x86_64.rpm
perl-threads-shared-1.43-6.el7.x86_64.rpm
perl-Time-HiRes-1.9725-3.el7.x86_64.rpm
perl-Time-Local-1.2300-2.el7.noarch.rpm
pigz-2.3.4-1.el7.x86_64.rpm
pinentry-0.8.1-17.el7.x86_64.rpm
pkgconfig-0.27.1-4.el7.x86_64.rpm
popt-1.13-16.el7.x86_64.rpm
procps-ng-3.3.10-23.el7.x86_64.rpm
pth-2.0.7-23.el7.x86_64.rpm
pygpgme-0.3-9.el7.x86_64.rpm
pyliblzma-0.5.3-11.el7.x86_64.rpm
# python-2.7.5-76.el7.x86_64.rpm
# python-devel-2.7.5-76.el7.x86_64.rpm
python2-rpm-macros-3-25.el7.noarch.rpm
python-iniparse-0.4-9.el7.noarch.rpm
# python-libs-2.7.5-76.el7.x86_64.rpm
python-pycurl-7.19.0-19.el7.x86_64.rpm
python-rpm-macros-3-25.el7.noarch.rpm
python-srpm-macros-3-25.el7.noarch.rpm
python-urlgrabber-3.10-9.el7.noarch.rpm
pyxattr-0.5.1-5.el7.x86_64.rpm
qrencode-libs-3.4.1-3.el7.x86_64.rpm
readline-6.2-10.el7.x86_64.rpm
redhat-rpm-config-9.1.0-87.el7.centos.noarch.rpm
# rpm-4.11.3-35.el7.x86_64.rpm
# rpm-build-4.11.3-35.el7.x86_64.rpm
# rpm-build-libs-4.11.3-35.el7.x86_64.rpm
# rpm-libs-4.11.3-35.el7.x86_64.rpm
# rpm-python-4.11.3-35.el7.x86_64.rpm
sed-4.2.2-5.el7.x86_64.rpm
setup-2.8.71-10.el7.noarch.rpm
shadow-utils-4.1.5.1-25.el7.x86_64.rpm
shared-mime-info-1.8-4.el7.x86_64.rpm
sqlite-3.7.17-8.el7.x86_64.rpm
tar-1.26-35.el7.x86_64.rpm
tzdata-2018g-1.el7.noarch.rpm
unzip-6.0-19.el7.x86_64.rpm
ustr-1.0.4-16.el7.x86_64.rpm
which-2.20-7.el7.x86_64.rpm
xz-5.2.2-1.el7.x86_64.rpm
xz-libs-5.2.2-1.el7.x86_64.rpm
yum-3.4.3-161.el7.centos.noarch.rpm
yum-metadata-parser-1.1.4-10.el7.x86_64.rpm
yum-plugin-fastestmirror-1.1.31-50.el7.noarch.rpm
zip-3.0-11.el7.x86_64.rpm
zlib-1.2.7-18.el7.x86_64.rpm

19
centos-mirror-tools/config/centos/mock/rpms_centos3rdparties.lst Normal file

@ -0,0 +1,19 @@
audit-libs-2.8.4-4.el7.x86_64.rpm
elfutils-0.176-2.el7.x86_64.rpm
elfutils-default-yama-scope-0.176-2.el7.noarch.rpm
elfutils-devel-0.176-2.el7.x86_64.rpm
elfutils-libelf-0.176-2.el7.x86_64.rpm
elfutils-libelf-devel-0.176-2.el7.x86_64.rpm
elfutils-libs-0.176-2.el7.x86_64.rpm
glib2-2.56.1-2.el7.x86_64.rpm
ima-evm-utils-1.1-2.el7.x86_64.rpm
libblkid-2.23.2-59.el7.x86_64.rpm
libcom_err-1.42.9-13.el7.x86_64.rpm
libsemanage-2.5-14.el7.x86_64.rpm
nss-softokn-3.36.0-5.el7_5.x86_64.rpm
nss-softokn-freebl-3.36.0-5.el7_5.x86_64.rpm
systemd-219-67.el7.x86_64.rpm
systemd-devel-219-67.el7.x86_64.rpm
systemd-libs-219-67.el7.x86_64.rpm
systemd-sysv-219-67.el7.x86_64.rpm
util-linux-2.23.2-59.el7.x86_64.rpm

6
centos-mirror-tools/dl_other_from_centos_repo.sh

@ -151,6 +151,12 @@ for ff in $all; do
let i+=1
echo "$file_name is downloaded successfully"
mkdir -p $save_path/$sub_path
if [ $? -ne 0 ]; then
echo "Error: mkdir -p '$save_path/$sub_path'"
error_count=$((error_count + 1))
fi
\mv -f ./$file_name $save_path/$_name
if [ $? -ne 0 ]; then
echo "Error: mv -f './$file_name' '$save_path/$_name'"

14
centos-mirror-tools/dl_rpms.sh

@ -153,13 +153,15 @@ fi
timestamp=$(date +%F_%H%M)
echo $timestamp
DESTDIR="output"
MDIR_SRC=$DESTDIR/stx-r1/CentOS/pike/Source
mkdir -p $MDIR_SRC
MDIR_BIN=$DESTDIR/stx-r1/CentOS/pike/Binary
mkdir -p $MDIR_BIN
export DL_MIRROR_LOG_DIR="${DL_MIRROR_LOG_DIR:-./logs}"
export DL_MIRROR_OUTPUT_DIR="${DL_MIRROR_OUTPUT_DIR:-./output/stx-r1/CentOS/pike}"
LOGSDIR="logs"
MDIR_SRC="${DL_MIRROR_OUTPUT_DIR}/Source"
mkdir -p "$MDIR_SRC"
MDIR_BIN="${DL_MIRROR_OUTPUT_DIR}/Binary"
mkdir -p "$MDIR_BIN"
LOGSDIR="${DL_MIRROR_LOG_DIR}"
from=$(get_from $rpms_list)
LOG="$LOGSDIR/${match_level}_failmoved_url_${from}.log"
MISSING_SRPMS="$LOGSDIR/${match_level}_srpms_missing_${from}.log"

179
centos-mirror-tools/dl_rpms_from_url.sh Executable file

@ -0,0 +1,179 @@
#!/bin/bash -e
#
# SPDX-License-Identifier: Apache-2.0
#
# download RPMs/SRPMs from a base url.
# this script was originated by Scott Little
set -o errexit
set -o nounset
# By default, we use "sudo" and we don't use a local yum.conf. These can
# be overridden via flags.
DL_RPMS_DIR="$(dirname "$(readlink -f "${BASH_SOURCE[0]}" )" )"
source $DL_RPMS_DIR/utils.sh
BASE_URL=""
usage() {
echo "$0 -u <base url> <rpms_list> "
echo ""
echo "Options:"
echo " -u: Use provided base url"
echo ""
echo "Returns: 0 = All files downloaded successfully"
echo " 1 = Some files could not be downloaded"
echo " 2 = Bad arguements or other error"
echo ""
}
CLEAN_LOGS_ONLY=0
dl_rc=0
distro="centos"
# Parse option flags
while getopts "u:h" o; do
case "${o}" in
u)
# Use an alternate yum.conf
BASE_URL="$OPTARG"
;;
h)
# Help
usage
exit 0
;;
*)
usage
exit 2
;;
esac
done
shift $((OPTIND-1))
if [ $# -lt 1 ]; then
usage
exit 2
fi
if [ "$1" == "" ]; then
echo "Need to supply the rpm file list"
exit 2;
else
rpms_list=$1
echo "using $rpms_list as the download name lists"
fi
if [ ! -f "${rpms_list}" ]; then
echo "Error: File not found: ${rpms_list}"
usage
exit 2
fi
timestamp=$(date +%F_%H%M)
echo $timestamp
export DL_MIRROR_LOG_DIR="${DL_MIRROR_LOG_DIR:-./logs}"
export DL_MIRROR_OUTPUT_DIR="${DL_MIRROR_OUTPUT_DIR:-./output/stx-r1/CentOS/pike}"
MDIR_SRC="${DL_MIRROR_OUTPUT_DIR}/Source"
mkdir -p "$MDIR_SRC"
MDIR_BIN="${DL_MIRROR_OUTPUT_DIR}/Binary"
mkdir -p "$MDIR_BIN"
LOGSDIR="${DL_MIRROR_LOG_DIR}"
from=$(get_from $rpms_list)
LOG="$LOGSDIR/L1_failmoved_url_${from}.log"
MISSING_SRPMS="$LOGSDIR/srpms_missing_${from}.log"
MISSING_RPMS="$LOGSDIR/rpms_missing_${from}.log"
FOUND_SRPMS="$LOGSDIR/srpms_found_${from}.log"
FOUND_RPMS="$LOGSDIR/rpms_found_${from}.log"
cat /dev/null > $LOG
cat /dev/null > $MISSING_SRPMS
cat /dev/null > $MISSING_RPMS
cat /dev/null > $FOUND_SRPMS
cat /dev/null > $FOUND_RPMS
if [ $CLEAN_LOGS_ONLY -eq 1 ];then
exit 0
fi
if [ "$BASE_URL" == "" ]; then
BASE_URL=file://$(readlink -f $(dirname ${rpms_list}))
fi
# Function to download different types of RPMs in different ways
download () {
local _file=$1
local _url=$2
local _list=""
local _from=""
local _arch=""
local rc=0
local download_cmd=""
local download_url=""
local rpm_name=""
local SFILE=""
local lvl
local dl_result
_list=$(cat $_file)
_from=$(get_from $_file)
echo "now the rpm will come from: $_from"
for ff in $_list; do
_arch=$(get_arch_from_rpm $ff)
rpm_name="$(get_rpm_name $ff)"
dest_dir="$(get_dest_directory $_arch)"
if [ ! -e $dest_dir/$rpm_name ]; then
dl_result=1
download_url="$_url/$rpm_name"
download_cmd="curl --silent --output $rpm_name ${download_url}"
echo "Looking for $rpm_name"
echo "--> run: $download_cmd"
if $download_cmd ; then
SFILE="$(get_rpm_level_name $rpm_name L1)"
process_result "$_arch" "$dest_dir" "$download_url" "$SFILE"
dl_result=0
else
echo "Warning: $rpm_name not found"
fi
if [ $dl_result -eq 1 ]; then
echo "Error: $rpm_name not found"
echo "missing_srpm:$rpm_name" >> $LOG
echo $rpm_name >> $MISSING_SRPMS
rc=1
fi
else
echo "Already have $dest_dir/$rpm_name"
fi
echo
done
return $rc
}
# Download files
if [ -s "$rpms_list" ];then
echo "--> start searching $rpms_list"
download $rpms_list $BASE_URL
if [ $? -ne 0 ]; then
dl_rc=1
fi
fi
echo "Done!"
exit $dl_rc

25
centos-mirror-tools/dl_tarball.sh

@ -16,15 +16,16 @@
# input files:
# The file tarball-dl.lst contains the list of packages and artifacts for
# building this sub-mirror.
script_path="$(dirname $(readlink -f $0))"
tarball_file="$script_path/tarball-dl.lst"
tarball_file=""
set -x
DL_TARBALL_DIR="$(dirname "$(readlink -f "${BASH_SOURCE[0]}" )" )"
source $DL_TARBALL_DIR/url_utils.sh
source $DL_TARBALL_DIR/utils.sh
usage () {
echo "$0 [-D <distro>] [-s|-S|-u|-U] [-h] <other_download_list.ini> <save_path> [<force_update>]"
echo "$0 [-D <distro>] [-s|-S|-u|-U] [-h] <path_to_tarball_dl.lst>"
}
# Permitted values of dl_source
@ -96,6 +97,8 @@ while getopts "D:hsSuU" o; do
esac
done
shift $((OPTIND-1))
tarball_file="${1}"
shift
if [ ! -e $tarball_file ]; then
@ -108,10 +111,12 @@ fi
# - Puppet hosted under "downloads/puppet" output directory.
# to be populated under $MY_REPO/addons/wr-cgcs/layers/cgcs/downloads/puppet
logs_dir="$script_path/logs"
output_main="$script_path/output"
export DL_MIRROR_LOG_DIR="${DL_MIRROR_LOG_DIR:-./logs}"
export DL_MIRROR_OUTPUT_DIR="${DL_MIRROR_OUTPUT_DIR:-./output/stx-r1/CentOS/pike}"
logs_dir="${DL_MIRROR_LOG_DIR}"
output_log="$logs_dir/log_download_tarball_missing.txt"
output_path=$output_main/stx-r1/CentOS/pike
output_path="${DL_MIRROR_OUTPUT_DIR}"
output_tarball=$output_path/downloads
output_puppet=$output_tarball/puppet
@ -122,10 +127,12 @@ if [ ! -d "$logs_dir" ]; then
fi
is_tarball() {
tarball_name="$1"
local tarball_name="$1"
local mime_type
local types=("gzip" "x-bzip2" "x-rpm" "x-xz" "x-gzip" "x-tar")
local FOUND=1
mime_type=$(file --mime-type -b $tarball_name | cut -d "/" -f 2)
types=("gzip" "x-bzip2" "x-rpm" "x-xz" "x-gzip" "x-tar")
FOUND=1
for t in "${types[@]}"; do
if [ "$mime_type" == "$t" ]; then
FOUND=0

333
centos-mirror-tools/download_mirror.sh

@ -3,20 +3,47 @@
# SPDX-License-Identifier: Apache-2.0
#
DOWNLOAD_MIRROR_DIR="$(dirname "$(readlink -f "${BASH_SOURCE[0]}" )" )"
source $DOWNLOAD_MIRROR_DIR/../toCOPY/lst_utils.sh
export DL_MIRROR_LOG_DIR="${DL_MIRROR_LOG_DIR:-./logs}"
export DL_MIRROR_OUTPUT_DIR="${DL_MIRROR_OUTPUT_DIR:-./output/stx-r1/CentOS/pike}"
usage() {
echo "$0 [-n] [-c <yum.conf>] [-g] [-s|-S|-u|-U]"
echo ""
echo "Options:"
echo " -n: Do not use sudo when performing operations (option passed on to"
echo " subscripts when appropriate)"
echo " -c: Use an alternate yum.conf rather than the system file (option passed"
echo " on to subscripts when appropriate)"
echo " -g: do not change group IDs of downloaded artifacts"
echo "$0 [options]"
echo
echo "Common Options:"
echo " -c <yum.conf>: Use an alternate yum.conf rather than the system file"
echo " Suggested valur is 'yum.conf.sample' in this directory."
echo " (option passed on to subscripts when appropriate)"
echo " -d <distro>: Download package to build designated distro. Default 'centos'"
echo " -g: Do not change group IDs of downloaded artifacts"
echo " -l <layer>: Download only packages required to build a given layer."
echo " Default: use the LAYER environmnet variable, or 'all'."
echo " -n: Do not use sudo when performing operations."
echo " (option passed on to subscripts when appropriate)"
echo
echo "Download Source Options: Only select one of these."
echo " -s: Download from StarlingX mirror only"
echo " -S: Download from StarlingX mirror, upstream as backup (default)"
echo " -u: Download from original upstream sources only"
echo " -U: Download from original upstream sources, StarlingX mirror as backup"
echo ""
echo
echo "Layered Build Options: For use when building multiple layers locally."
echo " -C <config_dir>: Use an alternate config directory rather than the system"
echo " defined one"
echo " -I <lower_layer>,<build_type>,<url>:"
echo " Override the url for the image include file of a lower"
echo " layer's build type. Normally the url(s) is read from"
echo " <config_dir>/<distro>/<layer>/required_layer_iso_inc.cfg"
echo " This option can be used more than once."
echo " -L <lower_layer>,<build_type>,<url>:"
echo " Override the url for the package list of a lower"
echo " layer's build type. Normally the url(s) is read from"
echo " <config_dir>/<distro>/<layer>/required_layer_pkgs.cfg."
echo " This option can be used more than once."
echo
}
generate_log_name() {
@ -35,6 +62,15 @@ need_file(){
done
}
make_if_needed_file(){
for f in $*; do
if [ ! -f $f ]; then
echo "Creating empty file '$f'"
touch $f
fi
done
}
need_dir(){
for d in $*; do
if [ ! -d $d ]; then
@ -45,10 +81,11 @@ need_dir(){
}
# Downloader scripts
rpm_downloader="./dl_rpms.sh"
tarball_downloader="./dl_tarball.sh"
other_downloader="./dl_other_from_centos_repo.sh"
make_stx_mirror_yum_conf="./make_stx_mirror_yum_conf.sh"
rpm_downloader="${DOWNLOAD_MIRROR_DIR}/dl_rpms.sh"
rpm_from_url_downloader="${DOWNLOAD_MIRROR_DIR}/dl_rpms_from_url.sh"
tarball_downloader="${DOWNLOAD_MIRROR_DIR}/dl_tarball.sh"
other_downloader="${DOWNLOAD_MIRROR_DIR}/dl_other_from_centos_repo.sh"
make_stx_mirror_yum_conf="${DOWNLOAD_MIRROR_DIR}/make_stx_mirror_yum_conf.sh"
# track optional arguments
change_group_ids=1
@ -57,13 +94,18 @@ alternate_yum_conf=""
alternate_repo_dir=""
rpm_downloader_extra_args=""
tarball_downloader_extra_args=""
distro="centos"
make_stx_mirror_yum_conf_extra_args=""
# lst files to use as input
rpms_from_3rd_parties="./rpms_3rdparties.lst"
rpms_from_centos_repo="./rpms_centos.lst"
rpms_from_centos_3rd_parties="./rpms_centos3rdparties.lst"
other_downloads="./other_downloads.lst"
rpms_from_3rd_parties_template="rpms_3rdparties.lst"
rpms_from_centos_repo_template="rpms_centos.lst"
rpms_from_centos_3rd_parties_template="rpms_centos3rdparties.lst"
rpms_from_layer_build_dir=${DL_MIRROR_OUTPUT_DIR}/layer_pkg_lists
image_inc_from_layer_build_dir=${DL_MIRROR_OUTPUT_DIR}/layer_image_inc
build_info_from_layer_build_dir=${DL_MIRROR_OUTPUT_DIR}/layer_build_info
tarball_downloads_template="tarball-dl.lst"
other_downloads_template="other_downloads.lst"
# Overall success
success=1
@ -103,22 +145,41 @@ multiple_dl_flag_check () {
fi
}
# Parse out optional arguments
while getopts "c:nghsSuU" o; do
while getopts "c:Cd:ghI:sl:L:nSuU" o; do
case "${o}" in
n)
# Pass -n ("no-sudo") to rpm downloader
rpm_downloader_extra_args="${rpm_downloader_extra_args} -n"
;;
c)
# Pass -c ("use alternate yum.conf") to rpm downloader
use_system_yum_conf=0
alternate_yum_conf="${OPTARG}"
;;
C)
# Alternate config directory
set_and_validate_config_dir "${OPTARG}"
;;
d)
# Alternate distro
set_and_validate_distro "${OPTARG}"
;;
g)
# Do not attempt to change group IDs on downloaded packages
change_group_ids=0
;;
I)
set_layer_image_inc_urls "${OPTARG}"
;;
l)
# layer
set_and_validate_layer "${OPTARG}"
;;
L)
set_layer_pkg_urls "${OPTARG}"
;;
n)
# Pass -n ("no-sudo") to rpm downloader
rpm_downloader_extra_args="${rpm_downloader_extra_args} -n"
;;
s)
# Download from StarlingX mirror only. Do not use upstream sources.
multiple_dl_flag_check
@ -156,6 +217,21 @@ while getopts "c:nghsSuU" o; do
done
shift $((OPTIND-1))
TMP_LST_DIR=$(mktemp -d /tmp/tmp_lst_dir_XXXXXX)
mkdir -p $TMP_LST_DIR
rpms_from_3rd_parties="$TMP_LST_DIR/${rpms_from_3rd_parties_template}"
rpms_from_centos_repo="$TMP_LST_DIR/${rpms_from_centos_repo_template}"
rpms_from_centos_3rd_parties="$TMP_LST_DIR/${rpms_from_centos_3rd_parties_template}"
tarball_downloads="$TMP_LST_DIR/${tarball_downloads_template}"
other_downloads="$TMP_LST_DIR/${other_downloads_template}"
merge_lst ${config_dir} ${distro} ${rpms_from_3rd_parties_template} > ${rpms_from_3rd_parties}
merge_lst ${config_dir} ${distro} ${rpms_from_centos_repo_template} > ${rpms_from_centos_repo}
merge_lst ${config_dir} ${distro} ${rpms_from_centos_3rd_parties_template} > ${rpms_from_centos_3rd_parties}
merge_lst ${config_dir} ${distro} ${tarball_downloads_template} > ${tarball_downloads}
merge_lst ${config_dir} ${distro} ${other_downloads_template} > ${other_downloads}
echo "--------------------------------------------------------------"
echo "WARNING: this script HAS TO access internet (http/https/ftp),"
@ -168,14 +244,92 @@ mkdir -p $LOGSDIR
# Check extistence of prerequisites files
need_file ${rpm_downloader} ${other_downloader} ${tarball_downloader}
need_file ${rpms_from_3rd_parties}
need_file ${rpms_from_centos_3rd_parties}
need_file ${rpms_from_centos_repo}
need_file ${other_downloads}
need_file tarball-dl.lst
make_if_needed_file ${rpms_from_3rd_parties}
make_if_needed_file ${rpms_from_centos_3rd_parties}
make_if_needed_file ${rpms_from_centos_repo}
make_if_needed_file ${other_downloads}
make_if_needed_file ${tarball_downloads}
#download RPMs/SRPMs from 3rd_party websites (not CentOS repos) by "wget"
echo "step #1: start downloading RPMs/SRPMs from 3rd-party websites..."
#
# Dowlnoad package lst files, image inc files and build info files for lower layers.
#
# Also it may set up extra arguements for make_stx_mirror_yum_conf that
# will exploy yum repos co-resident with the lst file.
#
\rm -rf ${rpms_from_layer_build_dir}
mkdir -p ${rpms_from_layer_build_dir}
for key in "${!layer_pkg_urls[@]}"; do
lower_layer="${key%,*}"
build_type="${key#*,}"
url="${layer_pkg_urls[${key}]}"
name_from_url=$(url_to_file_name $url)
list="${rpms_from_layer_build_dir}/${name_from_url}"
curl --silent --fail ${url} > ${list} ||
if [ $? -ne 0 ]; then
echo "ERROR: Failed to download from url: ${url}"
exit 1
fi
#
# If the lst file is co-resident with a yum repodata directory,
# then add arguements for our call to make_stx_mirror_yum_conf
# so that we'll use that repo.
#
url_type=${url%%:*}
if [ "${url_type}" == "file" ]; then
base_url=$(dirname $url)
repomod_url=${base_url}/repodata/repomd.xml
curl --silent --fail --output /dev/null ${repomod_url} ||
if [ $? -ne 0 ]; then
echo "ERROR: Failed to download from url: ${url}"
exit 1
fi
make_stx_mirror_yum_conf_extra_args+=" -u ${lower_layer},${build_type},${base_url}"
fi
done
\rm -rf ${image_inc_from_layer_build_dir}
mkdir -p ${image_inc_from_layer_build_dir}
for key in "${!layer_image_inc_urls[@]}"; do
lower_layer="${key%,*}"
inc_type="${key#*,}"
url="${layer_image_inc_urls[${key}]}"
name_from_url=$(url_to_file_name $url)
list="${image_inc_from_layer_build_dir}/${name_from_url}"
curl --silent --fail ${url} > ${list} ||
if [ $? -ne 0 ]; then
echo "ERROR: Failed to download from url: ${url}"
exit 1
fi
done
\rm -rf ${build_info_from_layer_build_dir}
mkdir -p ${build_info_from_layer_build_dir}
# Borrow std image.inc url as a proxy for the BUILD_INFO with a simple substitution
for key in "${!layer_image_inc_urls[@]}"; do
lower_layer="${key%,*}"
inc_type="${key#*,}"
if [ "${inc_type}" != "std" ]; then
continue
fi
if [ "$(basename ${layer_image_inc_urls[${key}]})" != "image.inc" ]; then
continue
fi
url=$( echo ${layer_image_inc_urls[${key}]} | sed 's#image.inc$#BUILD_INFO#' )
name_from_url=$(url_to_file_name $url)
dest="${build_info_from_layer_build_dir}/${name_from_url}"
curl --silent --fail ${url} > ${dest} ||
if [ $? -ne 0 ]; then
echo "ERROR: Failed to download from url: ${url}"
exit 1
fi
done
echo "step #0: Configuring yum repos ..."
if [ ${use_system_yum_conf} -ne 0 ]; then
# Restore StarlingX_3rd repos from backup
@ -184,6 +338,12 @@ if [ ${use_system_yum_conf} -ne 0 ]; then
if [ -d $REPO_SOURCE_DIR ] && [ -d $REPO_DIR ]; then
\cp -f $REPO_SOURCE_DIR/*.repo $REPO_DIR/
fi
if [ $layer != "all" ]; then
if [ -d ${config_dir}/${distro}/${layer}/yum.repos.d ]; then
\cp -f ${config_dir}/${distro}/${layer}/yum.repos.d/*.repo $REPO_DIR
fi
fi
fi
if [ $use_system_yum_conf -eq 0 ]; then
@ -193,6 +353,10 @@ if [ $use_system_yum_conf -eq 0 ]; then
if [ "$alternate_repo_dir" == "" ]; then
alternate_repo_dir="$(dirname "${alternate_yum_conf}"/yum.repos.d)"
fi
if [[ $alternate_repo_dir != /* ]]; then
# Path is relative, so prefix with directory where yum.conf lives
alternate_repo_dir=$(dirname ${alternate_yum_conf})/${alternate_repo_dir}
fi
need_dir "${alternate_repo_dir}"
fi
fi
@ -224,10 +388,10 @@ else
# point to the repos that need to be modified as well.
if dl_from_upstream; then
# add
${make_stx_mirror_yum_conf} -R -d $TEMP_DIR -y $alternate_yum_conf -r $alternate_repo_dir -D $distro
${make_stx_mirror_yum_conf} -R -d $TEMP_DIR -y $alternate_yum_conf -r $alternate_repo_dir -D $distro -l $layer ${make_stx_mirror_yum_conf_extra_args}
else
# substitute
${make_stx_mirror_yum_conf} -d $TEMP_DIR -y $alternate_yum_conf -r $alternate_repo_dir -D $distro
${make_stx_mirror_yum_conf} -d $TEMP_DIR -y $alternate_yum_conf -r $alternate_repo_dir -D $distro -l $layer ${make_stx_mirror_yum_conf_extra_args}
fi
else
# Modify system yum.conf and yum.repos.d. Remember that we expect to run this
@ -235,16 +399,58 @@ else
# in these scripts.
if dl_from_upstream; then
# add
${make_stx_mirror_yum_conf} -R -d $TEMP_DIR -y /etc/yum.conf -r /etc/yum.repos.d -D $distro
${make_stx_mirror_yum_conf} -R -d $TEMP_DIR -y /etc/yum.conf -r /etc/yum.repos.d -D $distro -l $layer ${make_stx_mirror_yum_conf_extra_args}
else
# substitute
${make_stx_mirror_yum_conf} -d $TEMP_DIR -y /etc/yum.conf -r /etc/yum.repos.d -D $distro
${make_stx_mirror_yum_conf} -d $TEMP_DIR -y /etc/yum.conf -r /etc/yum.repos.d -D $distro -l $layer ${make_stx_mirror_yum_conf_extra_args}
fi
fi
rpm_downloader_extra_args="${rpm_downloader_extra_args} -c $TEMP_CONF"
fi
#download RPMs/SRPMs from lower layer builds
echo "step #1: start downloading RPMs/SRPMs from lower layer builds..."
retcode=0
for key in "${!layer_pkg_urls[@]}"; do
lower_layer="${key%,*}"
build_type="${key#*,}"
url="${layer_pkg_urls[${key}]}"
name_from_url=$(url_to_file_name $url)
list="${rpms_from_layer_build_dir}/${name_from_url}"
url_type=${url%%:*}
if [ "${url_type}" == "file" ]; then
level=L1
logfile=$(generate_log_name $list level)
$rpm_from_url_downloader -u $(dirname $url) $list |& tee $logfile
local_retcode=${PIPESTATUS[0]}
else
#download RPMs/SRPMs from CentOS repos by "yumdownloader"
level=L1
logfile=$(generate_log_name $list $level)
$rpm_downloader ${rpm_downloader_extra_args} $list $level |& tee $logfile
local_retcode=${PIPESTATUS[0]}
fi
if [ $local_retcode -ne 0 ]; then
echo "ERROR: Something wrong with downloading files listed in $list."
echo " Please check the log at $(pwd)/$logfile !"
echo ""
success=0
retcode=$local_retcode
fi
done
if [ $retcode -eq 0 ];then
echo "step #1: done successfully"
else
echo "step #1: finished with errors"
fi
#download RPMs/SRPMs from 3rd_party websites (not CentOS repos) by "wget"
echo "step #2: start downloading RPMs/SRPMs from 3rd-party websites..."
list=${rpms_from_3rd_parties}
level=L1
logfile=$(generate_log_name $list $level)
@ -263,23 +469,28 @@ level=L1
logfile=$(generate_log_name $list $level)
$rpm_downloader ${rpm_downloader_extra_args} $list $level |& tee $logfile
retcode=${PIPESTATUS[0]}
if [ $retcode -ne 0 ];then
if [ $retcode -eq 0 ];then
echo "step #2: done successfully"
else
echo "step #2: finished with errors"
echo "ERROR: Something wrong with downloading files listed in $list."
echo " Please check the log at $(pwd)/$logfile !"
echo ""
echo "step #2: finished with errors"
success=0
fi
if [ ${use_system_yum_conf} -eq 1 ]; then
# deleting the StarlingX_3rd to avoid pull centos packages from the 3rd Repo.
\rm -f $REPO_DIR/StarlingX_3rd*.repo
\rm -f $REPO_DIR/StarlingX_cengn*.repo
if [ "$TEMP_DIR" != "" ]; then
\rm -f $TEMP_DIR/yum.repos.d/StarlingX_3rd*.repo
\rm -f $TEMP_DIR/yum.repos.d/StarlingX_cengn*.repo
fi
fi
echo "step #2: start 1st round of downloading RPMs and SRPMs with L1 match criteria..."
echo "step #3: start 1st round of downloading RPMs and SRPMs with L1 match criteria..."
#download RPMs/SRPMs from CentOS repos by "yumdownloader"
list=${rpms_from_centos_repo}
level=L1
@ -314,7 +525,7 @@ elif [ $retcode -eq 1 ]; then
# Remove files found by K1 download from L1_rpms_missing_centos.txt to prevent
# false reporting of missing files.
grep -v -x -F -f $LOGSDIR/K1_rpms_found_centos.log $LOGSDIR/L1_rpms_missing_centos.log > $LOGSDIR/L1_rpms_missing_centos.tmp
grep -v -x -F -f $LOGSDIR/K1_rpms_found_centos.log $LOGSDIR/L1_rpms_missing_centos.log > $LOGSDIR/L1_rpms_missing_centos.tmp || true
mv -f $LOGSDIR/L1_rpms_missing_centos.tmp $LOGSDIR/L1_rpms_missing_centos.log
@ -334,27 +545,34 @@ elif [ $retcode -eq 1 ]; then
fi
fi
if [ $retcode -ne 0 ]; then
if [ $retcode -eq 0 ];then
echo "step #3: done successfully"
else
echo "ERROR: Something wrong with downloading files listed in ${rpms_from_centos_repo}."
echo " Please check the logs at $(pwd)/$logfile"
echo " and $(pwd)/logs/$K1_logfile !"
echo ""
echo "step #3: finished with errors"
success=0
fi
## verify all RPMs SRPMs we download for the GPG keys
find ./output -type f -name "*.rpm" | xargs rpm -K | grep -i "MISSING KEYS" > $LOGSDIR/rpm-gpg-key-missing.txt
find ./output -type f -name "*.rpm" | xargs rpm -K | grep -i "MISSING KEYS" > $LOGSDIR/rpm-gpg-key-missing.txt || true
# remove all i686.rpms to avoid pollute the chroot dep chain
find ./output -name "*.i686.rpm" | tee $LOGSDIR/all_i686.txt
find ./output -name "*.i686.rpm" | xargs rm -f
line1=`wc -l ${rpms_from_3rd_parties} | cut -d " " -f1-1`
line2=`wc -l ${rpms_from_centos_repo} | cut -d " " -f1-1`
line3=`wc -l ${rpms_from_centos_3rd_parties} | cut -d " " -f1-1`
let total_line=$line1+$line2+$line3
# Count unique rpms. Strip extra fields from 'rpms_from_3rd_partiesIgnore',
# commented out entries, and blank lines.
total_line=$(sed 's/#.*//' ${rpms_from_3rd_parties} \
${rpms_from_centos_repo} \
${rpms_from_centos_3rd_parties} \
| grep -v '^$' \
| sort --unique \
| wc -l)
echo "We expected to download $total_line RPMs."
num_of_downloaded_rpms=`find ./output -type f -name "*.rpm" | wc -l | cut -d" " -f1-1`
num_of_downloaded_rpms=$(find ./output -type f -name "*.rpm" | wc -l | cut -d" " -f1-1)
echo "There are $num_of_downloaded_rpms RPMs in output directory."
if [ "$total_line" != "$num_of_downloaded_rpms" ]; then
echo "WARNING: Not the same number of RPMs in output as RPMs expected to be downloaded, need to check outputs and logs."
@ -365,16 +583,15 @@ if [ $change_group_ids -eq 1 ]; then
chown 751:751 -R ./output
fi
echo "step #3: start downloading other files ..."
echo "step #4: start downloading other files ..."
logfile=$LOGSDIR"/otherfiles_centos_download.log"
${other_downloader} ${dl_flag} -D "$distro" ${other_downloads} ./output/stx-r1/CentOS/pike/Binary/ |& tee $logfile
${other_downloader} ${dl_flag} -D "$distro" ${other_downloads} ${DL_MIRROR_OUTPUT_DIR}/Binary/ |& tee $logfile
retcode=${PIPESTATUS[0]}
if [ $retcode -eq 0 ];then
echo "step #3: done successfully"
echo "step #4: done successfully"
else
echo "step #3: finished with errors"
echo "step #4: finished with errors"
echo "ERROR: Something wrong with downloading from ${other_downloads}."
echo " Please check the log at $(pwd)/$logfile!"
echo ""
@ -384,33 +601,35 @@ fi
# StarlingX requires a group of source code pakages, in this section
# they will be downloaded.
echo "step #4: start downloading tarball compressed files"
echo "step #5: start downloading tarball compressed files"
logfile=$LOGSDIR"/tarballs_download.log"
${tarball_downloader} ${dl_flag} -D "$distro" ${tarball_downloader_extra_args} |& tee $logfile
${tarball_downloader} ${dl_flag} -D "$distro" ${tarball_downloader_extra_args} ${tarball_downloads} |& tee $logfile
retcode=${PIPESTATUS[0]}
if [ $retcode -eq 0 ];then
echo "step #4: done successfully"
echo "step #5: done successfully"
else
echo "step #4: finished with errors"
echo "step #5: finished with errors"
echo "ERROR: Something wrong with downloading tarballs."
echo " Please check the log at $(pwd)/$logfile !"
echo ""
success=0
fi
#
# Clean up the mktemp directory, if required.
#
if [ "$TEMP_DIR" != "" ]; then
echo "rm -rf $TEMP_DIR"
\rm -rf "$TEMP_DIR"
fi
echo "IMPORTANT: The following 3 files are just bootstrap versions. Based"
echo "on them, the workable images for StarlingX could be generated by"
echo "running \"update-pxe-network-installer\" command after \"build-iso\""
echo " - out/stx-r1/CentOS/pike/Binary/LiveOS/squashfs.img"
echo " - out/stx-r1/CentOS/pike/Binary/images/pxeboot/initrd.img"
echo " - out/stx-r1/CentOS/pike/Binary/images/pxeboot/vmlinuz"
echo " - ${DL_MIRROR_OUTPUT_DIR}/Binary/LiveOS/squashfs.img"
echo " - ${DL_MIRROR_OUTPUT_DIR}/Binary/images/pxeboot/initrd.img"
echo " - ${DL_MIRROR_OUTPUT_DIR}/Binary/images/pxeboot/vmlinuz"
echo ""
if [ $success -ne 1 ]; then

57
centos-mirror-tools/make_stx_mirror_yum_conf.sh

@ -24,7 +24,7 @@ RETAIN_REPODIR=0
usage () {
echo ""
echo "$0 -d <dest_dir> [-D <distro>] [-y <src_yum_conf>] [-r <src_repos_dir>] [-R]"
echo "$0 -d <dest_dir> [-D <distro>] [-y <src_yum_conf>] [-r <src_repos_dir>] [-R] [-l <layer>] [-u <lower-layer>,<repo_url>]"
echo ""
echo "Replicate a yum.conf and yum.repo.d under a new directory and"
echo "then modify the files to point to equivalent repos in the StarlingX"
@ -36,13 +36,32 @@ usage () {
echo " 'yum.conf.sample' in same directory as this script"
echo "-r <repos_dir> = Path to yum.repos.d that we will modify. Default is"
echo " 'yum.repos.d' in same directory as this script"
echo "-l <layer> = Download only packages required to build a given layer"
echo "-u <lower-layer>,<build-type>,<repo_url> = Add/change the repo baseurl for a lower layer"
}
declare -A layer_urls
set_layer_urls () {
local option="${1}"
local layer_and_build_type="${option%,*}"
local layer="${layer_and_build_type%,*}"
local build_type="${layer_and_build_type#*,}"
local layer_url="${option##*,}"
# Enforce trailing '/'
if [ "${layer_url:${#layer_url}-1:1}" != "/" ]; then
layer_url+="/"
fi
layer_urls["${layer_and_build_type}"]="${layer_url}"
}
#
# option processing
#
while getopts "D:d:Rr:y:" o; do
while getopts "D:d:l:Rr:u:y:" o; do
case "${o}" in
D)
DISTRO="${OPTARG}"
@ -50,12 +69,18 @@ while getopts "D:d:Rr:y:" o; do
d)
TEMP_DIR="${OPTARG}"
;;
l)
LAYER="${OPTARG}"
;;
r)
SRC_REPO_DIR="${OPTARG}"
;;
R)
RETAIN_REPODIR=1
;;
u)
set_layer_urls "${OPTARG}"
;;
y)
SRC_YUM_CONF="${OPTARG}"
;;
@ -136,11 +161,18 @@ ARCH=$(get_arch)
#
# Copy as yet unmodified yum.conf and yum.repos.d from source to dest.
#
echo "\cp -r '$SRC_REPO_DIR' '$CENGN_REPOS_DIR'"
\cp -r "$SRC_REPO_DIR" "$CENGN_REPOS_DIR"
mkdir -p "$CENGN_REPOS_DIR"
echo "\cp -r '$SRC_REPO_DIR/*' '$CENGN_REPOS_DIR/'"
\cp -r "$SRC_REPO_DIR"/* "$CENGN_REPOS_DIR/"
echo "\cp '$SRC_YUM_CONF' '$CENGN_YUM_CONF'"
\cp "$SRC_YUM_CONF" "$CENGN_YUM_CONF"
if [ "$LAYER" != "all" ]; then
if [ -d ${MAKE_STX_MIRROR_YUM_CONF_DIR}/config/${DISTRO}/${LAYER}/yum.repos.d ]; then
\cp -f ${MAKE_STX_MIRROR_YUM_CONF_DIR}/config/${DISTRO}/${LAYER}/yum.repos.d/*.repo $CENGN_REPOS_DIR
fi
fi
#
# Add or modify reposdir= value in our new yum.conf
#
@ -229,4 +261,21 @@ for REPO in $(find "$CENGN_REPOS_DIR" -type f -name '*repo'); do
sed "s#^\[\([^]]*\)\]#[CENGN_\1]#" -i "$REPO"
done
for key in "${!layer_urls[@]}"; do
lower_layer="${key%,*}"
build_type="${key#*,}"
REPO="$CENGN_REPOS_DIR/StarlingX_cengn_${lower_layer}_layer.repo"
if [ -f "$REPO" ]; then
sed "s#^baseurl=.*/${lower_layer}/.*/${build_type}/\$#baseurl=${layer_urls[${key}]}#" -i "$REPO"
else
REPO="$CENGN_REPOS_DIR/StarlingX_local_${lower_layer}_${build_type}_layer.repo"
(
echo "[Starlingx-local_${lower_layer}_${build_type}_layer]"
echo "name=Starlingx-cengn_${lower_layer}_${build_type}_layer"
echo "baseurl=${layer_urls[${key}]}"
echo "enabled=1"
) > "$REPO"
fi
done
echo $TEMP_DIR

82
centos-mirror-tools/tarball-dl.lst

@ -1,82 +0,0 @@
blkin-f24ceec055ea236a093988237a9821d145f5f7c8.tar.gz#blkin#https://api.github.com/repos/ceph/blkin/tarball/f24ceec055ea236a093988237a9821d145f5f7c8#https##
boost_1_67_0.tar.bz2#boost_1_67_0#https://dl.bintray.com/boostorg/release/1.67.0/source/boost_1_67_0.tar.bz2#https##
ceph-erasure-code-corpus-2d7d78b9cc52e8a9529d8cc2d2954c7d375d5dd7.tar.gz#ceph-erasure-code-corpus#https://api.github.com/repos/ceph/ceph-erasure-code-corpus/tarball/2d7d78b9cc52e8a9529d8cc2d2954c7d375d5dd7#https##
ceph-object-corpus-e32bf8ca3dc6151ebe7f205ba187815bc18e1cef.tar.gz#ceph-object-corpus#https://api.github.com/repos/ceph/ceph-object-corpus/tarball/e32bf8ca3dc6151ebe7f205ba187815bc18e1cef#https##
civetweb-ff2881e2cd5869a71ca91083bad5d12cccd22136.tar.gz#civetweb#https://api.github.com/repos/ceph/civetweb/tarball/ff2881e2cd5869a71ca91083bad5d12cccd22136#https##
containerd-v1.3.0.tar.gz#containerd#https://github.com/containerd/containerd/archive/v1.3.0.tar.gz#https##
crictl-v1.16.0-linux-amd64.tar.gz#crictl#https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.16.0/crictl-v1.16.0-linux-amd64.tar.gz#https##
dep-v0.5.0.tar.gz#dep-v0.5.0#https://github.com/golang/dep/archive/v0.5.0.tar.gz#http##
docker-distribution-48294d928ced5dd9b378f7fd7c6f5da3ff3f2c89.tar.gz#docker-distribution#github.com/docker/distribution/archive/48294d928ced5dd9b378f7fd7c6f5da3ff3f2c89.tar.gz#http##
docker-libtrust-fa567046d9b14f6aa788882a950d69651d230b21.tar.gz#docker-libtrust#https://github.com/docker/libtrust/archive/fa567046d9b14f6aa788882a950d69651d230b21.tar.gz#http##
dpdk-18.11.tar.xz#dpdk-18.11#http://fast.dpdk.org/rel/dpdk-18.11.tar.xz#http##
dpdk-6ece49ad5a26f5e2f5c4af6c06c30376c0ddc387.tar.gz#dpdk#https://api.github.com/repos/spdk/dpdk/tarball/6ece49ad5a26f5e2f5c4af6c06c30376c0ddc387#https##
dpkg_1.18.24.tar.xz#dpkg-1.18.24#http://http.debian.net/debian/pool/main/d/dpkg/dpkg_1.18.24.tar.xz#http##
drbd-8.4.11-1.tar.gz#drbd-8.4.11-1#http://www.linbit.com/downloads/drbd/8.4/drbd-8.4.11-1.tar.gz#http##
drbd-8.4.3.tar.gz#drbd-8.4.3#http://www.linbit.com/downloads/drbd/8.4/archive/drbd-8.4.3.tar.gz#http##
dtc-1.4.4.tar.gz#dtc-1.4.4#https://www.kernel.org/pub/software/utils/dtc/dtc-1.4.4.tar.gz#http##
e1000e-3.4.2.1.tar.gz#e1000e-3.4.2.1#https://sourceforge.net/projects/e1000/files/e1000e%20stable/3.4.2.1/e1000e-3.4.2.1.tar.gz#http##
etcd-v3.3.15.tar.gz#etcd-v3.3.15#https://github.com/etcd-io/etcd/archive/v3.3.15.tar.gz#http##
gf-complete-7e61b44404f0ed410c83cfd3947a52e88ae044e1.tar.gz#gf-complete#https://api.github.com/repos/ceph/gf-complete/tarball/7e61b44404f0ed410c83cfd3947a52e88ae044e1#https##
gnocchi-4.2.5.tar.gz#gnocchi-4.2.5#https://pypi.io/packages/source/g/gnocchi/gnocchi-4.2.5.tar.gz#http##
gnocchiclient-7.0.1.tar.gz#gnocchiclient-7.0.1#https://pypi.io/packages/source/g/gnocchiclient/gnocchiclient-7.0.1.tar.gz#http##
gnulib-ffc927e.tar.gz#gnulib-ffc927e#http://git.savannah.gnu.org/cgit/gnulib.git/snapshot/gnulib-ffc927eef29016a5219cd969daad8928af6a1f4d.tar.gz#http##
go1.12.10.src.tar.gz#go#https://dl.google.com/go/go1.12.10.src.tar.gz##https##
googletest-0a439623f75c029912728d80cb7f1b8b48739ca4.tar.gz#googletest#https://api.github.com/repos/google/googletest/tarball/0a439623f75c029912728d80cb7f1b8b48739ca4#https##
googletest-fdb850479284e2aae047b87df6beae84236d0135.tar.gz#googletest#https://api.github.com/repos/ceph/googletest/tarball/fdb850479284e2aae047b87df6beae84236d0135#https##
gophercloud-gophercloud-aa00757ee3ab58e53520b6cb910ca0543116400a.tar.gz#gophercloud-gophercloud#github.com/gophercloud/gophercloud/archive/aa00757ee3ab58e53520b6cb910ca0543116400a.tar.gz#http##
gorilla-context-08b5f424b9271eedf6f9f0ce86cb9396ed337a42.tar.gz#gorilla-context#https://github.com/gorilla/context/archive/08b5f424b9271eedf6f9f0ce86cb9396ed337a42.tar.gz#http##
gorilla-mux-456bcfa82d672db7cae587c9b541463f65bc2718.tar.gz#gorilla-mux#https://github.com/gorilla/mux/archive/456bcfa82d672db7cae587c9b541463f65bc2718.tar.gz#http##
helm-charts-92b6289ae93816717a8453cfe62bad51cbdb8ad0.tar.gz#helm-charts#https://github.com/helm/charts/archive/92b6289ae93816717a8453cfe62bad51cbdb8ad0.tar.gz#http##
helm-charts-elastic-2bd7616ceddbdf2eee88965e2028ee37d304c79c.tar.gz#helm-charts-elastic#https://github.com/elastic/helm-charts/archive/2bd7616ceddbdf2eee88965e2028ee37d304c79c.tar.gz#http##
helm-v2.13.1-linux-amd64.tar.gz#linux-amd64#https://storage.googleapis.com/kubernetes-helm/helm-v2.13.1-linux-amd64.tar.gz#http##
i40e-2.10.19.30.tar.gz#i40e-2.10.19.30#https://sourceforge.net/projects/e1000/files/i40e%20stable/2.10.19.30/i40e-2.10.19.30.tar.gz/download#http##
iavf-3.7.61.20.tar.gz#iavf-3.7.61.20#https://sourceforge.net/projects/e1000/files/iavf%20stable/3.7.61.20/iavf-3.7.61.20.tar.gz/download#http##
!integrity-kmod-e6aef069.tar.gz#integrity#http://git.infradead.org/users/jjs/linux-tpmdd.git/snapshot/e6aef069b6e97790cb127d5eeb86ae9ff0b7b0e3.tar.gz#http_script##post-dl-script/integrity-kmod.sh
isa-l-7e1a337433a340bc0974ed0f04301bdaca374af6.tar.gz#isa-l#https://api.github.com/repos/ceph/isa-l/tarball/7e1a337433a340bc0974ed0f04301bdaca374af6#https##
isa-l_crypto-603529a4e06ac8a1662c13d6b31f122e21830352.tar.gz#isa-l_crypto#https://api.github.com/repos/01org/isa-l_crypto/tarball/603529a4e06ac8a1662c13d6b31f122e21830352#https##
ixgbe-5.5.5.tar.gz#ixgbe-5.5.5#https://sourceforge.net/projects/e1000/files/ixgbe%20stable/5.5.5/ixgbe-5.5.5.tar.gz/download#http##
ixgbevf-4.5.3.tar.gz#ixgbevf-4.5.3#https://sourceforge.net/projects/e1000/files/ixgbevf%20stable/4.5.3/ixgbevf-4.5.3.tar.gz/download#http##
jerasure-96c76b89d661c163f65a014b8042c9354ccf7f31.tar.gz#jerasure#https://api.github.com/repos/ceph/jerasure/tarball/96c76b89d661c163f65a014b8042c9354ccf7f31#https##
keycodemapdb-16e5b07.tar.gz#keycodemapdb#https://github.com/CendioOssman/keycodemapdb/tarball/16e5b0787687d8904dad2c026107409eb9bfcb95#http##
kubernetes-contrib-v1.16.2.tar.gz#kubernetes-contrib-1.16.2#https://github.com/kubernetes-retired/contrib/tarball/89f6948e24578fed2a90a87871b2263729f90ac3#http##
kubernetes-v1.16.2.tar.gz#kubernetes-1.16.2#https://github.com/kubernetes/kubernetes/archive/v1.16.2.tar.gz#http##
kvm-unit-tests.git-4ea7633.tar.bz2#kvm-unit-tests#https://git.kernel.org/pub/scm/virt/kvm/kvm-unit-tests.git/snapshot/kvm-unit-tests-20171020.tar.gz#http##
ldapscripts-2.0.8.tgz#ldapscripts-2.0.8#https://sourceforge.net/projects/ldapscripts/files/ldapscripts/ldapscripts-2.0.8/ldapscripts-2.0.8.tgz/download#http##
libtpms-0.6.0-4f0d59d.tar.gz#libtpms-0.6.0#https://github.com/stefanberger/libtpms/tarball/c421ca0f4d00c0caceeda8d62c1efb2b7e47db04#http##
lldpd-0.9.0.tar.gz#lldpd-0.9.0#https://media.luffy.cx/files/lldpd/lldpd-0.9.0.tar.gz#http##
lua-1fce39c6397056db645718b8f5821571d97869a4.tar.gz#lua#https://api.github.com/repos/ceph/lua/tarball/1fce39c6397056db645718b8f5821571d97869a4#https##
!mariadb-10.1.28.tar.gz#mariadb-10.1.28#https://github.com/MariaDB/server/archive/mariadb-10.1.28.tar.gz#http_script##post-dl-script/mariadb.sh
!MLNX_OFED_SRC-4.5-1.0.1.0.tgz#MLNX_OFED_SRC-4.5-1.0.1.0#http://content.mellanox.com/ofed/MLNX_OFED-4.5-1.0.1.0/MLNX_OFED_SRC-4.5-1.0.1.0.tgz#http_script##post-dl-script/MLNX_OFEDqat1.7.sh
opae-intel-fpga-driver-1.3.0-2.tar.gz#opae-intel-fpga-driver-1.3.0-2#https://github.com/OPAE/opae-sdk/releases/download/1.3.0-2/opae-intel-fpga-driver-1.3.0-2.tar.gz#http##
openstack-helm-82c72367c85ca94270f702661c7b984899c1ae38.tar.gz#openstack-helm#https://github.com/openstack/openstack-helm/archive/82c72367c85ca94270f702661c7b984899c1ae38.tar.gz#http##
openstack-helm-infra-c9d6676bf9a5aceb311dc31dadd07cba6a3d6392.tar.gz#openstack-helm-infra#https://github.com/openstack/openstack-helm-infra/archive/c9d6676bf9a5aceb311dc31dadd07cba6a3d6392.tar.gz#http##
openvswitch-2.11.0.tar.gz#openvswitch-2.11.0#https://github.com/openvswitch/ovs/archive/v2.11.0.tar.gz#http##
puppet-boolean-22b726dd78b0a60a224cc7054aebbf28e9306f62.tar.gz#puppet-boolean#https://github.com/voxpupuli/puppet-boolean/tarball/22b726dd78b0a60a224cc7054aebbf28e9306f62#http##
puppet-dnsmasq-cff07e90890662972c97684a2baee964f68ff3ed.tar.gz#packstack/puppet/modules/dnsmasq/#https://github.com/procore/puppet-dnsmasq/tarball/cff07e90890662972c97684a2baee964f68ff3ed#http##
puppet-filemapper-9b53310278e76827bbe12a36cc6470d77071abb2.tar.gz#packstack/puppet/modules/filemapper#https://github.com/voxpupuli/puppet-filemapper/tarball/9b53310278e76827bbe12a36cc6470d77071abb2#http##
puppetlabs-create_resources-4639819a7f3a4fa9310d2ba583c63e467df7e2c3.tar.gz#packstack/puppet/modules/create_resources#https://github.com/puppetlabs/puppetlabs-create_resources/tarball/4639819a7f3a4fa9310d2ba583c63e467df7e2c3#http##
puppetlabs-drbd-496b3ba9cd74a2d12636f9e90a718739a5451169.tar.gz#puppetlabs-drbd#https://github.com/voxpupuli/puppet-drbd/tarball/496b3ba9cd74a2d12636f9e90a718739a5451169#http##
puppetlabs-lvm-d0283da637ae24550fb4ba109a48ef8d5d8c8b84.tar.gz#packstack/puppet/modules/lvm#https://github.com/puppetlabs/puppetlabs-lvm/tarball/d0283da637ae24550fb4ba109a48ef8d5d8c8b84#http##
puppetlabs-postgresql-d022a56b28b2174456fc0f6adc51a4b54493afad.tar.gz#puppetlabs-postgresql#https://github.com/puppetlabs/puppetlabs-postgresql/tarball/d022a56b28b2174456fc0f6adc51a4b54493afad#http##
puppet-ldap-480f13af6d17d1d3fcf0dc7b4bd04b49fa4099e1.tar.gz#puppet-ldap-master#https://github.com/torian/puppet-ldap/tarball/480f13af6d17d1d3fcf0dc7b4bd04b49fa4099e1#http##
puppet-network-7deacd5fdc22c0543455878a8d1872f2f5417c1d.tar.gz#packstack/puppet/modules/network#https://github.com/voxpupuli/puppet-network/tarball/7deacd5fdc22c0543455878a8d1872f2f5417c1d#http##
puppet-nslcd-b8c19b1ada89865f2e50758e054583798ad8011a.tar.gz#packstack/puppet/modules/nslcd#https://github.com/jlyheden/puppet-nslcd/tarball/b8c19b1ada89865f2e50758e054583798ad8011a#http##
puppi-c1c47f4edfd761d1bbde32a75da0c3fa7cc93a81.tar.gz#puppi-master#https://github.com/example42/puppi/tarball/c1c47f4edfd761d1bbde32a75da0c3fa7cc93a81#http##
python-cephclient-v0.1.0.5.tar.gz#python-cephclient-0.1.0.5#https://github.com/dmsimard/python-cephclient/archive/v0.1.0.5.tar.gz#http##
python-setuptools-v38.5.1.tar.gz#setuptools-38.5.1#https://github.com/pypa/setuptools/archive/v38.5.1.tar.gz#http##
!qat1.7.l.4.5.0-00034.tar.gz#quickassist#https://01.org/sites/default/files/downloads/qat1.7.l.4.5.0-00034.tar.gz#http_script##post-dl-script/qat1.7.sh
rapidjson-f54b0e47a08782a6131cc3d60f94d038fa6e0a51.tar.gz#rapidjson#https://api.github.com/repos/ceph/rapidjson/tarball/f54b0e47a08782a6131cc3d60f94d038fa6e0a51#https##
Redfishtool-1.1.0.tar.gz#Redfishtool-1.1.0#https://github.com/DMTF/Redfishtool/archive/1.1.0.tar.gz#http##
requests-toolbelt-0.9.1.tar.gz#requests-toolbelt-0.9.1#https://github.com/requests/toolbelt/archive/0.9.1.tar.gz#http##
rocksdb-f4a857da0b720691effc524469f6db895ad00d8e.tar.gz#rocksdb#https://api.github.com/repos/ceph/rocksdb/tarball/f4a857da0b720691effc524469f6db895ad00d8e#https##
rook-1.1.1.tar.gz#rook-1.1.1#https://github.com/rook/rook/archive/v1.1.1.tar.gz##
rpm-4.14.0.tar.bz2#rpm-4.14.0#https://ftp.osuosl.org/pub/rpm/releases/rpm-4.14.x/rpm-4.14.0.tar.bz2#http##
runc-1.0.0-rc8.tar.gz#runc#https://github.com/opencontainers/runc/archive/v1.0.0-rc8.tar.gz#https##
Sirupsen-logrus-55eb11d21d2a31a3cc93838241d04800f52e823d.tar.gz#Sirupsen-logrus#github.com/Sirupsen/logrus/archive/55eb11d21d2a31a3cc93838241d04800f52e823d.tar.gz#http##
spdk-f474ce6930f0a44360e1cc4ecd606d2348481c4c.tar.gz#spdk#https://api.github.com/repos/ceph/spdk/tarball/f474ce6930f0a44360e1cc4ecd606d2348481c4c#https##
spectre-meltdown-checker-0.37+-5cc77741.tar.gz#spectre-meltdown-checker#https://github.com/speed47/spectre-meltdown-checker/tarball/5cc77741af1d2f52140aa9f89339f56b6c4b6783#http##
swtpm-0.1.0-253eac5.tar.gz#swtpm-0.1.0#https://github.com/stefanberger/swtpm/tarball/1303be7d03294fb02204cb8242f366cbf0da076d#http##
!tpm-kmod-e6aef069.tar.gz#tpm#http://git.infradead.org/users/jjs/linux-tpmdd.git/snapshot/e6aef069b6e97790cb127d5eeb86ae9ff0b7b0e3.tar.gz#http_script#post-dl-script/tpm-kmod.sh
!tss2-930.tar.gz#tss2-930#https://git.code.sf.net/p/ibmtpm20tss/tss#git#v930#
xxHash-1f40c6511fa8dd9d2e337ca8c9bc18b3e87663c9.tar.gz#xxHash#https://api.github.com/repos/ceph/xxHash/tarball/1f40c6511fa8dd9d2e337ca8c9bc18b3e87663c9#https##
zstd-f4340f46b2387bc8de7d5320c0b83bb1499933ad.tar.gz#zstd#https://api.github.com/repos/facebook/zstd/tarball/f4340f46b2387bc8de7d5320c0b83bb1499933ad#https##

2
centos-mirror-tools/utils.sh

@ -20,7 +20,7 @@ get_yum_command() {
if [ "$arch" == "src" ]; then
yumdownloader_extra_opts="--source"
else
yumdownloader_extra_opts="--archlist=noarch,x86_64"
yumdownloader_extra_opts="--exclude='*.i686' --archlist=noarch,x86_64"
fi
echo "yumdownloader -q -C ${YUMCONFOPT} ${RELEASEVER} $yumdownloader_extra_opts $rpm_name"

1
tb.sh

@ -89,6 +89,7 @@ case $CMD in
echo "HOST_MIRROR_DIR=${HOST_MIRROR_DIR}"
echo "MY_TC_RELEASE=${MY_TC_RELEASE}"
echo "MY_REPO_ROOT_DIR=${MY_REPO_ROOT_DIR}"
echo "LAYER=${LAYER}"
;;
create)
create_container

335
toCOPY/generate-cgcs-centos-repo.sh

@ -5,14 +5,64 @@
# Copyright (C) 2019 Intel Corporation
#
GENERATE_CGCS_CENTOS_REPO_DIR="$(dirname "$(readlink -f "${BASH_SOURCE[0]}" )" )"
source $GENERATE_CGCS_CENTOS_REPO_DIR/lst_utils.sh
mirror_dir=""
layer_dirs=""
CREATEREPO=$(which createrepo_c)
if [ $? -ne 0 ]; then
CREATEREPO="createrepo"
fi
usage () {
echo "$0 <mirror-path>"
echo
echo "Create a virtual rpm repo containing only rpms listed in various lst files."
echo "The virtual repo contains only symlinks to to previously downloaded or built rpms."
echo
echo "Usage"
echo
echo "$0 [Options] [ --mirror-dir=<mirror-path> | <mirror-path> ]"
echo
echo "Commin Options:"
echo " --distro=<distro>: Create repo for the designated distro."
echo " Default 'centos'"
echo " --layer=<layer>: Create a smaller repo, sufficient to build"
echo " only the given layer."
echo " Default: use the LAYER environmnet valiable, or 'all'."
echo " --mirror-dir=<dir>: Set the mirror directory. This is where"
echo " the previously downloaded rpms are located."
echo
echo "Override options: For use when working on a multi-layer change"
echo " --config-dir=<dir>: Use an alternate config directory rather than the"
echo " system defined one"
echo " --layer-inc-url=<lower_layer>,<build_type>,<url>:"
echo " Override the url for the image include file of a lower"
echo " layer's build type. Normally the url(s) is read from"
echo " <config_dir>/<distro>/<layer>/required_layer_iso_inc.cfg"
echo " This option can be used more than once."
echo " --layer-pkg-url=<lower_layer>,<build_type>,<url>:"
echo " Override the url for the package list of a lower"
echo " layer's build type. Normally the url(s) is read from"
echo " <config_dir>/<distro>/<layer>/required_layer_pkgs.cfg."
echo " This option can be used more than once."
echo " --layer-dir=<dir>: Look in provided dir for packages to link to."
echo " This option can be used more than once."
echo
}
if [ $# -ne 1 ]; then
usage
exit -1
fi
cleanup () {
if [ -e "${mirror_content}" ]; then
\rm -f ${mirror_content}
fi
if [ -e "${TMP_LST_DIR}" ]; then
\rm -rf ${TMP_LST_DIR}
fi
}
trap "cleanup ; exit 1" INT
if [ -z "$MY_REPO" ]; then
echo "\$MY_REPO is not set. Ensure you are running this script"
@ -21,7 +71,50 @@ if [ -z "$MY_REPO" ]; then
exit -1
fi
mirror_dir=$1
TEMP=$(getopt -o h --long help,config-dir:,distro:,layer:,layer-dir:,layer-inc-url:,layer-pkg-url:,mirror-dir: -n 'generate-cgcs-centos-repo' -- "$@")
if [ $? -ne 0 ]; then
echo "getopt error"
usage
exit 1
fi
eval set -- "$TEMP"
while true ; do
case "$1" in
--mirror-dir) mirror_dir=$2 ; shift 2 ;;
--layer-dir) layer_dirs+=" ${2/,/ }" ; shift 2 ;;
--layer-inc-url) set_layer_image_inc_urls "${2}" ; shift 2 ;;
--layer-pkg-url) set_layer_pkg_urls "${2}" ; shift 2 ;;
--config-dir) set_and_validate_config_dir "${2}"; shift 2 ;;
--distro) set_and_validate_distro "${2}"; shift 2 ;;
--layer) set_and_validate_layer "${2}"; shift 2 ;;
-h|--help) echo "help"; usage; exit 0 ;;
--) shift ; break ;;
*) usage; exit 1 ;;
esac
done
if [ "$mirror_dir" == "" ]; then
if [ $# -ne 1 ]; then
usage
exit -1
fi
mirror_dir=$1
fi
echo "mirror_dir=${mirror_dir}"
echo "config_dir=${config_dir}"
echo "distro=${distro}"
echo "layer=${layer}"
echo
echo "layer_pkg_urls=${layer_pkg_urls[@]}"
echo
echo "layer_image_inc_urls=${layer_image_inc_urls[@]}"
echo
dest_dir=$MY_REPO/cgcs-centos-repo
timestamp="$(date +%F_%H%M)"
mock_cfg_file=$MY_REPO/build-tools/repo_files/mock.cfg.proto
@ -29,12 +122,24 @@ comps_xml_file=$MY_REPO/build-tools/repo_files/comps.xml
mock_cfg_dest_file=$MY_REPO/cgcs-centos-repo/mock.cfg.proto
comps_xml_dest_file=$MY_REPO/cgcs-centos-repo/Binary/comps.xml
lst_file_dir="$MY_REPO_ROOT_DIR/stx-tools/centos-mirror-tools"
TMP_LST_DIR=$(mktemp -d /tmp/tmp_lst_dir_XXXXXX)
mkdir -p $TMP_LST_DIR
lst_file_dir="$TMP_LST_DIR"
inc_file_dir="${dest_dir}/layer_image_inc"
build_info_file_dir="${dest_dir}/layer_build_info"
rpm_lst_files="rpms_3rdparties.lst rpms_centos3rdparties.lst rpms_centos.lst"
rpm_lst_files_rt=""
other_lst_file="other_downloads.lst"
for template in $rpm_lst_files $other_lst_file; do
lst="$lst_file_dir/${template}"
merge_lst ${config_dir} ${distro} ${template} > ${lst}
done
missing_rpms_file=missing.txt
rm -f ${missing_rpms_file}
\rm -f ${missing_rpms_file}
# Strip trailing / from mirror_dir if it was specified...
mirror_dir=$(echo ${mirror_dir} | sed "s%/$%%")
@ -45,6 +150,13 @@ if [[ ( ! -d ${mirror_dir}/Binary ) || ( ! -d ${mirror_dir}/Source ) ]]; then
exit -1
fi
for layer_dir in ${layer_dirs}; do
if [ ! -d ${layer_dir} ]; then
echo "The layer-dir ${layer_dir} doesn't exist"
exit -1
fi
done
if [ ! -d "${dest_dir}" ]; then
mkdir -p "${dest_dir}"
fi
@ -59,15 +171,149 @@ for t in "Binary" "Source" ; do
fi
done
mirror_content=$(mktemp -t centos-repo-XXXXXX)
find -L ${mirror_dir} -type f > ${mirror_content}
#
# Dowload image inc files from layer_image_inc_urls
#
\rm -rf ${inc_file_dir}
mkdir -p ${inc_file_dir}
for key in "${!layer_image_inc_urls[@]}"; do
lower_layer="${key%,*}"
inc_type="${key#*,}"
url="${layer_image_inc_urls[${key}]}"
name_from_url=$(url_to_file_name "${url}")
if [ "${inc_type}" == "std" ]; then
ideal_name="${lower_layer}_${image_inc_from_layer_build_template}"
elif [ "${inc_type}" == "dev" ]; then
ideal_name="${lower_layer}_${dev_image_inc_from_layer_build_template}"
else
ideal_name="${lower_layer}_${inc_type}_${image_inc_from_layer_build_template}"
fi
list="${ideal_name}"
for f in $(find -L ${layer_dirs} ${mirror_dir} -type f -name "${name_from_url}"); do
cp $f ${inc_file_dir}/${list}
if [ $? -ne 0 ]; then
echo "WARNING: Failed to copy from cached file '$f' to satisfy url '${url}'"
fi
done
if [ ! -f ${inc_file_dir}/${list} ]; then
curl --silent --fail ${url} > ${inc_file_dir}/${list}
if [ $? -ne 0 ]; then
echo "ERROR: Failed to download from url '${url}'"
exit 1
fi
fi
done
#
# Dowload build info files
#
build_info_from_layer_build_template="BUILD_INFO"
\rm -rf ${build_info_file_dir}
mkdir -p ${build_info_file_dir}
for key in "${!layer_image_inc_urls[@]}"; do
lower_layer="${key%,*}"
inc_type="${key#*,}"
if [ "${inc_type}" != "std" ]; then
continue
fi
if [ "$(basename ${layer_image_inc_urls[${key}]})" != "image.inc" ]; then
continue
fi
url=$( echo ${layer_image_inc_urls[${key}]} | sed 's#image.inc$#BUILD_INFO#' )
name_from_url=$(url_to_file_name "${url}")
ideal_name="${lower_layer}_${build_info_from_layer_build_template}"
list="${ideal_name}"
for f in $(find -L ${layer_dirs} ${mirror_dir} -type f -name "${name_from_url}"); do
cp $f ${build_info_file_dir}/${list}
if [ $? -ne 0 ]; then
echo "WARNING: Failed to copy from cached file '$f' to satisfy url '${url}'"
fi
done
if [ ! -f ${build_info_file_dir}/${list} ]; then
curl --silent --fail ${url} > ${build_info_file_dir}/${list}
if [ $? -ne 0 ]; then
echo "WARNING: Failed to download from url '${url}'"
fi
fi
done
#
# Dowload lst files from layer_pkg_urls
#
for key in "${!layer_pkg_urls[@]}"; do
lower_layer="${key%,*}"
build_type="${key#*,}"
url="${layer_pkg_urls[${key}]}"
name_from_url=$(url_to_file_name "${url}")
ideal_name="${lower_layer}_${build_type}_${rpms_from_layer_build_template}"
list="${ideal_name}"
for f in $(find -L ${layer_dirs} ${mirror_dir} -type f -name "${name_from_url}"); do
cp $f ${lst_file_dir}/${list}
if [ $? -ne 0 ]; then
echo "WARNING: Failed to copy from cached file '$f' to satisfy url '${url}'"
fi
done
if [ ! -f ${lst_file_dir}/${list} ]; then
curl --silent --fail ${url} > ${lst_file_dir}/${list}
if [ $? -ne 0 ]; then
echo "ERROR: Failed to download from url '${url}'"
exit 1
fi
fi
if [ "${build_type}" == "rt" ]; then
rpm_lst_files_rt+=" ${list}"
else
rpm_lst_files+=" ${list}"
fi
url_type=${url%%:*}
if [ "${url_type}" == "file" ]; then
url_dir=$(dirname ${url#file://})
if [ ! -d ${url_dir} ]; then
echo "ERROR: No such directory '${url_dir}' derived from url: ${url}"
exit 1
fi
layer_dirs+=" ${url_dir}"
fi
done
echo "rpm_lst_files=${rpm_lst_files}"
echo "rpm_lst_files_rt=${rpm_lst_files_rt}"
echo "layer_dirs=${layer_dirs}"
mirror_content=$(mktemp -t centos-repo-XXXXXX)
find -L ${layer_dirs} ${mirror_dir} -type f -name '*.rpm' > ${mirror_content}
sed_expression=""
for d in ${mirror_dir} ${layer_dirs}; do
sed_expression+=" -e s%^${d}/%%"
done
process_lst_file () {
local lst_file="${1}"
local dest_dir="${2}"
for lst_file in ${rpm_lst_files} ; do
grep -v "^#" ${lst_file_dir}/${lst_file} | while IFS="#" read rpmname extrafields; do
if [ -z "${rpmname}" ]; then
continue
fi
mirror_file=$(grep "/${rpmname}$" ${mirror_content})
mirror_file=$(grep "/${rpmname}$" ${mirror_content} | head -n 1)
if [ -z "${mirror_file}" ]; then
echo "Error -- could not find requested ${rpmname} in ${mirror_dir}"
echo ${rpmname} >> ${missing_rpms_file}
@ -75,9 +321,14 @@ for lst_file in ${rpm_lst_files} ; do
fi
# Great, we found the file! Let's strip the mirror_dir prefix from it...
ff=$(echo ${mirror_file} | sed "s%^${mirror_dir}/%%")
ff=$(echo ${mirror_file} | sed ${sed_expression})
f_name=$(basename "$ff")
sub_dir=$(dirname "$ff")
arch=$(echo ${f_name} | rev | cut -d '.' -f 2 | rev)
if [ "${arch}" == "src" ]; then
sub_dir="Source"
else
sub_dir="Binary/${arch}"
fi
# Make sure we have a subdir (so we don't symlink the first file as
# the subdir name)
@ -85,36 +336,68 @@ for lst_file in ${rpm_lst_files} ; do
# Link it!
echo "Creating symlink for ${dest_dir}/${sub_dir}/${f_name}"
ln -sf "${mirror_dir}/$ff" "${dest_dir}/${sub_dir}"
ln -sf "${mirror_file}" "${dest_dir}/${sub_dir}/${f_name}"
if [ $? -ne 0 ]; then
echo "Failed ${mirror_file}: ln -sf \"${mirror_dir}/$ff\" \"${dest_dir}/${sub_dir}\""
echo "Failed ${mirror_file}: ln -sf \"${mirror_file}\" \"${dest_dir}/${sub_dir}\""
fi
done
}
for lst_file in ${rpm_lst_files} ; do
process_lst_file "${lst_file}" "${dest_dir}" || exit 1
done
rm -f ${mirror_content}
for lst_file in ${rpm_lst_files_rt} ; do
process_lst_file "${lst_file}" "${dest_dir}/rt" || exit 1
done
if [ ! -f "$mock_cfg_file" ]; then
echo "Cannot find mock.cfg.proto file!"
exit 1
fi
echo "Copying comps.xml file."
if [ ! -f "$comps_xml_file" ]; then
echo "Cannot find comps.xml file!"
exit 1
fi
echo "Copying mock.cfg.proto and comps.xml files."
if [ -f "$comps_xml_dest_file" ]; then
\cp -f "$comps_xml_dest_file" "$comps_xml_dest_file-backup-$timestamp"
fi
cp "$comps_xml_file" "$comps_xml_dest_file"
echo "Createing yum repodata."
for subdir in Source Binary; do
repo_dir="${dest_dir}/${subdir}"
mkdir -p "${repo_dir}"
if [ -f "${repo_dir}/comps.xml" ]; then
${CREATEREPO} -g "${repo_dir}/comps.xml" -d "${repo_dir}"
else
${CREATEREPO} -d "${repo_dir}"
fi
repo_dir="${dest_dir}/rt/${subdir}"
mkdir -p "${repo_dir}"
if [ -f "${repo_dir}/comps.xml" ]; then
${CREATEREPO} -g "${repo_dir}/comps.xml" -d "${repo_dir}"
else
${CREATEREPO} -d "${repo_dir}"
fi
done
echo "Copying mock.cfg.proto file."
if [ ! -f "$mock_cfg_file" ]; then
echo "Cannot find mock.cfg.proto file!"
exit 1
fi
if [ -f "$mock_cfg_dest_file" ]; then
\cp -f "$mock_cfg_dest_file" "$mock_cfg_dest_file-backup-$timestamp"
fi
cp "$mock_cfg_file" "$mock_cfg_dest_file"
if [ -f "$comps_xml_dest_file" ]; then
\cp -f "$comps_xml_dest_file" "$comps_xml_dest_file-backup-$timestamp"
fi
cp "$comps_xml_file" "$comps_xml_dest_file"
# Populate the contents from other list files
cat ${lst_file_dir}/${other_lst_file} | grep -v "#" | while IFS=":" read targettype item extrafields; do

207
toCOPY/lst_utils.sh Normal file

@ -0,0 +1,207 @@
#
# Copyright (c) 2020 Wind River Systems, Inc.
#
# SPDX-License-Identifier: Apache-2.0
#
rpms_from_layer_build_template="rpm.lst"
image_inc_from_layer_build_template="image.inc"
dev_image_inc_from_layer_build_template="image-dev.inc"
config_dir=${MY_REPO}/../stx-tools/centos-mirror-tools/config
distro="centos"
layer="all"
# Store urls for package list files of the various layers in an associative array
declare -A layer_pkg_urls
# Store urls for image include files of the various layers in an associative array
declare -A layer_image_inc_urls
url_to_file_name () {
echo "${1}" | sed 's#[:/ ]#-#g'
}
merge_lst () {
local cfg_dir=$1
local distro=$2
local template=$3
local cfg_name="${distro}_build_layer.cfg"
local layer_cfgs
local layers
local layer
if [ "$cfg_dir" == "" ] || [ "$distro" == "" ] || [ "$template" == "" ]; then
echo "ERROR: merge_lst: missing argument" >&2
return 1
fi
if [ "$MY_REPO" == "" ]; then
echo "ERROR: merge_lst: environment variable MY_REPO must be set" >&2
return 1
fi
layer_cfgs=$(find ${MY_REPO} -maxdepth 3 -name ${cfg_name})
if [ "$layer_cfgs" == "" ]; then
echo "ERROR: merge_lst: Could not find any '${cfg_name}' files" >&2
return 1
fi
layers=$(cat ${layer_cfgs} | sort --unique)
layers+=" mock"
(
for layer in ${layers}; do
for f in $(find ${cfg_dir}/${distro}/${layer} -name ${template} ); do
grep -v '^#' $f || true
done
done
for f in $(find ${MY_REPO} -maxdepth 3 -name ${distro}_${template}; \
find ${MY_REPO} -maxdepth 3 -name ${distro}_s${template}; \
); do
grep -v '^#' $f || true
done
) | sort --unique
}
set_layer_image_inc_urls () {
local option="${1}"
if [ "${option}" == "" ]; then
return
fi
local layer_and_inc_type="${option%,*}"
local layer="${layer_and_build_type%,*}"
local inc_type="${layer_and_inc_type#*,}"
local layer_image_inc_url="${option##*,}"
layer_image_inc_urls["${layer_and_inc_type}"]="${layer_image_inc_url}"
}
set_layer_pkg_urls () {
local option="${1}"
if [ "${option}" == "" ]; then
return
fi
local layer_and_build_type="${option%,*}"
local layer="${layer_and_build_type%,*}"
local build_type="${layer_and_build_type#*,}"
local layer_pkg_url="${option##*,}"
layer_pkg_urls["${layer_and_build_type}"]="${layer_pkg_url}"
}
read_layer_image_inc_urls () {
local layer="${1}"
local cfg="${config_dir}/${distro}/${layer}/required_layer_iso_inc.cfg"
local line=""
local key
if [ ! -f "${cfg}" ]; then
return 0;
fi
# Clear all pre-existing entries
for key in "${!layer_image_inc_urls[@]}"; do
unset layer_image_inc_urls[${key}]
done
while read line; do
line=$(echo "${line}" | sed 's/^[ \t]*//;s/[ \t]*$//' | grep '^[^#]')
if [ "${line}" == "" ]; then
continue
fi
set_layer_image_inc_urls "${line}"
done < "${cfg}"
}
read_layer_pkg_urls () {
local layer="${1}"
local cfg="${config_dir}/${distro}/${layer}/required_layer_pkgs.cfg"
local line=""
local key
if [ ! -f "${cfg}" ]; then
return 0;
fi
# Clear all pre-existing entries
for key in "${!layer_pkg_urls[@]}"; do
unset layer_pkg_urls[${key}]
done
while read line; do
line=$(echo "${line}" | sed 's/^[ \t]*//;s/[ \t]*$//' | grep '^[^#]')
if [ "${line}" == "" ]; then
continue
fi
set_layer_pkg_urls "${line}"
done < "${cfg}"
}
set_and_validate_config_dir () {
# Note: Setting the global 'config_dir' here. Not local!
config_dir=${1}
if [ ! -d ${config_dir} ]; then
echo "Error: Invalid config_dir '$config_dir'"
echo " Please select one of: $(find ${config_dir} -maxdepth 1 ! -path ${config_dir} -type d -exec basename {} \;)"
echo
usage
exit 1
fi
}
set_and_validate_distro () {
# Note: Setting the global 'distro' here. Not local!
distro=${1}
if [ ! -d ${config_dir}/${distro} ]; then
echo "Error: Invalid distro '$distro'"
echo " Please select one of: $(find ${config_dir} -maxdepth 1 ! -path ${config_dir} -type d -exec basename {} \;)"
echo
usage
exit 1
fi
if [ -d ${config_dir}/${distro}/${layer} ]; then
read_layer_pkg_urls ${layer}
read_layer_image_inc_urls ${layer}
else
echo "Warning: layer ${layer} not defined for distro '${distro}', please provide a valid layer via '-l <layer>'"
fi
}
set_and_validate_layer () {
# Note: Setting the global 'layer' here. Not local!
layer=${1}
if [ ${layer} != "all" ] && [ ! -d ${config_dir}/${distro}/${layer} ]; then
echo "Error: Invalid layer '$layer'"
echo " Please select one of: all $(find ${config_dir}/${distro} -maxdepth 1 ! -path ${config_dir}/${distro} -type d -exec basename {} \;)"
echo
usage
exit 1
fi
read_layer_pkg_urls ${layer}
read_layer_image_inc_urls ${layer}
}
# Pick up value of the config_dir from environment if set
if [ "$STX_CONFIG_DIR" != "" ]; then
set_and_validate_config_dir "$STX_CONFIG_DIR"
fi
# Pick up value of layer from environment if set
if [ "$LAYER" != "" ]; then
set_and_validate_layer "$LAYER"
fi
read_layer_pkg_urls ${layer}
read_layer_image_inc_urls ${layer}

78
toCOPY/populate_downloads.sh

@ -3,14 +3,24 @@
# SPDX-License-Identifier: Apache-2.0
#
POPULATE_DOWNLOADS_DIR="$(dirname "$(readlink -f "${BASH_SOURCE[0]}" )" )"
source $POPULATE_DOWNLOADS_DIR/lst_utils.sh
usage () {
echo "$0 <mirror-path>"
echo
echo "Create a virtual downloads directory containing only files (likely tarballs) listed in various lst files."
echo "The virtual directory contains only symlinks to to previously downloaded tarballs/files."
echo
echo "$0 [--config-dir=<dir>] [--distro=<distro>] [--layer=<layer>] [ --mirror-dir=<mirror-path> | <mirror-path> ]"
echo
echo " --config-dir=<dir>: Use an alternate config directory rather than the system defined one"
echo " --distro=<distro>: Set distro we intend to build. Default 'centos'"
echo " --layer=<layer>: Set layer we intend to build. Default: use the LAYER environmnet valiable, or 'all'."
echo " --mirror-dir=<dir>: Set the mirror directory. This is where the previously download tarballs are located."
}
if [ $# -ne 1 ]; then
usage
exit -1
fi
mirror_dir=""
if [ -z "$MY_REPO" ]; then
echo "\$MY_REPO is not set. Ensure you are running this script"
@ -19,10 +29,57 @@ if [ -z "$MY_REPO" ]; then
exit -1
fi
mirror_dir=$1
tarball_lst=${MY_REPO_ROOT_DIR}/stx-tools/centos-mirror-tools/tarball-dl.lst
TEMP=$(getopt -o h --long help,config-dir:,distro:,layer:,mirror-dir: -n 'populate_downloads' -- "$@")
if [ $? -ne 0 ]; then
echo "getopt error"
usage
exit 1
fi
eval set -- "$TEMP"
while true ; do
case "$1" in
--mirror-dir) mirror_dir=$2 ; shift 2 ;;
--config-dir) config_dir="${2}"; shift 2 ;;
--distro) set_and_validate_distro "${2}"; shift 2 ;;
--layer) set_and_validate_layer "${2}"; shift 2 ;;
-h|--help) echo "help"; usage; exit 0 ;;
--) shift ; break ;;
*) usage; exit 1 ;;
esac
done
if [ "$mirror_dir" == "" ]; then
if [ $# -ne 1 ]; then
usage
exit -1
fi
mirror_dir=$1
fi
echo "mirror_dir=${mirror_dir}"
echo "config_dir=${config_dir}"
echo "distro=${distro}"
echo "layer=${layer}"
tarball_downloads_template="tarball-dl.lst"
extra_downloads_template="extra_downloads.lst"
TMP_LST_DIR=$(mktemp -d /tmp/tmp_lst_dir_XXXXXX)
mkdir -p $TMP_LST_DIR
tarball_lst="$TMP_LST_DIR/${tarball_downloads_template}"
extra_downloads_lst="$TMP_LST_DIR/${extra_downloads_template}"
merge_lst ${config_dir} ${distro} ${tarball_downloads_template} > ${tarball_lst}
merge_lst ${config_dir} ${distro} ${extra_downloads_template} > ${extra_downloads_lst}
downloads_dir=${MY_REPO}/stx/downloads
extra_downloads="mlnx-ofa_kernel-4.5-OFED.4.5.1.0.1.1.gb4fdfac.src.rpm libibverbs-41mlnx1-OFED.4.5.0.1.0.45101.src.rpm rdma-core-45mlnx1-1.45101.src.rpm"
extra_downloads=""
if [ -f ${extra_downloads_lst} ]; then
extra_downloads="$(grep -v '^#' ${extra_downloads_lst})"
fi
mkdir -p ${MY_REPO}/stx/downloads
@ -43,7 +100,10 @@ grep -v "^#" ${tarball_lst} | while read x; do
rel_dir_name=$(dirname ${rel_path})
if [ ! -e ${downloads_dir}/${rel_dir_name}/${tarball_file} ]; then
mkdir -p ${downloads_dir}/${rel_dir_name}
echo "Creating symlink for $(basename ${source_file})"
ln -sf ${source_file} ${downloads_dir}/${rel_dir_name}/
else
echo "Already have symlink for $(basename ${source_file})"
fi
fi
done
@ -51,3 +111,5 @@ done
for x in ${extra_downloads}; do
ln -sf ${mirror_dir}/downloads/$x ${downloads_dir}
done
\rm -rf ${TMP_LST_DIR}