generate-cgcs-centos-repo.sh was renamed generate-centos-repo.sh
in a prior update. This commit restores the old command
as a wrapper for the renamed command. This will provide
backward compatability for a time.
Story: 2006387
Task: 36912
Depends-On: https://review.opendev.org/#/c/687403
Change-Id: I69714d46ab20b52828f31625cc4e7955fbd10387
Signed-off-by: Scott Little <scott.little@windriver.com>
1) Add config for the container layer.
2) Add support for the new config file 'required_layer_wheel_inc.cfg',
Which will specify the default urls for lower layer wheels.inc files.
These files must be downloaded by download_mirror.sh, and linked by
generate-cgcs-centos-repo.sh.
3) Add options to download_mirror.sh and generate-cgcs-centos-repo.sh
allowing overrides or additions to the set of url's for lower layer
wheels.inc
4) Changes to allow build scripts running within a docker container
to launch a sister (not daughter) container. This will allow
build-wheels to run within a container.
Story: 2006166
Task: 38980
Depends-On: https://review.opendev.org/711773
Depends-On: https://review.opendev.org/717755
Change-Id: I95c71a0327dc9d919ed835d3661a6f1ec7ce30c7
Signed-off-by: Scott Little <scott.little@windriver.com>
The default yumdownloader approach to downloading files
does so one at a time, and is very slow. Too slow for layered
build which is built around the assumption that files can
be downloaded faster than they can be built.
This update will switch from yumdownloader to reposync
for the download of lower layer rpms. It exploits the
fact that each layer and build type publishes it's own lst file
of rpms to download. The lst file is transformed into an'
includepkgs directive in a custom yum.conf which is passed
to reposync, so we only sync the desited rpms.
Reposync won't redownload rpms that it already has, even if the
upstream repodata indicates that the files checksum has changed.
Forceing the redownload of these rpms requires that we manually
download the upstream repodata and that we use verifytree to
identify and delete the new obsolete rpms.
Also including two small bug fixes found while investivgating an
alternative solution to launchpad 1864221.
- incorrect userid in a chown
- Bug in get_url, --url pass being passed in the wrong place
Story: 2006166
Task: 39307
Closes-Bug: 1864221
Change-Id: If12b98ff4f5f24d9318250356920f397419f0f80
Signed-off-by: Scott Little <scott.little@windriver.com>
Script changes to download content by layer.
Valid options are 'all', 'compiler', 'distro', 'flock'.
Current .lst and yum files under directory centos-mirror-tools
are relocated. Generic package dependencies are relocated to
centos-mirror-tools/config/<os>/<layer>/ .
Lst entries for compilable content have been relocated to other
git repos by prior updates. i.e. those that list tarballs or
srpms to be compiled within that repo.
The original .lst files are deleted to make it easier
to identify new content during development.
Layer 'all' builds all layers in a single workspace. The
lst files are identical to current content, minus the src.rpm and
tarball entries.
Other layers get only a subset of packages download. The minimum
required to build the layer. The 'flock' layer will have additional
content to satisfy the run time requirements as well as the build
time requirements.
An upper layer does not need to list rpms known to be provided by
a lower layer. Instead the config file 'required_layer_pkgs.cfg'
lists urls for lst files for lower layer build outputs.
These build outputs are generated and published by cengn for
each layer.
A second layer config file, 'required_layer_iso_inc.cfg' lists
image.inc files for lower layer builds. These build outputs are
generated and published by cengn for each layer, summarizing
the image.inc files found in individual git repos.
Image.inc files inform the build-iso process, listing rpms that
that provide services and commands that need to be included in
the iso. The transitive list of required rpms need not be listed.
Finally the layer config should include a yum.repos.d
directory in which supplementary yum repos are defined to
pick up cengn built content from lower layers.
To allow a designer to do cross-layer building using local sources
rather than those provided by CENGN, there are several options.
The designer can modify the urls for lower layer build outputs,
as found in the .cfg and .repo files within the config directory
'stx-tools/centos-mirror-tools/config/<distro>/<layer-to-build>'
directly within the git. Substitute urls can use the file:///
syntax. Just be sure to remove these changes before submitting.
Alternatively new args have been added to download_mirror.sh,
generate-cgcs-centos-repo.sh and commands that override the
normal config.
The easiest to use is a command argurement that substitutes a new
config directory, replacing stx-tools/centos-mirror-tools/config.
The intent is for the designer to do a recursive copy of that
directory into a side location. make his changes there, outside of
git, and provide the path to that directory as an extra arguement
to download_mirror.sh and generate-cgcs-centos-repo.sh.
e.g. For simplicity I'll only list the 'extra' arguements
download_mirror.sh -C <my-config-dir> \
-l <layer> \
...
generate-cgcs-centos-repo.sh --config-dir=<my-config-dir> \
--layer=<layer> \
...
populate_downloads.sh --config-dir=<my-config-dir> \
--layer=<layer> \
...
These arguements can also be suplied via the environment.
For the purpose of containerized builds, these arguements
should be defined in your localrc.
e.g.
export STX_CONFIG_DIR=<my-config-dir>
export LAYER=<layer>
The final alternative is to override things at a more granular level,
replacing a single lst file of image.inc file. Here you can replace
a single line found in a required_layer_pkgs.cfg or
required_layer_iso_inc.cfg file.
e.g. We are doing a flock build and want to modify the content picked up
from the distro layer's rt build, and that content delivers a service
we want in the iso. For simplicity I'll only list the 'extra' arguments
./download_mirror.sh -l flock \
-L distro,rt,file:///<my-distro-workspace>/rt/rpmbuild/RPMS/rpm.lst \
-I distro,std,file:///<my-distro-workspace>/rt/image.inc \
...
generate-cgcs-centos-repo.sh --layer=flock \
--layer-pkg-url=distro,rt,file:///<my-distro-workspace>/rt/rpmbuild/RPMS/rpm.lst \
--layer-inc-url=distro,std,file:////<my-distro-workspace>/rt/image.inc \
...
NOTE: The triplet syntax for a package list url is
<lower-layer>,<build-type>,<url-to-rpm.lst>
lower-layer: 'compiler', 'distro'
build-type: 'std', 'rt', 'installer'
Also if 'file:///' syntax is used, a matching change is made to
the yum *.repo file. This assumes that the rpm.lst is co-resident with
repodata directory, as is the norm for our build outputs.
NOTE: The triplet syntax for a image inc url is
<lower-layer>,<include-type>,<url-to-image.inc>
lower-layer: 'compiler', 'distro'
build-type: 'std', 'dev'
A typical user is likely only working in the flock layer on the master
branch. He should be content to use the compiler and distro layer
outputs from cengn.
His workflow looks like ...
1, sync code for flock layer
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m flock.xml
$ repo sync
2, download rpms for flock layer, and populate a local mirror
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S -l flock
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
3, Prepare a virtual repo and downloads directory for building
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --layer=flock $LOCAL_MIRROR
$ ./populate_downloads.sh --layer=flock $LOCAL_MIRROR
4, rpm package and iso building
$ build-pkgs && build-iso && build-helm-charts.sh
Building all layers in a single workspace is still supported, and
looks identical to the previous workflow.
1, sync code
$ repo init -u https://opendev.org/starlingx/manifest.git -b master
$ repo sync
2, download rpms for flock layer, and populate a local mirror
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
3, create repo named "StxCentos7Distro" for building
$ cd ../toCOPY
$ generate-cgcs-centos-repo.sh $LOCAL_MIRROR
$ populate_downloads.sh $LOCAL_MIRROR
4, rpm package and iso building
$ build-pkgs && build-iso && build-helm-charts.sh
Only a cross-layer developer should setup two or three copies of the
building environment, one per layer. We suggest you use seperate shells
for each layer, as the various paths (MY_REPO, MY_WORKSPACE ...) need to
be unique,
Shell 1, compiler layer
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ LOCAL_CONFIG=<some-dir>/config
$ MY_REPO_ROOT_DIR=<some-dir>/layer-compiler
$ MY_REPO=$MY_REPO_ROOT_DIR/cgcs-root
$ MY_WORKSPACE=$MY_REPO_ROOT_DIR/workspace
...
$ mkdir -p $MY_REPO_ROOT_DIR
$ cd $MY_REPO_ROOT_DIR
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m compiler.xml
$ cd stx-tools/centos-mirror-tools
$ cp -r config/* $LOCAL_CONFIG
... edit urls in *.cfg and *.repo files under $LOCAL_CONFIG ...
$ ./download_mirror.sh -n -g -c yum.conf.sample -S -C $LOCAL_CONFIG -l compiler
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --config-dir=$LOCAL_CONFIG --layer=compiler $LOCAL_MIRROR
$ ./populate_downloads.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ build-pkgs
Shell 2, distro layer
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ LOCAL_CONFIG=<some-dir>/config
$ MY_REPO_ROOT_DIR=<some-dir>/layer-distro
$ MY_REPO=$MY_REPO_ROOT_DIR/cgcs-root
$ MY_WORKSPACE=$MY_REPO_ROOT_DIR/workspace
...
$ mkdir -p $MY_REPO_ROOT_DIR
$ cd $MY_REPO_ROOT_DIR
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m distro.xml
$ repo sync
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S -C $LOCAL_CONFIG -l distro
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --config-dir=$LOCAL_CONFIG --layer=distro $LOCAL_MIRROR
$ ./populate_downloads.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ build-pkgs
Shell 3, flock layer
$ LOCAL_MIRROR=/import/mirrors/starlingx
$ LOCAL_CONFIG=<some-dir>/config
$ MY_REPO_ROOT_DIR=<some-dir>/layer-flock
$ MY_REPO=$MY_REPO_ROOT_DIR/cgcs-root
$ MY_WORKSPACE=$MY_REPO_ROOT_DIR/workspace
...
$ mkdir -p $MY_REPO_ROOT_DIR
$ cd $MY_REPO_ROOT_DIR
$ repo init -u https://opendev.org/starlingx/manifest.git -b master -m flock.xml
$ repo sync
$ cd stx-tools/centos-mirror-tools
$ ./download_mirror.sh -n -g -c yum.conf.sample -S -C $LOCAL_CONFIG -l flock
$ cp -r output/stx-r1/CentOS/pike/* $LOCAL_MIRROR/
$ cd ../toCOPY
$ ./generate-cgcs-centos-repo.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ ./populate_downloads.sh --config-dir=$LOCAL_CONFIG --layer=flock $LOCAL_MIRROR
$ build-pkgs && build-iso && build-helm-charts.sh
Story: 2006166
Task: 37103
Depends-On: https://review.opendev.org/698756
Depends-On: https://review.opendev.org/700819
Depends-On: https://review.opendev.org/700821
Change-Id: I088020b81f08656e50aa29b5584bbc1dd1378f12
Signed-off-by: Scott Little <scott.little@windriver.com>
The generate-cgcs-centos-repo.sh utility sets up symlinks
in the cgcs-centos-repo directory for each entry in the
defined LST files, pointing to files from a mirror. In the
processing of the LST files, a complete "find" of the
specified mirror is done for each file, looking for a match.
Given there are more than 2000 such entries, this causes
the tool to take a long time to do its job.
This update enhances the tool to do a single find upfront,
caching the result in a temporary file. During processing of
the LST files, then, the tool can grep this file to find the
corresponding file, rather than repeating the find.
In addition, this update fixes an issue with error reporting,
where a flag was used inside the processing loop to indicate
missing files. However, because the loop runs in a subshell,
this status flag is no longer set when checked outside the loop.
This is replaced with a check of the missing file report.
Change-Id: I0eb41f9da3d5afd010ade9308474e537c409f540
Closes-Bug: 1831488
Signed-off-by: Don Penney <don.penney@windriver.com>
The script was updated as the filenames were changed.
Change-Id: I1ef3dae3752cc377a9c9a087628f741b30cb31b5
Signed-off-by: Erich Cordoba <erich.cordoba.malibran@intel.com>
This change improves support for organizational or personal local
mirrors. Download tools work in a wider variety of environmnets,
and the population of build environments is now driven by .lst
files.
The download_mirror.sh (and related files) now have options to
better support running in non-containerized environments. This
includes options to run using local yum.conf files (rather than
system files) and not executing sudo or chgrp commands. These
options are enabled by flags. Default behaviour remains
unchanged.
The generate-cgcs-centos-repo.sh tool now populates build
environment as driven by lst files. This allows a mirror to be
maintained (and grown) at an organizational level, while allowing
a developer to produce exactly the environment they need as per
the lst files they have checked out.
generate-cgcs-centos-repo.sh now also populates the downloads
directory by invoking the populate_downloads.sh script (also
driven by lst files)
Change-Id: I8aae32c776df7618a5ef545a0ce43903b41112d0
Story: 2003422
Task: 24586
Signed-off-by: jmckenna <jason.mckenna@windriver.com>
There was a bug in the formation of the symlink that doesn't
create the correct folder structure.
Change-Id: I4d0215dc0f1c31588f337323a336c1ac4b70f651
Signed-off-by: Erich Cordoba <erich.cordoba.malibran@intel.com>
The build system needs the cgcs-centos-repo symlinks pointing to
the files in the mirror. Also, the mock.cfg.proto file is needed
to provide configurations to the mock environment.
With this script we eliminate the need of having a separate
repository just to hold these symlinks. Now, the developers can
recreate this folder whenever is needed.
Change-Id: Ic4ca21516a1e1271ed15666d5d46ce7f4f5b68b3
Signed-off-by: Erich Cordoba <erich.cordoba.malibran@intel.com>
Scott Little
Bin Yang
Don Penney
Erich Cordoba
jmckenna