The build-stx-images.sh allows for building images using a git repo as
the source, which is cloned into the workspace as part of the build
procedure. In some cases, there may be source changes required for
fixes or enhancements that have not yet merged into the referenced
repo. This update adds support in build-stx-images to allow the
designer to specify patches in the image directives file, which will
be applied after cloning.
Designer workflow:
* clone git repo (may need to apply any existing git patches from
previous fixes)
* make changes
* commit, with a beautiful and admirable commit message
* git format-patch
* copy source patch to stx repo, somewhere alongside the docker_image
directives file (ie. a docker_patches dir)
* update docker_image directives file to add DOCKER_REPO_PATCHES or
SOURCE_PATCHES, as appropriate, with DOCKER_REF or SOURCE_REF set to
the current upstream SHA used as the base for the patch
Story: 2006711
Task: 38499
Change-Id: Icac2b4dc2ebeb32946de6e5adcbf59a10c56a30c
Signed-off-by: Don Penney <don.penney@windriver.com>
This commit updates the old_image_reference regex to correctly handle
images records using a hostname with hyphens and/or a port specified.
Closes-bug: 1860465
Change-Id: Ibf88d1c675277d4b133d0fd68029555ceea3e18d
Signed-off-by: Joseph Richard <joseph.richard@windriver.com>
Update build-helm-charts to allow application to optionally specify app_name.
The script was ignoring app_name if it was already specified in the metadata;
this updates it to accept the app_name if specified, otherwise default to
APP_NAME as specified in command line for the tarball.
Change-Id: I9368b3a8a7c3d50981f02fe7c4a5e4be8255cfc3
Partial-Bug: 1859882
Signed-off-by: John Kung <john.kung@windriver.com>
As the python-smartpm module is no longer used, the reference in
source_lookup.txt can now be removed.
Depends-On: https://review.opendev.org/702791
Change-Id: I604ff62614d0f8872c9f0db07a66df26718d3c1e
Story: 2006227
Task: 38137
Signed-off-by: Don Penney <don.penney@windriver.com>
If a custom docker build file sets up a non-shell entrypoint,
build-stx-images.sh fails during the build while trying to collect
some information by running the newly built image.
This update enhances the handling to check the entrypoint and ensure
it is maintained, while using bash to run any needed commands.
Closes-Bug: 1857349
Change-Id: I1f798476eb4976a4c81cdd2defe5d767a4681f5b
Signed-off-by: Don Penney <don.penney@windriver.com>
This is to support kata container in StarlingX.
Story: 2006145
Task: 36833
Depends-On: https://review.opendev.org/685211
Change-Id: I0ca5aa13ad9acb9449d2dc637d1a02604c9fb757
Signed-off-by: Shuicheng Lin <shuicheng.lin@intel.com>
This commit updates the stable-wheels.cfg to align with the
stable/train upper-constraints.txt file, to ensure the train images
built have the proper content:
- If an appropriate prebuilt wheel is available at pypi.org, remove
the entry from stable-wheels.cfg to avoid changing the configured
train constraint.
- If building from a tar or zip and the configured train constraint
has changed, update the entry with the new version.
- Added entres to build pyrsistent, psycopg2, and thriftpy2 from
tarballs, as they are needed for various images and no prebuilt wheels
are available.
Change-Id: I6cebb9a9867c37746500e3d6d4762ed7668d60df
Story: 2006544
Task: 37427
Signed-off-by: Don Penney <don.penney@windriver.com>
The previous commit for openstack/train upgrade modified the rjsmin
entry to download a prebuilt wheel file from pypi, rather than
building from a tar file. However, the type field was left as "tar"
rather than updating to "pypi", and the build fails as a result.
This update corrects that field.
Change-Id: I9ad77054f41925d3dfdd10889e0c1ef7683c711b
Story: 2006544
Task: 36618
Signed-off-by: Don Penney <don.penney@windriver.com>
The update-iso.sh utility has been moved to starlingx/utilities as
utilities/platform-util/scripts/update-iso.sh and packaged for the
controller.
This update removes it from starlingx/root.
Change-Id: I76564f77db5cb70a1782214485545ba41b912e28
Story: 2006595
Task: 37167
Signed-off-by: Don Penney <don.penney@windriver.com>
Update openstack version in build-wheel-tarball.sh so that it
can fetch packages with required version.
Story: 2006544
Task: 36618
Depends-on: https://review.opendev.org/#/c/684166/
Change-Id: I8d3bfe2a6fdf976de6774229b0c33e3355d55f80
Signed-off-by: zhipengl <zhipengs.liu@intel.com>
The recently released mock-1.4.20 replaces mockchain with a bash
wrapper that calls mock --chain. However, the StarlingX
mockchain-parallel utility assumes mockchain is python, and fails to
determine the mockchain version as a result. This causes build
failures when mock-1.4.20 is installed.
This update enhances mockchain-parallel to use the version from the
installed mock rpm if the version cannot be determined from mockchain
itself. This allows mockchain-parallel to work with mock-1.4.20, as
well as previously supported versions of mock.
Change-Id: Ic0360fc64e277f1816036b6259e6e51d94c7f368
Closes-Bug: 1849147
Signed-off-by: Don Penney <don.penney@windriver.com>
If an application RPM contains a plugins directory, stage the contents
for packaging with tarball.
Change-Id: Idc47f39c7180407b85ff50d4e7cd7ab6444905ec
Story: 2006537
Task: 36979
Signed-off-by: Robert Church <robert.church@windriver.com>
This commit adds a new tool, update-iso.sh, that allows
a user to customize an ISO to add or modify installation
boot parameters or to add a custom kickstart post-install
addon.
For example, to modify the boot_device and rootfs_device
flags to use a different disk from the default, or to use
the persistent by-path symlink, the user can run:
update-iso.sh -i export/bootimage.iso \
-o updated/bootimage.iso \
-p rootfs_device=/dev/disk/by-path/pci-0000:00:0d.0-ata-1.0 \
-p boot_device=/dev/disk/by-path/pci-0000:00:0d.0-ata-1.0
update-iso.sh -i export/bootimage.iso \
-o updated/bootimage.iso \
-p rootfs_device=nvme0n1 \
-p boot_device=nvme0n1
Or to add a custom addon:
update-iso.sh -i export/bootimage.iso \
-o updated/bootimage.iso \
-a updated/ks-addon.cfg
Story: 2006595
Task: 36739
Change-Id: I09cd0447a35406f27fdf0762ff596d35f3d8dc8f
Signed-off-by: Don Penney <don.penney@windriver.com>
There may be cases where the image tags for the platform
and application side remote clients may be different. This
mostly covers post release scenarios where, we only make
changes to one of the clients.
Note: still leaving the "-t" parameter intact to not
impact CENGN build which uses this parameter
for remote-cli packaging.
Story: 2005312
Task: 36759
Change-Id: I35f80d174eb1c71292b457d6416c98f9bd17e4ca
Depends-On: Id6419fdb1409c57e97866de50ec1f637f4e9b71b
Signed-off-by: Stefan Dinescu <stefan.dinescu@windriver.com>
The packaging script for remote clients was still referencing
the old path to the clients. Chaning the path to reflect the
new location
Change-Id: I309279d58c82436839bf5b7b31d4d2ee61a97104
Closes-bug: 1843897
Signed-off-by: Stefan Dinescu <stefan.dinescu@windriver.com>
Build tools need to be made ready for the code
restructuring:
1) new repos
2) relocation of release-info.inc and bsp files
3) Removal of the stx- prefix from sub-repos.
Changes:
1) Create new functions to find the release-info.inc and
bsp file under both pre and post restructure paths.
2) Update .gitignore to ignore all the new repos that
are to be created.
3) Remove an argument sanity check from build-helm-charts.sh.
It is making invalid assumptions about where applications
can be found in the directory structure.
4) build-iso will need to look to additional repos to find
packages from lower layers.
Change-Id: If62444390d0a5a363974c6ff8cdaceca35978bda
Story: 2006166
Task: 35687
Signed-off-by: Scott Little <scott.little@windriver.com>
This script creates a tarball with the remote CLI wrapper
scripts present in stx-clients.
This also allows configuring the docker image tag for the
remote cli containers so we can distribute a remote cli
tarball compatible with the platform and application
deployment.
Change-Id: I156172a6ed208d6fcf9bb8f37182daea73b2856c
Partial-bug: 1840133
Depends-On: I02ceee5d20aefe5fcc68f3059b41167bf1fa6a94
Signed-off-by: Stefan Dinescu <stefan.dinescu@windriver.com>
A failure is STX_build_docker_images [0], in second step:
1. STX_build_docker_base_image [1] SUCCESS
2. STX_build_wheels [2] FAILURE
A new Python Wheel is being used, modifying the name of certain packages by
changing "none" to "cp27mu".
[Test Steps]
$ export MY_REPO=/home/user/starlingx/workspace/localdisk/designer/user/starlingx/cgcs-root/
$ export MY_WORKSPACE=/home/user/starlingx/workspace/localdisk/loadbuild/user/starlingx/
$ $MY_REPO/build-tools/build-wheels/build-wheel-tarball.sh --attempts 5 --os centos --os-version 7.5.1804 --stream stable
[Test Results]
...
Adding tsconfig-1.0.0...
Creating stx-centos-stable-wheels.tar...
Done.
[0] <cengn>/20190812T033004Z/logs/jenkins-STX_build_docker_images-112.log.html
[1] <cengn>/20190812T033004Z/logs/jenkins-STX_build_docker_base_image-118.log.html
[2] <cengn>/20190812T033004Z/logs/jenkins-STX_build_wheels-112.log.html
Closes-Bug: 1840175
Change-Id: I838de699d49936631b33ef4acb28f5cad6c35089
Cc: Scott Little <scott.little@windriver.com>
Signed-off-by: Abraham Arce <abraham.arce.moreno@intel.com>
If an application RPM contains a metadata file, copy it into the staging
area and use it as a baseline for further modifications.
This enables specifying 'disabled_charts' metadata to allow specific
charts to be disabled automatically on application upload
Change-Id: I418f0fe4978946a44e512c3025817fb27216c078
Partial-Bug: #1833746
Signed-off-by: Robert Church <robert.church@windriver.com>
adding BUILDER option "script" to support upstream projects
with more than one Dockerfiles,like
https://github.com/intel/intel-device-plugins-for-kubernetes.
it also has ability to support more complex cases.
Options supported by BUILD=script include:
LABEL: the image name
SOURCE_REPO: main project source git repo
SOURCE_REF: git branch or tag for main project source repo
COMMAND: command to run script
SCRIPT: script path
ARGS: args of script (op)
adding OS option "distroless" to support upstream projects
whoes base image is distroless, such as "gcr.io/distroless/base"
Change-Id: Idc9ca03bf6113e554fd23ac9b6bce570866bafbc
Story: 2005937
Task: 35665
Signed-off-by: SidneyAn <ran1.an@intel.com>
This also changes the group wrs_protected to sys_protected
to de-brand the user and group names.
Change-Id: I887464a20fc17d66529caea03be2b445156f9426
Story: 2004716
Task: 30925
Signed-off-by: Saul Wold <sgw@linux.intel.com>
This update modifies build-iso to adapt to changes made to
the platform_comps.py utility. These changes allow build-iso
to specify package lists to be used to generate a comps.xml
file with appropriate software groups.
Change-Id: Ia16207db86ace6d6d1fff0b6f36e4683df5dbc25
Story: 2004764
Task: 33603
Depends-On: https://review.opendev.org/663761
Signed-off-by: Don Penney <don.penney@windriver.com>
Sysinv wheel requires the amqplib wheel.
This update adds amqplib to the base wheels.cfg
so that we can download and build a wheel
to be includeed in the wheels tarball.
Change-Id: If6e486739ff00f3c4adb53d4e345e5bd50b7bc50
Story: 2004766
Task: 30454
Signed-off-by: Tao Liu <tao.liu@windriver.com>
Problem:
Mockchain-parallel requires python2-requests, but it is not found.
Mockchain-parallel runs inside docker. the Dockerfile does not
explicitly list python2-requests as a requires package. The Dockerfile
does list mock, which used to resolve to mock-1.4.15-1.el7.noarch,
which pulled in python2-requests. Centos/EPEL upgraded to
mock-1.4.16-1.el7.noarch 2019-06-04, and it now pulls in
python36-requests, rather than python2-requests.
Mockchain-parallel also uses functions from yum and mock.
Yum remains at python2, mock is now at python3.
There is also one api change in the new mock.
Whatever solution should allow mockchain-parallel function
in both a new or an old container. The difference being the
mock version, and the verion of python required to run
mock/mockchain. Supporting the old mock will also allow more
time for conversion of old build systems that pre-date
the conversion to docker.
Solution:
1) For now, ship two versions of mock-parallel. One
compatible with the new mock (python 3, new api). The
second compatible with the older mock (python 2, old api).
The old version can eventually be retired, so not worth
trying to break out common functions.
2) Remove the dependency on yum. We only needed
one function to split an rpm file name into components
(package-name, version, release, architecture, epoch).
We'll write our own.
3) mockchain-parallel becomes a wrapper that detects
which mock is present, and from that determine which
version of mock-parallel-* to use, and which python to use.
Change-Id: Ia56dfd39e349096b09351e89ad3d4f024e60e4e4
Depends-On: I7c5d9b52eed49cdce02224603b9bfd17e4426558
Closes-Bug: 1831768
Signed-off-by: Scott Little <scott.little@windriver.com>
This commit adds support for manually specifying rpms when building helm charts.
This is specified with a --rpm option that takes a comma-separated list of rpms
providing manifests for building the helm application tarball
These are then merged to generate APP_NAME.yaml, which is included in the
application tarball.
Change-Id: I8bbc5a06a369cfe5db5ed3667d76d7d44d5b52e3
Story: 2005350
Task: 33526
Signed-off-by: Joseph Richard <joseph.richard@windriver.com>
This update introduces a tool for incrementally updating images,
update-stx-image.sh. This allows for small customizations to
images, such as updating specific python modules.
Additionally, this update corrects how docker-build-wheel.sh checks
out a specific git branch, fetching the branch and checking out
the FETCH_HEAD, where previously it was just checking out the branch.
Change-Id: I09c1b2d7a3664a3af5fd887087522b1186eb1ddd
Story: 2005248
Task: 33440
Signed-off-by: Don Penney <don.penney@windriver.com>
A recent update to build-stx-images.sh broke the building
of images with BUILDER=docker. A bash syntax error was
introduced that leads to invalid function calls, resulting
in the following error:
get_git '' ''
basename: missing operand
Try 'basename --help' for more information.
git repo appears to be invalid: . Aborting...
Failed to clone or update . Aborting...
This update corrects the bash syntax.
Change-Id: I09487869350af82b13d9670ff7035078c4a21d4a
Closes-Bug: 1831093
Signed-off-by: Don Penney <don.penney@windriver.com>
Sriov-cni and sriov-network-device-plugin are the first two in what
might be a long list of third party packages that provide their own
Dockerfile.
This update extend the images directives file syntax by introducing
'DOCKER_REPO=<url>' and 'DOCKER_REF=<git_ref>'. The DOCKER_REPO will
be cloned into a working directory, the DOCKER_REF checked out,
the Dockerfile found and passed to docker build.
Change-Id: Iea5eae32bd245557a4b02c9825297343a001e778
Story: 2005208
Task: 33485
Signed-off-by: Scott Little <scott.little@windriver.com>
When loci hits a failure building an image, it leaves behind
a stopped docker container and a <none>:<none> image. This
commit updates build-stx-images.sh to look for and remove
these leftovers on failures.
Change-Id: I5c61b03b7f9b0f69f81e80e5c9fc787852f4441b
Story: 2005248
Task: 31050
Signed-off-by: Don Penney <don.penney@windriver.com>
A recent change to stx-metal/mtce-common broke
the nightly StarlingX build due to stx-nfv/mtce-guest
build failure.
This occurred after a stx-metal/mtce-common change
removed a structure member that the stx-nfv/mtce-guest
compile required.
Since mtce-guest was not rebuilt automatically
the pending compile error was not noticed during UT and
the update was delivered only to show
up as a nightly build failure.
This commit recreates $MY_REPO/cgcs-tis-repo/dependancy-cache
running create_dependancy_cache.py every time when
any rpms and srpms changes, to keep the stx cache updated.
The rpms/srpms changes are detected using the repodata files:
The repodata filenames includes it’s own sha256 in their name,
this is useful to detect rpm/srpms by only comparing the filenames.
Closes-Bug: #1804687
Change-Id: Ieb6bf59542451a0cb4cc275bd9d59b532c5b6189
Signed-off-by: Felipe De Jesus Ruiz Garcia <felipe.de.jesus.ruiz.garcia@intel.com>
Update the yum repo config for the stx-centos dev base image
to point to the openstack-stein repo.
Change-Id: Ia84d49deca85b8b32e6925e8de1b0768f6334196
Story: 2005248
Task: 30694
Signed-off-by: Don Penney <don.penney@windriver.com>
This commit updates the chart build to build a metadata file into
an application tarball. The metadata file includes the app name,
app version and patch dependencies if there are patches specified
via option --patch-dependencies. The app name can be passed via
option --app, the app version is constructed with the app rpm version
and label.
The name of application tarball is renamed to the app name plus
the app version.
Change-Id: I8ce54c3beac3d671f580fec6c2e4c37153e163f4
Depends-On: https://review.opendev.org/#/c/655789/
Story: 2005350
Task: 30420
Signed-off-by: Angie Wang <angie.wang@windriver.com>
Add support for setting TIS_PATCH_VER in the image directives
files to allow for incremental version changes. This tag is
optional, defaulting to 0, and is appended to the versioned
image tag.
Change-Id: Ibd8d552a814689118701e58588fb2ffa63177249
Story: 2005248
Task: 30566
Signed-off-by: Don Penney <don.penney@windriver.com>
python-gunicorn srpm is replaced with rpm package.
Explicitly require the package to be included in ISO.
Story: 2005484
Task: 30578
Depends-On: https://review.opendev.org/653908
Change-Id: I7fe1e33427a4a6dae719744019492712f022ebdc
Signed-off-by: Shuicheng Lin <shuicheng.lin@intel.com>
