This commit updates the stable-wheels.cfg to align with the
stable/train upper-constraints.txt file, to ensure the train images
built have the proper content:
- If an appropriate prebuilt wheel is available at pypi.org, remove
the entry from stable-wheels.cfg to avoid changing the configured
train constraint.
- If building from a tar or zip and the configured train constraint
has changed, update the entry with the new version.
- Added entres to build pyrsistent, psycopg2, and thriftpy2 from
tarballs, as they are needed for various images and no prebuilt wheels
are available.
Change-Id: I6cebb9a9867c37746500e3d6d4762ed7668d60df
Story: 2006544
Task: 37427
Signed-off-by: Don Penney <don.penney@windriver.com>
The previous commit for openstack/train upgrade modified the rjsmin
entry to download a prebuilt wheel file from pypi, rather than
building from a tar file. However, the type field was left as "tar"
rather than updating to "pypi", and the build fails as a result.
This update corrects that field.
Change-Id: I9ad77054f41925d3dfdd10889e0c1ef7683c711b
Story: 2006544
Task: 36618
Signed-off-by: Don Penney <don.penney@windriver.com>
The update-iso.sh utility has been moved to starlingx/utilities as
utilities/platform-util/scripts/update-iso.sh and packaged for the
controller.
This update removes it from starlingx/root.
Change-Id: I76564f77db5cb70a1782214485545ba41b912e28
Story: 2006595
Task: 37167
Signed-off-by: Don Penney <don.penney@windriver.com>
Update openstack version in build-wheel-tarball.sh so that it
can fetch packages with required version.
Story: 2006544
Task: 36618
Depends-on: https://review.opendev.org/#/c/684166/
Change-Id: I8d3bfe2a6fdf976de6774229b0c33e3355d55f80
Signed-off-by: zhipengl <zhipengs.liu@intel.com>
The recently released mock-1.4.20 replaces mockchain with a bash
wrapper that calls mock --chain. However, the StarlingX
mockchain-parallel utility assumes mockchain is python, and fails to
determine the mockchain version as a result. This causes build
failures when mock-1.4.20 is installed.
This update enhances mockchain-parallel to use the version from the
installed mock rpm if the version cannot be determined from mockchain
itself. This allows mockchain-parallel to work with mock-1.4.20, as
well as previously supported versions of mock.
Change-Id: Ic0360fc64e277f1816036b6259e6e51d94c7f368
Closes-Bug: 1849147
Signed-off-by: Don Penney <don.penney@windriver.com>
If an application RPM contains a plugins directory, stage the contents
for packaging with tarball.
Change-Id: Idc47f39c7180407b85ff50d4e7cd7ab6444905ec
Story: 2006537
Task: 36979
Signed-off-by: Robert Church <robert.church@windriver.com>
This commit adds a new tool, update-iso.sh, that allows
a user to customize an ISO to add or modify installation
boot parameters or to add a custom kickstart post-install
addon.
For example, to modify the boot_device and rootfs_device
flags to use a different disk from the default, or to use
the persistent by-path symlink, the user can run:
update-iso.sh -i export/bootimage.iso \
-o updated/bootimage.iso \
-p rootfs_device=/dev/disk/by-path/pci-0000:00:0d.0-ata-1.0 \
-p boot_device=/dev/disk/by-path/pci-0000:00:0d.0-ata-1.0
update-iso.sh -i export/bootimage.iso \
-o updated/bootimage.iso \
-p rootfs_device=nvme0n1 \
-p boot_device=nvme0n1
Or to add a custom addon:
update-iso.sh -i export/bootimage.iso \
-o updated/bootimage.iso \
-a updated/ks-addon.cfg
Story: 2006595
Task: 36739
Change-Id: I09cd0447a35406f27fdf0762ff596d35f3d8dc8f
Signed-off-by: Don Penney <don.penney@windriver.com>
There may be cases where the image tags for the platform
and application side remote clients may be different. This
mostly covers post release scenarios where, we only make
changes to one of the clients.
Note: still leaving the "-t" parameter intact to not
impact CENGN build which uses this parameter
for remote-cli packaging.
Story: 2005312
Task: 36759
Change-Id: I35f80d174eb1c71292b457d6416c98f9bd17e4ca
Depends-On: Id6419fdb1409c57e97866de50ec1f637f4e9b71b
Signed-off-by: Stefan Dinescu <stefan.dinescu@windriver.com>
The packaging script for remote clients was still referencing
the old path to the clients. Chaning the path to reflect the
new location
Change-Id: I309279d58c82436839bf5b7b31d4d2ee61a97104
Closes-bug: 1843897
Signed-off-by: Stefan Dinescu <stefan.dinescu@windriver.com>
Adding the following to .gitignore for new repos
added to the manifest
- docs
- test
- zuul-jobs
Depends-On: https://review.opendev.org/681508
Change-Id: If1a1dedac5a750d5cbbae25f7c0ea90645b36edc
Signed-off-by: Don Penney <don.penney@windriver.com>
Remove entries from .gitignore for pre-restructure directories.
Also 'kubernetes' was renamed as 'containers'.
Finally add an entry for distcloud-client. The gui tox
targets expect the user to create a distcloud-client symlink
in order to be run locally.
Change-Id: I4ea03ec3cb517b1d26215d0fee2ba0dd0e25e598
Story: 2006166
Task: 35687
Signed-off-by: Scott Little <scott.little@windriver.com>
Build tools need to be made ready for the code
restructuring:
1) new repos
2) relocation of release-info.inc and bsp files
3) Removal of the stx- prefix from sub-repos.
Changes:
1) Create new functions to find the release-info.inc and
bsp file under both pre and post restructure paths.
2) Update .gitignore to ignore all the new repos that
are to be created.
3) Remove an argument sanity check from build-helm-charts.sh.
It is making invalid assumptions about where applications
can be found in the directory structure.
4) build-iso will need to look to additional repos to find
packages from lower layers.
Change-Id: If62444390d0a5a363974c6ff8cdaceca35978bda
Story: 2006166
Task: 35687
Signed-off-by: Scott Little <scott.little@windriver.com>
This script creates a tarball with the remote CLI wrapper
scripts present in stx-clients.
This also allows configuring the docker image tag for the
remote cli containers so we can distribute a remote cli
tarball compatible with the platform and application
deployment.
Change-Id: I156172a6ed208d6fcf9bb8f37182daea73b2856c
Partial-bug: 1840133
Depends-On: I02ceee5d20aefe5fcc68f3059b41167bf1fa6a94
Signed-off-by: Stefan Dinescu <stefan.dinescu@windriver.com>
A failure is STX_build_docker_images [0], in second step:
1. STX_build_docker_base_image [1] SUCCESS
2. STX_build_wheels [2] FAILURE
A new Python Wheel is being used, modifying the name of certain packages by
changing "none" to "cp27mu".
[Test Steps]
$ export MY_REPO=/home/user/starlingx/workspace/localdisk/designer/user/starlingx/cgcs-root/
$ export MY_WORKSPACE=/home/user/starlingx/workspace/localdisk/loadbuild/user/starlingx/
$ $MY_REPO/build-tools/build-wheels/build-wheel-tarball.sh --attempts 5 --os centos --os-version 7.5.1804 --stream stable
[Test Results]
...
Adding tsconfig-1.0.0...
Creating stx-centos-stable-wheels.tar...
Done.
[0] <cengn>/20190812T033004Z/logs/jenkins-STX_build_docker_images-112.log.html
[1] <cengn>/20190812T033004Z/logs/jenkins-STX_build_docker_base_image-118.log.html
[2] <cengn>/20190812T033004Z/logs/jenkins-STX_build_wheels-112.log.html
Closes-Bug: 1840175
Change-Id: I838de699d49936631b33ef4acb28f5cad6c35089
Cc: Scott Little <scott.little@windriver.com>
Signed-off-by: Abraham Arce <abraham.arce.moreno@intel.com>
If an application RPM contains a metadata file, copy it into the staging
area and use it as a baseline for further modifications.
This enables specifying 'disabled_charts' metadata to allow specific
charts to be disabled automatically on application upload
Change-Id: I418f0fe4978946a44e512c3025817fb27216c078
Partial-Bug: #1833746
Signed-off-by: Robert Church <robert.church@windriver.com>
adding BUILDER option "script" to support upstream projects
with more than one Dockerfiles,like
https://github.com/intel/intel-device-plugins-for-kubernetes.
it also has ability to support more complex cases.
Options supported by BUILD=script include:
LABEL: the image name
SOURCE_REPO: main project source git repo
SOURCE_REF: git branch or tag for main project source repo
COMMAND: command to run script
SCRIPT: script path
ARGS: args of script (op)
adding OS option "distroless" to support upstream projects
whoes base image is distroless, such as "gcr.io/distroless/base"
Change-Id: Idc9ca03bf6113e554fd23ac9b6bce570866bafbc
Story: 2005937
Task: 35665
Signed-off-by: SidneyAn <ran1.an@intel.com>
This also changes the group wrs_protected to sys_protected
to de-brand the user and group names.
Change-Id: I887464a20fc17d66529caea03be2b445156f9426
Story: 2004716
Task: 30925
Signed-off-by: Saul Wold <sgw@linux.intel.com>
This update modifies build-iso to adapt to changes made to
the platform_comps.py utility. These changes allow build-iso
to specify package lists to be used to generate a comps.xml
file with appropriate software groups.
Change-Id: Ia16207db86ace6d6d1fff0b6f36e4683df5dbc25
Story: 2004764
Task: 33603
Depends-On: https://review.opendev.org/663761
Signed-off-by: Don Penney <don.penney@windriver.com>
Sysinv wheel requires the amqplib wheel.
This update adds amqplib to the base wheels.cfg
so that we can download and build a wheel
to be includeed in the wheels tarball.
Change-Id: If6e486739ff00f3c4adb53d4e345e5bd50b7bc50
Story: 2004766
Task: 30454
Signed-off-by: Tao Liu <tao.liu@windriver.com>
Problem:
Mockchain-parallel requires python2-requests, but it is not found.
Mockchain-parallel runs inside docker. the Dockerfile does not
explicitly list python2-requests as a requires package. The Dockerfile
does list mock, which used to resolve to mock-1.4.15-1.el7.noarch,
which pulled in python2-requests. Centos/EPEL upgraded to
mock-1.4.16-1.el7.noarch 2019-06-04, and it now pulls in
python36-requests, rather than python2-requests.
Mockchain-parallel also uses functions from yum and mock.
Yum remains at python2, mock is now at python3.
There is also one api change in the new mock.
Whatever solution should allow mockchain-parallel function
in both a new or an old container. The difference being the
mock version, and the verion of python required to run
mock/mockchain. Supporting the old mock will also allow more
time for conversion of old build systems that pre-date
the conversion to docker.
Solution:
1) For now, ship two versions of mock-parallel. One
compatible with the new mock (python 3, new api). The
second compatible with the older mock (python 2, old api).
The old version can eventually be retired, so not worth
trying to break out common functions.
2) Remove the dependency on yum. We only needed
one function to split an rpm file name into components
(package-name, version, release, architecture, epoch).
We'll write our own.
3) mockchain-parallel becomes a wrapper that detects
which mock is present, and from that determine which
version of mock-parallel-* to use, and which python to use.
Change-Id: Ia56dfd39e349096b09351e89ad3d4f024e60e4e4
Depends-On: I7c5d9b52eed49cdce02224603b9bfd17e4426558
Closes-Bug: 1831768
Signed-off-by: Scott Little <scott.little@windriver.com>
This commit adds support for manually specifying rpms when building helm charts.
This is specified with a --rpm option that takes a comma-separated list of rpms
providing manifests for building the helm application tarball
These are then merged to generate APP_NAME.yaml, which is included in the
application tarball.
Change-Id: I8bbc5a06a369cfe5db5ed3667d76d7d44d5b52e3
Story: 2005350
Task: 33526
Signed-off-by: Joseph Richard <joseph.richard@windriver.com>
This update introduces a tool for incrementally updating images,
update-stx-image.sh. This allows for small customizations to
images, such as updating specific python modules.
Additionally, this update corrects how docker-build-wheel.sh checks
out a specific git branch, fetching the branch and checking out
the FETCH_HEAD, where previously it was just checking out the branch.
Change-Id: I09c1b2d7a3664a3af5fd887087522b1186eb1ddd
Story: 2005248
Task: 33440
Signed-off-by: Don Penney <don.penney@windriver.com>
A recent update to build-stx-images.sh broke the building
of images with BUILDER=docker. A bash syntax error was
introduced that leads to invalid function calls, resulting
in the following error:
get_git '' ''
basename: missing operand
Try 'basename --help' for more information.
git repo appears to be invalid: . Aborting...
Failed to clone or update . Aborting...
This update corrects the bash syntax.
Change-Id: I09487869350af82b13d9670ff7035078c4a21d4a
Closes-Bug: 1831093
Signed-off-by: Don Penney <don.penney@windriver.com>
Sriov-cni and sriov-network-device-plugin are the first two in what
might be a long list of third party packages that provide their own
Dockerfile.
This update extend the images directives file syntax by introducing
'DOCKER_REPO=<url>' and 'DOCKER_REF=<git_ref>'. The DOCKER_REPO will
be cloned into a working directory, the DOCKER_REF checked out,
the Dockerfile found and passed to docker build.
Change-Id: Iea5eae32bd245557a4b02c9825297343a001e778
Story: 2005208
Task: 33485
Signed-off-by: Scott Little <scott.little@windriver.com>
When loci hits a failure building an image, it leaves behind
a stopped docker container and a <none>:<none> image. This
commit updates build-stx-images.sh to look for and remove
these leftovers on failures.
Change-Id: I5c61b03b7f9b0f69f81e80e5c9fc787852f4441b
Story: 2005248
Task: 31050
Signed-off-by: Don Penney <don.penney@windriver.com>
A recent change to stx-metal/mtce-common broke
the nightly StarlingX build due to stx-nfv/mtce-guest
build failure.
This occurred after a stx-metal/mtce-common change
removed a structure member that the stx-nfv/mtce-guest
compile required.
Since mtce-guest was not rebuilt automatically
the pending compile error was not noticed during UT and
the update was delivered only to show
up as a nightly build failure.
This commit recreates $MY_REPO/cgcs-tis-repo/dependancy-cache
running create_dependancy_cache.py every time when
any rpms and srpms changes, to keep the stx cache updated.
The rpms/srpms changes are detected using the repodata files:
The repodata filenames includes it’s own sha256 in their name,
this is useful to detect rpm/srpms by only comparing the filenames.
Closes-Bug: #1804687
Change-Id: Ieb6bf59542451a0cb4cc275bd9d59b532c5b6189
Signed-off-by: Felipe De Jesus Ruiz Garcia <felipe.de.jesus.ruiz.garcia@intel.com>
Update the yum repo config for the stx-centos dev base image
to point to the openstack-stein repo.
Change-Id: Ia84d49deca85b8b32e6925e8de1b0768f6334196
Story: 2005248
Task: 30694
Signed-off-by: Don Penney <don.penney@windriver.com>
This commit updates the chart build to build a metadata file into
an application tarball. The metadata file includes the app name,
app version and patch dependencies if there are patches specified
via option --patch-dependencies. The app name can be passed via
option --app, the app version is constructed with the app rpm version
and label.
The name of application tarball is renamed to the app name plus
the app version.
Change-Id: I8ce54c3beac3d671f580fec6c2e4c37153e163f4
Depends-On: https://review.opendev.org/#/c/655789/
Story: 2005350
Task: 30420
Signed-off-by: Angie Wang <angie.wang@windriver.com>
Add support for setting TIS_PATCH_VER in the image directives
files to allow for incremental version changes. This tag is
optional, defaulting to 0, and is appended to the versioned
image tag.
Change-Id: Ibd8d552a814689118701e58588fb2ffa63177249
Story: 2005248
Task: 30566
Signed-off-by: Don Penney <don.penney@windriver.com>
python-gunicorn srpm is replaced with rpm package.
Explicitly require the package to be included in ISO.
Story: 2005484
Task: 30578
Depends-On: https://review.opendev.org/653908
Change-Id: I7fe1e33427a4a6dae719744019492712f022ebdc
Signed-off-by: Shuicheng Lin <shuicheng.lin@intel.com>
The wheel and image build tools hit intermittent issues such as:
* Network or server issues that cause file download failures
* Repo access failures due to repo updates or other issues
To avoid formal build failures on such issues, this update adds
retries around specific commands, like wget and docker build.
Change-Id: Ifafdc6201872f43b2dd77efd4dcb033456477c2e
Closes-Bug: 1823986
Signed-off-by: Don Penney <don.penney@windriver.com>
The build-stx-images.sh has been updated to store image
versions to files after building and pushing images.
This commit updates the chart build to allow specifying
the path to the image files/urls as input for building
image versions into armada manifest and the label of the
application tarball which is used to construct the tarball
name.
The default stable latest images in the manifest will be
used if there's no image file passed.
Change-Id: Ideea6ef3b9298a182be26fc1295456a978338f45
Story: 2005350
Task: 30419
Signed-off-by: Angie Wang <angie.wang@windriver.com>
The openstack/requirements master/upper-constraints.txt was
recently updated to point to pecan-1.3.3, but pypi.org does
not have a wheel for this newly released version. This, in
turn, caused a dev image build failure.
This update adds pecan-1.3.3 to the dev-wheels.cfg to build
a wheel for this version and add it to the dev wheels tarball.
Change-Id: I165a3b3e31dad275459d015936e4c9ea5a0e2d78
Closes-Bug: 1823984
Signed-off-by: Don Penney <don.penney@windriver.com>
/usr/bin/getfattr is provided by attr-2.4.46-13.el7.x86_64, which
was being pulled into the ISO by openstack-nova-compute.
openstack-nova-compute is removed from ISO since openstack is
containerized.
Add attr rpm to the image.inc lst, to explicitly require it be
included in ISO.
Test:
Pass build and confirm attr rpm package is included in ISO.
Closes-Bug: 1823075
Change-Id: Ie40af4417ef48151eba5ed89bb0c0311bfbcc155
Signed-off-by: Shuicheng Lin <shuicheng.lin@intel.com>
Update the base stable-wheels.cfg to align with the
Openstack Stein release upper-constraints.txt, to
support building stable stein Openstack service images.
Change-Id: I6852620b05f58509c93877d9d8c5754e9aa64c0e
Story: 2005248
Task: 30353
Signed-off-by: Don Penney <don.penney@windriver.com>
This update removes pike and master wheel and/or image files,
as these have been replaced with stable/dev.
Change-Id: I1f44dc69de2819eba8dadf1639c6a4ce8f6f2031
Story: 2005248
Task: 30328
Signed-off-by: Don Penney <don.penney@windriver.com>