266 Commits

Author SHA1 Message Date
Zuul
b153028880 Merge "Preserve quotes in building helm charts" 2020-10-13 18:06:37 +00:00
Zuul
f85a9447f7 Merge "Build script readiness for renaming cgcs-tis-repo to local-repo" 2020-10-13 18:06:27 +00:00
Zuul
c20ea0822c Merge "Build script readiness for renaming cgcs-centos-repo to centos-repo" 2020-10-13 18:06:22 +00:00
Yuxing Jiang
e9fbe29f11 Preserve quotes in building helm charts
RoundTripLoader in ruamel.yaml will omit quotation marks. In some
cases, the result by loading a yaml with RoundTripLoader and
dumping it out will be unreadable by PYYAML. For example, the
pattern:
  hosts: ["mon-logstash:port"]
will be transformed to:
  hosts: [mon-logstash:port]
Currently, the Armada project is using PYYAML. It's better to keep
the  quotes to allow the following process have to use PYYAML.

Tested with rebuild the helm chart with the yaml has the pattern
in the example, and the quotation marks are preserved.

Change-Id: I7375a1ad498336baa337cca5af8f90028b753110
Partial-Bug: 1896530
Signed-off-by: Yuxing Jiang <yuxing.jiang@windriver.com>
2020-09-28 10:27:15 -04:00
Scott Little
d81920c349 Build script readiness for renaming cgcs-tis-repo to local-repo
I want the build to work with either local-repo or cgcs-tis-repo.
In many places we will be testing for the existance local-repo as
the prefered path, then fall back to cgcs-tis-repo as an
alternative.  If neither are present, either exit or continue but
assuming the new path is intended.

Story: 2006387
Task: 36910
Change-Id: I2e97dc8cd5d4a54158d4b1cefeecd2000ec11cf2
Signed-off-by: Scott Little <scott.little@windriver.com>
2020-09-24 12:03:00 -04:00
Scott Little
01c5009d7d Build script readiness for renaming cgcs-centos-repo to centos-repo
I want the build to work with either centos-repo or cgcs-centos-repo.
In many places we will be testing for the existance centos-repo as
the prefered path, then fall back to cgcs-centos-repo as an alternative.
If neither are present, either exit or continue but assuming the new
path is intended.

NOTE: The patch_rebase_1/2/3/4 scripts remain broken, but I hope
to salvage them one day.  The current coding assumes content under
centos-repo/cgcs-centos-repo is managed by a git, which is not
currently true.

Story: 2006387
Task: 36912
Change-Id: I8f694814c41957c5b37eb2e64b653b7d42f2e2c9
Signed-off-by: Scott Little <scott.little@windriver.com>
2020-09-24 11:59:55 -04:00
Zuul
e5f5b25e95 Merge "helm_chart_modify.py: remove break" 2020-09-19 04:19:50 +00:00
Scott Little
53a1537cde helm_chart_modify.py: remove break
Closes-Bug: 1893261
Change-Id: Id4c3863ba7eddeb77e8dcbe6cf6caefa5252957d
Signed-off-by: Scott Little <scott.little@windriver.com>
2020-09-19 00:07:08 -04:00
Zuul
5a3d6b6a34 Merge "build-helm-charts.sh support for multi-line url substitution" 2020-09-18 20:06:12 +00:00
Scott Little
b2ff8cd529 build-helm-charts.sh support for multi-line url substitution
One of the steps under build-helm-charts.sh is to modify
upstream manifests by substituting docker image locations and tags
with ones that reference the images we have built and published.

Some newer manifests we would like to include are now using a
multi-line format that isn't supported by the current script.

Examples would be ...
     image: <host>[:<port>]/<path>/<image-name>
     imageTag: <tag>

or
     image:
        repository: <host>[:<port>]/<path>/<image-name>
        tag: <tag>

Our fix is to abandon sed, and substitute a routine
that walks the data structures looking substitutions
in either one or two line format.

Closes-Bug: 1893261
Depends-On: https://review.opendev.org/750710
Change-Id: I81be885e0c2460be9319fe403ef8fe938fe35549
Signed-off-by: Scott Little <scott.little@windriver.com>
2020-09-18 14:48:28 -04:00
Teresa Ho
dbb2a0c34f Update tag for stx-oidc-client to stx.5.0-v1.0.3
Closes-Bug: 1877172

Change-Id: Ic6aa7317ac9a18e19dcaab1572359d1060a532ce
Signed-off-by: Teresa Ho <teresa.ho@windriver.com>
2020-09-09 11:39:24 -04:00
Zuul
39c1730c88 Merge "Eliminate intermediate .tox files during build" 2020-09-02 16:51:30 +00:00
Zuul
3d1bdb2812 Merge "Build contexts should not include .repo/repo" 2020-08-31 19:36:03 +00:00
Scott Little
2dd5d9f07c Build contexts should not include .repo/repo
Build avoidance does not work reliably because directory
.repo/repo is being included in the build context.
This directory is the repo tool itself, not any of the
gits managed by the repo manifest.  The sha reported for
.repo/repo might not be available in users environmnet,
causing build avoidance to reject an otherwise valid
reference context.

Fix is in two parts.
1) Do not include .repo/repo in future build contexts.
2) Filter .repo/repo out of downloaded contexts before
use in build avoidance calculations.  This addresses
contexts that are already published.

Closes-Bug: 1893243
Change-Id: I7bd20597cb7bc5ee93ae49728176791c51e89c53
Signed-off-by: Scott Little <scott.little@windriver.com>
2020-08-27 15:54:44 -04:00
albailey
e2bcb4c6c8 Eliminate intermediate .tox files during build
The symlinks under .tox were being included for md5sum
calculations.

The md5sum calculations would report an error when a file containing
a space was encountered (which can happen in some .tox libraries such
as setuptools

Change-Id: Ib9bfe55a66e1f27b1e39596b14303c0ef00852c6
Signed-off-by: albailey <Al.Bailey@windriver.com>
2020-08-27 11:01:45 -05:00
Zuul
4ce7faa4bc Merge "Fix the bug introduced by adding proxy for image build scripts" 2020-08-26 20:12:55 +00:00
zhipengl
3f9912bd53 Fix the bug introduced by adding proxy for image build scripts
Below commit will introduce a bug in image build script.
bb472a9f37b5acb5c8bf1544f854584f90b5dd88

Closes-bug: 1890383

Change-Id: I6551a33bf099549eb9a66a65730b884c4a85baef
Signed-off-by: zhipengl <zhipengs.liu@intel.com>
2020-08-27 01:46:21 +08:00
Zuul
b7873b51a2 Merge "build-img: default IP route and account passwords" 2020-08-26 16:28:37 +00:00
Davlet Panech
b215134dc7 build-img: default IP route and account passwords
- Options to set default IP v4 & v6 routes
- Option to set root & sysadmin passwords and disable password change on
  1st login

Story: 2007858
Task: 40733

Change-Id: I59a7f112e8a01b00693aabbb804a80ef0cb8253e
Signed-off-by: Davlet Panech <davlet.panech@windriver.com>
2020-08-25 09:31:19 -04:00
Zuul
3bb8edb85e Merge "Add build date of the app tarball" 2020-08-24 20:48:12 +00:00
Zuul
b5bc6fe8dc Merge "build-img: support graphical boot & virtio disks" 2020-08-21 18:48:00 +00:00
Zuul
360ea5fc5d Merge "Fix patch-iso pxe-network-installer handling" 2020-08-21 18:47:59 +00:00
Scott Little
58e55e9400 Fix url's in base-image-build-centos-*.cfg
build.starlingx.cengn.ca is not a public web server.  It's use
results in 'HTTP Error 502 - Bad Gateway' outside of CENGN.

The correct url should refer to mirror.starlingx.cengn.ca.

Closes-Bug: 1892538
Change-Id: I6021ee3d97815e70ee461c2c1a8297dfdefc7769
Signed-off-by: Scott Little <scott.little@windriver.com>
2020-08-21 14:25:18 -04:00
Don Penney
466ce2c528 Fix patch-iso pxe-network-installer handling
Changes made previously to upversion grub2 in starlingx resulted in
some restructuring of the pxe-network-installer package, and
corresponding changes in build-iso. These changes were not reflected
in patch-iso, however. This update fixes the handling of
pxe-network-installer extraction, and includes additional grub2
packages in the patch support.

Change-Id: Ib72a44d2d5a39fe41023be8feb83b88e5f39435b
Closes-Bug: 1892424
Signed-off-by: Don Penney <don.penney@windriver.com>
2020-08-20 17:52:54 -04:00
Davlet Panech
24274fcba2 build-img: support graphical boot & virtio disks
- Added new option --graphical to generate images that boot into a
  graphics card, rather than the serial port
- Added "_graphical" suffix to the generated image file name when
  --graphical is in effect
- Comment out global_filter parameter in lvm.conf, otherwise LVM fails
  to start when image is booted in different emulation environment

Story: 2007858
Task: 40440

Change-Id: I33e4899873c4f915417e0ab24f60a540d5a68642
Signed-off-by: Davlet Panech <davlet.panech@windriver.com>
2020-08-19 14:36:38 -04:00
Robert Church
2e01fe8df3 Add build date of the app tarball
Include the build date of the tarball as underlying RPMs may not have
changed between builds.

Change-Id: I719f9f0224f00a408daf8385c3d5a346dc282521
Story: 2007960
Task: 40686
Signed-off-by: Robert Church <robert.church@windriver.com>
2020-08-19 11:00:17 -04:00
Zuul
a3e83a6c02 Merge "Add proxy setting for container image build scripts" 2020-08-19 14:49:48 +00:00
Liu, ZhipengS
bb472a9f37 Add proxy setting for container image build scripts
Users in some places like China need to use proxy during building image.
So, it is better to let user be able to use proxy by adding these proxy
arguments.

Closes-bug: 1890383

Change-Id: If9ef7d09a5f53148252b4ab62c40ab0f143e1c3f
Signed-off-by: Liu, ZhipengS <zhipengs.liu@intel.com>
2020-08-18 19:55:46 +08:00
Zuul
6e7074bb6c Merge "Fix use of 'let -i' in scripts" 2020-08-13 06:24:53 +00:00
Don Penney
f36db7e207 Fix use of 'let -i' in scripts
Unlike "declare -i" and "local -i", the bash "let" does not support a
"-i" option. Rather, it takes it as a variable reference. If no "i"
variable is defined in scope, it does not cause an issue. If "i" has
been defined somewhere, however, it may cause a syntax issue, as the i
is evaluated.

A recent update to build-stx-images.sh added a loop that defines an
"i" variable without limiting its scope. In a current image build,
this loop ends with having "i" defined as a URL. As a result, a
"syntax error in expression" occurs, causing the "with_retries"
function to fail to increment the counter. Should a build error occur,
the "with_retries" will never hit the retry limit, looping until it
has a successful result.

This update removes the -i from all "let -i" occurrences in the build
scripts.

Change-Id: I34ad49f8872a81659ff4caf8087b256ea9fb3d32
Closes-Bug: 1891189
Signed-off-by: Don Penney <don.penney@windriver.com>
2020-08-11 10:36:43 -04:00
Teresa Ho
a41c56cf12 Update stx-platformclients tag to stx.5.0-v1.4.0
Story: 2007875
Task: 40633

Change-Id: I690890692e1716ecc8a600cac3a0cd2c1ffc6810
Signed-off-by: Teresa Ho <teresa.ho@windriver.com>
2020-08-07 17:22:42 -04:00
Davlet Panech
b21cacbffc build-img: support centos8 & non-tty environments
- when looking for an error message in qemu log, accept more patterns,
  because QEMU executable may be named slightly differently than
  literally "qemu"
- disable QEMU's terminal features when STDOUT is not a terminal
- allow mounting the ISO via sudo, because the default method (udisks)
  needs additional PolicyKit configuration

Story: 2007858
Task: 40441

Change-Id: I19ec7e55aaa80b6ed98162b65e41ce1a63f6da40
2020-08-04 19:48:09 -04:00
Zuul
7576ea5213 Merge "Add --config-file argument for docker image build script" 2020-07-29 06:56:59 +00:00
zhipengl
4d6af626b1 Add --config-file argument for docker image build script
build-stx-base.sh and build-stx-images.sh need to accept a
new argument (--config-file <path>) specifying a path to a
config file.
These commands automatically look for the config files at
the predetermined location, and apply them if found. If not
found, continue without error. If optional argument
--config-file is added, These commands will use specified
configfile instead of predetermined one.

Test pass on local build enviroment.

Closes-Bug: 1886062

Change-Id: I476e38064d1515d3d0a0d49c789f1d472d8ca1cc
Signed-off-by: zhipengl <zhipengs.liu@intel.com>
2020-07-28 13:49:47 +00:00
Joseph Richard
6a2d0b73c2 portieris image tagging
Tag docker.io/starlingx/portieris image.
master-centos-stable-20200726T013628Z.0 is retagged as stx.5.0-v0.7.0

Story: 2007348
Task: 40415
Change-Id: I93472bcd030fec782717f9db5cc3b2223e9c654a
Signed-off-by: Joseph Richard <joseph.richard@windriver.com>
2020-07-26 12:33:40 -04:00
Don Penney
962c4ac21c Reorganize tag management files
Rather than maintain tag management files for each release, use a
single file that is updated as needed. This update drops the r3
image-tags file and renames the r4 file to drop the release number.

Change-Id: I37d6d4b805d488db6702c48170805df54fca7300
Signed-off-by: Don Penney <don.penney@windriver.com>
2020-07-22 16:42:55 -04:00
Zuul
3f32ab7e05 Merge "Add vault-manager image to tag management" 2020-07-22 19:41:41 +00:00
Cole Walker
2d316fc64a Add vault-manager image to tag management
Story: 2007718
Task: 40418

Change-Id: I578ab06c2561b3b819eae683e4930b16036ade69
Signed-off-by: Cole Walker <cole.walker@windriver.com>
2020-07-22 14:31:14 -04:00
Davlet Panech
0c7d394f60 build-img: generate a QCOW2 image
Rewrote script (build-img) that generates a QCOW2 (virtual machine)
image with StarlingX pre-installed. This will allow users to create
virtual machines more easily using a disk where verything is already
installed.

This script updates a standard iso file with a default menu choice
(e.g., AIO/serial console) and adds a kickstart script that configures
the network; then boots the ISO in QEMU and lets it install to a disk
image.

The generated image only works in QEMU because it has a few things that
are specific to that emulation:
* /etc/lvm/lvm.conf restricts LVM to only the bus/port/device type that
  QEMU emulates SATA as
* It includes configuration for the network device named "ens3", which
  is QEMU's default

Story: 2007858
Task: 40163

Change-Id: I9fa6960eadbeca9481dc91c0878153d3fe95f0c1
Signed-off-by: Davlet Panech <davlet.panech@windriver.com>
2020-07-20 13:20:58 -04:00
Matt Peters
26658d920b Add configurable docker build context and file
Add support for building docker images with an explicit
configuration for specifying the docker context and Dockerfile.
This is required for projects that build from 3rd party source
that maintains a Dockerfile outside of the project repo and
builds from source that is cloned by the repo tool into the
project path.

New Configuration Parameters:

DOCKER_CONTEXT=<relative-path-to-source>
DOCKER_FILE=<relative-path-to-docker-file>

NOTE: Paths are relative to the image build file.

Story: 2006537
Task: 40286
Change-Id: I6520fd31218aaf849f4d1e803bb1e0bfb614b29f
Signed-off-by: Matt Peters <matt.peters@windriver.com>
2020-07-06 16:29:37 -04:00
zhipengl
43323eb3fa Add docker image build config file
Add python2 based services to docker_image_build_cfg_file.
The config file could then be modified to individually shift
images from python2 to python3 build method without having
to sync with cengn build scripts when there is a change.
Add --python2-wheels opt for build-stx-images.sh to specify link
to python2 based wheels tarball.

Story: 2007474
Task: 40139

Depends-on: https://review.opendev.org/#/c/712880/
Change-Id: Ic4313a2ac6f7ec2c8b87bef960f4a796f6a10738
Signed-off-by: zhipengl <zhipengs.liu@intel.com>
2020-06-26 04:43:30 +08:00
yuchengde
ee5eb24836 Modify build-tools and stable-wheels for Ussuri upgrading
story: 2007474
task: 39178

Depends-on: https://review.opendev.org/#/c/712862/
Change-Id: Ib383cd77da5ef7393ed2281861cccffd03986850
Signed-off-by: Yu Chengde <yu.chengde@99cloud.com>
2020-06-23 18:29:31 +08:00
Zuul
aa31732371 Merge "use single '=' for string equality in repo forall" 2020-06-18 14:15:43 +00:00
Zuul
6522abb866 Merge "remove leading path from branch name" 2020-06-18 14:09:10 +00:00
Saul Wold
5e85a4bb54 remove leading path from branch name
Remove the f/ or r/ from a branch name so we can create valid
prefix for the new manifest that gets created with the --manifest
flag.

Change-Id: I4eb8f3c70f85566463fb114944309597241a7a89
Signed-off-by: Saul Wold <sgw@linux.intel.com>
2020-06-17 11:02:38 -07:00
Teresa Ho
7d2a8a4497 Update stx-platformclients tag to stx.4.0-v1.3.0
Story: 2006740
Task: 39498

Change-Id: I9d597d0d408aef57044fb50e1d7a7ba75401069d
Signed-off-by: Teresa Ho <teresa.ho@windriver.com>
2020-06-17 11:06:37 -04:00
Saul Wold
4cb7539a9f use single '=' for string equality in repo forall
The sub-script fails when a double '==' is used inside of a repo
forall -c option. The single '=' is preferred when used in 'test'
or '['.

The error is similar to this:
if [ $REPO_PATH == cgcs-root ]; then echo $REPO_PROJECT; fi: 1: [: cgcs-root: unexpected operator

Closes-Bug: 1883772
Change-Id: If17de30a03939240f0cc329179d452820a691e32
Signed-off-by: Saul Wold <sgw@linux.intel.com>
2020-06-16 11:44:44 -07:00
Scott Little
5003bb100d Extend GITREVCOUNT to inputs other than SRC_DIR and PKG_BASE
Changes in brief:

1) TIS_PATCH_VER will now accept new automatic variables
'OTHER_GITREVCOUNT' and 'SRC_GITREVCOUNT'.

2) New BASE_SRCREV variables are supported, 'SRC_BASE_SRCREV' and
'BASE_SRCREV_FOR_PATH'

3) OPT_DEP_LIST_FOR_BUILD_TYPE[<build-type>] joins OPT_DEP_LIST as
a way to list files that affect the build, but are not included
in the assembled src.rpm.

4) Some code relating to interpreting build_srpm.data files,
and producing a list of files that affect the build, are relocated
to 'srpm-util' and restructured for greater utility.

The details:

TIS_PATCH_VER=<expression>
                     # An integer, or one of the supported
                     # variables listed below, or the sum of
                     # variables and integers.
                     #  e.g.
                     #  TIS_PATCH_VER=PKG_GITREVCOUNT+SRC_GITREVCOUNT+5

   PKG_GITREVCOUNT   # Count git revisions relative to PKG_BASE.
                     # Optionally only count from PKG_BASE_SRCREV

   SRC_GITREVCOUNT   # Count git revisions relative to SRC_DIR.
                     # Optionally only count from SRC_BASE_SRCREV

   GITREVCOUNT       # Deprecated, please use SRC_GITREVCOUNT instead.
                     # Count git revisions relative to SRC_DIR.
                     # Optionally only count from TIS_BASE_SRCREV

   OTHER_GITREVCOUNT # count git revisions from all sources excluding
                     # PKG_BASE and SRC_DIR
                     # Optionally only count from
                     # BASE_SRCREV_FOR_PATH[<path>]

PKG_BASE_SRCREV=<sha>   # Limit PKG_GITREVCOUNT revision count to
                        # commits since <sha>

SRC_BASE_SRCREV=<sha>   # Limit SRC_GITREVCOUNT revision count to
                        # commits since <sha>

TIS_BASE_SRCREV=<sha>   # Deprecated, please use SRC_BASE_SRCREV
                        # instead
                        # Limit GITREVCOUNT revision count to commits
                        # since <sha>

BASE_SRCREV_FOR_PATH[<path>]=[<sha>|OTHER_PKG_BASE_SRCREV]
                        # Limit OTHER_GITREVCOUNT revision count for
                        # commits under <path> to commits since <sha>.
                        # If <path> is the PKG_BASE of another package
                        # (not the current package) then the keyword
                        # 'OTHER_PKG_BASE_SRCREV' can be used to extract
                        # the 'PKG_BASE_SRCREV' value of the other
                        # package.
                        #
                        # The <path> can reference variables like
                        # $STX_BASE and $GIT_BASE.

OPT_DEP_LIST=<path-list>  # Add a space separated list of paths that
                          # don't contribute to the content of a src.rpm
                          # but do contribute to triggering a rebuild,
                          # and possibly modifying the TIS_PATCH_VER via
                          # use of OTHER_GITREVCOUNT.

OPT_DEP_LIST_FOR_BUILD_TYPE[<build-type>]=<path-list>
                          # For a specific build type only, add a space
                          # separated list of paths that don't
                          # contribute to the content of src.rpm,
                          # but do contribute to triggering a
                          # rebuild, and possibly modifying the
                          # TIS_PATCH_VER via use of OTHER_GITREVCOUNT.

Story: 2006166
Task: 39765
Change-Id: I9e7d409d4eefdb41a7083db1b801d531c443b678
Signed-off-by: Scott Little <scott.little@windriver.com>
2020-06-10 17:48:26 -04:00
Bin Yang
e01eb44f2c enable pbr version
Currently all flock packages have a common base version of 1.0 or 1.0.0
defined in specfiles. The build system creates tarballs and whl files
with the base version (without TIS_PATCH_VER) and the version is also
attached the generated RPM file (with TIS_PATCH_VER).

This patch addes PBR (OpenStack's Python Build Reasonableness) support,
which can generate the version by parsing semantically-versioned Git
tags and walking the Git history back to the last release tag.

A new variable named PBR_VERSION is supported in build_srpm.data. If its
is not set, the build system still uses TIS_PATCH_VER like before.
If PBR_VERSION=auto, the build system will generate the version by
PBR and create tarballs, whl files and RPM files with this version.
If PBR_VERSION is set manually, the build system will use it directly.

Story: 2006703
Task: 37014

Change-Id: Ic377e3ae3344b291d2d9c0d0fca8f681d1a007b5
Signed-off-by: Bin Yang <bin.yang@intel.com>
Signed-off-by: Yang, Bin <bin.yang@intel.com>
2020-06-08 16:54:14 +08:00
Scott Little
c70efec479 Add a dependency checker to build-pkgs
Builds a single package in a clean environment to detect missing
build dependencies.

Usage:
build-pkgs --dep-test <pkg>

Note:
Should only be run after a full 'build-pkgs'.
We want the build to fail because a dependency wasn't
been listed, not because a dependency hasn't been built yet.

Closes-Bug: 1880248
Change-Id: Icf64be37a77de0992647c37ce392fd5a2a458fba
Signed-off-by: Scott Little <scott.little@windriver.com>
2020-06-05 18:42:21 -04:00