17 Commits

Author SHA1 Message Date
Scott Little
5003bb100d Extend GITREVCOUNT to inputs other than SRC_DIR and PKG_BASE
Changes in brief:

1) TIS_PATCH_VER will now accept new automatic variables
'OTHER_GITREVCOUNT' and 'SRC_GITREVCOUNT'.

2) New BASE_SRCREV variables are supported, 'SRC_BASE_SRCREV' and
'BASE_SRCREV_FOR_PATH'

3) OPT_DEP_LIST_FOR_BUILD_TYPE[<build-type>] joins OPT_DEP_LIST as
a way to list files that affect the build, but are not included
in the assembled src.rpm.

4) Some code relating to interpreting build_srpm.data files,
and producing a list of files that affect the build, are relocated
to 'srpm-util' and restructured for greater utility.

The details:

TIS_PATCH_VER=<expression>
                     # An integer, or one of the supported
                     # variables listed below, or the sum of
                     # variables and integers.
                     #  e.g.
                     #  TIS_PATCH_VER=PKG_GITREVCOUNT+SRC_GITREVCOUNT+5

   PKG_GITREVCOUNT   # Count git revisions relative to PKG_BASE.
                     # Optionally only count from PKG_BASE_SRCREV

   SRC_GITREVCOUNT   # Count git revisions relative to SRC_DIR.
                     # Optionally only count from SRC_BASE_SRCREV

   GITREVCOUNT       # Deprecated, please use SRC_GITREVCOUNT instead.
                     # Count git revisions relative to SRC_DIR.
                     # Optionally only count from TIS_BASE_SRCREV

   OTHER_GITREVCOUNT # count git revisions from all sources excluding
                     # PKG_BASE and SRC_DIR
                     # Optionally only count from
                     # BASE_SRCREV_FOR_PATH[<path>]

PKG_BASE_SRCREV=<sha>   # Limit PKG_GITREVCOUNT revision count to
                        # commits since <sha>

SRC_BASE_SRCREV=<sha>   # Limit SRC_GITREVCOUNT revision count to
                        # commits since <sha>

TIS_BASE_SRCREV=<sha>   # Deprecated, please use SRC_BASE_SRCREV
                        # instead
                        # Limit GITREVCOUNT revision count to commits
                        # since <sha>

BASE_SRCREV_FOR_PATH[<path>]=[<sha>|OTHER_PKG_BASE_SRCREV]
                        # Limit OTHER_GITREVCOUNT revision count for
                        # commits under <path> to commits since <sha>.
                        # If <path> is the PKG_BASE of another package
                        # (not the current package) then the keyword
                        # 'OTHER_PKG_BASE_SRCREV' can be used to extract
                        # the 'PKG_BASE_SRCREV' value of the other
                        # package.
                        #
                        # The <path> can reference variables like
                        # $STX_BASE and $GIT_BASE.

OPT_DEP_LIST=<path-list>  # Add a space separated list of paths that
                          # don't contribute to the content of a src.rpm
                          # but do contribute to triggering a rebuild,
                          # and possibly modifying the TIS_PATCH_VER via
                          # use of OTHER_GITREVCOUNT.

OPT_DEP_LIST_FOR_BUILD_TYPE[<build-type>]=<path-list>
                          # For a specific build type only, add a space
                          # separated list of paths that don't
                          # contribute to the content of src.rpm,
                          # but do contribute to triggering a
                          # rebuild, and possibly modifying the
                          # TIS_PATCH_VER via use of OTHER_GITREVCOUNT.

Story: 2006166
Task: 39765
Change-Id: I9e7d409d4eefdb41a7083db1b801d531c443b678
Signed-off-by: Scott Little <scott.little@windriver.com>
2020-06-10 17:48:26 -04:00
Bin Yang
e01eb44f2c enable pbr version
Currently all flock packages have a common base version of 1.0 or 1.0.0
defined in specfiles. The build system creates tarballs and whl files
with the base version (without TIS_PATCH_VER) and the version is also
attached the generated RPM file (with TIS_PATCH_VER).

This patch addes PBR (OpenStack's Python Build Reasonableness) support,
which can generate the version by parsing semantically-versioned Git
tags and walking the Git history back to the last release tag.

A new variable named PBR_VERSION is supported in build_srpm.data. If its
is not set, the build system still uses TIS_PATCH_VER like before.
If PBR_VERSION=auto, the build system will generate the version by
PBR and create tarballs, whl files and RPM files with this version.
If PBR_VERSION is set manually, the build system will use it directly.

Story: 2006703
Task: 37014

Change-Id: Ic377e3ae3344b291d2d9c0d0fca8f681d1a007b5
Signed-off-by: Bin Yang <bin.yang@intel.com>
Signed-off-by: Yang, Bin <bin.yang@intel.com>
2020-06-08 16:54:14 +08:00
Scott Little
61ef1d4124 Build avoidance for all build-type values
Two improvements.

1) Extend build avoidance to work with any build type,
and built in any order.

Previously build avoidance did not support build types other
than 'std' and 'rt', and only supported thos properly
if both 'std' and 'rt' are built at the same time.

2) Allow for a build avoidance config that does not specify
a host.  In this mode the reference build must be on the same
machine.  The envisioned use case is to allow a build server
to use build avoidance internally for faster builds, without
potentially exposing an rsync port on the network.

Story: 2006166
Task: 39207
Signed-off-by: Scott Little <scott.little@windriver.com>
Change-Id: Idbd87bcc7683746d7b2ff8ac7488919c4a406b50
2020-03-30 16:52:13 -04:00
Scott Little
b2286535c9 Update to build tools to support Build layering.
1: Building rpms now produces an rpm.lst file.  This file serves to
list all rpms produced by the build.  A seperate rpm.lst file
is producede for each build type (std, rt, installer). The file is
co-resident with the matching repodata directory.

The rpm.lst files will need to be published by cengn for each layer build.

The download tools in stx-tools will need to find the rpm.lst files
of lower layer builds, and use it to direct the download of rpms
from the lower layer.

2: Building rpms now produces an image.inc file.  This file serves
to list all rpms that the layer recommends be built into an iso.
The file is stored under the $MY_WORKSPACE/<build-type> subdirectory,
although it has identical content for any build-type.

The image.inc file will need to be published by cengn for each layer
build.

The download tools in stx-tools will need to download the per-layer
image.inc files to the $MY_REPO/cgcs-centos-repo/layer_image_inc/
sub-directory, renaming the file in some layer specific way.

The build-iso tool will process any *.inc files found under
$MY_REPO/cgcs-centos-repo/layer_image_inc/ ,
adding those rpms to the iso.

3) Add a mechanism for layer specific mock.cfg prototypes.
This will allow special handling of the real-time (aka 'rt'),
repositories for layer builds that need to do a 'rt' build.

4) Add support for a $MY_REPO/cgcs-centos-repo/rt subdirectory,
a place to build a repo of real-time rpms originating from
lower layer builds.

The download tools in stx-tools will need to populate the new
rt repos.

As of this writing, non-rt rpms remain in $MY_REPO/cgcs-centos-repo/.
i.e. there is not a new $MY_REPO/cgcs-centos-repo/std/ directory.

5) Some changes to make us more flexible about where we find
realease and bsp files.

6) Found that kernel mudules were not reliably building against
the hearnel-headers of our modified kernel.  Found that adding '--update'
to our mock build command was not working.  Does mock expect '--update'
to only be used independently of a build command?  It does work when
used in that manner, so that's what we will do.

7) The build-pkgs, build-srpms, build-rpms family of commands
can take a layer argument and/or will read the LAYER environment
variable.  Current the only use of this variable is to modify
the build-info.  It does NOT limit the compile to packages
for a specific layer.

Story: 2006166
Task: 37094

Depends-On: https://review.opendev.org/698756
Depends-On: https://review.opendev.org/700819
Change-Id: I817e08a19cdabe08b3fcc47dee63a36b461c13c0
Co-Authored-by: Martin Chen <haochuan.z.chen@intel.com>
Signed-off-by: Scott Little <scott.little@windriver.com>
2020-02-07 16:36:40 -05:00
Scott Little
83d4446d74 Prepare the build tools for code restucturing
Build tools need to be made ready for the code
restructuring:

1) new repos
2) relocation of release-info.inc and bsp files
3) Removal of the stx- prefix from sub-repos.

Changes:
1) Create new functions to find the release-info.inc and
bsp file under both pre and post restructure paths.

2) Update .gitignore to ignore all the new repos that
are to be created.

3) Remove an argument sanity check from build-helm-charts.sh.
It is making invalid assumptions about where applications
can be found in the directory structure.

4) build-iso will need to look to additional repos to find
packages from lower layers.

Change-Id: If62444390d0a5a363974c6ff8cdaceca35978bda
Story: 2006166
Task: 35687
Signed-off-by: Scott Little <scott.little@windriver.com>
2019-09-05 15:18:10 -04:00
Al Bailey
aa15df5887 Provide a new build mode for containers
Providing a new build target --containers.

Container build targets typically need network access to
build, and do not need to be built as part of a regular build.

Container build targets are defined by: centos_pkg_dirs_containers
Container build targets can utilize the std and rt repos for
their build requirements.
Container target enables the mock flag: rpmbuild_networking

This also includes a fix where build-pkgs --installer in parallel
mode was building in serial mode instead.

Story: 2003909
Task: 27632
Change-Id: I3727a15fbdadc345a104c2af3446aee845171309
Signed-off-by: Al Bailey <Al.Bailey@windriver.com>
2018-11-08 14:01:44 -06:00
Scott Little
1aee4a42e6 Refusal to build libvirt src.rpm after CTRL-C
Problem:
An incremental build was interrupted by CTRL-C.
Libvirt was one of the packages interrupted mid build.
Subsequent incremental builds failed to rebuild libvirt.

The BUILD_NEEDED logic fails to trigger if the working
directory has been created, but is empty of src.rpms's.

Solution:
Add additional tests to ensure empty working directories
trigger a BUILD_NEEDED condition.

Closes-Bug: 1798191
Change-Id: I99e1fe76b016a5779d623d2ff91cc924b45a9338
Signed-off-by: Scott Little <scott.little@windriver.com>
2018-10-18 10:17:09 -04:00
Scott Little
b20ac0164d Build Avoidance
Purpose:
   Reduce build times after a repo sync by pulling in pre-generated
srpms and rpms and other build products created by a local reference build.

Usage:
  repo sync
  generate-cgcs-centos-repo.sh ...
  populate_downloads.sh ...
  build-pkgs --build-avoidance [--build-avoidance-user <user> \
     --build-avoidance-host <addr> --build-avoidance-dir <dir>]

Reference builds:
- A server performs a regular (daily?), automated builds using
  existing methods. Call these the reference builds.

- The builds are timestamped, and preserved for some time. (weeks?)
  The MY_WORKSPACE directory for the build shall have a common root
  directory, and a leaf directory that is a UTC time stamp of format
  YYYYMMDDThhmmssZ.
  e.g.
  MY_WORKSPACE=/localdisk/loadbuild/jenkins/StarlingX/20180719T113021Z

  Alternative formats are possible by setting values in ...
  "$MY_REPO/local-build-data/build_avoidance_source"
  e.g.
  BUILD_AVOIDANCE_DATE_FORMAT="%Y-%m-%d"
  BUILD_AVOIDANCE_TIME_FORMAT="%H-%M-%S"
  BUILD_AVOIDANCE_DATE_TIME_DELIM="_"
  BUILD_AVOIDANCE_DATE_TIME_POSTFIX=""
  BUILD_AVOIDANCE_DATE_UTC=0

  Which results in YYYY-MM-DD_hh-mm-ss format using local time.
  The one property that the timestamp must have is that they
  are sortable, and that the reference build and the consumer of
  the reference builds agree on the format.

- A build CONTEXT is captured, consisting of the SHA of each and every
  git that contributed to the build.

- For each package built, a file shall capture he md5sums of all the
  source code inputs to the build of that package.

- All these build products are accessible locally (e.g. a regional
  office) via rsync (other protocols can be added later).  ssh
  is also required to run remote query commands on the reference build.

  Initial ground work to support a selection variable ....
  BUILD_AVOIDANCE_FILE_TRANSFER="my-transfer-protocol"
  in $MY_REPO/local-build-data/build_avoidance_source"
  has been created, but "rsync" is the only valid value at this time.

- Location of the reference build can be specified via command line, or
  defaults can be put in $MY_REPO/local-build-data/build_avoidance_source.
  The local-build-data directory is gitignored by stx-root and so can be
  customized for local needs.
  e.g.
  cat $MY_REPO/local-build-data/build_avoidance_source
  BUILD_AVOIDANCE_USR="jenkins"
  BUILD_AVOIDANCE_HOST="stx-build-server.myco.com"
  BUILD_AVOIDANCE_DIR="/localdisk/loadbuild/jenkins/StarlingX"

Notes:
- Build avoidance is only used if requested.
- Build avoidance does not necessarily use the latest reference build.
  It compares the git context of all available reference builds vs your
  own git context, and chooses the most recent for which you gits have
  all the conent.  i.e. all your gits will be same or newer than that
  used by the reference build.  This also meens that some packages might
  still need to be rebuilt after the download step.
- Normally build avoidance remembers the last download context and will only
  consider reference builds newer than the last download.   You can reset
  using 'build-pkgs --build-avoidance --clear' to erase the download history.
  When might this matter to me?  If you change to an old branch that
  hasn't been synced recently and want to build in that context.
- The primary assumtion of Build Avoidance is that it is faster to
  download packages than to build them.  This is typically true of a
  good LAN, but likely not true of a WAN. This is why we emphasize the
  local nature of your reference build server.

Also in this update:
- reworked context generation to be relative to 'dirname $MY_REPO'
- Moved md5sum calculation to a common file, and fixed case where
  symlinks where canonacalized to paths outside of $MY_REPO.
  We'll make an exception to canonacalization to keep paths
  relative to $MY_REPO.
- In future other functions could be moved to the common file.

Story: 2002835
Task: 22754
Change-Id: I757780190cc6063d0a2d3ad9d0a6020ab5169e99
Signed-off-by: Scott Little <scott.little@windriver.com>
2018-09-17 16:41:31 -04:00
Scott Little
f6cbdb5af0 Use md5sums of inputs to determine if a package needs a rebuild
During the build-srpm phase of build-pkgs
- for each package
  - determine the unique sorted list of all input files
  - calculate md5 sums of all input files and save to file
     $MY_WORKSPACE/std/rpmbuild/SOURCES/<pkg-name>/srpm_input.md5
  - Compare vs reference md5sums of previous builds
     $MY_WORKSPACE/std/rpmbuild/SOURCES/<pkg-name>/srpm_reference.md5
  - If different, rebuild package
  - If build successful, cp srpm_input.md5 -> srpm_reference.md5

Build-rpms still uses timestamps, but we will switch from
'find --newer' to 'find -newermm' to identify updates files
which is compatable with how rsync sets the timestamps in the
next build-avoidance update.

Finally bring build-rpms-serial up to date, relative to
build-rpms-parallel, with respect to spec cache, signal handling,
process cleanup, and mock environment cleanup.  The remaining
delta between the two should reflect just the differences for
parallel building.

Story: 2002835
Task: 22754
Change-Id: I19c5594c84f026861e913a3af8af7dd0dc676ced
Signed-off-by: Scott Little <scott.little@windriver.com>
2018-08-31 17:02:00 -04:00
Scott Little
58b942baf1 Capture the git context of the build
First step in build avoidance is the creation of a reference build.
We need to capture the state of all the git trees that contribute
to the reference build.

This update will save the git context to $MY_WORKSPACE/CONTEXT

The context file format is actually a shell script that can be run to
recreate the build context.

   repo init ...
   repo sync
   cd $MY_REPO/..
   source .../CONTEXT

Sample CONTEXT file ....
(cd ./cgcs-root && git checkout -f 12a159594130d75aedf40da3f3b1e6bd7f9ff375)
(cd ./cgcs-root/stx/git/calico && git checkout -f 12a159594130d75aedf40da3f3b1e6bd7f9ff375)
(cd ./cgcs-root/stx/git/ceilometer && git checkout -f 12a159594130d75aedf40da3f3b1e6bd7f9ff375)
...
(cd ./stx-tools && git checkout -f 12a159594130d75aedf40da3f3b1e6bd7f9ff375)
(cd ./.repo/manifests && git checkout -f 12a159594130d75aedf40da3f3b1e6bd7f9ff375)
(cd ./.repo/repo && git checkout -f 12a159594130d75aedf40da3f3b1e6bd7f9ff375)

This update also the git_list function, which populates the previously
introduced GIT_LIST variable, and can be used in several other code
paths where we want to find all gits in a directory tree.

Tested build/clean/edit  parallel/serial

Story: 2002835
Task: 22754
Change-Id: I5009b8a360bdb4a03e5a4e83430b7f2ef135115e
Signed-off-by: Scott Little <scott.little@windriver.com>
2018-08-31 12:17:47 -04:00
Scott Little
b0bcb59c15 Build: Improved logging, error handling, and other small fixes
Add function and line number to error logs.

Log git commands that are issured when using --edit.

Replace global DIR variable with a more unique variable name.
DIR is used in many scripts that get sourced, possibly overwriting
the value in the calling script.

Make sure scripts won't fail if a redundant --parallel or --serial
argurement is seen.

Source files once at top of script, rather than in multiple places.

Source build_srpm.data once per package, and handle case where
build_srpm.data file is missing.

Fix handling of '<build-cmd> --installer <pkg-name>'.

Add handling of 'BUILD_IS_BIG' and 'UILD_IS_SLOW' to serial build.

Fix handling of packages that are built twice, once with a -rt
extension, during srpm audit.

Fix some exit code handling.

Prioritize Name over Service when searching for a package.

Delete some commented out code paths.

Change-Id: Ib5153cecf7b586d68aa382d382bc5a1a03a6b326
Story: 2002835
Task: 24519
Signed-off-by: Scott Little <scott.little@windriver.com>
2018-08-29 16:31:09 -04:00
Scott Little
cbc9c29c5c Incremental parallel build failure on drbd and drbd-kernel.
Background:
drbd-kernel compiles from source directory drbd.
drbd compiles from source directory drbd-tools.

In one cleanup path the build tools incorrectly assume
package name = source directory name.

As a result "build-pkgs --clean drbd" was incorrectly deleting
a temporary directory named 'drbd' that was in fact the temporary
directory for package drbd-kernel.

For incremental parallel builds this can lead to build failures
if drbd is cleaned in paraller with drbd-kernel being built.

Change-Id: Ia0f5e9e226ce3ea0f771c2a4dc899a2b606f561f
Story: 2002835
Task: 24519
Signed-off-by: Scott Little <scott.little@windriver.com>
2018-08-21 14:57:23 -04:00
Scott Little
8744c5f598 Build hangs on macro resolution in spec file
Problem:
Hung on a spec file that contains a seemingly self referential macro ...
   Version: %{_version}

Normally 'Version: x.y.z' has the effect of setting macro %{_version}.
In effect we have 'x=x'.  Not sure how rpm handles this, but we need to
prevent a recursion in our evaluation scripts.

Solution:
1) Test for recursion in spec_evaluate() and fail rather than
entering into a recursion.

2) Record the version from the srpm file name into an environment variable.
If we fail to read the version from the spec file directly, substitute
the one from the environment variable, if defined.

Note: Tripped up on mismatched '`' during testing, so substituted
all occurances with $().  Now a good browser should be able to
highlight an future mismatches.

Story: 2002839
Task: 22778
Change-Id: Ib69f879e531d842ff007d473bd3ad34db46cabf1
Signed-off-by: Scott Little <scott.little@windriver.com>
2018-08-13 13:52:35 -04:00
Scott Little
a39749ff68 Move content from stx-utils into stx-integ or stx-update
Package build-info is referenced in man build scripts.
It's relocation requires that the build scripts be modified
to reflect the new location.

from: stx/middleware/recipes-common/build-info/release-info.inc
to:   stx/stx-integ/utilities/build-info/release-info.inc

Change-Id: I82a9c12cd1e5607c1c903fc942a3c2007aca4920
Story: 2002801
Task: 22687
Signed-off-by: Scott Little <scott.little@windriver.com>
2018-08-01 16:12:17 -04:00
Scott Little
d2d96619a2 Shorten "addons/wr-cgcs/layers/cgcs" to just "stx"
Part of the project to remove cgcs references.
Replace and shorten the path the needlessly long and
complex "addons/wr-cgcs/layers/cgcs" path with just "stx".

This update just fixes up paths found in scripts, comments
and config files.

Depends-On: https://review.openstack.org/579954
Depends-On: https://review.openstack.org/579957
Depends-On: https://review.openstack.org/580170
Depends-On: https://review.openstack.org/579984
Change-Id: I04d653f740f17d8a9b2732f26d36907f635d8950
Signed-off-by: Scott Little <scott.little@windriver.com>
2018-07-04 15:12:58 -04:00
Scott Little
de12eed9d1 Support PATCHES_BASE, GIT_BASE, STX_BASE in build_srpm.data
Define PATCHES_BASE, GIT_BASE and STX_BASE within the build tools,
so that the can be refereced within build_srpm.data files.

STX_BASE: Synonym for i$CGCS_BASE.
          CGCS_BASE will be obsoleted in a future update.
GIT_BASE: New, points to the root of the git that contains the package.
          This will facilitate inter-package references that are
          co-located in the same git.
PATCHES_BASE: New, same as $PKG_BASE/$DISTRO/patches

Change-Id: I0636bb38b347374fa5ffe03ea67fd9d3422e2eea
Signed-off-by: Scott Little <scott.little@windriver.com>
2018-07-03 15:15:22 -04:00
Dean Troyer
cfe45dadae StarlingX open source release updates
Signed-off-by: Dean Troyer <dtroyer@gmail.com>
2018-06-01 07:45:23 -07:00