7 Commits

Author SHA1 Message Date
Davlet Panech
f5985aa83a Use private yum cache dir in build scripts
YUM & friends use a cache directory named /tmp/yum-$USER-xxx or
similar, even when used concurrently e.g. when bulding unrelated
source trees. This causes interference between CI pipelines.

This patch sets TMPDIR=$MY_WORKSPACE/tmp when calling repoquery,
etc.

Change-Id: Ieeb4d6fd9447f1c2988380c3427975893be365a5
Closes-Bug: 1867817
Signed-off-by: Davlet Panech <davlet.panech@windriver.com>
2020-03-19 10:21:00 -04:00
Scott Little
83d4446d74 Prepare the build tools for code restucturing
Build tools need to be made ready for the code
restructuring:

1) new repos
2) relocation of release-info.inc and bsp files
3) Removal of the stx- prefix from sub-repos.

Changes:
1) Create new functions to find the release-info.inc and
bsp file under both pre and post restructure paths.

2) Update .gitignore to ignore all the new repos that
are to be created.

3) Remove an argument sanity check from build-helm-charts.sh.
It is making invalid assumptions about where applications
can be found in the directory structure.

4) build-iso will need to look to additional repos to find
packages from lower layers.

Change-Id: If62444390d0a5a363974c6ff8cdaceca35978bda
Story: 2006166
Task: 35687
Signed-off-by: Scott Little <scott.little@windriver.com>
2019-09-05 15:18:10 -04:00
Felipe De Jesus Ruiz Garcia
e9bbaaf8a0 Fix mtce-guest not rebuilt after mtce-common
A recent change to stx-metal/mtce-common broke
the nightly StarlingX build due to stx-nfv/mtce-guest
build failure.

This occurred after a stx-metal/mtce-common change
removed a structure member that the stx-nfv/mtce-guest
compile required.

Since mtce-guest was not rebuilt automatically
the pending compile error was not noticed during UT and
the update was delivered only to show
up as a nightly build failure.

This commit recreates $MY_REPO/cgcs-tis-repo/dependancy-cache
running create_dependancy_cache.py every time when
any rpms and srpms changes, to keep the stx cache updated.

The rpms/srpms changes are detected using the repodata files:
The repodata filenames includes it’s own sha256 in their name,
this is useful to detect rpm/srpms by only comparing the filenames.

Closes-Bug: #1804687

Change-Id: Ieb6bf59542451a0cb4cc275bd9d59b532c5b6189
Signed-off-by: Felipe De Jesus Ruiz Garcia <felipe.de.jesus.ruiz.garcia@intel.com>
2019-05-15 11:55:32 -05:00
Scott Little
b20ac0164d Build Avoidance
Purpose:
   Reduce build times after a repo sync by pulling in pre-generated
srpms and rpms and other build products created by a local reference build.

Usage:
  repo sync
  generate-cgcs-centos-repo.sh ...
  populate_downloads.sh ...
  build-pkgs --build-avoidance [--build-avoidance-user <user> \
     --build-avoidance-host <addr> --build-avoidance-dir <dir>]

Reference builds:
- A server performs a regular (daily?), automated builds using
  existing methods. Call these the reference builds.

- The builds are timestamped, and preserved for some time. (weeks?)
  The MY_WORKSPACE directory for the build shall have a common root
  directory, and a leaf directory that is a UTC time stamp of format
  YYYYMMDDThhmmssZ.
  e.g.
  MY_WORKSPACE=/localdisk/loadbuild/jenkins/StarlingX/20180719T113021Z

  Alternative formats are possible by setting values in ...
  "$MY_REPO/local-build-data/build_avoidance_source"
  e.g.
  BUILD_AVOIDANCE_DATE_FORMAT="%Y-%m-%d"
  BUILD_AVOIDANCE_TIME_FORMAT="%H-%M-%S"
  BUILD_AVOIDANCE_DATE_TIME_DELIM="_"
  BUILD_AVOIDANCE_DATE_TIME_POSTFIX=""
  BUILD_AVOIDANCE_DATE_UTC=0

  Which results in YYYY-MM-DD_hh-mm-ss format using local time.
  The one property that the timestamp must have is that they
  are sortable, and that the reference build and the consumer of
  the reference builds agree on the format.

- A build CONTEXT is captured, consisting of the SHA of each and every
  git that contributed to the build.

- For each package built, a file shall capture he md5sums of all the
  source code inputs to the build of that package.

- All these build products are accessible locally (e.g. a regional
  office) via rsync (other protocols can be added later).  ssh
  is also required to run remote query commands on the reference build.

  Initial ground work to support a selection variable ....
  BUILD_AVOIDANCE_FILE_TRANSFER="my-transfer-protocol"
  in $MY_REPO/local-build-data/build_avoidance_source"
  has been created, but "rsync" is the only valid value at this time.

- Location of the reference build can be specified via command line, or
  defaults can be put in $MY_REPO/local-build-data/build_avoidance_source.
  The local-build-data directory is gitignored by stx-root and so can be
  customized for local needs.
  e.g.
  cat $MY_REPO/local-build-data/build_avoidance_source
  BUILD_AVOIDANCE_USR="jenkins"
  BUILD_AVOIDANCE_HOST="stx-build-server.myco.com"
  BUILD_AVOIDANCE_DIR="/localdisk/loadbuild/jenkins/StarlingX"

Notes:
- Build avoidance is only used if requested.
- Build avoidance does not necessarily use the latest reference build.
  It compares the git context of all available reference builds vs your
  own git context, and chooses the most recent for which you gits have
  all the conent.  i.e. all your gits will be same or newer than that
  used by the reference build.  This also meens that some packages might
  still need to be rebuilt after the download step.
- Normally build avoidance remembers the last download context and will only
  consider reference builds newer than the last download.   You can reset
  using 'build-pkgs --build-avoidance --clear' to erase the download history.
  When might this matter to me?  If you change to an old branch that
  hasn't been synced recently and want to build in that context.
- The primary assumtion of Build Avoidance is that it is faster to
  download packages than to build them.  This is typically true of a
  good LAN, but likely not true of a WAN. This is why we emphasize the
  local nature of your reference build server.

Also in this update:
- reworked context generation to be relative to 'dirname $MY_REPO'
- Moved md5sum calculation to a common file, and fixed case where
  symlinks where canonacalized to paths outside of $MY_REPO.
  We'll make an exception to canonacalization to keep paths
  relative to $MY_REPO.
- In future other functions could be moved to the common file.

Story: 2002835
Task: 22754
Change-Id: I757780190cc6063d0a2d3ad9d0a6020ab5169e99
Signed-off-by: Scott Little <scott.little@windriver.com>
2018-09-17 16:41:31 -04:00
Scott Little
b0bcb59c15 Build: Improved logging, error handling, and other small fixes
Add function and line number to error logs.

Log git commands that are issured when using --edit.

Replace global DIR variable with a more unique variable name.
DIR is used in many scripts that get sourced, possibly overwriting
the value in the calling script.

Make sure scripts won't fail if a redundant --parallel or --serial
argurement is seen.

Source files once at top of script, rather than in multiple places.

Source build_srpm.data once per package, and handle case where
build_srpm.data file is missing.

Fix handling of '<build-cmd> --installer <pkg-name>'.

Add handling of 'BUILD_IS_BIG' and 'UILD_IS_SLOW' to serial build.

Fix handling of packages that are built twice, once with a -rt
extension, during srpm audit.

Fix some exit code handling.

Prioritize Name over Service when searching for a package.

Delete some commented out code paths.

Change-Id: Ib5153cecf7b586d68aa382d382bc5a1a03a6b326
Story: 2002835
Task: 24519
Signed-off-by: Scott Little <scott.little@windriver.com>
2018-08-29 16:31:09 -04:00
Scott Little
354ee9e9c8 Slow builds due to sub-optimal build order
Problem:
   In the absence of a dependency cache, build-pkgs has
not information on which packages should be built first.
All it can do is try to build each package in a arbitrary order.
If the build fails, it is set aside and will be retired on
the next iteration.  Builds fail when a BuildRequired package
can't be found, because it hasn't been built yet.
   This was not an issue when cgcs-tis-repo, the home of the cache,
was stored in a git.

Solution:
   Have build-pkgs invoke create_dependancy_cache.py if the
cache is not present.

Note: We only run this automatically once.  If the user is
adding packages, or modifying the BuildRequires or Requires of
an existing package, they should re-run create_dependancy_cache.py
manually.

Change-Id: Id63e42903d9c0884470a990fb62fe1bafacd1849
Story: 2002835
Task: 24519
Signed-off-by: Scott Little <scott.little@windriver.com>
2018-08-22 10:31:19 -04:00
Dean Troyer
cfe45dadae StarlingX open source release updates
Signed-off-by: Dean Troyer <dtroyer@gmail.com>
2018-06-01 07:45:23 -07:00