To generate the pre-patched ISO, take some content
from a secondary input ISO.
Specifically, all ISO content should be taken from 24.09 GA,
expect for the "patches", "kickstart" and "upgrades" folders.
The ostree_repo remains the same as it is, being taken
from "ostree_base"
Test Plan:
pass - Mock pipeline
Story: 2011318
Task: 51506
Depends-On: https://review.opendev.org/c/starlingx/root/+/938229
Change-Id: I3cc5151c97cf374cf04586c29f76528f83bd6658
Signed-off-by: Leonardo Fagundes Luz Serrano <Leonardo.FagundesLuzSerrano@windriver.com>
This change adds the stage build-iso to the patch pipeline, change the
ISO used by the create-prepatched-iso script and specify the ostree
repository to be use as base.
Test plan:
PASS: Run complete patch pipeline with success
Depends-On: https://review.opendev.org/c/starlingx/root/+/936975
Closes-Bug: 2090871
Change-Id: Ia0b47fb9a18e4747c4983e09b144691d27c173e2
Signed-off-by: Dostoievski Batista <dostoievski.albinobatista@windriver.com>
This change allows the user to select if he wants to sign the patch
using the SIGN_PATCH variable, when set to True it will automatically
add the "--remote-sign" option to the patch-builder script and export
the env variables, SIGNING_USER and SIGNING_SERVER, from the build.conf
Test plan:
PASS: Run entire patch pipeline with SIGN_PATCH enable
Story: 2010676
Task: 51404
Change-Id: I7e895f30ea64a517a3555117250a4f00f8b6dfe1
Signed-off-by: Dostoievski Batista <dostoievski.albinobatista@windriver.com>
This changes remove from Jenkins Pipeline the post step
archive-prerequisites.sh from the step "download-prerequisites".
TESTS
========================
* Run entire monolithic Jenkins Pipeline with success.
* Re-run monolithic with a fix and make sure it finished success.
Closes-Bug: 2028880
Change-Id: I157e532b86ff10df9e13463eca732f58845b3db4
This change adds support for creating a patch on top of specified build
state using Jenkins:
- patch-monolithic: Main job where all the stages are listed and called.
- ostree-pull: Part where we pull the ostree from the remote build home.
- patch-make: Part where "patch-builder" is executed and the patch gets
created.
- patch-iso: Part where the "create-prepatched-iso" is executed and the
pre-patched ISO is created.
- publish-patch: Part where the publish directory for the pre-patched
ISO and patch is created.
We also update the build-packages part to support "reuse_maximum" option
when running build-pkgs.
Test plan:
PASS: Run full monolithic pipeline
PASS: Run full patch-monolithic pipeline
Story: 2010226
Task: 50700
Change-Id: I7c7d688f2c568532a0b23844dcfd81349ca96476
Signed-off-by: Dostoievski Batista <dostoievski.albinobatista@windriver.com>
To publish realtime log to mirror.starlingx.windriver.com.
BUILD_RT information is required, so adding this parameter.
Test-Plan:
PASS - Run jenkins job to verify publish-log part.
PASS - Check if package log file exists.
Closes-bug: 2071355
Change-Id: Ibb185df1b4c6ad7f5f488062f4f489744a03a6b8
Signed-off-by: suyash <suyash.srivastava@windriver.com>
We need to publish sufficient logs to mirror.starlingx.windriver.com
to allow debugging of a build problem. The current publish-logs step of
jenkins pipeline only concerns itself with jenkins logs.
This new script will be executed as part of publish-logs step.
Test-Plan:
PASS - Run jenkins job to verify publish-log part.
PASS - Check if package log file exists.
Closes-bug: 2071355
Change-Id: I0c884e4817134e7b11dc13a0364ba90af18a4d5d
Signed-off-by: suyash <suyash.srivastava@windriver.com>
Separate the build step from startup for environment containers. Build
step is very slow, and we want to be able to start the containers
without rebuilding when troubleshooting Jenkins jobs.
TESTS
==============================
Run both steps from Jenkins
Story: 2010226
Task: 49212
Depends-On: https://review.opendev.org/c/starlingx/tools/+/902792
Signed-off-by: Davlet Panech <davlet.panech@windriver.com>
Change-Id: I3462de0a9550310c931cbed2555de1b3a6bc53f9
This step was missed when we created these scripts for Debian.
Story: 2010226
Task: 48670
Change-Id: I73f90b5a018fc32b39ab6d265acfbbb7fc275415
Signed-off-by: Davlet Panech <davlet.panech@windriver.com>
This patch contains improvements that avoid hitting DockerHub download
rate limits. Add logic to login to DockerHub at various stages in the
build.
* New config option DOCKER_CONFIG specifies a directory with docker CLI
configuration files and defaults to $HOME/.docker. This env var is
recognized by docker CLI and is used whenever Jenkins scripts in this
repo access docker on the host, eg when running rsync within a
container.
* Old config option DOCKER_CONFIG_FILE is now deprecated in favor of
$DOCKER_CONFIG/config.json ; but still supported with a warning. If
specified, it will be used only to authenticate app container builds.
* New sub-job: docker-host-login: execute "docker login" before doing
anything else. This makes sure host docker daemon and subsequent
"docker build" etc are authenticated. This step, and all subsequent
docket operations, used configuration specified by $DOCKER_CONFIG.
Builds using different credentials will not interfere with each
other.
* Rename DOCKER_REGISTRY_PUSH_LOGIN_LIST => DOCKER_REGISTRY_LOGIN_LIST
Old name is still supported with a warning. Always login to these
registries within the builder container, rather than only before
pushing app containers. The intent is for this parameter to contain
"docker.io" in most situations; otherwise we will use anonymous access
to DockerHub, which may hit the download rate limit more frequently.
* If DOCKER_REGISTRY_LOGIN_LIST contains "docker.io", add
"--dockerhub-login" when calling "stx-init-env". This makes sure we
additionally authenticate minikube's embedded docker daemon used
to build/pull builder images in that environment.
LIMITATIONS
=================
K8s itself may pull images anonymously from docker hub. This can happen
only with the images that we don't build ourselves, used for running
some of the builder containers, ie "nginx" and "docker" images, and only
if they are not already in the local docker's cache.
TESTING
=================
Set DOCKER_CONFIG to a custom directory with a config.json file.
Set DOCKER_REGISTRY_LOGIN_LIST="docker.io SOME_OTHER_REPO". Run a build
and make sure docker credentials are copied and applied inside the build
container.
Story: 2010055
Task: 48436
Change-Id: I23bf9817cce3ed833fa605d6448f9ef600e056b0
Signed-off-by: Davlet Panech <davlet.panech@windriver.com>
This patch creates or updates the symlink "latest_iso_build" in the
archive and publish areas, immediately after the ISO build succeeds, but
before the other stages (docker images etc).
This allows external users to reference that path to download the latest
ISO, even if the overall build had failed due to docker image build
errors.
TESTS
================================================
Run a build where on of the docker images fails to compile
Make sure the new ISO build symlink gets created, even though the
overall build fails.
Story: 2010226
Task: 47978
Signed-off-by: Davlet Panech <davlet.panech@windriver.com>
Change-Id: Idd57b4e0bbe0f231f83304a600496a48a71fba9f
* Jenkins scripts:
- remove POST_ISO_SIGNING job parameter, as ISO signing is controled
by build.conf, as originally intended.
* build.conf:
- rename SIGN_ISO => SIGN_ISO_FORMAL to better reflect purpose. ISOs
are always signed, with developer keys (SIGN_ISO_FORMAL=false) or
the signing server (SIGN_ISO_FORMAL=true).
- add SECUREBOOT_FORMAL - whether to generate secureboot signatures
using the signing server (true), or not to generate them at all
(false)
* Added code in job_utils.sh to set the defaults for these new config
options as necessary, in case the job runs against an older build.conf
that still has the obsolete BUILD_ISO option.
TESTS
========================
* Make sure SIGN_ISO_FORMAL==true calls "build-image --no-sign"
followed by "sign_iso_formal.sh"
* Make sure SIGN_ISO_FORMAL==false calls "build-image" not
followed by "sign_iso_formal.sh" and the dev-key based ISO signature
gets created
* Make sure SECUREBOOT_FORMAL==true calls calls the secureboot script
* Make sure SECUREBOOT_FORMAL==false does not call the secureboot script
* Test with both the new parameters undefined, but SIGN_ISO defined,
and make sure they aquire expected defaults
Story: 2010226
Task: 47777
Depends-On: https://review.opendev.org/c/starlingx/root/+/879206
Signed-off-by: Davlet Panech <davlet.panech@windriver.com>
Change-Id: I928de97fefc70b3062820547d1256c2a3ce106e8
1) Register an ssh-key allowing access to the signing server
from within LAT.
2) Provide a means to bypass poat iso build signing, as the
signing should have oddured within LAT.
Story: 2009221
Task: 47358
Signed-off-by: Scott Little <scott.little@windriver.com>
Change-Id: If450ed2f67ef3e094f8370d9857c8bdf506a9571
Currently Jenkinsfile ignores failures in sub-jobs related to docker
images (base image, wheels, etc). If they fail, the overall build
remains successful. This allows docker-related build problems remain
unnoticed.
Make this behavior conditional on a job parameter.
TESTS
==============================================
* A build where there are no docker-related errors (overall: SUCCESS)
* A build where wheels fail and SUPPRESS_DOCKER_IMAGE_BUILD_ERRORS is
true (overall: SUCCESS)
* A build where wheels fail and SUPPRESS_DOCKER_IMAGE_BUILD_ERRORS is
false (overall: FAIL)
Story: 2009895
Task: 47280
Signed-off-by: Davlet Panech <davlet.panech@windriver.com>
Change-Id: I16ac4f63033ecbc78c647c4da105d550fe2facc4
Some versions of Jenkins crash when a multi-line parameter is empty.
Solution: set PATCH_LIST to "-" by default, and treat it as empty.
Signed-off-by: Davlet Panech <davlet.panech@windriver.com>
Change-Id: I7bd8f994debbfe86bc914ce469b81518ca13ea4a
Many docker images failed to compile in StarlingX, and we never publish
the latest_docker_image_build symlink & the last build's image lists on
CENGN. As a result there's no fixed URL where we could download the
latest sucessfully-built image tags.
This patch always publishes the symlink + lists, even if some docker
images fail to compile. This is temporary until most/all problems with
stx images are resolved.
Also: try building both distroless & debian images before giving up, so
that as many build errors as possible show up in the logs
Signed-off-by: Davlet Panech <davlet.panech@windriver.com>
Change-Id: If18ca2c7d83db520201b6027b970431c4161ac51
Main job resolves this repo's current commit, then passes that exact
commit SHA to each sub-job. This is to make sure all executed sub-jobs
are consistent with each other.
Signed-off-by: Davlet Panech <davlet.panech@windriver.com>
Change-Id: I20842b29accc9102fc98f63ca775b9988746e975
This is required for the CENGN build cleanup scripts to work reliably.
Signed-off-by: Davlet Panech <davlet.panech@windriver.com>
Change-Id: Iec8f58a4c9ff995ebff5563b4817843cd7467f19
This makes sure we have (partial) image lists in the publish dir,
even if some images failed to build.
Signed-off-by: Davlet Panech <davlet.panech@windriver.com>
Change-Id: I1ffe33ac305d617dfca84fed2dad55322cfe5b47
New set of scripts for generating & publlishing the "export" directory.
It allows one to specify a custom command to populate a directory with
arbitrary files, at the end of the build.
Signed-off-by: Davlet Panech <davlet.panech@windriver.com>
Change-Id: Ic071f395b4c1ef5ffe8a373564074e0a399ae6b8
Currently errors in docker image builds fail the entire build. Docker
images are in flux, and are likely to cause every build to fail.
Workaround: don't fail the main build if some docker images failed (but
print out a warning in Jenkins log).
Changes:
- Ignore docker image build errors with a warning
- Misc changes to scripts to make sure various steps happen in the right
order
Story: 2010226
Task: 46146
Signed-off-by: Davlet Panech <davlet.panech@windriver.com>
Change-Id: Ib8869ce263731f7bce3157890c303ec5cec59fde
- New parameter BUILD_HELM_CHARTS
- build.conf: new parameter HELM_CHART_APPS
- monolithic.Jenkinsfile: separate out docker image steps into
individual stages
Signed-off-by: Davlet Panech <davlet.panech@windriver.com>
This commit declares the BUILD_ISO variable in the
publish-package and build-packages sub jobs.
Signed-off-by: Luis Sampaio <luis.sampaio@windriver.com>
