26 Commits

Author SHA1 Message Date
Leonardo Fagundes Luz Serrano
c6b4ff0e88 Workaround for resolving deb.debian.org
The stx-init-env script sets up a minikube profile
and immediately starts building the container images.

Occasionally, the minikube network has not yet finished setup
when the build image step starts, which causes some URLs
to fail to resolve, most often deb.debian.org

While the root cause investigation is pending,
this commit adds a short pause in between these steps
to prevent the issue.

In addition, some extra logging is added for clarity.

Test Plan:
pass - ./stx-init-env --rebuild

Partial-Bug: 2093371

Change-Id: I4f5ce7849799b3fdd3a0bb4e64bab244b95cca94
Signed-off-by: Leonardo Fagundes Luz Serrano <Leonardo.FagundesLuzSerrano@windriver.com>
2025-01-09 20:11:13 +00:00
Davlet Panech
44ccc58f55 stx-init-env: get LAT installer from $STX_MIRROR
LAT docker file downloads the installer from a hard-coded URL that
points to https://mirror.starlingx.windriver.com . Allow users to
override this location by defining STX_MIRROR_URL in the host
environment. By default, guess mirror location from stx.conf.

TESTS
==========================
* Rebuild LAT container and make sure it uses the mirror URL from
  stx.conf
* Set STX_MIRROR_URL prior to calling stx-init-env and make sure it
  gets picked up by the docker file

Story: 2010055
Task: 49883

Signed-off-by: Davlet Panech <davlet.panech@windriver.com>
Change-Id: Id8ea88407f74003db934337efd574451658633d8
2024-04-17 16:23:05 +00:00
Davlet Panech
4df5160bbd stx-init-env: faster minikube status check
Avoid "minikube profile list" when checking whether the profile exists.
The list command attempts to connect to each profile and is quite slow.

Use "minikube status -p $MINIKUBENAME" instead.

Story: 2011038
Task: 49570

Signed-off-by: Davlet Panech <davlet.panech@windriver.com>
Change-Id: If799840d749de00af907de7867ec68fb9908afa3
2024-02-14 13:25:08 -05:00
Davlet Panech
4187e73f86 Commands to reset the build environment
* stx script:
- New command "stx control is-started" to complement start/stop
- New option "stx control {start,stop} --wait"

* stx-init-env:
- new option --reset: delete chroots + restart pods
- new option --reset-hard: stop pods, delete local workspaces,
  chroots, aptly, docker & minikube profile
- rename option "--nuke" to "--delete-minikube-profile"; old spelling
  is still accepted with a warning
- renamed & refactored some functions

* import-stx:
- new env var STX_RM_METHOD: may be optionally set to "docker" for
  deleting root-owned files via "docker run", rather than "sudo"

TESTS
=========================
* Misc sanity checks using minikube & k8s
* Manually tested blacklist checks in safe_rm()
* rm via "sudo" vs "docker run"
* Using minikube:
- stx-init-env
- stx-init-env --rebuild
- stx start, build all packages, --reset, build all packages
- stx start, build all packages, --reset-hard, stx-init-env,
  build all packages

Story: 2011038
Task: 49549

Signed-off-by: Davlet Panech <davlet.panech@windriver.com>
Change-Id: Ife4172ae9fa7b58332ac7ad65beb99525bc2a1a3
2024-02-14 13:19:31 -05:00
Davlet Panech
5b62b7ef1e stx-init-env: new option --no-start
New option to skip (re-)starting the pods. This will allow us to better
control build stages in Jenkins.

TESTS
=====================
Run with and without the new option

Story: 2010226
Task: 49211

Signed-off-by: Davlet Panech <davlet.panech@windriver.com>
Change-Id: I367a6db9503a13d376966376a8b25cbedccd1f35
2023-12-06 11:25:46 -05:00
Willy Njomen
f44dfffbe5 Updated minikube profile name
I fixed the issue with minikube_started() and helm_started() where the
grep command was not grepping the correct docker image and helm chart
due to matching substring. It was solved using helm ls command and the
option --kube-context when minikube eixist to pass minikube profile
and helm ls with the option --namespace when minikube is not exist.

TESTS
========================================

* setup a use profile names "MINIKUBENAME" Eg "stx" and "stx1"
* setup a similar environment to reproduce the issue

Closes-Bug: 1995400
Change-Id: Ic9b2224240e0e02be30957a96e7deb24da122136
Signed-off-by: Willy Njomen <willy.njomen@windriver.com>
2023-09-21 11:54:41 -04:00
Davlet Panech
0227d889de stx-init-env: remove -L option
Previous commit added a new option "--dockerhub-login/-L". Remove the
"-L" alias since we want to reserve it for future use.

TESTS
=======================
Run script with the new option and with/without --rebuild. Make sure
"docker login" is executed against the host docker daemon and minikube's
docker daemon.

Story: 2010055
Task: 48435

Signed-off-by: Davlet Panech <davlet.panech@windriver.com>
Change-Id: I06703f58e1598e35e53b12160743516ae7b20c14
2023-07-21 10:47:32 -04:00
Davlet Panech
dead08978a stx-init-env: login to DockerHub before starting
New option -L/--dockerhub-login: authenticate local docker daemon and
minikube's docker daemon before building or downloading docker images.
This makes sure DockerHub's download rate limit is counted towards the
current user's DockerHub account.

TESTS
=======================
Run script with the new option and with/without --rebuild. Make sure
"docker login" is executed against the host docker daemon and minikube's
docker daemon.

Story: 2010055
Task: 48435

Change-Id: Ia87d54a383f164c36b2380b1683a56407b1f5974
Signed-off-by: Davlet Panech <davlet.panech@windriver.com>
2023-07-20 16:01:34 -04:00
Willy Njomen
6d2105ae47 Override builder images registry path
Stx-tool always uses "stx-init-env --rebuild" from the docker hub
master branch. This causes random problems with people trying to
set up an environment

I used "stx-init-env --rebuild" to make sure the images are built
locally, it does not make sense to always use the latest tag for a
release.

TESTS
========================================

* Start stx-tools containers
* Pull an image inside "stx shell"
* Stop stx-tools container
* Run "stx-init-env" without the "--rebuild"
* to make sure the images are
  built locally

Story: 2010055
Task: 47970

Signed-off-by: Willy Njomen <willy.njomen@windriver.com>
Change-Id: Id466fbd243df1d77330473850fa4ba7d649941b7
2023-05-15 12:43:36 -04:00
Davlet Panech
5e70ce6c7e Environment overrides for builder image tags
New environment variables to set docker image tags for the images used
by the helm chart, as well as the images we download from Docker Hub.

When these new vars are missing, the scripts behave as before this
patch.

TESTS
============================
* Re-create the environment with/without the new env vars, with/without
  "--rebuild"
* Make sure "stx control start/stop" works
* Make sure images created by stx-init-env use tags matching new env
  vars

Signed-off-by: Davlet Panech <davlet.panech@windriver.com>
Change-Id: I66e061aef52fdb26c99303ecfbcd1fa9b43bc6bc
2022-10-07 10:49:26 -04:00
Zhixiong Chi
1211204a42 Debian: stx-init-env: fix --nuke option KeyError exception
Add the checking for minikube_exists to avoid the KeyError error,
when the command './stx-init-env --nuke' is executed and the minikube
cluster doesn't exist.

Story: 2009897
Task: 45824

Test Plan:
Pass: export MINIKUBENAME=abc ......
      './stx-init-env --nuke' works without the KeyError exception.

Signed-off-by: Zhixiong Chi <zhixiong.chi@windriver.com>
Change-Id: I2cf46e01df312604ca2a0afa38a498e7397747b8
2022-07-20 00:52:40 -04:00
Davlet Panech
c7e4e6b236 debian: --cache avoid pulling builder images
Changed the "--cache" option to skip pulling images that already exist
locally. Otherwise a locally-built image would be overwritten by a
downloaded one.

TESTS
===============================
Call the script with various permutations of --cache & --rebuild, with &
without deleting cached images prior.

Story: 2009897
Task: 45225

Signed-off-by: Davlet Panech <davlet.panech@windriver.com>
Change-Id: I37435d3d7a6d1a20b87794530cc937d759674685
2022-04-29 16:08:15 -04:00
Zuul
466bc3d795 Merge "debian: option to rebuild specific images" 2022-04-26 21:37:02 +00:00
Davlet Panech
0ed3e55da6 debian: auto-upgrade stx.conf on pod restart
This patch adds any missing keys to stx.conf while restarting pods via
stx-init-env. This should minimize disruption to developers when new
keys are added to the reference file, stx.conf.sample.

TESTS
========================
- Remove some required keys from stx.conf, then run stx-init-env and
  make sure they get re-added

Story: 2009897
Task: 45179

Signed-off-by: Davlet Panech <davlet.panech@windriver.com>
Change-Id: Ifbed10e01dc2e5653b827274a556f6dad166f1d8
2022-04-26 10:52:05 -04:00
Davlet Panech
d7d0843530 debian: option to rebuild specific images
This patch allows stx-init-env to rebuild specific images, rather than
all.

TESTS
===============
Run the script with various --rebuild options:
* without --rebuild
* with --rebuild
* with --rebuild=builder,lat-sdk
* with --rebuild=builder,INVALID_IMAGE_NAME

Story: 2009897
Task: 45177

Signed-off-by: Davlet Panech <davlet.panech@windriver.com>
Change-Id: I96801d7f44842303d3b9e4374abcae677873d8a7
2022-04-25 14:52:33 -04:00
Davlet Panech
cfe2abcb6b debian: stx-init-env: option to use docker cache
By default we use the "--no-cache" option when building containers. Add
a command-line option to avoid this, to make testing this script easier.

TESTS
=====================
Rebuild containers with and without the "--cache" option

Story: 2008846
Task: 45124

Signed-off-by: Davlet Panech <davlet.panech@windriver.com>
Change-Id: I79d61cc9a7606d6de65478ced5bab325f7264278
2022-04-20 15:23:32 -04:00
Luis Sampaio
52ef35d1bf Update Debian build system to support Kubernetes
Extends the build tools to add support to Kubernetes. For kubernetes,
it supports single node cluster and host path for the volumes.

New environment variables:
- PROJECT: build project name
- STX_PLATFORM:  minikube or kubernetes
- STX_BUILD_CPUS: replaces MINIKUBECPUS
- STX_K8S_NAMESPACE: kubernetes namespace name
* Default values are set to minikube, more details added in
import-stx.README.

To deploy stx on k8s you need to follow the below steps:

- create k8s namespace
- export PROJECT, included to support multiproject environments it is
used by the new default build home and also for docker tagging.
- export STX_BUILD_HOME, e.g: /localdisk/user/$PROJECT
- export STX_K8S_NAMESPACE="namespace_name"
- export STX_PLATFORM="kubernetes"
- export KUBECONFIG to your kubernetes config file
- STX_BUILD_CPUS replaces MINIKUBECPUS, this variable is used
by build-pkgs parallel jobs ($MAX_CPUS)
- Create your build home $STX_BUILD_HOME
- Init repo & repo sync
- source import-stx, check the env variables
- stx-init-env
- stx control status/start/stop/enter

Test Plan:

Pass: Create env on minikube
Pass: Create env on Kubernetes
Pass: Apply patch on current minikube env and continue to work on the
environment without issues
Pass: build package on Debian
Pass: build Debian image

Story: 2009812
Task: 44391

Signed-off-by: Luis Sampaio <luis.sampaio@windriver.com>
Change-Id: I7b760fbf1454f6aa90dd93dd9ff3a61d5fbd1b5c
2022-02-11 11:05:12 -08:00
Zuul
bdec843d04 Merge "stx tool: Add the cleanup for config file and configmap data" 2021-12-02 21:54:49 +00:00
Zhixiong Chi
13350756b5 stx tool: Support to allocate cpu and memory resources for minikube
Add the support to allow to define the cpu and memory resources
when we start the minikube.

Story: 2008862
Task: 44079

Signed-off-by: Zhixiong Chi <zhixiong.chi@windriver.com>
Change-Id: I4ca4f7b5bb6d6eb59ceafbb8114b2fd4a856cb0d
2021-11-30 02:41:27 -05:00
Zhixiong Chi
cd956cf61f stx tool: Add the cleanup for config file and configmap data
Add the '--clean' option for stx-init-env to support to refresh
the config file stx.conf, and to drop the configmap data and the
intermediate files.

Story: 2008862
Task: 44080

Signed-off-by: Zhixiong Chi <zhixiong.chi@windriver.com>
Change-Id: I53cdb50e7f3244940ec4ab26cd68e62aa15ba64f
2021-11-29 03:52:12 -05:00
Zhixiong Chi
01d2929e49 stx tool: Create mirrors directory and record the minikube variables
Create the mirror directory before the minikube cluster is created
otherwise it will be automatically created with root permission.

Log the minikube varibales MINIKUBE_HOME, MINIKUBENAME and
STX_BUILD_HOME into minikube_history.log.

Story: 2008862
Task: 43862

Signed-off-by: Zhixiong Chi <zhixiong.chi@windriver.com>
Change-Id: I72c877cc171f49633d8b1802834cf7da48a9e7b5
2021-11-07 21:15:51 -05:00
Davlet Panech
74653b6298 stx-init-env: download builder images
* stx-init-env
  - convert to executable form & prohibit sourcing
  - add command-line processing
  - download pre-built images from docker hub by default,
    instead of rebuilding; option to force a rebuild
  - option to delete minikube cluster

* import-stx: remove executable permission

* stx/lib/stx/stx_control.py:
  - stx control start: don't fail if helm chart is already installed
  - stx control stop: don't fail if helm chart is already uninstalled

Story: 2008846
Task: 43819
Signed-off-by: Davlet Panech <davlet.panech@windriver.com>
Change-Id: Idf301c4809ba747da08281ea5a735f3f2c4a3f8e
2021-11-03 13:44:39 -04:00
Davlet Panech
fcfa8b61ac Debian env scripts: allow user overrides
- import-stx: allow users to override MINIKUBENAME
- stx-init-env: push VolumePath via the environment. This script appears
  to have been designed to be sourced (it uses "return" statements
  rather than "exit"), yet it attempts to process a command-line
  argument ($1), which doesn't work in sourced files. Solution: define
  docker mount point in import-stx as an environment variable.

Story: 2008846
Task:  43753

Signed-off-by: Davlet Panech <davlet.panech@windriver.com>
Change-Id: I489be2276bc2f0798affaf6c334614648f5922c2
2021-10-27 15:53:45 -04:00
Zhixiong Chi
98b68bd82a stx tool: Setup the minikube env for starlingx debian building
Start the minikube env for stx tool, so that we can implement
the four containers then begin to execute the subtasks.

Story: 2008862
Task: 43583

Change-Id: I9941395ac2853f5edcbd306941d14f690477e9e7
Signed-off-by: Zhixiong Chi <zhixiong.chi@windriver.com>
2021-10-11 05:53:22 -04:00
Zhixiong Chi
8bcee26f1a stx tool: control: Add the support of control module
Implement the stx control module so that the developer could
manage the four containers with the 'stx control xxx' command.
Containers: (stx-builder|stx-pkgbuilder|stx-lat-tool|stx-repomgr)

Now we assume that we have a minikube env to be ready for this
module, we can use it directly. For the minikube env installation
we can build it with the stx-init-env script later.

Now support the action: [status|start|stop|enter|upgrade]

stx control status:  Check status of the containers
stx control start:   Boot the containers up
stx control stop:    Shutdown the containers
stx control enter:   Login the container(default builder)
stx control upgrade: Upgrade the containers

This module will be used after the developer have a minikube env,

Please refer to the more help information with the command
'stx control --help'

Story: 2008862
Task: 42514

Signed-off-by: Zhixiong Chi <zhixiong.chi@windriver.com>
Change-Id: I0c25e0f784671e90c56d5d367b12a27c3f3fa7cf
2021-09-15 21:35:47 -04:00
Zhixiong Chi
189200e17d stx-tool: config: Add the support of config module
Implement the stx config module so that the developer
could define the customized configuration before the
starlingx building.
Now support the action: show|add|get|unset
The usage of stx config module is like the git command.
Please refer to the more help information with the command
'stx config --help'

Story: 2008862
Task: 42515

Signed-off-by: Zhixiong Chi <zhixiong.chi@windriver.com>
Change-Id: I0310d1b591415f111e30efdf5bd76b555f725416
2021-09-01 03:54:44 -04:00