Add sources.list for non-default layers

After reestructuring `deb-local-binary` into multiple binary
repositories, one for each layer, the image build now has access to
three repositories by default:

  * deb-local-binary;
  * deb-local-binary-containers;
  * deb-local-build.

Although these repositories are sufficient for the vast majority of
images, there are some, such as `stx-openstackclients`, that require
dependencies that are found in another repository:
`deb-local-binary-openstack`.

This image, as the name suggests, contains all the OpenStack clients
currently supported by the StarlingX OpenStack application.

As one can see in [1], although most clients are installed in the form
of pip packages, i.e., through wheel files collected from the upstream
indexes, there are two clients that are installed in the form of
distribution packages:

  * python3-cinderclient;
  * python3-openstackclient.

The reason why they are installed as distribution packages is because
they have patches that are specific to StarlingX OpenStack -- [2] and
[3] -- and also because the build procedure can easily identify them in
`deb-local-build` and use the patched versions as expected by us (of
course, it they have been built previously).

The problem, however, is that although this image is currently being
built, there is a detail in it that can cause problems at run time.

Despite the build locating (and using) our patched packages, when it
searches for their dependencies, it always ends up using binaries from
repositories `deb-local-binary`, `deb-local-binary-containers` or
`deb-local-build`.

Which means that, if we have a dependency on
`deb-local-binary-openstack` meant for OpenStack Antelope, and the same
dependency on `deb-local-binary` meant for OpenStack Victoria, the image
build will use the latter, for the simple fact that it do not consider
the `openstack` layer (and, therefore, `deb-local-binary-openstack`).

To get around this problem, this change seeks to add a `sources.list`
for each build layer. By default, they are all disabled. However, if
necessary, they can be enabled and used in image builds based on Docker
or LOCI.

Note: This change is the prerequisite of [4].
      The test plan with the `stx-openstackclients` image will be done
      there.

[1] https://opendev.org/starlingx/openstack-armada-app/src/branch/f/antelope/upstream/openstack/python-openstackclient/debian/stx-openstackclient.stable_docker_image#L26-L27
[2] https://opendev.org/starlingx/openstack-armada-app/src/branch/f/antelope/upstream/openstack/python-cinderclient/debian/patches
[3] https://opendev.org/starlingx/openstack-armada-app/src/branch/f/antelope/upstream/openstack/python-openstackclient/debian/patches
[4] https://review.opendev.org/c/starlingx/openstack-armada-app/+/896462

Test Plan:
PASS - Build `stx-debian` base image
PASS - Build all container images with --clean and confirm that:
       1. All container images were built successfully;
       2. All container images were removed successfully.

Partial-Bug: 2037339

Change-Id: I36d1af990afad9fd69b723bf3a13b88673dd08ad
Signed-off-by: Luan Nunes Utimura <LuanNunes.Utimura@windriver.com>
(cherry picked from commit 5fa60a5421)

build-tools/build-docker-images/build-stx-base.sh

@@ -1,6 +1,6 @@
 #!/bin/bash
 #
-# Copyright (c) 2018-2019 Wind River Systems, Inc.
+# Copyright (c) 2018-2023 Wind River Systems, Inc.
 #
 # SPDX-License-Identifier: Apache-2.0
 #
@@ -10,6 +10,7 @@
 MY_SCRIPT_DIR=$(dirname $(readlink -f $0))
 
 source ${MY_SCRIPT_DIR}/../utils.sh
+source ${MY_SCRIPT_DIR}/../git-utils.sh
 
 # Required env vars
 if [ -z "${MY_WORKSPACE}" -o -z "${MY_REPO}" ]; then
@@ -356,6 +357,7 @@ else
             -e "s!@DEBIAN_DISTRIBUTION@!${DEBIAN_DISTRIBUTION}!g" \
             -e "s!@REPOMGR_DEPLOY_URL@!${REPOMGR_DEPLOY_URL}!g" \
             -e "s!@REPOMGR_ORIGIN@!${REPOMGR_ORIGIN}!g" \
+            -e "s!@LAYER@!${LAYER}!g" \
             "$@"
     }
 
@@ -365,6 +367,22 @@ else
     # debian.sources.list
     replace_vars "${SRC_DOCKER_DIR}/apt/debian.sources.list.in" >"${BUILDDIR}/apt/debian.sources.list"
 
+    # <layer>.sources.list
+    # These can be optionally used if it is necessary to build an image that
+    # requires dependencies that are in repositories not listed in
+    # `stx.sources.list`.
+    layer_cfg_name="${OS}_build_layer.cfg"
+    layer_cfgs=($(find ${GIT_LIST} -maxdepth 1 -name ${layer_cfg_name}))
+    LAYERS=($(
+      for layer_cfg in "${layer_cfgs[@]}"; do
+        echo $(cat "${layer_cfg}")
+      done | sort --unique
+    ))
+
+    for LAYER in "${LAYERS[@]}"; do
+        replace_vars "${SRC_DOCKER_DIR}/apt/layer.sources.list.in" >"${BUILDDIR}/apt/${LAYER}.layer.sources.list"
+    done
+
     # stx.sources: if user provided any --repo's use them instead of the template
     if [[ "${#REPO_LIST[@]}" -gt 0 ]] ; then
         rm -f "${BUILDDIR}/apt/stx.sources.list"

build-tools/build-docker-images/stx-debian/Dockerfile.stable

@@ -18,6 +18,15 @@ COPY apt/debian.sources.list /etc/apt/sources.list.d/debian.list.disabled
 COPY apt/stx.sources.list /etc/apt/sources.list.d/stx.list.disabled
 COPY apt/stx.preferences /etc/apt/preferences.d/stx
 
+# Install layer-specific binary repositories.
+# Note: They are all supposed to be disabled by default, but can be optionally
+#       enabled if it is necessary to build an image that requires
+#       dependencies that are in repositories not listed in `stx.sources.list`.
+COPY apt/*.layer.sources.list /etc/apt/sources.list.d/
+RUN for layer in /etc/apt/sources.list.d/*.layer.sources.list; do \
+      mv "${layer}" "$(echo "${layer}" | sed s/.layer.sources.list/.list.disabled/)"; \
+    done
+
 # repo templates:
 #   /etc/apt/sources.list.d/
 #     debian.list.disabled         - vanilla debian repos

build-tools/build-docker-images/stx-debian/apt/layer.sources.list.in

@@ -0,0 +1 @@
+deb [trusted=yes] @REPOMGR_DEPLOY_URL@/deb-local-binary-@LAYER@ @DEBIAN_DISTRIBUTION@ main