starlingx
/
docs
Code Issues Proposed changes
docs/doc/source/developer_resources/build_docker_image.rst

644 lines
25 KiB
ReStructuredText
Raw Permalink Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

=============================
Build StarlingX Docker Images
=============================
Building StarlingX Docker images consists of three components:
* StarlingX base image, which provides the operating system for the
Docker image
* Python wheels, providing an installation source for pip when
installing python modules via loci
* Docker image build, using image directives files
The images are currently built using either a Docker file or via loci
(https://github.com/openstack/loci), which is an OpenStack image
builder.
.. contents::
:local:
:depth: 1
----------
Base image
----------
The StarlingX base image is the operating system image that provides the
base for the StarlingX Docker images. This is built using the
build-stx-base.sh tool in stx-root/build-tools/build-docker-images.
Currently, we build two base images with CentOS - one for use with
the stable build stream images, and one for the dev build stream
(bleeding edge) images:
* The stable base image is configured with repo commands to point to
the StarlingX build output as the source for packages to be installed
in the images. After setting up the repo configuration, a yum upgrade
is performed to update installed packages with versions from the
StarlingX build, to try to align with the host OS as much as
possible.
* The dev base image does not point to the StarlingX build, as packages
will come primarily from upstream sources. It currently installs
centos-release-openstack-stein in order to add repo configuration to
point to the Stein release, which is currently the latest available
release.
The base image is passed into the StarlingX Docker image build command
as an argument.
Example stable build command:
::
OS=centos
BUILD_STREAM=stable
BRANCH=master
IMAGE_VERSION=${BRANCH}-${BUILD_STREAM}-${USER}
LATEST=${BRANCH}-${BUILD_STREAM}-latest
DOCKER_USER=${USER}
DOCKER_REGISTRY=192.168.0.1:9001 # Some private registry you've setup for your testing, for example
time $MY_REPO/build-tools/build-docker-images/build-stx-base.sh \
--os ${OS} \
--stream ${BUILD_STREAM} \
--version ${IMAGE_VERSION} \
--user ${DOCKER_USER} --registry ${DOCKER_REGISTRY} \
--push \
--latest-tag ${LATEST} \
--repo local-stx-build,http://${HOSTNAME}:8088/${MY_WORKSPACE}/std/rpmbuild/RPMS \
--repo stx-distro,http://${HOSTNAME}:8088/${MY_REPO}/cgcs-${OS}-repo/Binary \
--clean
Example dev build command:
::
OS=centos
BUILD_STREAM=dev
BRANCH=master
IMAGE_VERSION=${BRANCH}-${BUILD_STREAM}-${USER}
LATEST=${BRANCH}-${BUILD_STREAM}-latest
DOCKER_USER=${USER}
DOCKER_REGISTRY=192.168.0.1:9001 # Some private registry you've setup for your testing, for example
time $MY_REPO/build-tools/build-docker-images/build-stx-base.sh \
--os ${OS} \
--stream ${BUILD_STREAM} \
--version ${IMAGE_VERSION} \
--user ${DOCKER_USER} --registry ${DOCKER_REGISTRY} \
--push \
--latest-tag ${LATEST} \
--clean
If you are not making changes to any source packages (ie. RPMs) that
need to be installed in your designer-built images, you can use the
official StarlingX built stx-base image. For example:
https://hub.docker.com/r/starlingx/stx-centos/tags
* Stable base image: starlingx/stx-centos:master-stable-latest
* Dev base image: starlingx/stx-centos:master-dev-latest
---------------
Building wheels
---------------
A wheel is a package format that provides a pre-built python module. We
collect or build a set of python modules in wheel format and store them
in a tarball, which can be passed to loci when building the StarlingX
Docker images. We have two groups of wheels in the StarlingX build:
* Base wheels - wheels that come from upstream source
* StarlingX wheels - wheels produced by the StarlingX build
The build-wheel-tarball.sh tool in stx-root/build-tools/build-wheels is
used to build and collect wheels and generate the wheels tarball. It
uses two sub-tools (located in the same directory) to build and/or
collect the two groups of wheels.
If you are not modifying any python modules, you can use the official
StarlingX built wheels tarball:
* Stable wheels:
https://mirror.starlingx.windriver.com/mirror/starlingx/master/centos/latest_docker_image_build/outputs/wheels/stx-centos-stable-wheels.tar
* Dev wheels:
https://mirror.starlingx.windriver.com/mirror/starlingx/master/centos/latest_docker_image_build/outputs/wheels/stx-centos-dev-wheels.tar
***********
Base wheels
***********
The base wheels are built and/or collected by the build-base-wheels.sh
script, which is called from build-wheel-tarball.sh. It uses a Docker
file in stx-root/build-tools/build-wheels/docker to setup a
wheel-builder container and runs the docker-build-wheel.sh script. This
script uses a wheels.cfg file as input (eg. dev-wheels.cfg), which
provides a list of wheels and build/download directives. The wheels.cfg
file can specify wheel/module sources as:
* pre-built wheel file to be downloaded
* source git repo
* source tarball
* source zip
In addition, when building the “dev” wheels tarball, the
build-base-wheels.sh script will pull the loci/requirements:master-${OS}
image, extracting the wheels from that image to provide the initial set.
This allows us to keep the dev wheels tarball at the latest upstream
versions, with the exception of wheels that we explicitly build.
Example build command:
::
OS=centos
BUILD_STREAM=stable
${MY_REPO}//build-tools/build-wheels/build-wheel-tarball.sh \
--os ${OS} \
--stream ${BUILD_STREAM}
| This will produce a wheels tarball in your workspace:
| ${MY_WORKSPACE}/std/build-wheels-${OS}-${BUILD_STREAM}/stx-${OS}-${BUILD_STREAM}-wheels.tar
****************
StarlingX wheels
****************
The StarlingX build provides support for producing python wheels during
the build. For CentOs, this means updating the package rpm specfile to
build the wheel and package it in a -wheels package. The names of the
wheels packages to be included in the tarball are listed in the
wheels.inc files in the corresponding repo (ie.
centos_stable_wheels.inc).
---------------
Building images
---------------
The StarlingX Docker images are built using a set of image directives
files, with the base image and wheels tarball as input. The images are
built by the build-stx-images.sh tool, in
stx-root/build-tools/build-docker-images. The build-stx-images.sh tool
will search the StarlingX repos for a corresponding docker_images.inc
file (ie. centos_dev_docker_images.inc) which contains a list of
subdirectories that contain the associated image directives files, which
are processed and built.
The following diff provides an example of changes made to a specfile to
add building a wheel to a package:
::
diff --git a/openstack/distributedcloud-client/centos/distributedcloud-client.spec b/openstack/distributedcloud-client/centos/distributedcloud-client.spec
index c6e17f6..7dc83f5 100644
--- a/openstack/distributedcloud-client/centos/distributedcloud-client.spec
+++ b/openstack/distributedcloud-client/centos/distributedcloud-client.spec
@@ -20,6 +20,8 @@ BuildArch: noarch
BuildRequires: python2-devel
BuildRequires: python-setuptools
+BuildRequires: python2-pip
+BuildRequires: python2-wheel
BuildRequires: python-jsonschema >= 2.0.0
BuildRequires: python-keystonemiddleware
BuildRequires: python-oslo-concurrency
@@ -75,10 +77,13 @@ rm -rf {test-,}requirements.txt tools/{pip,test}-requires
%build
export PBR_VERSION=%{version}
%{__python2} setup.py build
+%py2_build_wheel
%install
export PBR_VERSION=%{version}
%{__python2} setup.py install --skip-build --root %{buildroot}
+mkdir -p $RPM_BUILD_ROOT/wheels
+install -m 644 dist/*.whl $RPM_BUILD_ROOT/wheels/
# prep SDK package
mkdir -p %{buildroot}/usr/share/remote-clients
@@ -94,3 +99,11 @@ tar zcf %{buildroot}/usr/share/remote-clients/%{pypi_name}-%{version}.tgz --excl
%files sdk
/usr/share/remote-clients/%{pypi_name}-%{version}.tgz
+%package wheels
+Summary: %{name} wheels
+
+%description wheels
+Contains python wheels for %{name}
+
+%files wheels
+/wheels/*
The get-stx-wheels.sh script, called by build-wheel-tarball.sh, will
gather the set of -wheels packages, defined by the corresponding
wheels.inc files, and extract the wheel files, making them available to
the build-wheel-tarball.sh tool.
**************
Wheels tarball
**************
The build-wheel-tarball.sh tool, after successfully calling
build-base-wheels.sh and get-stx-wheels.sh, will collect the wheels
built or downloaded and prep the tarball. It will also download the
OpenStack requirements.txt and upper-constraints.txt files, which are
used by loci when installing the python modules. The
upper-constraints.txt file is modified based on the collected/built
wheels, allowing us to override or append module specifications. The
upper-constraints.txt file in the StarlingX wheels tarball then reflects
the content of the tarball, to ensure the desired module versions are
installed.
---------------
Building images
---------------
The StarlingX Docker images are built using the build-stx-images.sh
tool, in stx-root/build-tools/build-docker-images, using the image
directives files for build instructions, with the base image and wheels
as input.
**********************
Image directives files
**********************
The image directives files provide the build arguments necessary for
building a specific image. The first required option is BUILDER, which
can be either “docker” or “loci”.
^^^^^^^^
"docker"
^^^^^^^^
Images with BUILDER set to “docker” are built using a Docker file. The
only other required option in the image directives file for “docker”
builds is the LABEL, or image name (ie. stx-libvirt). The Docker file
can use the StarlingX base image as its “FROM” by including the
following at the top:
::
ARG BASE
FROM ${BASE}
The BASE is passed by build-stx-images.sh as a build argument.
Options supported by BUILDER=docker image directives files include:
* LABEL: the image name
* PROJECT: main project name
* DOCKER_REPO: main project source git repo
* DOCKER_REF: git branch or tag for main project source repo (default "master")
* DOCKER_PATCHES: list of patch files to apply to DOCKER_REPO, relative to the local dir
* DOCKER_CONTEXT: path to build context source, relative to the local dir (default "docker")
* DOCKER_FILE: path to Dockerfile, relative to the local dir (default "docker/Dockerfile")
.. note::
DOCKER_CONTEXT and DOCKER_FILE are mutually exclusive to DOCKER_REPO, DOCKER_REF and DOCKER_PATCHES.
For an example of a BUILDER=docker image, see
https://opendev.org/starlingx/oidc-auth-armada-app/src/branch/master/dex/centos/dex.stable_docker_image
^^^^^^
"loci"
^^^^^^
The loci project (https://github.com/openstack/loci) provides a
mechanism for building images using a python module as the main project
source. The image directives file for BUILDER=loci images allows you to
specify supporting python modules or packages to be installed, in
addition to specifying the main project source repo and/or branch. In
addition, the build-stx-images.sh supports specifying an additional
customization command that is applied to the loci-built image.
Options supported by BUILDER=loci image directives files that are passed on to
loci include:
* LABEL: the image name
* PROJECT: main project name
* PROJECT_REPO: main project source git repo
* PROJECT_REF: git branch or tag for main project source repo
* PIP_PACKAGES: list of python modules to be installed, beyond those
specified by project dependencies or requirements
* DIST_PACKAGES: additional packages to be installed (eg. RPMs from
repo, configured by base image)
* PROFILES: bindep profiles supported by project to be installed (eg.
apache)
In addition, you can specify a bash command in the CUSTOMIZATION option,
in order to do a modification on the loci-built image.
Example:
stx-upstream/openstack/python-nova/centos/stx-nova.dev_docker_image
::
BUILDER=loci
LABEL=stx-nova
PROJECT=nova
PROJECT_REPO=https://github.com/openstack/nova.git
PIP_PACKAGES="pycrypto httplib2 pylint"
DIST_PACKAGES="openssh-clients openssh-server libvirt e2fsprogs"
PROFILES="fluent nova ceph linuxbridge openvswitch configdrive qemu apache"
CUSTOMIZATION="yum install -y openssh-clients"
In a case where the image is built without a main project source git
repo, where the main project source is just coming from a wheel, you can
set PROJECT to infra, and loci skips the git clone steps. For example,
stx-nova-api-proxy:
stx-nfv/nova-api-proxy/centos/stx-nova-api-proxy.dev_docker_image
::
BUILDER=loci
LABEL=stx-nova-api-proxy
# Set PROJECT=infra and PROJECT_REPO=nil because we are not cloning a repo
PROJECT=infra
PROJECT_REPO=nil
PIP_PACKAGES="api_proxy eventlet oslo.config oslo.log \
paste PasteDeploy routes webob keystonemiddleware pylint"
^^^^^^^^^^^^^^^^^^^
Image build command
^^^^^^^^^^^^^^^^^^^
Example image build command, using the official StarlingX base image and wheels:
::
OS=centos
BUILD_STREAM=stable
BRANCH=master
CENTOS_BASE=starlingx/stx-centos:${BRANCH}-${BUILD_STREAM}-latest
WHEELS=https://mirror.starlingx.windriver.com/mirror/starlingx/master/centos/latest_docker_image_build/outputs/wheels//stx-centos-${BUILD_STREAM}-wheels.tar
DOCKER_USER=${USER}
DOCKER_REGISTRY=192.168.0.1:9001 # Some private registry you've setup for your testing, for example
time $MY_REPO/build-tools/build-docker-images/build-stx-images.sh \
--os centos \
--stream ${BUILD_STREAM} \
--base ${CENTOS_BASE} \
--wheels ${WHEELS} \
--user ${DOCKER_USER} --registry ${DOCKER_REGISTRY} \
--push --latest \
--clean
If I want to build using the wheels tarball from my build, instead:
::
WHEELS=http://${HOSTNAME}:8088/${MY_WORKSPACE}/std/build-wheels-${OS}-${BUILD_STREAM}/stx-${OS}-${BUILD_STREAM}-wheels.tar
Note: To specify a local wheels tarball, loci needs to be able to access
it via wget from a Docker container. This could mean changes to your
http server and iptables rules to allow “external” access, to allow
access from the Docker containers.
::
## Note: Verify that lighttpd is not bound to "localhost"
vi /etc/lighttpd/lighttpd.conf
# server.bind = "localhost"
systemctl restart lighttpd
## Note: You may need to add an iptables rule to allow the Docker
## containers to access the http server on your host. For example:
iptables -I INPUT 6 -i docker0 -p tcp --dport ${HOST_PORT} -m state --state NEW,ESTABLISHED -j ACCEPT
If you only want to build specify images, the build-stx-images.sh
provides --only and --skip options (ie. --only stx-nova).
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Testing image on running system
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Now that you've built an image, you may be wondering how to test it on
an already running system. First, locate the specific image of interest
you just built
::
## Note: You may need to be root to run Docker commands on your build system. If so, "sudo -s"
docker images
# Locate the image of interest you just built in the output, should be at or near the top of the list, then
# save the image of interest as a compressed tarball. It could be rather large.
docker save <image id> | gzip -9 >container_filename.tgz
# scp container_filename.tgz to the active controller, then
# ssh to active controller, then run the following instructions there:
# become root:
sudo -s
# load the transferred container compressed tarball into the image list
zcat container_filename.tgz | docker load
# find the newly loaded container image in the list and make note of it
docker images
Now that you have the container image loaded, proceed to use it for
test/debug as detailed in the `use container for
debugging <https://wiki.openstack.org/wiki/StarlingX/Containers/FAQ#How_do_I_make_changes_to_the_code_or_configuration_in_a_pod_for_debugging_purposes.3F>`_
FAQ. The next step is to tag the container before pushing it, so scroll
down to the point in the FAQ instructions where you are directed to tag
the container and continue from there.
-------------------------
Incremental image updates
-------------------------
The update-stx-image.sh utility (introduced by
https://review.opendev.org/661641) provides designers with a mechanism
to add minor updates to an existing image without rebuilding the entire
image. This results in a small customization layer appended to the
existing image, which reduces storage requirements over a full rebuild
of the entire image. The tool allows for updates via:
* installing or updating Python modules, via:
* wheel file
* Python module source directory
* Python module source git repository
* software packages (currently only supporting RPM packages on CentOS)
* customization script: a bash script that the updater will run that
can make necessary modifications in the image that can't be handled
by updating software
*******************************
Specifying Python module source
*******************************
The --module-src command-line option (or MODULE_SRC in an update
directives file) allows a designer to specify python module source from
either a directory or git repository. If specifying a git repository,
you can also specify a branch or tag to be fetched, as well as
optionally hardcode a version number. For example:
`` --module-src ``\ “```https://github.com/starlingx-staging/stx-nova|stx/stein.1`` <https://github.com/starlingx-staging/stx-nova%7Cstx/stein.1>`__”
This will clone the stx-nova repo and fetch/checkout the stx/stein.1
branch, installing or updating the python module in the image.
********************
Customization script
********************
You can optionally provide a customization script to make changes to the
image that cannot be handled by updating software, using the --customize
command-line option (or CUSTOMIZATION_SCRIPT in an update directives
file). You can also provide supporting files with the --extra
command-line option (or EXTRA_FILES in an update directives file),
which will be accessible to the customization script in the
/image-update/extras directory within the update container.
**********************
Update directives file
**********************
You can optionally specify an updates directives file with the --file
command-line option to provide the various update directives. Options
that can be set from the update directives file include:
::
FROM= # Specify the base image (equivalent to --from)
IMAGE_UPDATE_VER= # Specify image update version (equivalent to --version)
CUSTOMIZATION_SCRIPT= # Specify customization script (equivalent to --customize)
WHEELS= # Specify one or more wheel files (equivalent to --wheel)
DIST_PACKAGES= # Specify one or more software packages (equivalent to --pkg)
MODULE_SRC= # Specify one or more python module source locations (equivalent to --module-src)
EXTRA_FILES= # Specify one or more extra files to be accessible to customization script (equivalent to --extra)
********
Examples
********
::
## Upstream image:
STX_MIRROR_DOCKER_URL="https://mirror.starlingx.windriver.com/mirror/starlingx/master/centos/latest_docker_image_build/outputs/docker-images"
DOCKER_USER=mydockerid # My docker ID
DOCKER_REGISTRY=docker.io # The docker registry to use for pushing. This can be a private registry, like 192.168.2.10:9001
# For this example, I've setup a directory with files I'm using for updates,
# under ${MY_WORKSPACE}/updates:
#
# updates/pip-packages/modules/aodh - a git clone of upstream aodh, where I've made changes
# updates/pip-packages/wheels - a directory with multiple wheel files
# updates/dist-packages - a directory with multiple RPM files
#
# Additionally, there's a basic updates/customize.sh bash script that creates
# a couple of files in /etc:
#
# #!/bin/bash
#
# # Sample
# echo example-update > /etc/example-update.txt
# find /image-update > /etc/example-update-dirlst.txt
#
# Get the latest versioned stx-aodh image tag
STX_AODH=$(curl ${STX_MIRROR_DOCKER_URL}/images-centos-stable-versioned.lst 2>/dev/null | grep stx-aodh:)
echo ${STX_AODH}
BASE_VERSION=$(echo ${STX_AODH} | sed 's/.*://')
# For the purposes of the first few examples, each update builds on the previous one.
# So the --from argument points to the image built in the previous step (or .0 in the first case)
# Build the first update using the customization script and a couple of random files
time bash -x ${MY_REPO}/build-tools/build-docker-images/update-stx-image.sh \
--user ${DOCKER_USER} --registry ${DOCKER_REGISTRY} \
--from ${STX_AODH} \
--customize ${MY_WORKSPACE}/updates/customize.sh \
--extra ${MY_REPO}/build-tools/build-docker-images/build-stx-base.sh \
--extra ${MY_REPO}/build-tools/build-wheels/build-base-wheels.sh \
--push \
--update-id example.1
# To see the content of one of the files created by the customization script:
# $ docker run --rm \
# ${DOCKER_REGISTRY}/${DOCKER_USER}/stx-aodh:${BASE_VERSION%.0}.1 \
# cat /etc/example-update-dirlst.txt
# /image-update
# /image-update/internal-update-stx-image.sh
# /image-update/customize.sh
# /image-update/extras
# /image-update/extras/build-base-wheels.sh
# /image-update/extras/build-stx-base.sh
# To see the size of the customization layer we just added to the image
# $ docker history ${DOCKER_REGISTRY}/${DOCKER_USER}/stx-aodh:${BASE_VERSION%.0}.1 | head -2
# IMAGE CREATED CREATED BY SIZE COMMENT
# 8735dde77f9c 3 minutes ago bash -x -c bash -x /image-update/internal-u… 201B
# Install/update wheels from the directory we've setup:
time bash -x ${MY_REPO}/build-tools/build-docker-images/update-stx-image.sh \
--user ${DOCKER_USER} --registry ${DOCKER_REGISTRY} \
--from ${DOCKER_REGISTRY}/${DOCKER_USER}/stx-aodh:${BASE_VERSION%.0}.1 \
--wheel "${MY_WORKSPACE}/updates/pip-packages/wheels/*.whl" \
--push \
--update-id example.2
# We can now do a diff of the piplst files from the updates and verify the change (which can also be seen in output of the tool):
# $ diff std/update-images/example.1/stx-aodh-${BASE_VERSION%.0}.1.piplst std/update-images/example.2/stx-aodh-${BASE_VERSION%.0}.2.piplst
# 14a15
# > cgcs-patch==1.0
# 130a132
# > tsconfig==1.0.0
# Update the aodh module from the dir we cloned and modified:
time bash -x ${MY_REPO}/build-tools/build-docker-images/update-stx-image.sh \
--user ${DOCKER_USER} --registry ${DOCKER_REGISTRY} \
--from ${DOCKER_REGISTRY}/${DOCKER_USER}/stx-aodh:${BASE_VERSION%.0}.2 \
--module-src ${MY_WORKSPACE}/updates/pip-packages/modules/aodh \
--push \
--update-id example.3
# We can now do a diff of the piplst files from the updates and verify the change (which can also be seen in output of the tool):
# $ diff std/update-images/example.2/stx-aodh-${BASE_VERSION%.0}.2.piplst std/update-images/example.3/stx-aodh-${BASE_VERSION%.0}.3.piplst
# 3c3
# < aodh==8.0.1.dev4
# ---
# > aodh==8.1.0.dev28
# Install/update RPMs from the dir we setup:
time bash -x ${MY_REPO}/build-tools/build-docker-images/update-stx-image.sh \
--user ${DOCKER_USER} --registry ${DOCKER_REGISTRY} \
--from ${DOCKER_REGISTRY}/${DOCKER_USER}/stx-aodh:${BASE_VERSION%.0}.3 \
--pkg "${MY_WORKSPACE}/updates/dist-packages/*.rpm" \
--push \
--update-id example.4
# We can now do a diff of the piplst files from the updates and verify the change (which can also be seen in output of the tool):
# $ diff std/update-images/example.3/stx-aodh-${BASE_VERSION%.0}.3.rpmlst std/update-images/example.4/stx-aodh-${BASE_VERSION%.0}.4.rpmlst
# 156a157
# > perl-Data-Dumper-2.145-3.el7.x86_64
# All of the above can also be done in a single command:
time bash -x ${MY_REPO}/build-tools/build-docker-images/update-stx-image.sh \
--user ${DOCKER_USER} --registry ${DOCKER_REGISTRY} \
--from ${STX_AODH} \
--wheel "${MY_WORKSPACE}/updates/pip-packages/wheels/*.whl" \
--pkg "${MY_WORKSPACE}/updates/dist-packages/*.rpm" \
--module-src ${MY_WORKSPACE}/updates/pip-packages/modules/aodh \
--customize ${MY_WORKSPACE}/updates/customize.sh \
--extra ${MY_REPO}/build-tools/build-docker-images/build-stx-base.sh \
--extra ${MY_REPO}/build-tools/build-wheels/build-base-wheels.sh
# Update stx-nova with the latest update from the stx-nova staging repo
STX_NOVA=$(curl ${STX_MIRROR_DOCKER_URL}/images-centos-stable-versioned.lst 2>/dev/null | grep stx-nova:)
echo ${STX_NOVA}
time bash -x ${MY_REPO}/build-tools/build-docker-images/update-stx-image.sh \
--user ${DOCKER_USER} --registry ${DOCKER_REGISTRY} \
--from ${STX_NOVA} \
--module-src "https://github.com/starlingx-staging/stx-nova.git|stx/stein.1" \
--update-id example.nova
View Git Blame Copy Permalink