Browse Source

Add spec for simple container generation

This spec has been drafted to highlight an effort to improve the container
generation process and life cycle management of our applications.

Change-Id: I369c9a9f450c220d8cdc2827460de56ff1e4fb57
Co-authored-by: Emilien Macchi <>
Signed-off-by: Kevin Carter <>
Kevin Carter 1 year ago
No known key found for this signature in database GPG Key ID: CE94BD890A47B20A
  1. 427


@ -0,0 +1,427 @@
This work is licensed under a Creative Commons Attribution 3.0 Unported
Simple Container Generation
Simple container generation is an initiative to reduce complexity in the
TripleO container build, deployment, and distribution process by reducing the
size and scope of the TripleO container build tools.
The primary objective of this initiative is to replace Kolla, and our
associated Kolla customization tools, as the selected container generation
tool-kit. The TripleO community has long desired an easier solution for
deployers and integrators alike and this initiative is making that desire a
The Simple container generation initiative is wanting to pivot from a
tool-chain mired between a foundational component of Kolla-Ansible and a
general purpose container build system, to a vertically integrated solution
that is only constructing what TripleO needs, in a minimally invasive, and
simple to understand way.
Problem Description
TripleO currently leverages Kolla to produce container images. These images are
built for Kolla-Ansible using an opinionated build process which has general
purpose features. While our current images work, they're large and not well
suited for the TripleO use-case, especially in distributed data-centers. The
issue of container complexity and size impacts three major groups, deployers,
third party integrators, and maintainers. As the project is aiming to simplify
interactions across the stack, the container life cycle and build process has
been identified as something that needs to evolve. The TripleO project needs
something vertically integrated which produces smaller images, that are easier
to maintain, with far fewer gyrations required to tailor images to our needs.
Proposed Change
Implement a container file generation role, and a set of statically defined
override variable files which are used to generate our required
container files. [#f2]_
.. code-block:: text
| |
| +-->openstack-${SERVICE}-1-b
| |
| +-->openstack-${SERVICE}-1-c
User Experience
Building the standard set of images will be done through a simple command line
interface using the TripleO python client.
.. code-block:: shell
$ openstack tripleo container image build [opts] <args>
This simple sub-command will provide users the ability to construct images as
needed, generate container files, and debug runtime issues.
CLI Options
The python TripleO client options for the new container image build entry point.
=========== =============================== =================================================================
Option Default Description
=========== =============================== =================================================================
config-file $PATH/overcloud_containers.yaml Configuration file setting the list of containers to build.
exclude [] Container type exclude. Can be specified multiple times.
work-dir /tmp/container-builds Container builds directory, storing the container files and
logs for each image and its dependencies.
skip-push False Skip pushing images to the registry
skip-build False Only generates container files without producing a local build.
base centos Base image name.
type binary Image type.
tag latest Image tag.
registry localhost Container registry URL.
namespace tripleomaster Container registry namespace.
volume [] Container bind mount used when building the image. Should be
specified multiple times if multiple volumes.
=========== =============================== =================================================================
Container Image Build Tools
Container images will be built using Buildah_, The required Buildah
functionality will leverage `BuildahBuilder` via `python-tripleoclient`
integration and be exposed though CLI options.
.. _Buildah:
Image layout
Each image will have its own YAML file which has access to the following
parameters. Each YAML file will have one required parameter (tcib_from for the
source image to build from) and optional parameters.
================= ============================= ==================== ======== ===================================================
Option Default Type Required Description
================= ============================= ==================== ======== ===================================================
tcib_path `{{ lookup('env', 'HOME') }}` String Path to generated the container file(s) for a given
tcib_args Dict[str, str] Single level `key:value` pairs. Implements arg_.
tcib_from `centos:8` Str True Container image to deploy from. Implements from_.
tcib_labels Dict[str, str] Single level `key:value` pairs. Implements label_.
tcib_envs Dict[str, str] Single level `key:value` pairs. Implements env_.
tcib_onbuilds List[str] <item>=String. Implements onbuild_.
tcib_volumes List[str] <item>=String. Implements volume_.
tcib_workdir Str Implements workdir_.
tcib_adds List[str] <item>=String. Implements add_.
tcib_copies List[str] <item>=String. Implements copy_.
tcib_exposes List[str] <item>=String. Implements expose_.
tcib_user Str Implements user_.
tcib_shell Str Implements shell_.
tcib_runs List[str] <item>=String. Implements run_.
tcib_healthcheck Str Implements healthcheck_.
tcib_stopsignal Str Implements stopsignal_.
tcib_entrypoint Str Implements entrypoint_.
tcib_cmd Str Implements cmd_.
tcib_actions List[Dict[str, str]] Each item is a Single level Dictionary `key:value`
pairs. Allows for arbitrary verbs which maintains
tcib_gather_files List[str] Each item is a String. Collects files from the
host and stores them in the build directory.
================= ============================= ==================== ======== ===================================================
.. _arg:
.. _from:
.. _label:
.. _env:
.. _onbuild:
.. _volume:
.. _workdir:
.. _add:
.. _copy:
.. _expose:
.. _user:
.. _shell:
.. _run:
.. _healthcheck:
.. _stopsignal:
.. _entrypoint:
.. _cmd:
Application packages are sorted within each container configuration file.
This provides a programmatic interface to derive package sets, allows
overrides, and is easily visualized. While the package option is not
processes by the `tripleo_container_image_build` role, it will serve as a
standard within our templates.
================ ====================================================
Option Description
================ ====================================================
tcib_packages Dictionary of packages to install.
.. code-block:: yaml
- openstack-${SERVICE}-common
- openstack-${SERVICE}-proprietary
- $dep-x86_64
- $dep-power
- openstack-${SERVICE}
- $dep
================ ====================================================
This option is then captured and processed by a simple `RUN` action.
.. code-block:: yaml
- run: "dnf install -y {{ tcib_packages['common'] }} {{ tcib_packages[ansible_distribution][ansible_architecture] }}"
Example Container Variable File
.. code-block:: yaml
tcib_from: ubi8
tcib_path: "{{ lookup('env', 'HOME') }}/example-image"
maintainer: MaintainerX
tcib_entrypoint: dumb-init --single-child --
tcib_stopsignal: SIGTERM
- mkdir -p /etc/ssh && touch /etc/ssh/ssh_known_host
- /etc/hosts /opt/hosts
- /etc
- curl
- wget
- run: "dnf install -y {{ tcib_packages['common'] }} {{ tcib_packages[ansible_distribution][ansible_architecture] }}"
- copy: /etc/resolv.conf /resolv.conf
- run: ["/bin/bash", "-c", "echo hello world"]
Container File Structure
The generated container file(s) will follow a simple directory structure
which provide an easy way to view, and understand, build relationships and
dependencies throughout the stack.
.. code-block:: shell
* Use Ansible Bender
Ansible Bender was evaluated as a tool which could help to build the container
images. However it has not been productized downstream; which would make it
difficult to consume. It doesn't generate Dockerfiles and there is a strong
dependency on Bender tool; the container image build process would therefore be
more difficult to do in a standalone environment where Bender isn't available.
* Leave the container image build process untouched.
We could leave the container image generate process untouched. This keeps us a
consumer of Kolla and requires we maintain our complex ancillary tooling to
ensure Kolla containers work for TripleO.
Security Impact
While security is not a primary virtue in the simple container generation
initiative, security will be improved by moving to simplified containers. If
the simple container generation initiative is ratified, all containers used
within TripleO will be vertically integrated into the stack, making it possible
to easily audit the build tools and all applications, services, and files
installed into our containerized runtimes. With simplification we'll improve
the ease of understanding and transparency which makes our project more
sustainable, thereby more secure. The proposed solution must provide layers
where we know what command has been run exactly; so we can quickly figure out
how an image was built.
Upgrade Impact
There is no upgrade impact because the new container images will provide
feature parity with the previous ones; they will have the same or similar
injected scripts that are used when the containers start.
Other End User Impact
Performance Impact
We should expect better performance out of our containers, as they will be
smaller. While the runtime will act the same, the software delivery will be
faster as the size of each container will smaller, with better constructed
layers. Smaller containers will decrease the mean time to ready which will have
a positive performance impact and generally improve the user experience.
Other Deployer Impact
The simplified container generation initiative will massively help third party
integrators. With simplified container build tools we will be able to easily
articulate requirements to folks looking to build on-top of TripleO. Our
tool-chain will be capable of bootstrapping applications where required, and
simple enough to integrate with a wide variety of custom applications
constructed in bespoke formats.
Developer Impact
In the first phase, there won't be any developer impact because the produced
images will be providing the same base layers as before. For example, they will
contain all the Kolla scripts that are required to merge configuration files or
initialize the container at startup.
These scripts will be injected in the container images for backward
* kolla_extend_start
In a second phase, we will simplify these scripts to remove what isn't needed
by TripleO. The interface in the composable services will likely evolve over
time. For example kolla_config will become container_config. There is no plan
at this time to rewrite the configuration file merge logic.
Primary assignee:
* Cloudnull
* EmilienM
Work Items
First phase
* Ansible role to generate container file(s) -
* Container images layouts -
* Deprecate "openstack overcloud container image build"
* Implement "openstack tripleo container image build" which will reuse the
`BuildahBuilder` and the same logic as the deprecated command but without Kolla.
* Build new images and publish them.
* Switch the upstream CI to use the new images.
Second phase:
* Simplifying the injected scripts to only do what we need in TripleO.
* Rename the configuration interfaces in TripleO Heat Templates.
The tooling will be in existing repositories so there is no new dependency. It
will mainly be in tripleo-ansible, tripleo-common, python-tripleoclient and
tripleo-heat-templates. Like before, Buildah will be required to build the
* The tripleo-build-containers-centos-8 job will be switched to be using
the new "openstack tripleo container image build" command.
* A molecule job will exercise the container image build process using
the new role.
* Some end-to-end job will also be investigated to build and deploy
a container into a running deployment.
Documentation Impact
Much of the documentation impact will be focused on cleanup of the existing
documentation which references Kolla, and the creation of documentation that
highlights the use of the vertically integrated stack.
Since the changes should be transparent for the end-users who just pull images
without rebuilding it, the manuals will still be updated with the new command
and options if anyone wants to build the images themselves.
.. [#f1]
.. [#f2]
.. [#f3]