Containers Service Spec

Containers share many features in common with Nova instances. For the common
features, virt drivers for Nova can be used to surface basic instance
functionality. For features that go beyond what can be naturally fit within
a virt driver, we propose a new API service that allows for advanced features
to be added without conflating the worlds of instances and containers.

Re-post of the spec originally posted at:

Change-Id: Ic5d95f5d6ba0d6845023bf59f9bad4377d50578f
Adrian Otto 2014-11-20 19:28:10 +00:00
parent 94f3ede485
commit f2c4c3eafc
1 changed files with 370 additions and 0 deletions

View File

@ -0,0 +1,370 @@
This work is licensed under a Creative Commons Attribution 3.0 Unported
Containers Service
Launchpad blueprint:
Containers share many features in common with Nova instances. For the common
features, virt drivers for Nova can be used to surface basic instance
functionality. For features that go beyond what can be naturally fit within
a virt driver, we propose a new API service that allows for advanced features
to be added without conflating the worlds of instances and containers.
Some examples of containers specific features are setting of shell environment
variables, and accepting a shell command to execute at runtime. Capturing the
STDIO of the process(es) within a container, and tracking the return status
of processes are all beyond the scope of what was contemplated for Nova. All
of these features will be implemented in the Containers Service.
Problem description
Container technology is rapidly gaining popularity as a way to bundle and
deploy applications. Recognizing and adapting to this trend will position
OpenStack to be useful not only to clouds that employ bare metal and virtual
machine instances, but can remain competitive in offering container services
as well.
Nova's concepts of an instance, and the actions that may be taken on it do not
match completely with containers.
Use cases
1. App Consolidation. End-user wants to run multiple small applications in
separate operating system environments, but wants to optimize for efficiency
to control hosting costs. Each application belongs to the same tenant, so
security isolation between applications is nice-to-have but not critical.
Isolation is desired primarily for simplified management of the execution
environment for each application.
2. App Portability. End-user wants to create a single container image, and
deploy the same image to multiple hosting environments, including OpenStack.
Other environments may include local servers, dedicated servers, private
clouds, and public clouds. Switching environments requires passing database
connection strings by environment variables at the time a container starts
to allow the application to use the services available in each environment
without changing the container image.
3. Docker Compatibility. End-user has a Dockerfile used to build an application
and its runtime environment and dependencies in a Docker container image.
They want an easy way to run the Docker resulting image on an OpenStack
4. LXC Compatibility. End-user wants an easy way to remotely create multiple
LXC containers within a single Nova instance.
5. OpenVZ Compatibility. End-user wants an easy way to remotely create multiple
OpenVZ containers within a single Nova instance.
6. Containers-Centric World View. End-user wants to communicate with a single
OpenStack API, and request the addition of containers, without the need to
be concerned with keeping track of how many containers are already running
on a given Nova instance, and when more need to be created. They want to
simply create and remove containers, and allow the appropriate resource
scheduling to happen automatically.
7. Platform Integration. Cloud operator already has an OpenStack cloud, and
wants to add a service/application centric management system on top.
Examples of such systems are Cloud Foundry, Kubernetes, Apache Mesos, etc.
The selected system is already Docker compatible. Allow this cloud operator
easy integration with OpenStack to run applications in containers. The
Cloud Operator now harnesses the power of both the management system, and
OpenStack, and does not need to manage a second infrastructure for his/her
application hosting needs. All details involving the integration of
containers with Nova instances is managed by OpenStack.
Proposed change
Add a new API service for CRUD and advanced management of containers.
If cloud operators only want to offer basic instance features for their
containers, they may use nova with an alternate virt-driver, such as
libvirt/lxc or nova-docker. For those wanting a full-featured container
experience, they may offer the Containers Service API as well, in combination
with Nova instances that contain an OpenStack agent that connects to the
containers service through a security controlled agent (daemon) that allows
the OpenStack control plane to provision and control containers running on
Compute Hosts.
The Containers Service will call the Nova API to create one or more Nova
instances inside which containers will be created. The Nova instances may
be of any type, depending on the virt driver(s) chosen by the cloud operator.
This includes bare-metal, virtual machines, containers, and potentially other
instance types.
This allows the following configurations of containers in OpenStack.
* Containers in Virtual Machine Instances
* Containers in Bare Metal Instances
* Containers in Container Instances (nested)
The concept of nesting containers is currently possible if the parent container
runs in privileged mode. Patches to the linux kernel are being developed to
allow nesting of non-privileged containers as well, which provides a higher
level of security.
The spirit of this plan aims to duplicate as little as possible between Nova
and the Containers Service. Common components like the scheduler are expected
to be abstracted into modules, such as Gantt that can be shared by multiple
projects. Until Gantt is ready for use by the Containers Service, we will
implement only two provisioning schemes for containers:
1. Create a container on a specified instance by using a nova instance guid.
2. Auto-create instances (applies only until the Gantt scheduler is used)
2.1. Fill them sequentially until full.
2.2. Remove them automatically when they become empty.
The above orchestration will be implemented using Heat. This requires some
kind of hypervisor painting (such as host aggregates) for security reasons.
The diagram below offers an overview of the system architecture. The OSC box
indicates an OpenStack client, which will communicate with the Containers
Service through a REST API. The containers service may silently create Nova
instances if one with enough capacity to host the requested container is not
already known to the Containers service. The containers service will maintain
a database "Map" of containers, and what Nova instance each belongs to. Nova
creates instances. Instances are created in Nova, and contaienrs belong only
to the Containers Service, and run witin a Nova instance. If the instance
includes the agent software "A", then it may be included in the inventory of
the Containers service. Instances that do not contain an agent may not interact
with the Containers Service, and can be controlled only by a Nova virt driver.
| |
| A | = Agent
| |
| |
| C | = Container
| |
Design Principles
1. Leverage existing OpenStack projects for what they are good at. Do not
duplicate functionality, or copy code that can be otherwise accessed through
API calls.
2. Keep modifications to Nova to a minimum.
3. Make the user experience for end users simple and familiar.
4. Allow for implementation of all features containers are intended to offer.
1. Extending Nova's existing featureset to offer container features
1.1. Container features don't fit into Nova's idea of compute (VM/Server)
2. A completely separate containers service forked from Nova.
2.1. Would result in large overlap and duplication in features and code
Data model impact
For Nova, None. All new data planned will be in the Containers Service.
REST API impact
For Nova, none. All new API calls will be implemented in the Containers
Service. The OpenStack Containers Service API will be a superset of
functionality offered by the The `Docker API:
with additionls to make is suitable for general use regardless of the backend
container technology used, and to be compatible with OpenStack multi-tenancy
and Keystone authentication.
Specific Additions:
1. Support for the X-Auth-Project-Id HTTP request header to allow for
multi-tenant use.
2. Support for the X-Auth-Token HTTP request header to allow for authentication
with keystone.
If either of the above headers are missing, a 401 Unathorized response will
be generated.
Docker CLI clients may communicate with a Swarmd instance that is configured
to use the OpenStack Containers API as the backend for libswarm. This will
allow for tool compatability with the Docker ecosystem using the officially
supported means for integration of a distributed system.
The scope of the full API will cause this spec to be too long to review, so
the intent is to deal with the specific API design as a series of Gerrit
reviews that submit API code as Not Implemented stubs with docstrings that
clearly document the design, so allow for approval, and further implementation.
Security impact
Because Nova will not be changed, there should be no security impacts to Nova.
The Containers Service implementation, will have the following security related
* Need to authenticate against keystone using python-keystoneclient.
* A trust token from Nova will be needed in order for the Containers Service
to call the Nova API on behalf of a user.
* Limits must be implemented to control resource consumption in accordance with
* Providing STDIO access may generate a considerable amount of network chatter
between containers and clients through the relay. This could lead to
bandwidth congestion at the relays, or API nodes. An approach similar to
how we handle serial console access today will need to be considered to
mitigate this concern.
Using containers implies a range of security considerations for cloud
operators. These include:
* Containers in the same instance share an operating system. If the kernel is
exploited using a security vulnerability, processes in once container may
escape the constraints of the container and potentially access other
resources on the host, including contents of other containers.
* Output of processes may be persisted by the containers service in order to
allow asynchronous collection of exit status, and terminal output. Such
content may include sensitive information. Features may be added to mitgate
the risk of this data being replicated in log messages, including errors.
* Creating containers usually requires root access. This means that the Agent
may need to be run with special privileges, or be given a method to
escalate privileges using techniques such as sudo.
* User provided data is passed through the API. This will require sensuble
data input validation.
Notifications impact
Contemplated features (in subsequent release cycles):
* Notify the end user each time a Nova instance is created or deleted by
the Containers service, if (s)he has registered for such notifications.
* Notify the user each on CRUD of containers containing start and end
notifications. (compute.container.create/delete/etc)
* Notify user periodically of existence of container service managed
containers (ex compute.container.exists)
Other end user impact
The user interface will be a REST API. On top of that API will be an
implementation of the libswarm API to allow for tools designed to use Docker
to treat OpenStack as an upstream system.
Performance Impact
The Nova API will be used to create instances as needed. If the Container to
Instance ratio is 10, then the Nova API will be called at least once for every
10 calls to the Containers Service. Instances that are left empty will be
automatically deleted, so in the example of a 10:1 ratio, the Nova API will be
called to perform a delete for every 10 deletes in the Container Service.
Depending on the configuration, the ratio may be as low as 1:1.
The Containers Service will only access Nova through its API, not by accessing
its database.
Other deployer impact
Deployers may want to adjust the default flavor used for Nova Instances created
by the Containers Service.
There should be no impact on users of prior releases, as this introduces a new
Developer impact
Minimal. There will be minimal changes required in Nova, if any.
Primary assignee:
Other contributors:
Work Items
1. Agent
2. Relay
3. API Service
4. IO Relays
1. <Links to Agent Blueprint and Spec here, once ready>
2. Early implementations may use libswarm, or a python port of libswarm to
implement Docker API conpatibility.
Each commit will be accompanied with unit tests, and Tempest functional tests.
Documentation Impact
A set of documentation for this new service will be required.
* Link to high level draft proposal from the Nova Midcyle Meetup for Juno:
`PDF <>`_
* `Libswarm Source <>`_