magnum/specs/containers-service.rst
Florian Haas 11e7f330fb Document use of kube_tag label
Explain how to select a specific Kubernetes version by specifying the
kube_tag label.

In the process, also fix the broken list of images that must be
mirrored, immediately above kube_tag.

In addition, fix an unrelated whitespace error in
specs/containers-service.rst which would cause tox -e docs to fail.

Change-Id: Ieff1474b74e0b1595c05d945b69bec16bfef9c3b
2018-02-13 13:40:34 +00:00

401 lines
19 KiB
ReStructuredText
Raw Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==================
Containers Service
==================
Launchpad blueprint:
https://blueprints.launchpad.net/nova/+spec/containers-service
Containers share many features in common with Nova instances. For the common
features, virt drivers for Nova can be used to surface basic instance
functionality. For features that go beyond what can be naturally fit within
a virt driver, we propose a new API service that allows for advanced features
to be added without conflating the worlds of instances and containers.
Some examples of containers specific features are setting of shell environment
variables, and accepting a shell command to execute at runtime. Capturing the
STDIO of the process(es) within a container, and tracking the return status
of processes are all beyond the scope of what was contemplated for Nova. All
of these features will be implemented in the Containers Service.
Problem description
===================
Container technology is rapidly gaining popularity as a way to bundle and
deploy applications. Recognizing and adapting to this trend will position
OpenStack to be useful not only to clouds that employ bare metal and virtual
machine instances, but can remain competitive in offering container services
as well.
Nova's concepts of an instance, and the actions that may be taken on it do not
match completely with containers.
Use cases
---------
1. App Consolidation. End-user wants to run multiple small applications in
separate operating system environments, but wants to optimize for efficiency
to control hosting costs. Each application belongs to the same tenant, so
security isolation between applications is nice-to-have but not critical.
Isolation is desired primarily for simplified management of the execution
environment for each application.
2. App Portability. End-user wants to create a single container image, and
deploy the same image to multiple hosting environments, including OpenStack.
Other environments may include local servers, dedicated servers, private
clouds, and public clouds. Switching environments requires passing database
connection strings by environment variables at the time a container starts
to allow the application to use the services available in each environment
without changing the container image.
3. Docker Compatibility. End-user has a Dockerfile used to build an application
and its runtime environment and dependencies in a Docker container image.
They want an easy way to run the Docker resulting image on an OpenStack
cloud.
4. LXC Compatibility. End-user wants an easy way to remotely create multiple
LXC containers within a single Nova instance.
5. OpenVZ Compatibility. End-user wants an easy way to remotely create multiple
OpenVZ containers within a single Nova instance.
6. Containers-Centric World View. End-user wants to communicate with a single
OpenStack API, and request the addition of containers, without the need to
be concerned with keeping track of how many containers are already running
on a given Nova instance, and when more need to be created. They want to
simply create and remove containers, and allow the appropriate resource
scheduling to happen automatically.
7. Platform Integration. Cloud operator already has an OpenStack cloud, and
wants to add a service/application centric management system on top.
Examples of such systems are Cloud Foundry, Kubernetes, Apache Mesos, etc.
The selected system is already Docker compatible. Allow this cloud operator
easy integration with OpenStack to run applications in containers. The
Cloud Operator now harnesses the power of both the management system, and
OpenStack, and does not need to manage a second infrastructure for his/her
application hosting needs. All details involving the integration of
containers with Nova instances is managed by OpenStack.
8. Container network. End-user wants to define a custom overlay network for
containers, and wants to have admin privilege to manage the network
topology. Building a container network can decouple application deployment
and management from the underlying network infrastructure, and enable
additional usage scenario, such as (i) software-defined networking, and
(ii) extending the container network (i.e. connecting various resources from
multiple hosting environments). End-users want a single service that could
help them build the container network, and dynamically modify the network
topology by adding or removing containers to or from the network.
9. Permit secure use of native REST APIs. Provide two models of operation with
Magnum. The first model allows Magnum to manage the lifecycle of Pods,
ReplicationControllers, and Services. The second model allows end-users to
manage the lifecycle of Pods, ReplicationControllers, and Services by
providing direct secure access to the native ReST APIs in Kubernetes and
possibly Docker.
Long Term Use Cases
-------------------
These use cases have been identified by the community as important, but
unlikely to be tackled in short term (especially prior to incubation). We wish
to adapt to these use cases in long term, but this is not a firm project
commitment.
1. Multi-region/multi-cloud support. End-user wants to deploy applications to
multiple regions/clouds, and dynamically relocate deployed applications
across different regions/clouds. In particular, they want a single service
that could help them (i) provision nodes from multiple regions/clouds, thus
running containers on top of them, and (ii) dynamically relocate containers
(e.g. through container migration) between nodes regardless of the
underlying infrastructure.
Proposed change
===============
Add a new API service for CRUD and advanced management of containers.
If cloud operators only want to offer basic instance features for their
containers, they may use nova with an alternate virt-driver, such as
libvirt/lxc or nova-docker. For those wanting a full-featured container
experience, they may offer the Containers Service API as well, in combination
with Nova instances that contain an OpenStack agent that connects to the
containers service through a security controlled agent (daemon) that allows
the OpenStack control plane to provision and control containers running on
Compute Hosts.
The Containers Service will call the Nova API to create one or more Nova
instances inside which containers will be created. The Nova instances may
be of any type, depending on the virt driver(s) chosen by the cloud operator.
This includes bare-metal, virtual machines, containers, and potentially other
instance types.
This allows the following configurations of containers in OpenStack.
* Containers in Virtual Machine Instances
* Containers in Bare Metal Instances
* Containers in Container Instances (nested)
The concept of nesting containers is currently possible if the parent container
runs in privileged mode. Patches to the linux kernel are being developed to
allow nesting of non-privileged containers as well, which provides a higher
level of security.
The spirit of this plan aims to duplicate as little as possible between Nova
and the Containers Service. Common components like the scheduler are expected
to be abstracted into modules, such as Gantt that can be shared by multiple
projects. Until Gantt is ready for use by the Containers Service, we will
implement only two provisioning schemes for containers:
1. Create a container on a specified instance by using a nova instance guid.
2. Auto-create instances (applies only until the Gantt scheduler is used)
2.1. Fill them sequentially until full.
2.2. Remove them automatically when they become empty.
The above orchestration will be implemented using Heat. This requires some
kind of hypervisor painting (such as host aggregates) for security reasons.
The diagram below offers an overview of the system architecture. The OSC box
indicates an OpenStack client, which will communicate with the Containers
Service through a REST API. The containers service may silently create Nova
instances if one with enough capacity to host the requested container is not
already known to the Containers service. The containers service will maintain
a database "Map" of containers, and what Nova instance each belongs to. Nova
creates instances. Instances are created in Nova, and containers belong only
to the Containers Service, and run within a Nova instance. If the instance
includes the agent software "A", then it may be included in the inventory of
the Containers service. Instances that do not contain an agent may not interact
with the Containers Service, and can be controlled only by a Nova virt driver.
::
+---------+
|OSC|
+----+----+
|
+----+----+
+--------Nova-------++-+REST+--Containers-+
|||+---------+Service|
||||
|+-------++--++-----+|
||Gantt||||Map||
|+-------+||+-----+|
||||
+-----------+---------++---------------+-----------+
||
+-----------+----+ComputeHost---------|-----------+
|+---+---+|
|+----+Relay+---+|
||+-------+||
||||
|+--Instance--++--Instance|-++--Instance|-+|
||||||||||
||||+---+||+---+||
||||||||||||
|||||A||||A|||
||||||||||||
||||+---+||+---+||
||||||||
||||+---++---+||+---++---+||
||||||||||||||||
|||||C||C||||C||C|||
||||||||||||||||
||||+---++---+||+---++---+||
||||||||
|+--------------++--------------++--------------+|
||
+----------------------------------------------------+
+---+
| |
| A | = Agent
| |
+---+
+---+
| |
| C | = Container
| |
+---+
Design Principles
-----------------
1. Leverage existing OpenStack projects for what they are good at. Do not
duplicate functionality, or copy code that can be otherwise accessed through
API calls.
2. Keep modifications to Nova to a minimum.
3. Make the user experience for end users simple and familiar.
4. Allow for implementation of all features containers are intended to offer.
Alternatives
------------
1. Extending Nova's existing feature set to offer container features
1.1. Container features don't fit into Nova's idea of compute (VM/Server)
2. A completely separate containers service forked from Nova.
2.1. Would result in large overlap and duplication in features and code
Data model impact
-----------------
For Nova, None. All new data planned will be in the Containers Service.
REST API impact
---------------
For Nova, none. All new API calls will be implemented in the Containers
Service. The OpenStack Containers Service API will be a superset of
functionality offered by the, The `Docker Remote API:
<https://docs.docker.com/reference/api/docker_remote_api/>`_
with additionals to make is suitable for general use regardless of the backend
container technology used, and to be compatible with OpenStack multi-tenancy
and Keystone authentication.
Specific Additions:
1. Support for the X-Auth-Project-Id HTTP request header to allow for
multi-tenant use.
2. Support for the X-Auth-Token HTTP request header to allow for authentication
with keystone.
If either of the above headers are missing, a 401 Unauthorized response will
be generated.
Docker CLI clients may communicate with a Swarmd instance that is configured
to use the OpenStack Containers API as the backend for libswarm. This will
allow for tool compatibility with the Docker ecosystem using the officially
supported means for integration of a distributed system.
The scope of the full API will cause this spec to be too long to review, so
the intent is to deal with the specific API design as a series of Gerrit
reviews that submit API code as Not Implemented stubs with docstrings that
clearly document the design, so allow for approval, and further implementation.
Security impact
---------------
Because Nova will not be changed, there should be no security impacts to Nova.
The Containers Service implementation, will have the following security related
issues:
* Need to authenticate against keystone using python-keystoneclient.
* A trust token from Nova will be needed in order for the Containers Service
to call the Nova API on behalf of a user.
* Limits must be implemented to control resource consumption in accordance with
quotas.
* Providing STDIO access may generate a considerable amount of network chatter
between containers and clients through the relay. This could lead to
bandwidth congestion at the relays, or API nodes. An approach similar to
how we handle serial console access today will need to be considered to
mitigate this concern.
Using containers implies a range of security considerations for cloud
operators. These include:
* Containers in the same instance share an operating system. If the kernel is
exploited using a security vulnerability, processes in once container may
escape the constraints of the container and potentially access other
resources on the host, including contents of other containers.
* Output of processes may be persisted by the containers service in order to
allow asynchronous collection of exit status, and terminal output. Such
content may include sensitive information. Features may be added to mitigate
the risk of this data being replicated in log messages, including errors.
* Creating containers usually requires root access. This means that the Agent
may need to be run with special privileges, or be given a method to
escalate privileges using techniques such as sudo.
* User provided data is passed through the API. This will require sensible
data input validation.
Notifications impact
--------------------
Contemplated features (in subsequent release cycles):
* Notify the end user each time a Nova instance is created or deleted by
the Containers service, if (s)he has registered for such notifications.
* Notify the user each on CRUD of containers containing start and end
notifications. (compute.container.create/delete/etc)
* Notify user periodically of existence of container service managed
containers (ex compute.container.exists)
Other end user impact
---------------------
The user interface will be a REST API. On top of that API will be an
implementation of the libswarm API to allow for tools designed to use Docker
to treat OpenStack as an upstream system.
Performance Impact
------------------
The Nova API will be used to create instances as needed. If the Container to
Instance ratio is 10, then the Nova API will be called at least once for every
10 calls to the Containers Service. Instances that are left empty will be
automatically deleted, so in the example of a 10:1 ratio, the Nova API will be
called to perform a delete for every 10 deletes in the Container Service.
Depending on the configuration, the ratio may be as low as 1:1.
The Containers Service will only access Nova through its API, not by accessing
its database.
Other deployer impact
---------------------
Deployers may want to adjust the default flavor used for Nova Instances created
by the Containers Service.
There should be no impact on users of prior releases, as this introduces a new
API.
Developer impact
----------------
Minimal. There will be minimal changes required in Nova, if any.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
aotto
Other contributors:
andrew-melton
ewindisch
Work Items
----------
1. Agent
2. Relay
3. API Service
4. IO Relays
Dependencies
============
1. <Links to Agent Blueprint and Spec here, once ready>
2. Early implementations may use libswarm, or a python port of libswarm to
implement Docker API compatibility.
Testing
=======
Each commit will be accompanied with unit tests, and Tempest functional tests.
Documentation Impact
====================
A set of documentation for this new service will be required.
References
==========
* Link to high level draft proposal from the Nova Midcycle Meetup for Juno:
`PDF <https://wiki.openstack.org/w/images/5/51/Containers_Proposal.pdf>`_
* `Libswarm Source <https://github.com/docker/libswarm>`_