Merge "Containers Service Spec"
This commit is contained in:
commit
e9ff12b521
|
@ -0,0 +1,370 @@
|
|||
..
|
||||
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||
License.
|
||||
|
||||
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||
|
||||
==================
|
||||
Containers Service
|
||||
==================
|
||||
|
||||
Launchpad blueprint:
|
||||
|
||||
https://blueprints.launchpad.net/nova/+spec/containers-service
|
||||
|
||||
Containers share many features in common with Nova instances. For the common
|
||||
features, virt drivers for Nova can be used to surface basic instance
|
||||
functionality. For features that go beyond what can be naturally fit within
|
||||
a virt driver, we propose a new API service that allows for advanced features
|
||||
to be added without conflating the worlds of instances and containers.
|
||||
|
||||
Some examples of containers specific features are setting of shell environment
|
||||
variables, and accepting a shell command to execute at runtime. Capturing the
|
||||
STDIO of the process(es) within a container, and tracking the return status
|
||||
of processes are all beyond the scope of what was contemplated for Nova. All
|
||||
of these features will be implemented in the Containers Service.
|
||||
|
||||
|
||||
Problem description
|
||||
===================
|
||||
Container technology is rapidly gaining popularity as a way to bundle and
|
||||
deploy applications. Recognizing and adapting to this trend will position
|
||||
OpenStack to be useful not only to clouds that employ bare metal and virtual
|
||||
machine instances, but can remain competitive in offering container services
|
||||
as well.
|
||||
|
||||
Nova's concepts of an instance, and the actions that may be taken on it do not
|
||||
match completely with containers.
|
||||
|
||||
Use cases
|
||||
---------
|
||||
1. App Consolidation. End-user wants to run multiple small applications in
|
||||
separate operating system environments, but wants to optimize for efficiency
|
||||
to control hosting costs. Each application belongs to the same tenant, so
|
||||
security isolation between applications is nice-to-have but not critical.
|
||||
Isolation is desired primarily for simplified management of the execution
|
||||
environment for each application.
|
||||
2. App Portability. End-user wants to create a single container image, and
|
||||
deploy the same image to multiple hosting environments, including OpenStack.
|
||||
Other environments may include local servers, dedicated servers, private
|
||||
clouds, and public clouds. Switching environments requires passing database
|
||||
connection strings by environment variables at the time a container starts
|
||||
to allow the application to use the services available in each environment
|
||||
without changing the container image.
|
||||
3. Docker Compatibility. End-user has a Dockerfile used to build an application
|
||||
and its runtime environment and dependencies in a Docker container image.
|
||||
They want an easy way to run the Docker resulting image on an OpenStack
|
||||
cloud.
|
||||
4. LXC Compatibility. End-user wants an easy way to remotely create multiple
|
||||
LXC containers within a single Nova instance.
|
||||
5. OpenVZ Compatibility. End-user wants an easy way to remotely create multiple
|
||||
OpenVZ containers within a single Nova instance.
|
||||
6. Containers-Centric World View. End-user wants to communicate with a single
|
||||
OpenStack API, and request the addition of containers, without the need to
|
||||
be concerned with keeping track of how many containers are already running
|
||||
on a given Nova instance, and when more need to be created. They want to
|
||||
simply create and remove containers, and allow the appropriate resource
|
||||
scheduling to happen automatically.
|
||||
7. Platform Integration. Cloud operator already has an OpenStack cloud, and
|
||||
wants to add a service/application centric management system on top.
|
||||
Examples of such systems are Cloud Foundry, Kubernetes, Apache Mesos, etc.
|
||||
The selected system is already Docker compatible. Allow this cloud operator
|
||||
easy integration with OpenStack to run applications in containers. The
|
||||
Cloud Operator now harnesses the power of both the management system, and
|
||||
OpenStack, and does not need to manage a second infrastructure for his/her
|
||||
application hosting needs. All details involving the integration of
|
||||
containers with Nova instances is managed by OpenStack.
|
||||
|
||||
Proposed change
|
||||
===============
|
||||
Add a new API service for CRUD and advanced management of containers.
|
||||
If cloud operators only want to offer basic instance features for their
|
||||
containers, they may use nova with an alternate virt-driver, such as
|
||||
libvirt/lxc or nova-docker. For those wanting a full-featured container
|
||||
experience, they may offer the Containers Service API as well, in combination
|
||||
with Nova instances that contain an OpenStack agent that connects to the
|
||||
containers service through a security controlled agent (daemon) that allows
|
||||
the OpenStack control plane to provision and control containers running on
|
||||
Compute Hosts.
|
||||
|
||||
The Containers Service will call the Nova API to create one or more Nova
|
||||
instances inside which containers will be created. The Nova instances may
|
||||
be of any type, depending on the virt driver(s) chosen by the cloud operator.
|
||||
This includes bare-metal, virtual machines, containers, and potentially other
|
||||
instance types.
|
||||
|
||||
This allows the following configurations of containers in OpenStack.
|
||||
|
||||
* Containers in Virtual Machine Instances
|
||||
* Containers in Bare Metal Instances
|
||||
* Containers in Container Instances (nested)
|
||||
|
||||
The concept of nesting containers is currently possible if the parent container
|
||||
runs in privileged mode. Patches to the linux kernel are being developed to
|
||||
allow nesting of non-privileged containers as well, which provides a higher
|
||||
level of security.
|
||||
|
||||
The spirit of this plan aims to duplicate as little as possible between Nova
|
||||
and the Containers Service. Common components like the scheduler are expected
|
||||
to be abstracted into modules, such as Gantt that can be shared by multiple
|
||||
projects. Until Gantt is ready for use by the Containers Service, we will
|
||||
implement only two provisioning schemes for containers:
|
||||
|
||||
1. Create a container on a specified instance by using a nova instance guid.
|
||||
2. Auto-create instances (applies only until the Gantt scheduler is used)
|
||||
2.1. Fill them sequentially until full.
|
||||
2.2. Remove them automatically when they become empty.
|
||||
|
||||
The above orchestration will be implemented using Heat. This requires some
|
||||
kind of hypervisor painting (such as host aggregates) for security reasons.
|
||||
|
||||
The diagram below offers an overview of the system architecture. The OSC box
|
||||
indicates an OpenStack client, which will communicate with the Containers
|
||||
Service through a REST API. The containers service may silently create Nova
|
||||
instances if one with enough capacity to host the requested container is not
|
||||
already known to the Containers service. The containers service will maintain
|
||||
a database "Map" of containers, and what Nova instance each belongs to. Nova
|
||||
creates instances. Instances are created in Nova, and contaienrs belong only
|
||||
to the Containers Service, and run witin a Nova instance. If the instance
|
||||
includes the agent software "A", then it may be included in the inventory of
|
||||
the Containers service. Instances that do not contain an agent may not interact
|
||||
with the Containers Service, and can be controlled only by a Nova virt driver.
|
||||
|
||||
::
|
||||
|
||||
+---------+
|
||||
| OSC |
|
||||
+----+----+
|
||||
|
|
||||
+----+----+
|
||||
+-------- Nova -------+ +-+ REST +-- Containers -+
|
||||
| | | +---------+ Service |
|
||||
| | | |
|
||||
| +-------+ +--+ +-----+ |
|
||||
| | Gantt | | | | Map | |
|
||||
| +-------+ | | +-----+ |
|
||||
| | | |
|
||||
+-----------+---------+ +---------------+-----------+
|
||||
| |
|
||||
+-----------+----+ Compute Host ---------|-----------+
|
||||
| +---+---+ |
|
||||
| +----+ Relay +---+ |
|
||||
| | +-------+ | |
|
||||
| | | |
|
||||
| +-- Instance --+ +-- Instance |-+ +-- Instance |-+ |
|
||||
| | | | | | | | | |
|
||||
| | | | +---+ | | +---+ | |
|
||||
| | | | | | | | | | | |
|
||||
| | | | | A | | | | A | | |
|
||||
| | | | | | | | | | | |
|
||||
| | | | +---+ | | +---+ | |
|
||||
| | | | | | | |
|
||||
| | | | +---+ +---+ | | +---+ +---+ | |
|
||||
| | | | | | | | | | | | | | | |
|
||||
| | | | | C | | C | | | | C | | C | | |
|
||||
| | | | | | | | | | | | | | | |
|
||||
| | | | +---+ +---+ | | +---+ +---+ | |
|
||||
| | | | | | | |
|
||||
| +--------------+ +--------------+ +--------------+ |
|
||||
| |
|
||||
+----------------------------------------------------+
|
||||
+---+
|
||||
| |
|
||||
| A | = Agent
|
||||
| |
|
||||
+---+
|
||||
+---+
|
||||
| |
|
||||
| C | = Container
|
||||
| |
|
||||
+---+
|
||||
|
||||
|
||||
Design Principles
|
||||
-----------------
|
||||
1. Leverage existing OpenStack projects for what they are good at. Do not
|
||||
duplicate functionality, or copy code that can be otherwise accessed through
|
||||
API calls.
|
||||
2. Keep modifications to Nova to a minimum.
|
||||
3. Make the user experience for end users simple and familiar.
|
||||
4. Allow for implementation of all features containers are intended to offer.
|
||||
|
||||
|
||||
Alternatives
|
||||
------------
|
||||
|
||||
1. Extending Nova's existing featureset to offer container features
|
||||
1.1. Container features don't fit into Nova's idea of compute (VM/Server)
|
||||
2. A completely separate containers service forked from Nova.
|
||||
2.1. Would result in large overlap and duplication in features and code
|
||||
|
||||
|
||||
Data model impact
|
||||
-----------------
|
||||
For Nova, None. All new data planned will be in the Containers Service.
|
||||
|
||||
|
||||
REST API impact
|
||||
---------------
|
||||
For Nova, none. All new API calls will be implemented in the Containers
|
||||
Service. The OpenStack Containers Service API will be a superset of
|
||||
functionality offered by the The `Docker API:
|
||||
<https://docs.docker.com/reference/api/docker_remote_api_v1.13/>`_
|
||||
with additionls to make is suitable for general use regardless of the backend
|
||||
container technology used, and to be compatible with OpenStack multi-tenancy
|
||||
and Keystone authentication.
|
||||
|
||||
Specific Additions:
|
||||
|
||||
1. Support for the X-Auth-Project-Id HTTP request header to allow for
|
||||
multi-tenant use.
|
||||
2. Support for the X-Auth-Token HTTP request header to allow for authentication
|
||||
with keystone.
|
||||
|
||||
If either of the above headers are missing, a 401 Unathorized response will
|
||||
be generated.
|
||||
|
||||
Docker CLI clients may communicate with a Swarmd instance that is configured
|
||||
to use the OpenStack Containers API as the backend for libswarm. This will
|
||||
allow for tool compatability with the Docker ecosystem using the officially
|
||||
supported means for integration of a distributed system.
|
||||
|
||||
The scope of the full API will cause this spec to be too long to review, so
|
||||
the intent is to deal with the specific API design as a series of Gerrit
|
||||
reviews that submit API code as Not Implemented stubs with docstrings that
|
||||
clearly document the design, so allow for approval, and further implementation.
|
||||
|
||||
Security impact
|
||||
---------------
|
||||
Because Nova will not be changed, there should be no security impacts to Nova.
|
||||
The Containers Service implementation, will have the following security related
|
||||
issues:
|
||||
|
||||
* Need to authenticate against keystone using python-keystoneclient.
|
||||
* A trust token from Nova will be needed in order for the Containers Service
|
||||
to call the Nova API on behalf of a user.
|
||||
* Limits must be implemented to control resource consumption in accordance with
|
||||
quotas.
|
||||
* Providing STDIO access may generate a considerable amount of network chatter
|
||||
between containers and clients through the relay. This could lead to
|
||||
bandwidth congestion at the relays, or API nodes. An approach similar to
|
||||
how we handle serial console access today will need to be considered to
|
||||
mitigate this concern.
|
||||
|
||||
Using containers implies a range of security considerations for cloud
|
||||
operators. These include:
|
||||
|
||||
* Containers in the same instance share an operating system. If the kernel is
|
||||
exploited using a security vulnerability, processes in once container may
|
||||
escape the constraints of the container and potentially access other
|
||||
resources on the host, including contents of other containers.
|
||||
* Output of processes may be persisted by the containers service in order to
|
||||
allow asynchronous collection of exit status, and terminal output. Such
|
||||
content may include sensitive information. Features may be added to mitgate
|
||||
the risk of this data being replicated in log messages, including errors.
|
||||
* Creating containers usually requires root access. This means that the Agent
|
||||
may need to be run with special privileges, or be given a method to
|
||||
escalate privileges using techniques such as sudo.
|
||||
* User provided data is passed through the API. This will require sensuble
|
||||
data input validation.
|
||||
|
||||
|
||||
Notifications impact
|
||||
--------------------
|
||||
|
||||
Contemplated features (in subsequent release cycles):
|
||||
|
||||
* Notify the end user each time a Nova instance is created or deleted by
|
||||
the Containers service, if (s)he has registered for such notifications.
|
||||
* Notify the user each on CRUD of containers containing start and end
|
||||
notifications. (compute.container.create/delete/etc)
|
||||
* Notify user periodically of existence of container service managed
|
||||
containers (ex compute.container.exists)
|
||||
|
||||
|
||||
Other end user impact
|
||||
---------------------
|
||||
|
||||
The user interface will be a REST API. On top of that API will be an
|
||||
implementation of the libswarm API to allow for tools designed to use Docker
|
||||
to treat OpenStack as an upstream system.
|
||||
|
||||
|
||||
Performance Impact
|
||||
------------------
|
||||
|
||||
The Nova API will be used to create instances as needed. If the Container to
|
||||
Instance ratio is 10, then the Nova API will be called at least once for every
|
||||
10 calls to the Containers Service. Instances that are left empty will be
|
||||
automatically deleted, so in the example of a 10:1 ratio, the Nova API will be
|
||||
called to perform a delete for every 10 deletes in the Container Service.
|
||||
Depending on the configuration, the ratio may be as low as 1:1.
|
||||
The Containers Service will only access Nova through its API, not by accessing
|
||||
its database.
|
||||
|
||||
|
||||
|
||||
Other deployer impact
|
||||
---------------------
|
||||
|
||||
Deployers may want to adjust the default flavor used for Nova Instances created
|
||||
by the Containers Service.
|
||||
|
||||
There should be no impact on users of prior releases, as this introduces a new
|
||||
API.
|
||||
|
||||
Developer impact
|
||||
----------------
|
||||
|
||||
Minimal. There will be minimal changes required in Nova, if any.
|
||||
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Primary assignee:
|
||||
aotto
|
||||
|
||||
Other contributors:
|
||||
andrew-melton
|
||||
ewindisch
|
||||
|
||||
|
||||
Work Items
|
||||
----------
|
||||
|
||||
1. Agent
|
||||
2. Relay
|
||||
3. API Service
|
||||
4. IO Relays
|
||||
|
||||
|
||||
Dependencies
|
||||
============
|
||||
|
||||
1. <Links to Agent Blueprint and Spec here, once ready>
|
||||
2. Early implementations may use libswarm, or a python port of libswarm to
|
||||
implement Docker API conpatibility.
|
||||
|
||||
Testing
|
||||
=======
|
||||
|
||||
Each commit will be accompanied with unit tests, and Tempest functional tests.
|
||||
|
||||
|
||||
Documentation Impact
|
||||
====================
|
||||
|
||||
A set of documentation for this new service will be required.
|
||||
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
* Link to high level draft proposal from the Nova Midcyle Meetup for Juno:
|
||||
`PDF <https://wiki.openstack.org/w/images/5/51/Containers_Proposal.pdf>`_
|
||||
* `Libswarm Source <https://github.com/docker/libswarm>`_
|
Loading…
Reference in New Issue