Browse Source

Import all implemented specs from magnum repo

* Add all implemented specs to specs/implemented.
* Create ocata directory to store the corresponding specs.

Change-Id: I36da3a26ece3b6f7d27b7af80460303f4a6c4226
changes/74/395674/1
Spyros Trigazis 2 years ago
parent
commit
8b0a99e44b

+ 1
- 0
doc/source/specs View File

@@ -0,0 +1 @@
1
+../../specs

+ 452
- 0
specs/implemented/async-container-operation.rst View File

@@ -0,0 +1,452 @@
1
+=================================
2
+Asynchronous Container Operations
3
+=================================
4
+
5
+Launchpad blueprint:
6
+
7
+https://blueprints.launchpad.net/magnum/+spec/async-container-operations
8
+
9
+At present, container operations are done in a synchronous way, end-to-end.
10
+This model does not scale well, and incurs a penalty on the client to be
11
+stuck till the end of completion of the operation.
12
+
13
+Problem Description
14
+-------------------
15
+
16
+At present Magnum-Conductor executes the container operation as part of
17
+processing the request forwarded from Magnum-API. For
18
+container-create, if the image needs to be pulled down, it may take
19
+a while depending on the responsiveness of the registry, which can be a
20
+substantial delay. At the same time, experiments suggest that even for
21
+pre-pulled image, the time taken by each operations, namely
22
+create/start/delete, are in the same order, as it involves complete turn
23
+around between the magnum-client and the COE-API, via Magnum-API and
24
+Magnum-Conductor[1].
25
+
26
+Use Cases
27
+---------
28
+
29
+For wider enterprise adoption of Magnum, we need it to scale better.
30
+For that we need to replace some of these synchronous behaviors with
31
+suitable alternative of asynchronous implementation.
32
+
33
+To understand the use-case better, we can have a look at the average
34
+time spent during container operations, as noted at[1].
35
+
36
+Proposed Changes
37
+----------------
38
+
39
+The design has been discussed over the ML[6]. The conclusions have been kept
40
+on the 'whiteboard' of the Blueprint.
41
+
42
+The amount of code change is expected to be significant. To ease the
43
+process of adoption, code review, functional tests, an approach of phased
44
+implementation may be required. We can define the scope of the three phases of
45
+the implementation as follows -
46
+
47
+* Phase-0 will bring in the basic feature of asynchronous mode of operation in
48
+  Magnum - (A) from API to Conductor and (B) from Conductor to COE-API. During
49
+  phase-0, this mode will be optional through configuration.
50
+
51
+  Both the communications of (A) and (B) are proposed to be made asynchronous
52
+  to achieve the best of it. If we do (A) alone, it does not gain us much, as
53
+  (B) takes up the higher cycles of the operation. If we do (B) alone, it does
54
+  not make sense, as (A) will synchronously wait for no meaningful data.
55
+
56
+* Phase-1 will concentrate on making the feature persistent to address various
57
+  scenarios of conductor restart, worker failure etc. We will support this
58
+  feature for multiple Conductor-workers in this phase.
59
+
60
+* Phase-2 will select asynchronous mode of operation as the default mode. At
61
+  the same time, we can evaluate to drop the code for synchronous mode, too.
62
+
63
+
64
+Phase-0 is required as a meaningful temporary step, to establish the
65
+importance and tangible benefits of phase-1. This is also to serve as a
66
+proof-of-concept at a lower cost of code changes with a configurable option.
67
+This will enable developers and operators to have a taste of the feature,
68
+before bringing in the heavier dependencies and changes proposed in phase-1.
69
+
70
+A reference implementation for the phase-0 items, has been put for review[2].
71
+
72
+Following is the summary of the design -
73
+
74
+1. Configurable mode of operation - async
75
+-----------------------------------------
76
+
77
+For ease of adoption, the async_mode of communication between API-conductor,
78
+conductor-COE in magnum, can be controlled using a configuration option. So
79
+the code-path for sync mode and async mode would co-exist for now. To achieve
80
+this with minimal/no code duplication and cleaner interface, we are using
81
+openstack/futurist[4]. Futurist interface hides the details of type of executor
82
+being used. In case of async configuration, a greenthreadpool of configured
83
+poolsize gets created. Here is a sample of how the config would look
84
+like: ::
85
+
86
+      [DEFAULT]
87
+      async_enable = False
88
+
89
+      [conductor]
90
+      async_threadpool_max_workers = 64
91
+
92
+Futurist library is used in oslo.messaging. Thus, it is used by almost all
93
+OpenStack projects, in effect. Futurist is very useful to run same code
94
+under different execution model and hence saving potential duplication of
95
+code.
96
+
97
+
98
+2. Type of operations
99
+---------------------
100
+
101
+There are two classes of container operations - one that can be made async,
102
+namely create/delete/start/stop/pause/unpause/reboot, which do not need data
103
+about the container in return. The other type requires data, namely
104
+container-logs. For async-type container-operations, magnum-API will be
105
+using 'cast' instead of 'call' from oslo_messaging[5].
106
+
107
+'cast' from oslo.messaging.rpcclient is used to invoke a method and return
108
+immediately, whereas 'call' invokes a method and waits for a reply. While
109
+operating in asynchronous mode, it is intuitive to use cast method, as the
110
+result of the response may not be available immediately.
111
+
112
+Magnum-api first fetches the details of a container, by doing
113
+'get_rpc_resource'. This function uses magnum objects. Hence, this function
114
+uses a 'call' method underneath. Once, magnum-api gets back the details,
115
+it issues the container operation next, using another 'call' method.
116
+The above proposal is to replace the second 'call' with 'cast'.
117
+
118
+If user issues a container operation, when there is no listening
119
+conductor (because of process failure), there will be a RPC timeout at the
120
+first 'call' method. In this case, user will observe the request to
121
+get blocked at client and finally fail with HTTP 500 ERROR, after the RPC
122
+timeout, which is 60 seconds by default. This behavior is independent of the
123
+usage of 'cast' or 'call' for the second message, mentioned above. This
124
+behavior does not influence our design, but it is documented here for clarity
125
+of understanding.
126
+
127
+
128
+3. Ensuring the order of execution - Phase-0
129
+--------------------------------------------
130
+
131
+Magnum-conductor needs to ensure that for a given bay and given container,
132
+the operations are executed in sequence. In phase-0, we want to demonstrate
133
+how asynchronous behavior helps scaling. Asynchronous mode of container
134
+operations would be supported for single magnum-conductor scenario, in
135
+phase-0. If magnum-conductor crashes, there will be no recovery for the
136
+operations accepted earlier - which means no persistence in phase-0, for
137
+operations accepted by magnum-conductor. Multiple conductor scenario and
138
+persistence will be addressed in phase-1 [please refer to the next section
139
+for further details]. If COE crashes or does not respond, the error will be
140
+detected, as it happens in sync mode, and reflected on the container-status.
141
+
142
+Magnum-conductor will maintain a job-queue. Job-queue is indexed by bay-id and
143
+container-id. A job-queue entry would contain the sequence of operations
144
+requested for a given bay-id and container-id, in temporal order. A
145
+greenthread will execute the tasks/operations in order for a given job-queue
146
+entry, till the queue empties. Using a greenthread in this fashion saves us
147
+from the cost and complexity of locking, along with functional correctness.
148
+When request for new operation comes in, it gets appended to the corresponding
149
+queue entry.
150
+
151
+For a sequence of container operations, if an intermediate operation fails,
152
+we will stop continuing the sequence. The community feels more confident to
153
+start with this strictly defensive policy[17]. The failure will be logged
154
+and saved into the container-object, which will help an operator be informed
155
+better about the result of the sequence of container operations. We may revisit
156
+this policy later, if we think it is too restrictive.
157
+
158
+4. Ensuring the order of execution - phase-1
159
+--------------------------------------------
160
+
161
+The goal is to execute requests for a given bay and a given container in
162
+sequence. In phase-1, we want to address persistence and capability of
163
+supporting multiple magnum-conductor processes. To achieve this, we will
164
+reuse the concepts laid out in phase-0 and use a standard library.
165
+
166
+We propose to use taskflow[7] for this implementation. Magnum-conductors
167
+will consume the AMQP message and post a task[8] on a taskflow jobboard[9].
168
+Greenthreads from magnum-conductors would subscribe to the taskflow
169
+jobboard as taskflow-conductors[10]. Taskflow jobboard is maintained with
170
+a choice of persistent backend[11]. This will help address the concern of
171
+persistence for accepted operations, when a conductor crashes. Taskflow
172
+will ensure that tasks, namely container operations, in a job, namely a
173
+sequence of operations for a given bay and container, would execute in
174
+sequence. We can easily notice that some of the concepts used in phase-0
175
+are reused as it is. For example, job-queue maps to jobboard here, use of
176
+greenthread maps to the conductor concept of taskflow. Hence, we expect easier
177
+migration from phase-0 to phase-1, with the choice of taskflow.
178
+
179
+For taskflow jobboard[11], the available choices of backend are Zookeeper and
180
+Redis. But, we plan to use MySQL as default choice of backend, for magnum
181
+conductor jobboard use-case. This support will be added to taskflow. Later,
182
+we may choose to support the flexibility of other backends like ZK/Redis via
183
+configuration. But, phase-1 will keep the implementation simple with MySQL
184
+backend and revisit this, if required.
185
+
186
+Let's consider the scenarios of Conductor crashing -
187
+ - If a task is added to jobboard, and conductor crashes after that,
188
+   taskflow can assign a particular job to any available greenthread agents
189
+   from other conductor instances. If the system was running with single
190
+   magnum-conductor, it will wait for the conductor to come back and join.
191
+ - A task is picked up and magnum-conductor crashes. In this case, the task
192
+   is not complete from jobboard point-of-view. As taskflow detects the
193
+   conductor going away, it assigns another available conductor.
194
+ - When conductor picks up a message from AMQP, it will acknowledge AMQP,
195
+   only after persisting it to jobboard. This will prevent losing the message,
196
+   if conductor crashes after picking up the message from AMQP. Explicit
197
+   acknowledgement from application may use NotificationResult.HANDLED[12]
198
+   to AMQP. We may use the at-least-one-guarantee[13] feature in
199
+   oslo.messaging[14], as it becomes available.
200
+
201
+To summarize some of the important outcomes of this proposal -
202
+ - A taskflow job represents the sequence of container operations on a given
203
+   bay and given container. At a given point of time, the sequence may contain
204
+   a single or multiple operations.
205
+ - There will be a single jobboard for all conductors.
206
+ - Task-flow conductors are multiple greenthreads from a given
207
+   magnum-conductor.
208
+ - Taskflow-conductor will run in 'blocking' mode[15], as those greenthreads
209
+   have no other job than claiming and executing the jobs from jobboard.
210
+ - Individual jobs are supposed to maintain a temporal sequence. So the
211
+   taskflow-engine would be 'serial'[16].
212
+ - The proposed model for a 'job' is to consist of a temporal sequence of
213
+   'tasks' - operations on a given bay and a given container. Henceforth,
214
+   it is expected that when a given operation, namely container-create is in
215
+   progress, a request for container-start may come in. Adding the task to
216
+   the existing job is intuitive to maintain the sequence of operations.
217
+
218
+To fit taskflow exactly into our use-case, we may need to do two enhancements
219
+in taskflow -
220
+- Supporting mysql plugin as a DB backend for jobboard. Support for redis
221
+exists, so it will be similar.
222
+We do not see any technical roadblock for adding mysql support for taskflow
223
+jobboard. If the proposal does not get approved by taskflow team, we may have
224
+to use redis, as an alternative option.
225
+- Support for dynamically adding tasks to a job on jobboard. This also looks
226
+feasible, as discussed over the #openstack-state-management [Unfortunately,
227
+this channel is not logged, but if we agree in this direction, we can initiate
228
+discussion over ML, too]
229
+If taskflow team does not allow adding this feature, even though they have
230
+agreed now, we will use the dependency feature in taskflow. We will explore
231
+and elaborate this further, if it requires.
232
+
233
+
234
+5. Status of progress
235
+---------------------
236
+
237
+The progress of execution of a container operation is reflected on the status
238
+of a container as - 'create-in-progress', 'delete-in-progress' etc.
239
+
240
+Alternatives
241
+------------
242
+
243
+Without an asynchronous implementation, Magnum will suffer from complaints
244
+about poor scalability and slowness.
245
+
246
+In this design, stack-lock[3] has been considered as an alternative to
247
+taskflow. Following are the reasons for preferring taskflow over
248
+stack-lock, as of now,
249
+- Stack-lock used in Heat is not a library, so it will require making a copy
250
+for Magnum, which is not desirable.
251
+- Taskflow is relatively mature, well supported, feature-rich library.
252
+- Taskflow has in-built capacity to scale out[in] as multiple conductors
253
+can join in[out] the cluster.
254
+- Taskflow has a failure detection and recovery mechanism. If a process
255
+crashes, then worker threads from other conductor may continue the execution.
256
+
257
+In this design, we describe futurist[4] as a choice of implementation. The
258
+choice was to prevent duplication of code for async and sync mode. For this
259
+purpose, we could not find any other solution to compare.
260
+
261
+Data model impact
262
+-----------------
263
+
264
+Phase-0 has no data model impact. But phase-1 may introduce an additional
265
+table into the Magnum database. As per the present proposal for using taskflow
266
+in phase-1, we have to introduce a new table for jobboard under magnum db.
267
+This table will be exposed to taskflow library as a persistent db plugin.
268
+Alternatively, an implementation with stack-lock will also require an
269
+introduction of a new table for stack-lock objects.
270
+
271
+REST API impact
272
+---------------
273
+
274
+None.
275
+
276
+Security impact
277
+---------------
278
+
279
+None.
280
+
281
+Notifications impact
282
+--------------------
283
+
284
+None
285
+
286
+Other end user impact
287
+---------------------
288
+
289
+None
290
+
291
+Performance impact
292
+------------------
293
+
294
+Asynchronous mode of operation helps in scalability. Hence, it improves
295
+responsiveness and reduces the turn around time in a significant
296
+proportion. A small test on devstack, comparing both the modes,
297
+demonstrate this with numbers.[1]
298
+
299
+Other deployer impact
300
+---------------------
301
+
302
+None.
303
+
304
+Developer impact
305
+----------------
306
+
307
+None
308
+
309
+Implementation
310
+--------------
311
+
312
+Assignee(s)
313
+-----------
314
+
315
+Primary assignee
316
+  suro-patz(Surojit Pathak)
317
+
318
+Work Items
319
+----------
320
+
321
+For phase-0
322
+* Introduce config knob for asynchronous mode of container operations.
323
+
324
+* Changes for Magnum-API to use CAST instead of CALL for operations eligible
325
+  for asynchronous mode.
326
+
327
+* Implement the in-memory job-queue in Magnum conductor, and integrate futurist
328
+  library.
329
+
330
+* Unit tests and functional tests for async mode.
331
+
332
+* Documentation changes.
333
+
334
+For phase-1
335
+* Get the dependencies on taskflow being resolved.
336
+
337
+* Introduce jobboard table into Magnum DB.
338
+
339
+* Integrate taskflow in Magnum conductor to replace the in-memory job-queue
340
+  with taskflow jobboard. Also, we need conductor greenthreads to subscribe
341
+  as workers to the taskflow jobboard.
342
+
343
+* Add unit tests and functional tests for persistence and multiple conductor
344
+  scenario.
345
+
346
+* Documentation changes.
347
+
348
+For phase-2
349
+* We will promote asynchronous mode of operation as the default mode of
350
+operation.
351
+
352
+* We may decide to drop the code for synchronous mode and corresponding config.
353
+
354
+* Documentation changes.
355
+
356
+
357
+Dependencies
358
+------------
359
+
360
+For phase-1, if we choose to implement using taskflow, we need to get
361
+following two features added to taskflow first -
362
+* Ability to add new task to an existing job on jobboard.
363
+* mysql plugin support as persistent DB.
364
+
365
+Testing
366
+-------
367
+
368
+All the existing test cases are run to ensure async mode does not break them.
369
+Additionally more functional tests and unit tests will be added specific to
370
+async mode.
371
+
372
+Documentation Impact
373
+--------------------
374
+
375
+Magnum documentation will include a description of the option for asynchronous
376
+mode of container operations and its benefits. We will also add to
377
+developer documentation on guideline for implementing a container operation in
378
+both the modes - sync and async. We will add a section on 'how to debug
379
+container operations in async mode'. The phase-0 and phase-1 implementation
380
+and their support for single or multiple conductors will be clearly documented
381
+for the operators.
382
+
383
+References
384
+----------
385
+
386
+[1] - Execution time comparison between sync and async modes:
387
+
388
+https://gist.github.com/surojit-pathak/2cbdad5b8bf5b569e755
389
+
390
+[2] - Proposed change under review:
391
+
392
+https://review.openstack.org/#/c/267134/
393
+
394
+[3] - Heat's use of stacklock
395
+
396
+http://docs.openstack.org/developer/heat/_modules/heat/engine/stack_lock.html
397
+
398
+[4] - openstack/futurist
399
+
400
+http://docs.openstack.org/developer/futurist/
401
+
402
+[5] - openstack/oslo.messaging
403
+
404
+http://docs.openstack.org/developer/oslo.messaging/rpcclient.html
405
+
406
+[6] - ML discussion on the design
407
+
408
+http://lists.openstack.org/pipermail/openstack-dev/2015-December/082524.html
409
+
410
+[7] - Taskflow library
411
+
412
+http://docs.openstack.org/developer/taskflow/
413
+
414
+[8] - task in taskflow
415
+
416
+http://docs.openstack.org/developer/taskflow/atoms.html#task
417
+
418
+[9] - job and jobboard in taskflow
419
+
420
+http://docs.openstack.org/developer/taskflow/jobs.html
421
+
422
+[10] - conductor in taskflow
423
+
424
+http://docs.openstack.org/developer/taskflow/conductors.html
425
+
426
+[11] - persistent backend support in taskflow
427
+
428
+http://docs.openstack.org/developer/taskflow/persistence.html
429
+
430
+[12] - oslo.messaging notification handler
431
+
432
+http://docs.openstack.org/developer/oslo.messaging/notification_listener.html
433
+
434
+[13] - Blueprint for at-least-once-guarantee, oslo.messaging
435
+
436
+https://blueprints.launchpad.net/oslo.messaging/+spec/at-least-once-guarantee
437
+
438
+[14] - Patchset under review for at-least-once-guarantee, oslo.messaging
439
+
440
+https://review.openstack.org/#/c/229186/
441
+
442
+[15] - Taskflow blocking mode for conductor
443
+
444
+http://docs.openstack.org/developer/taskflow/conductors.html#taskflow.conductors.backends.impl_executor.ExecutorConductor
445
+
446
+[16] - Taskflow serial engine
447
+
448
+http://docs.openstack.org/developer/taskflow/engines.html
449
+
450
+[17] - Community feedback on policy to handle failure within a sequence
451
+
452
+http://eavesdrop.openstack.org/irclogs/%23openstack-containers/%23openstack-containers.2016-03-08.log.html#t2016-03-08T20:41:17

+ 344
- 0
specs/implemented/bay-drivers.rst View File

@@ -0,0 +1,344 @@
1
+..
2
+   This work is licensed under a Creative Commons Attribution 3.0 Unported
3
+ License.
4
+
5
+ http://creativecommons.org/licenses/by/3.0/legalcode
6
+
7
+======================================
8
+Container Orchestration Engine drivers
9
+======================================
10
+
11
+Launchpad blueprint:
12
+
13
+https://blueprints.launchpad.net/magnum/+spec/bay-drivers
14
+
15
+Container Orchestration Engines (COEs) are different systems for managing
16
+containerized applications in a clustered environment, each having their own
17
+conventions and ecosystems. Three of the most common, which also happen to be
18
+supported in Magnum, are: Docker Swarm, Kubernetes, and Mesos. In order to
19
+successfully serve developers, Magnum needs to be able to provision and manage
20
+access to the latest COEs through its API in an effective and scalable way.
21
+
22
+
23
+Problem description
24
+===================
25
+
26
+Magnum currently supports the three most popular COEs, but as more emerge and
27
+existing ones change, it needs an effective and scalable way of managing
28
+them over time.
29
+
30
+One of the problems with the current implementation is that COE-specific logic,
31
+such as Kubernetes replication controllers and services, is situated in the
32
+core Magnum library and made available to users through the main API. Placing
33
+COE-specific logic in a core API introduces tight coupling and forces
34
+operators to work with an inflexible design.
35
+
36
+By formalising a more modular and extensible architecture, Magnum will be
37
+in a much better position to help operators and consumers satisfy custom
38
+use-cases.
39
+
40
+Use cases
41
+---------
42
+
43
+1. Extensibility. Contributors and maintainers need a suitable architecture to
44
+   house current and future COE implementations. Moving to a more extensible
45
+   architecture, where core classes delegate to drivers, provides a more
46
+   effective and elegant model for handling COE differences without the need
47
+   for tightly coupled and monkey-patched logic.
48
+
49
+   One of the key use cases is allowing operators to customise their
50
+   orchestration logic, such as modifying Heat templates or even using their
51
+   own tooling like Ansible. Moreover, operators will often expect to use a
52
+   custom distro image with lots of software pre-installed and many special
53
+   security requirements that is extremely difficult or impossible to do with
54
+   the current upstream templates. COE drivers solves these problems.
55
+
56
+2. Maintainability. Moving to a modular architecture will be easier to manage
57
+   in the long-run because the responsibility of maintaining non-standard
58
+   implementations is shifted into the operator's domain. Maintaining the
59
+   default drivers which are packaged with Magnum will also be easier and
60
+   cleaner since logic is now demarcated from core codebase directories.
61
+
62
+3. COE & Distro choice. In the community there has been a lot of discussion
63
+   about which distro and COE combination to support with the templates.
64
+   Having COE drivers allows for people or organizations to maintain
65
+   distro-specific implementations (e.g CentOS+Kubernetes).
66
+
67
+4. Addresses dependency concerns. One of the direct results of
68
+   introducing a driver model is the ability to give operators more freedom
69
+   about choosing how Magnum integrates with the rest of their OpenStack
70
+   platform. For example, drivers would remove the necessity for users to
71
+   adopt Barbican for secret management.
72
+
73
+5. Driver versioning. The new driver model allows operators to modify existing
74
+   drivers or creating custom ones, release new bay types based on the newer
75
+   version, and subsequently launch news bays running the updated
76
+   functionality. Existing bays which are based on older driver versions would
77
+   be unaffected in this process and would still be able to have lifecycle
78
+   operations performed on them. If one were to list their details from the
79
+   API, it would reference the old driver version. An operator can see which
80
+   driver version a bay type is based on through its ``driver`` value,
81
+   which is exposed through the API.
82
+
83
+Proposed change
84
+===============
85
+
86
+1. The creation of new directory at the project root: ``./magnum/drivers``.
87
+   Each driver will house its own logic inside its own directory. Each distro
88
+   will house its own logic inside that driver directory. For example, the
89
+   Fedora Atomic distro using Swarm will have the following directory
90
+   structure:
91
+
92
+   ::
93
+
94
+      drivers/
95
+        swarm_atomic_v1/
96
+            image/
97
+              ...
98
+            templates/
99
+              ...
100
+            api.py
101
+            driver.py
102
+            monitor.py
103
+            scale.py
104
+            template_def.py
105
+            version.py
106
+
107
+
108
+   The directory name should be a string which uniquely identifies the driver
109
+   and provides a descriptive reference. The driver version number and name are
110
+   provided in the manifest file and will be  included in the bay metadata at
111
+   cluster build time.
112
+
113
+   There are two workflows for rolling out driver updates:
114
+
115
+   - if the change is relatively minor, they modify the files in the
116
+     existing driver directory and update the version number in the manifest
117
+     file.
118
+
119
+   - if the change is significant, they create a new directory
120
+     (either from scratch or by forking).
121
+
122
+   Further explanation of the three top-level files:
123
+
124
+   - an ``image`` directory is *optional* and should contain documentation
125
+     which tells users how to build the image and register it to glance. This
126
+     directory can also hold artifacts for building the image, for instance
127
+     diskimagebuilder elements, scripts, etc.
128
+
129
+   - a ``templates`` directory is *required* and will (for the forseeable
130
+     future) store Heat template YAML files. In the future drivers will allow
131
+     operators to use their own orchestration tools like Ansible.
132
+
133
+   - ``api.py`` is *optional*, and should contain the API controller which
134
+     handles custom API operations like Kubernetes RCs or Pods. It will be
135
+     this class which accepts HTTP requests and delegates to the Conductor. It
136
+     should contain a uniquely named class, such as ``SwarmAtomicXYZ``, which
137
+     extends from the core controller class. The COE class would have the
138
+     opportunity of overriding base methods if necessary.
139
+
140
+   - ``driver.py`` is *required*, and should contain the logic which maps
141
+     controller actions to COE interfaces. These include: ``bay_create``,
142
+     ``bay_update``, ``bay_delete``, ``bay_rebuild``, ``bay_soft_reboot`` and
143
+     ``bay_hard_reboot``.
144
+
145
+   - ``version.py`` is *required*, and should contain the version number of
146
+     the bay driver. This is defined by a ``version`` attribute and is
147
+     represented in the ``1.0.0`` format. It should also include a ``Driver``
148
+     attribute and should be a descriptive name such as ``swarm_atomic``.
149
+
150
+     Due to the varying nature of COEs, it is up to the bay
151
+     maintainer to implement this in their own way. Since a bay is a
152
+     combination of a COE and an image, ``driver.py`` will also contain
153
+     information about the ``os_distro`` property which is expected to be
154
+     attributed to Glance image.
155
+
156
+   - ``monitor.py`` is *optional*, and should contain the logic which monitors
157
+     the resource utilization of bays.
158
+
159
+   - ``template_def.py`` is *required* and should contain the COE's
160
+     implementation of how orchestration templates are loaded and matched to
161
+     Magnum objects. It would probably contain multiple classes, such as
162
+     ``class SwarmAtomicXYZTemplateDef(BaseTemplateDefinition)``.
163
+
164
+   - ``scale.py`` is *optional* per bay specification and should contain the
165
+     logic for scaling operations.
166
+
167
+2. Renaming the ``coe`` attribute of BayModel to ``driver``. Because this
168
+   value would determine which driver classes and orchestration templates to
169
+   load, it would need to correspond to the name of the driver as it is
170
+   registered with stevedore_ and setuptools entry points.
171
+
172
+   During the lifecycle of an API operation, top-level Magnum classes (such as
173
+   a Bay conductor) would then delegate to the driver classes which have been
174
+   dynamically loaded. Validation will need to ensure that whichever value
175
+   is provided by the user is correct.
176
+
177
+   By default, drivers are located under the main project directory and their
178
+   namespaces are accessible via ``magnum.drivers.foo``. But a use case that
179
+   needs to be looked at and, if possible, provided for is drivers which are
180
+   situated outside the project directory, for example in
181
+   ``/usr/share/magnum``. This will suit operators who want greater separation
182
+   between customised code and Python libraries.
183
+
184
+3. The driver implementations for the 3 current COE and Image combinations:
185
+   Docker Swarm Fedora, Kubernetes Fedora, Kubernetes CoreOS, and Mesos
186
+   Ubuntu. Any templates would need to be moved from
187
+   ``magnum/templates/{coe_name}`` to
188
+   ``magnum/drivers/{driver_name}/templates``.
189
+
190
+4. Removal of the following files:
191
+
192
+   ::
193
+
194
+    magnum/magnum/conductor/handlers/
195
+      docker_conductor.py
196
+      k8s_conducter.py
197
+
198
+Design Principles
199
+-----------------
200
+
201
+- Minimal, clean API without a high cognitive burden
202
+
203
+- Ensure Magnum's priority is to do one thing well, but allow extensibility
204
+  by external contributors
205
+
206
+- Do not force ineffective abstractions that introduce feature divergence
207
+
208
+- Formalise a modular and loosely coupled driver architecture that removes
209
+  COE logic from the core codebase
210
+
211
+
212
+Alternatives
213
+------------
214
+
215
+This alternative relates to #5 of Proposed Change. Instead of having a
216
+drivers registered using stevedore_ and setuptools entry points, an alternative
217
+is to use the Magnum config instead.
218
+
219
+
220
+Data model impact
221
+-----------------
222
+
223
+Since drivers would be implemented for the existing COEs, there would be
224
+no loss of functionality for end-users.
225
+
226
+
227
+REST API impact
228
+---------------
229
+
230
+Attribute change when creating and updating a BayModel (``coe`` to
231
+``driver``). This would occur before v1 of the API is frozen.
232
+
233
+COE-specific endpoints would be removed from the core API.
234
+
235
+
236
+Security impact
237
+---------------
238
+
239
+None
240
+
241
+
242
+Notifications impact
243
+--------------------
244
+
245
+None
246
+
247
+
248
+Other end user impact
249
+---------------------
250
+
251
+There will be deployer impacts because deployers will need to select
252
+which drivers they want to activate.
253
+
254
+
255
+Performance Impact
256
+------------------
257
+
258
+None
259
+
260
+
261
+
262
+Other deployer impact
263
+---------------------
264
+
265
+In order to utilize new functionality and bay drivers, operators will need
266
+to update their installation and configure bay models to use a driver.
267
+
268
+
269
+Developer impact
270
+----------------
271
+
272
+Due to the significant impact on the current codebase, a phased implementation
273
+approach will be necessary. This is defined in the Work Items section.
274
+
275
+Code will be contributed for COE-specific functionality in a new way, and will
276
+need to abide by the new architecture. Documentation and a good first
277
+implementation will play an important role in helping developers contribute
278
+new functionality.
279
+
280
+
281
+Implementation
282
+==============
283
+
284
+
285
+Assignee(s)
286
+-----------
287
+
288
+Primary assignee:
289
+murali-allada
290
+
291
+Other contributors:
292
+jamiehannaford
293
+strigazi
294
+
295
+
296
+Work Items
297
+----------
298
+
299
+1. New ``drivers`` directory
300
+
301
+2. Change ``coe`` attribute to ``driver``
302
+
303
+3. COE drivers implementation (swarm-fedora, k8s-fedora, k8s-coreos,
304
+   mesos-ubuntu). Templates should remain in directory tree until their
305
+   accompanying driver has been implemented.
306
+
307
+4. Delete old conductor files
308
+
309
+5. Update client
310
+
311
+6. Add documentation
312
+
313
+7. Improve user experience for operators of forking/creating new
314
+   drivers. One way we could do this is by creating new client commands or
315
+   scripts. This is orthogonal to this spec, and will be considered after
316
+   its core implementation.
317
+
318
+Dependencies
319
+============
320
+
321
+None
322
+
323
+
324
+Testing
325
+=======
326
+
327
+Each commit will be accompanied with unit tests, and Tempest functional tests.
328
+
329
+
330
+Documentation Impact
331
+====================
332
+
333
+A set of documentation for this architecture will be required. We should also
334
+provide a developer guide for creating a new bay driver and updating existing
335
+ones.
336
+
337
+
338
+References
339
+==========
340
+
341
+`Using Stevedore in your Application
342
+<http://docs.openstack.org/developer/stevedore/tutorial/index.html/>`_.
343
+
344
+.. _stevedore: http://docs.openstack.org/developer/stevedore/

+ 458
- 0
specs/implemented/container-networking-model.rst View File

@@ -0,0 +1,458 @@
1
+..
2
+ This work is licensed under a Creative Commons Attribution 3.0 Unported
3
+ License.
4
+
5
+ http://creativecommons.org/licenses/by/3.0/legalcode
6
+
7
+=================================
8
+Magnum Container Networking Model
9
+=================================
10
+
11
+Launchpad Blueprint:
12
+
13
+https://blueprints.launchpad.net/magnum/+spec/extensible-network-model
14
+
15
+For Magnum to prosper, the project must support a range of networking tools
16
+and techniques, while maintaining a simple, developer-focused user
17
+experience. The first step in achieving this goal is to standardize the
18
+process of allocating networking to containers, while providing an
19
+abstraction for supporting various networking capabilities through
20
+pluggable back-end implementations. This document recommends using Docker's
21
+libnetwork library to implement container networking abstractions and
22
+plugins. Since libnetwork is not a standard and the container ecosystem
23
+is rapidly evolving, the Magnum community should continue evaluating
24
+container networking options on a frequent basis.
25
+
26
+Problem Description
27
+===================
28
+
29
+The container networking ecosystem is undergoing rapid changes. The
30
+networking tools and techniques used in today's container deployments are
31
+different than twelve months ago and will continue to evolve. For example,
32
+Flannel [6]_, Kubernetes preferred networking implementation, was initially
33
+released in July of 2014 and was not considered preferred until early 2015.
34
+
35
+Furthermore, the various container orchestration engines have not
36
+standardized on a container networking implementation and may never. For
37
+example, Flannel is the preferred container networking implementation for
38
+Kubernetes but not for Docker Swarm. Each container networking implementation
39
+comes with its own API abstractions, data model, tooling, etc.. Natively
40
+supporting each container networking implementation can be a burden on the
41
+Magnum community and codebase. By supporting only a subset of container
42
+networking implementations, the project may not be widely adopted or may
43
+provide a suboptimal user experience.
44
+
45
+Lastly, Magnum has limited support for advanced container networking
46
+functionality. Magnum instantiates container networks behind the scenes
47
+through Heat templates, exposing little-to-no user configurability. Some
48
+users require the ability to customize their container environments,
49
+including networking details. However, networking needs to "just work" for
50
+users that require no networking customizations.
51
+
52
+Roles
53
+-----
54
+
55
+The following are roles that the Magnum Container Networking Model takes
56
+into consideration. Roles are an important reference point when creating
57
+user stories. This is because each role provides different functions and
58
+has different requirements.
59
+
60
+1. Cloud Provider (CP): Provides standard OpenStack cloud infrastructure
61
+   services, including the Magnum service.
62
+
63
+2. Container Service Provider (CSP): Uses Magnum to deliver
64
+   Containers-as-a-Service (CaaS) to users. CSPs are a consumer of CP
65
+   services and a CaaS provider to users.
66
+
67
+3. Users: Consume Magnum services to provision and manage clustered
68
+   container environments and deploy apps within the container clusters.
69
+
70
+The container ecosystem focuses on the developer user type. It is imperative
71
+that the Magnum Container Networking Model meets the need of this user type.
72
+
73
+These roles are not mutually exclusive. For example:
74
+
75
+1. A CP can also be a CSP. In this case, the CP/CSP provisions and manages
76
+   standard OpenStack services, the Magnum service, and provides CaaS
77
+   services to users.
78
+
79
+2. A User can also be a CSP. In this case, the user provisions their own
80
+   baymodels, bays, etc. from the CP.
81
+
82
+Definitions
83
+-----------
84
+
85
+COE
86
+  Container Orchestration Engine
87
+
88
+Baymodel
89
+  An object that stores template information about the bay which is
90
+  used to create new bays consistently.
91
+
92
+Bay
93
+  A Magnum resource that includes at least one host to run containers on,
94
+  and a COE to manage containers created on hosts within the bay.
95
+
96
+Pod
97
+  Is the smallest deployable unit that can be created, scheduled, and
98
+  managed within Kubernetes.
99
+
100
+Additional Magnum definitions can be found in the Magnum Developer
101
+documentation [2]_.
102
+
103
+Use Cases
104
+----------
105
+
106
+This document does not intend to address each use case. The use cases are
107
+provided as reference for the long-term development of the Magnum Container
108
+Networking Model.
109
+
110
+As a User:
111
+
112
+1. I need to easily deploy containerized apps in an OpenStack cloud.
113
+   My user experience should be similar to how I deploy containerized apps
114
+   outside of an OpenStack cloud.
115
+
116
+2. I need to have containers communicate with vm-based apps that use
117
+   OpenStack networking.
118
+
119
+3. I need the option to preserve the container's IP address so I can
120
+   manage containers by IP's, not just ports.
121
+
122
+4. I need to block unwanted traffic to/from my containerized apps.
123
+
124
+5. I need the ability for my containerized apps to be highly available.
125
+
126
+6. I need confidence that my traffic is secure from other tenants traffic.
127
+
128
+As a CSP:
129
+
130
+1. I need to easily deploy a bay for consumption by users. The bay must
131
+   support the following:
132
+
133
+   A. One or more hosts to run containers.
134
+   B. The ability to choose between virtual or physical hosts to run
135
+      containers.
136
+   C. The ability to automatically provision networking to containers.
137
+
138
+2. I need to provide clustering options that support different
139
+   container/image, formats and technologies.
140
+
141
+3. After deploying my initial cluster, I need the ability to provide ongoing
142
+   management, including:
143
+
144
+   A. The ability to add/change/remove networks that containers connect to.
145
+   B. The ability to add/change/remove nodes within the cluster.
146
+
147
+4. I need to deploy a Bay without admin rights to OpenStack services.
148
+
149
+5. I need the freedom to choose different container networking tools and
150
+   techniques offered by the container ecosystem beyond OpenStack.
151
+
152
+As a CP:
153
+
154
+1. I need to easily and reliably add the Magnum service to my existing
155
+   OpenStack cloud environment.
156
+
157
+2. I need to easily manage (monitor, troubleshoot, etc..) the Magnum
158
+   service. Including the ability to mirror ports to capture traffic
159
+   for analysis.
160
+
161
+3. I need to make the Magnum services highly-available.
162
+
163
+4. I need to make Magnum services highly performant.
164
+
165
+5. I need to easily scale-out Magnum services as needed.
166
+
167
+6. I need Magnum to be robust regardless of failures within the container
168
+   orchestration engine.
169
+
170
+Proposed Changes
171
+================
172
+
173
+1. Currently, Magnum supports Flannel [6]_ as the only multi-host container
174
+   networking implementation. Although Flannel has become widely accepted
175
+   for providing networking capabilities to Kubernetes-based container
176
+   clusters, other networking tools exist and future tools may develop.
177
+
178
+   This document proposes extending Magnum to support specifying a
179
+   container networking implementation through a combination of user-facing
180
+   baymodel configuration flags. Configuration parameters that are common
181
+   across Magnum or all networking implementations will be exposed as unique
182
+   flags. For example, a flag named network-driver can be used to instruct
183
+   Magnum which network driver to use for implementing a baymodel
184
+   container/pod network. network driver examples may include:
185
+
186
+     flannel, weave, calico, midonet, netplugin, etc..
187
+
188
+   Here is an example of creating a baymodel that uses Flannel as the
189
+   network driver: ::
190
+
191
+     magnum baymodel-create --name k8sbaymodel \
192
+                            --image-id fedora-21-atomic-5 \
193
+                            --keypair-id testkey \
194
+                            --external-network-id 1hsdhs88sddds889 \
195
+                            --dns-nameserver 8.8.8.8 \
196
+                            --flavor-id m1.small \
197
+                            --docker-volume-size 5 \
198
+                            --coe kubernetes \
199
+                            --network-driver flannel
200
+
201
+   If no network-driver parameter is supplied by the user, the baymodel is
202
+   created using the default network driver of the specified Magnum COE.
203
+   Each COE must support a default network driver and each driver must
204
+   provide reasonable default configurations that allow users to instantiate
205
+   a COE without supplying labels. The default network driver for each COE
206
+   should be consistent with existing Magnum default settings. Where current
207
+   defaults do not exist, the defaults should be consistent with upstream
208
+   network driver projects.
209
+
210
+2. Each network driver supports a range of configuration parameters that
211
+   should be observed by Magnum. This document suggests using an attribute
212
+   named "labels" for supplying driver-specific configuration parameters.
213
+   Labels consist of one or more arbitrary key/value pairs. Here is an
214
+   example of using labels to change default settings of the Flannel
215
+   network driver: ::
216
+
217
+     magnum baymodel-create --name k8sbaymodel \
218
+                            --image-id fedora-21-atomic-5 \
219
+                            --keypair-id testkey \
220
+                            --external-network-id ${NIC_ID} \
221
+                            --dns-nameserver 8.8.8.8 \
222
+                            --flavor-id m1.small \
223
+                            --docker-volume-size 5 \
224
+                            --coe kubernetes \
225
+                            --network-driver flannel \
226
+                            --labels flannel_network_cidr=10.0.0.0/8,\
227
+                                     flannel_network_subnetlen=22,\
228
+                                     flannel_backend=vxlan
229
+
230
+   With Magnum's current implementation, this document would support
231
+   labels for the Kubernetes COE type. However, labels are applicable
232
+   beyond Kubernetes, as the Docker daemon, images and containers now
233
+   support labels as a mechanism for providing custom metadata. The labels
234
+   attribute within Magnum should be extended beyond Kubernetes pods, so a
235
+   single mechanism can be used to pass arbitrary metadata throughout the
236
+   entire system. A blueprint [2]_ has been registered to expand the scope
237
+   of labels for Magnum. This document intends on adhering to the
238
+   expand-labels-scope blueprint.
239
+
240
+   Note: Support for daemon-labels was added in Docker 1.4.1. Labels for
241
+   containers and images were introduced in Docker 1.6.0
242
+
243
+   If the --network-driver flag is specified without any labels, default
244
+   configuration values of the driver will be used by the baymodel. These
245
+   defaults are set within the Heat template of the associated COE. Magnum
246
+   should ignore label keys and/or values not understood by any of the
247
+   templates during the baymodel operation.
248
+
249
+   Magnum will continue to CRUD bays in the same way:
250
+
251
+     magnum bay-create --name k8sbay --baymodel k8sbaymodel --node-count 1
252
+
253
+3. Update python-magnumclient to understand the new Container Networking
254
+   Model attributes. The client should also be updated to support passing
255
+   the --labels flag according to the expand-labels-scope blueprint [2]_.
256
+
257
+4. Update the conductor template definitions to support the new Container
258
+   Networking Model attributes.
259
+
260
+5. Refactor Heat templates to support the Magnum Container Networking Model.
261
+   Currently, Heat templates embed Flannel-specific configuration within
262
+   top-level templates. For example, the top-level Kubernetes Heat
263
+   template [8]_ contains the flannel_network_subnetlen parameter. Network
264
+   driver specific configurations should be removed from all top-level
265
+   templates and instead be implemented in one or more template fragments.
266
+   As it relates to container networking, top-level templates should only
267
+   expose the labels and generalized parameters such as network-driver.
268
+   Heat templates, template definitions and definition entry points should
269
+   be suited for composition, allowing for a range of supported labels. This
270
+   document intends to follow the refactor-heat-templates blueprint [3]_ to
271
+   achieve this goal.
272
+
273
+6. Update unit and functional tests to support the new attributes of the
274
+   Magnum Container Networking Model.
275
+
276
+7. The spec will not add support for natively managing container networks.
277
+   Due to each network driver supporting different API operations, this
278
+   document suggests that Magnum not natively manage container networks at
279
+   this time and instead leave this job to native tools. References [4]_ [5]_
280
+   [6]_ [7]_.
281
+   provide additional details to common labels operations.
282
+
283
+8. Since implementing the expand-labels-scope blueprint [2]_ may take a while,
284
+   exposing network functionality through baymodel configuration parameters
285
+   should be considered as an interim solution.
286
+
287
+Alternatives
288
+------------
289
+
290
+
291
+1. Observe all networking configuration parameters, including labels
292
+   within a configuration file instead of exposing the labels attribute to
293
+   the user.
294
+
295
+2. Only support a single networking implementation such as Flannel. Flannel
296
+   is currently supported for the Kubernetes COE type. It can be ported to
297
+   support the swarm COE type.
298
+
299
+3. Add support for managing container networks. This will require adding
300
+   abstractions for each supported network driver or creating an
301
+   abstraction layer that covers all possible network drivers.
302
+
303
+4. Use the Kuryr project [10]_ to provide networking to Magnum containers.
304
+   Kuryr currently contains no documentation or code, so this alternative
305
+   is highly unlikely if the Magnum community requires a pluggable
306
+   container networking implementation in the near future. However, Kuryr
307
+   could become the long-term solution for container networking within
308
+   OpenStack. A decision should be made by the Magnum community whether
309
+   to move forward with Magnum's own container networking model or to wait
310
+   for Kuryr to mature. In the meantime, this document suggests the Magnum
311
+   community become involved in the Kuryr project.
312
+
313
+Data Model Impact
314
+-----------------
315
+
316
+This document adds the labels and network-driver attribute to the baymodel
317
+database table. A migration script will be provided to support the attribute
318
+being added. ::
319
+
320
+    +-------------------+-----------------+---------------------------------------------+
321
+    |    Attribute      |     Type        |             Description                     |
322
+    +===================+=================+=============================================+
323
+    |     labels        | JSONEncodedDict | One or more arbitrary key/value pairs       |
324
+    +-------------------+-----------------+---------------------------------------------+
325
+    |    network-driver |    string       | Container networking backend implementation |
326
+    +-------------------+-----------------+---------------------------------------------+
327
+
328
+REST API Impact
329
+---------------
330
+
331
+This document adds the labels and network-driver attribute to the BayModel
332
+API class. ::
333
+
334
+    +-------------------+-----------------+---------------------------------------------+
335
+    |    Attribute      |     Type        |             Description                     |
336
+    +===================+=================+=============================================+
337
+    |     labels        | JSONEncodedDict | One or more arbitrary key/value pairs       |
338
+    +-------------------+-----------------+---------------------------------------------+
339
+    |    network-driver |    string       | Container networking backend implementation |
340
+    +-------------------+-----------------+---------------------------------------------+
341
+
342
+Security Impact
343
+---------------
344
+
345
+Supporting more than one network driver increases the attack
346
+footprint of Magnum.
347
+
348
+Notifications Impact
349
+--------------------
350
+
351
+None
352
+
353
+Other End User Impact
354
+---------------------
355
+
356
+Most end users will never use the labels configuration flag
357
+and simply use the default network driver and associated
358
+configuration options. For those that wish to customize their
359
+container networking environment, it will be important to understand
360
+what network-driver and labels are supported, along with their
361
+associated configuration options, capabilities, etc..
362
+
363
+Performance Impact
364
+------------------
365
+
366
+Performance will depend upon the chosen network driver and its
367
+associated configuration. For example, when creating a baymodel with
368
+"--network-driver flannel" flag, Flannel's default configuration
369
+will be used. If the default for Flannel is an overlay networking technique
370
+(i.e. VXLAN), then networking performance will be less than if Flannel used
371
+the host-gw configuration that does not perform additional packet
372
+encapsulation to/from containers. If additional performance is required
373
+when using this driver, Flannel's host-gw configuration option could be
374
+exposed by the associated Heat template and instantiated through the labels
375
+attribute.
376
+
377
+Other Deployer Impact
378
+---------------------
379
+
380
+Currently, container networking and OpenStack networking are different
381
+entities. Since no integration exists between the two, deployers/operators
382
+will be required to manage each networking environment individually.
383
+However, Magnum users will continue to deploy baymodels, bays, containers,
384
+etc. without having to specify any networking parameters. This will be
385
+accomplished by setting reasonable default parameters within the Heat
386
+templates.
387
+
388
+Developer impact
389
+----------------
390
+
391
+None
392
+
393
+Implementation
394
+==============
395
+
396
+Assignee(s)
397
+-----------
398
+
399
+Primary assignee:
400
+Daneyon Hansen (danehans)
401
+
402
+Other contributors:
403
+Ton Ngo (Tango)
404
+Hongbin Lu (hongbin)
405
+
406
+Work Items
407
+----------
408
+
409
+1. Extend the Magnum API to support new baymodel attributes.
410
+2. Extend the Client API to support new baymodel attributes.
411
+3. Extend baymodel objects to support new baymodel attributes. Provide a
412
+   database migration script for adding attributes.
413
+4. Refactor Heat templates to support the Magnum Container Networking Model.
414
+5. Update Conductor template definitions and definition entry points to
415
+   support Heat template refactoring.
416
+6. Extend unit and functional tests to support new baymodel attributes.
417
+
418
+Dependencies
419
+============
420
+
421
+Although adding support for these new attributes does not depend on the
422
+following blueprints, it's highly recommended that the Magnum Container
423
+Networking Model be developed in concert with the blueprints to maintain
424
+development continuity within the project.
425
+
426
+1. Common Plugin Framework Blueprint [1]_.
427
+
428
+2. Expand the Scope of Labels Blueprint [9]_.
429
+
430
+3. Refactor Heat Templates, Definitions and Entry Points Blueprint [3]_.
431
+
432
+Testing
433
+=======
434
+
435
+Each commit will be accompanied with unit tests. There will also be
436
+functional tests which will be used as part of a cross-functional gate
437
+test for Magnum.
438
+
439
+Documentation Impact
440
+====================
441
+
442
+The Magnum Developer Quickstart document will be updated to support the
443
+configuration flags introduced by this document. Additionally, background
444
+information on how to use these flags will be included.
445
+
446
+References
447
+==========
448
+
449
+.. [1] https://blueprints.launchpad.net/magnum/+spec/common-plugin-framework
450
+.. [2] http://docs.openstack.org/developer/magnum/
451
+.. [3] https://blueprints.launchpad.net/magnum/+spec/refactor-heat-templates
452
+.. [4] https://github.com/docker/libnetwork/blob/master/docs/design.md
453
+.. [5] https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/design/networking.md
454
+.. [6] https://github.com/coreos/flannel
455
+.. [7] https://github.com/coreos/rkt/blob/master/Documentation/networking.md
456
+.. [8] https://github.com/openstack/magnum/blob/master/magnum/templates/kubernetes/kubecluster.yaml
457
+.. [9] https://blueprints.launchpad.net/magnum/+spec/expand-labels-scope
458
+.. [10] https://github.com/openstack/kuryr

+ 500
- 0
specs/implemented/container-volume-integration-model.rst View File

@@ -0,0 +1,500 @@
1
+..
2
+   This work is licensed under a Creative Commons Attribution 3.0 Unported
3
+ License.
4
+
5
+ http://creativecommons.org/licenses/by/3.0/legalcode
6
+
7
+=========================================
8
+Magnum Container Volume Integration Model
9
+=========================================
10
+
11
+Launchpad Blueprint:
12
+
13
+https://blueprints.launchpad.net/magnum/+spec/magnum-integrate-with-cinder
14
+
15
+Storage is a key part of any computing system. Containers in particular have
16
+the interesting characteristic that local storage by default is ephemeral:
17
+any changes to the file system disappear when the container is deleted. This
18
+introduces the need for persistent storage to retain and share data between
19
+containers, and this is currently an active area of development in all
20
+container orchestration engines (COE).
21
+
22
+As the component in OpenStack for managing COE's, Magnum must fully enable the
23
+features for persistent storage in the COE's. To achieve this goal, we propose
24
+in this specification to generalize the process for utilizing persistent
25
+storage with containers so that it is applicable for different bay types.
26
+Despite the complexity, we aim to maintain a good user experience by a simple
27
+abstraction for working with various volume capabilities. For the rest of this
28
+specification, we will use the term Volume to refer to persistent storage, and
29
+Volume Driver as the plugin in a COE to support the particular persistent
30
+storage.
31
+
32
+Problem Description
33
+===================
34
+
35
+Containers requires full life cycle management such as create, run, stop,
36
+delete,... and a key operation is to manage the data - making the data
37
+persistent, reusing the data, sharing data between containers, etc.
38
+In this area, the support for container volume is undergoing rapid change
39
+to bring more integration with open source software and third party
40
+storage solutions.
41
+
42
+A clear evidence of this growth is the many plugin volume drivers [1]_ [4]_
43
+such as NFS, GlusterFS, EBS, etc. They provide different functionality, use
44
+different storage backend and have different requirements. The COE's are
45
+naturally motivated to be flexible and allow as many choices as possible for
46
+the users with respect to the storage backend. Since Magnum's role is to
47
+support the COE's within OpenStack, the goal is to be transparent and enable
48
+these same storage backends for the COE's through the COE's lifecycle
49
+operation.
50
+
51
+Currently, Magnum provides limited support for managing container volume
52
+. The only option available is to specify the docker-volume-size for a
53
+pre-allocated block storage in the COE to host the containers. Magnum
54
+instantiates container volumes through Heat templates, exposing no other
55
+mechanism to configure and operate on volumes. In practice, some users
56
+require the ability to manage volumes easily in the COEs .
57
+
58
+Note that we are not proposing to create a new volume management interface
59
+in Magnum. After the users create the baymodel and bays, we assume that the
60
+users would manage the volumes through existing techniques:
61
+
62
+1. Log in to the COE, use COE specific CLI or GUI to manage volumes.
63
+
64
+2. Use native tools to manage volumes.
65
+
66
+The initial implementation will focus on OpenStack Cinder integration; as
67
+other alternatives become available, contributors are welcome through
68
+3rd-party maintained projects.
69
+
70
+
71
+Definitions
72
+-----------
73
+
74
+COE
75
+  Container Orchestration Engine
76
+
77
+Baymodel
78
+  An object that stores template information about the bay which is
79
+  used to create new bays consistently.
80
+
81
+Bay
82
+  A Magnum resource that includes at least one host to run containers on,
83
+  and a COE to manage containers created on hosts within the bay.
84
+
85
+Pod
86
+  Is the smallest deployable unit that can be created, scheduled, and
87
+  managed within Kubernetes.
88
+
89
+Volume
90
+  storage that is persistent
91
+
92
+Volume plugin
93
+  COE specific code that supports the functionality of a type of volume.
94
+
95
+Additional Magnum definitions can be found in the Magnum Developer
96
+documentation[7]_ .
97
+
98
+Use Cases
99
+----------
100
+
101
+This document does not intend to address all use cases. We list below a number
102
+of use cases for 3 different roles; they should be useful as reference for the
103
+long-term development of the Magnum Container Volume Integration.
104
+
105
+As a User:
106
+
107
+As mentioned above, our goal is to preserve the user experience specific to
108
+the COE in managing the volumes.  Therefore, we expect the use cases for the
109
+users will be fulfilled by the COE's themselves; Magnum will simply ensure
110
+that the necessary supports are in place.
111
+
112
+1. I need to easily create volume for containers to use as persistent
113
+   data store.
114
+
115
+2. I need the ability to create and mount a data volume container for cross
116
+   container sharing.
117
+
118
+3. I need to mount a host directory as a data volume.
119
+
120
+4. I need to easily attach a known volume to container to use the
121
+   existing data.
122
+
123
+5. I need the ability to delete the volume.
124
+
125
+6. I need to list and view the details of the volume
126
+
127
+7. I need to modify the volume.
128
+
129
+
130
+As a CSP:
131
+
132
+1. I need to easily deploy a bay for consumption by users. The bay must
133
+   support the following:
134
+
135
+   A. One or more hosts to run containers.
136
+   B. The ability to choose between virtual or physical hosts to
137
+      run containers.
138
+   C. The ability to automatically enable volume plugins to containers.
139
+
140
+2. I need to provide clustering options that support different volume plugins
141
+   per COE.
142
+
143
+3. After deploying my initial cluster, I need the ability to provide lifecycle
144
+   management, including:
145
+
146
+   A. The ability to add/remove volumes that containers used.
147
+   B. The ability to add/remove nodes within the cluster with the necessary
148
+      adjustment to the volumes
149
+
150
+As a CP:
151
+
152
+1. I need to easily and reliably add the Magnum service to my existing
153
+   OpenStack cloud environment.
154
+
155
+2. I need to make the Magnum services highly-available.
156
+
157
+3. I need to make Magnum services highly performant.
158
+
159
+4. I need to easily scale-out Magnum services as needed.
160
+
161
+
162
+Proposed Changes
163
+================
164
+
165
+We propose extending Magnum as follows.
166
+
167
+
168
+
169
+1. The new attribute volume-driver for a baymodel specifies the volume backend
170
+   driver to use when deploying a bay.
171
+
172
+  Volume drivers may include:
173
+
174
+     rexray, flocker, nfs, glusterfs, etc..
175
+
176
+  Here is an example of creating a Docker Swarm baymodel that uses rexray [5]_
177
+  [6]_ as the volume driver: ::
178
+
179
+
180
+     magnum baymodel-create --name swarmbaymodel \
181
+                            --image-id fedora-21-atomic-5 \
182
+                            --keypair-id testkey \
183
+                            --external-network-id 1hsdhs88sddds889 \
184
+                            --dns-nameserver 8.8.8.8 \
185
+                            --flavor-id m1.small \
186
+                            --docker-volume-size 5 \
187
+                            --coe swarm\
188
+                            --network-driver flannel \
189
+                            --volume-driver rexray
190
+
191
+  When a Swarm bay is created with this bay model, the REX-Ray storage
192
+  subsystem will be installed, configured and started on the Swarm nodes,
193
+  then the REX-Ray volume plugin will be registered in Docker. When a container
194
+  is created with rexray as the volume driver, the container will have full
195
+  access to the REX-Ray capabilities such as creating, mounting, deleting
196
+  volumes [6]_. REX-Ray in turn will interface with Cinder to manage the
197
+  volumes in OpenStack.
198
+
199
+  Here is an example of creating a Kubernetes baymodel that uses Cinder [2]_
200
+  [3]_ as the volume driver: ::
201
+
202
+    magnum baymodel-create --name k8sbaymodel \
203
+                            --image-id fedora-21-atomic-5 \
204
+                            --keypair-id testkey \
205
+                            --external-network-id 1hsdhs88sddds889 \
206
+                            --dns-nameserver 8.8.8.8 \
207
+                            --flavor-id m1.small \
208
+                            --docker-volume-size 5 \
209
+                            --coe kubernetes\
210
+                            --network-driver flannel \
211
+                            --volume-driver cinder
212
+
213
+  When the Kubernetes bay is created using this bay model, the kubelet will be
214
+  configured so that an existing Cinder volume can be mounted in a pod by
215
+  specifying the volume ID in the pod manifest as follows: ::
216
+
217
+    volumes:
218
+    - name: mysql-persistent-storage
219
+      cinder:
220
+        volumeID: bd82f7e2-wece-4c01-a505-4acf60b07f4a
221
+        fsType: ext4
222
+
223
+
224
+
225
+Here is an example of creating a mesos baymodel that uses rexray as the
226
+volume driver: ::
227
+
228
+    magnum baymodel-create --name mesosbaymodel \
229
+                            --image-id ubuntu-mesos\
230
+                            --keypair-id testkey \
231
+                            --external-network-id 1hsdhs88sddds889 \
232
+                            --dns-nameserver 8.8.8.8 \
233
+                            --flavor-id m1.small \
234
+                            --coe mesos\
235
+                            --network-driver docker \
236
+                            --volume-driver rexray
237
+
238
+When the mesos bay is created using this bay model, the mesos bay will be
239
+configured so that an existing Cinder volume can be mounted in a container
240
+by configuring the parameters to mount the cinder volume in the json file. ::
241
+
242
+    "parameters": [
243
+       { "key": "volume-driver", "value": "rexray" },
244
+       { "key": "volume", "value": "redisdata:/data" }
245
+    ]
246
+
247
+If no volume-driver parameter is supplied by the user, the baymodel is
248
+created using the default volume driver of the particular COE.
249
+Magnum will provide a default volume driver for each COE as well as the
250
+reasonable default configuration for each driver so that
251
+users can instantiate a COE without supplying a volume driver and
252
+associated labels. Generally the defaults should be consistent with upstream
253
+volume driver projects.
254
+
255
+2. Each volume driver supports a range of configuration parameters that are
256
+   handled by the "labels" attribute.
257
+
258
+  Labels consist of one or more arbitrary key/value pairs.
259
+  Here is an example of using labels to choose ¡°storage-provider¡± for
260
+  rexray driver.
261
+  Volume driver: ::
262
+
263
+     magnum baymodel-create --name k8sbaymodel \
264
+                            --image-id fedora-21-atomic-5 \
265
+                            --keypair-id testkey \
266
+                            --external-network-id ${NIC_ID} \
267
+                            --dns-nameserver 8.8.8.8 \
268
+                            --flavor-id m1.small \
269
+                            --docker-volume-size 5 \
270
+                            --coe kubernetes \
271
+                            --volume-driver rexray \
272
+                            --labels storage-provider=openstack \
273
+                                [, key2=value2...]
274
+
275
+
276
+  If the --volume-driver flag is specified without any labels, default
277
+  configuration values of the driver will be used by the baymodel.
278
+
279
+  Magnum will validate the labels together with the driver specified before
280
+  creating the bay and will return an error if the validation fails.
281
+
282
+  Magnum will continue to CRUD bays in the same way:
283
+
284
+     magnum bay-create --name k8sbay --baymodel k8sbaymodel --node-count 1
285
+
286
+3. Update python-magnumclient to handle the new container volume-
287
+   driver attributes.
288
+
289
+4. Update the conductor template definitions to support the new container
290
+   volume-driver model attributes.
291
+
292
+5. Refactor Heat templates to support the Magnum volume driver plugin.
293
+   Configurations specific to volume drivers should be
294
+   implemented in one or more template fragments.
295
+   Top-level templates should only
296
+   expose the labels and generalized parameters such as volume-driver.
297
+   Heat templates, template definitions and definition entry points should
298
+   be designed for composition, allowing for a range of supported labels.
299
+
300
+6. Update unit and functional tests to support the new attributes of the
301
+   Magnum container volume driver.
302
+
303
+7. Preserve the user experience by ensuring that any operation on volume will
304
+   be identical between a COE deployed by Magnum and a COE deployed by other
305
+   methods.
306
+
307
+
308
+Alternatives
309
+------------
310
+
311
+1. Without the support proposed, the user will need to manually enable and
312
+   configure the volume plugin.  This will require the user to log into the
313
+   nodes in the cluster and understand the low level infrastructure of the
314
+   cluster as deployed by the heat templates.
315
+2. We can add full support for managing container volume in Magnum user
316
+   interface itself. This will require adding abstractions for each supported
317
+   COE volume plugins driver or creating an abstraction layer that covers all
318
+   possible COE volume drivers.
319
+
320
+Data Model Impact
321
+-----------------
322
+
323
+This document adds the volume-driver attribute to the baymodel
324
+database table. A migration script will be provided to support the attribute
325
+being added. ::
326
+
327
+    +-------------------+-----------------+---------------------------------------------+
328
+    |    Attribute      |     Type        |             Description                     |
329
+    +===================+=================+=============================================+
330
+    +-------------------+-----------------+---------------------------------------------+
331
+    | volume-driver     |    string       | Container volume backend implementation     |
332
+    +-------------------+-----------------+---------------------------------------------+
333
+
334
+REST API Impact
335
+---------------
336
+
337
+This document adds volume-driver attribute to the BayModel
338
+API class. ::
339
+
340
+    +-------------------+-----------------+---------------------------------------------+
341
+    |    Attribute      |     Type        |             Description                     |
342
+    +===================+=================+=============================================+
343
+    +-------------------+-----------------+---------------------------------------------+
344
+    | volume-driver     |    string       | Container volume backend implementation     |
345
+    +-------------------+-----------------+---------------------------------------------+
346
+
347
+Security Impact
348
+---------------
349
+
350
+Supporting volume drivers can potentially increase the attack surface
351
+on containers.
352
+
353
+Notifications Impact
354
+--------------------
355
+
356
+None
357
+
358
+Other End User Impact
359
+---------------------
360
+
361
+There is no impact if the user does not use a volume driver.
362
+We anticipate that most users would not use the labels for volume
363
+and would simply use the default volume driver and associated
364
+configuration options. For those who wish to customize their
365
+container volume driver environment, it will be important to understand
366
+what volume-driver and labels are supported, along with their
367
+associated configuration options, capabilities, etc..
368
+
369
+Performance Impact
370
+------------------
371
+
372
+There is no impact if the user does not use a volume driver.
373
+When a volume driver is used, the performance will depend upon the specific
374
+volume driver and its associated storage backends.  For example, Kubernetes
375
+supports Cinder and awsEBS; the two types of volumes can have different
376
+performance.
377
+
378
+An example of the second case is a docker swarm bay with
379
+"--volume-driver rexray" where the rexray driver's storage provider is
380
+OpenStack cinder. The resulting performance for container may vary depending
381
+on the storage backends. As listed in [8]_ , Cinder supports many storage
382
+drivers. Besides this, different container volume driver can also cause
383
+performance variance.
384
+
385
+
386
+High-Availability Impact
387
+------------------------------
388
+
389
+
390
+
391
++-----------------+--------------------+--------------------------+
392
+|       COE       |      Master HA     |   Pod/Container/App  HA  |
393
++=================+====================+==========================+
394
+|    Kubernetes   |         No         |          Yes             |
395
++-----------------+--------------------+--------------------------+
396
+|   Docker Swarm  |         No         |          Yes             |
397
++-----------------+--------------------+--------------------------+
398
+|       Mesos     |         No         |          No              |
399
++-----------------+--------------------+--------------------------+
400
+
401
+"No" means that  the volume doesn't affect the high-availability.
402
+"Yes" means that the volume affect the high-availability.
403
+
404
+Kubernetes does support pod high-availability through the replication
405
+controller, however, this doesn't work when a pod with volume attached
406
+fails. Refer the link [11]_  for details.
407
+
408
+Docker swarm doesn't support the containers rescheduling when a node fails, so
409
+volume can not be automatically detached by volume driver. Refer the
410
+link [12]_  for details.
411
+
412
+Mesos supports the application high-availability when a node fails, which
413
+means application would be started on new node, and volumes can be
414
+automatically attached to the new node by the volume driver.
415
+
416
+Other Deployer Impact
417
+---------------------
418
+
419
+Currently, both Kubernetes and Docker community have supported some volume
420
+plugins. The changes proposed will enable these volume plugins in Magnum.
421
+However, Magnum users will be able to continue to deploy baymodels, bays,
422
+containers, etc. without having to specify any parameters for volume.
423
+This will be accomplished by setting reasonable default parameters within
424
+the Heat templates.
425
+
426
+Developer impact
427
+----------------
428
+
429
+None
430
+
431
+Implementation
432
+==============
433
+
434
+Assignee(s)
435
+-----------
436
+
437
+Primary assignee:
438
+
439
+- Kai Qiang Wu (Kennan)
440
+
441
+Other contributors:
442
+
443
+- Qun Wang (wangqun)
444
+- Ton Ngo (Tango)
445
+
446
+
447
+Work Items
448
+----------
449
+
450
+1. Extend the Magnum API to support new baymodel attributes.
451
+2. Extend the Client API to support new baymodel attributes.
452
+3. Extend baymodel objects to support new baymodel attributes. Provide a
453
+   database migration script for adding attributes.
454
+4. Refactor Heat templates to support the Magnum container volume driver.
455
+5. Update Conductor template definitions and definition entry points to
456
+   support Heat template refactoring.
457
+6. Extend unit and functional tests to support new baymodel attributes.
458
+7. Document how to use the volume drivers with examples.
459
+
460
+Dependencies
461
+============
462
+
463
+Although adding support for these new attributes does not depend on the
464
+following blueprints, it's highly recommended that the Magnum Container
465
+Networking Model be developed in concert with the blueprints to maintain
466
+development continuity within the project.
467
+https://blueprints.launchpad.net/magnum/+spec/ubuntu-image-build
468
+
469
+Kubernetes with cinder support need Kubernetes version >= 1.1.1
470
+Swarm need version >= 1.8.3, as Kubernetes 1.1.1 upgraded to that version
471
+
472
+Testing
473
+=======
474
+
475
+Each commit will be accompanied with unit tests. There will also be
476
+functional tests which will be used as part of a cross-functional gate
477
+test for Magnum.
478
+
479
+Documentation Impact
480
+====================
481
+
482
+The Magnum Developer Quickstart document will be updated to support the
483
+configuration flags introduced by this document. Additionally, background
484
+information on how to use these flags will be included.
485
+
486
+References
487
+==========
488
+
489
+.. [1] http://kubernetes.io/v1.1/docs/user-guide/volumes.html
490
+.. [2] http://kubernetes.io/v1.1/examples/mysql-cinder-pd/
491
+.. [3] https://github.com/kubernetes/kubernetes/tree/master/pkg/volume/cinder
492
+.. [4] http://docs.docker.com/engine/extend/plugins/
493
+.. [5] https://github.com/emccode/rexray
494
+.. [6] http://rexray.readthedocs.org/en/stable/user-guide/storage-providers/openstack
495
+.. [7] http://docs.openstack.org/developer/magnum/
496
+.. [8] http://docs.openstack.org/liberty/config-reference/content/section_volume-drivers.html
497
+.. [9] http://docs.openstack.org/admin-guide-cloud/blockstorage_multi_backend.html#
498
+.. [10] http://docs.openstack.org/user-guide-admin/dashboard_manage_volumes.html
499
+.. [11] https://github.com/kubernetes/kubernetes/issues/14642
500
+.. [12] https://github.com/docker/swarm/issues/1488

+ 400
- 0
specs/implemented/containers-service.rst View File

@@ -0,0 +1,400 @@
1
+..
2
+   This work is licensed under a Creative Commons Attribution 3.0 Unported
3
+ License.
4
+
5
+ http://creativecommons.org/licenses/by/3.0/legalcode
6
+
7
+==================
8
+Containers Service
9
+==================
10
+
11
+Launchpad blueprint:
12
+
13
+https://blueprints.launchpad.net/nova/+spec/containers-service
14
+
15
+Containers share many features in common with Nova instances. For the common
16
+features, virt drivers for Nova can be used to surface basic instance
17
+functionality. For features that go beyond what can be naturally fit within
18
+a virt driver, we propose a new API service that allows for advanced features
19
+to be added without conflating the worlds of instances and containers.
20
+
21
+Some examples of containers specific features are setting of shell environment
22
+variables, and accepting a shell command to execute at runtime. Capturing the
23
+STDIO of the process(es) within a container, and tracking the return status
24
+of processes are all beyond the scope of what was contemplated for Nova. All
25
+of these features will be implemented in the Containers Service.
26
+
27
+
28
+Problem description
29
+===================
30
+Container technology is rapidly gaining popularity as a way to bundle and
31
+deploy applications. Recognizing and adapting to this trend will position
32
+OpenStack to be useful not only to clouds that employ bare metal and virtual
33
+machine instances, but can remain competitive in offering container services
34
+as well.
35
+
36
+Nova's concepts of an instance, and the actions that may be taken on it do not
37
+match completely with containers.
38
+
39
+Use cases
40
+---------
41
+1. App Consolidation. End-user wants to run multiple small applications in
42
+   separate operating system environments, but wants to optimize for efficiency
43
+   to control hosting costs. Each application belongs to the same tenant, so
44
+   security isolation between applications is nice-to-have but not critical.
45
+   Isolation is desired primarily for simplified management of the execution
46
+   environment for each application.
47
+2. App Portability. End-user wants to create a single container image, and
48
+   deploy the same image to multiple hosting environments, including OpenStack.
49
+   Other environments may include local servers, dedicated servers, private
50
+   clouds, and public clouds. Switching environments requires passing database
51
+   connection strings by environment variables at the time a container starts
52
+   to allow the application to use the services available in each environment
53
+   without changing the container image.
54
+3. Docker Compatibility. End-user has a Dockerfile used to build an application
55
+   and its runtime environment and dependencies in a Docker container image.
56
+   They want an easy way to run the Docker resulting image on an OpenStack
57
+   cloud.
58
+4. LXC Compatibility. End-user wants an easy way to remotely create multiple
59
+   LXC containers within a single Nova instance.
60
+5. OpenVZ Compatibility. End-user wants an easy way to remotely create multiple
61
+   OpenVZ containers within a single Nova instance.
62
+6. Containers-Centric World View. End-user wants to communicate with a single
63
+   OpenStack API, and request the addition of containers, without the need to
64
+   be concerned with keeping track of how many containers are already running
65
+   on a given Nova instance, and when more need to be created. They want to
66
+   simply create and remove containers, and allow the appropriate resource
67
+   scheduling to happen automatically.
68
+7. Platform Integration. Cloud operator already has an OpenStack cloud, and
69
+   wants to add a service/application centric management system on top.
70
+   Examples of such systems are Cloud Foundry, Kubernetes, Apache Mesos, etc.
71
+   The selected system is already Docker compatible. Allow this cloud operator
72
+   easy integration with OpenStack to run applications in containers. The
73
+   Cloud Operator now harnesses the power of both the management system, and
74
+   OpenStack, and does not need to manage a second infrastructure for his/her
75
+   application hosting needs. All details involving the integration of
76
+   containers with Nova instances is managed by OpenStack.
77
+8. Container network. End-user wants to define a custom overlay network for
78
+   containers, and wants to have admin privilege to manage the network
79
+   topology. Building a container network can decouple application deployment
80
+   and management from the underlying network infrastructure, and enable
81
+   additional usage scenario, such as (i) software-defined networking, and
82
+   (ii) extending the container network (i.e. connecting various resources from
83
+   multiple hosting environments). End-users want a single service that could
84
+   help them build the container network, and dynamically modify the network
85
+   topology by adding or removing containers to or from the network.
86
+9. Permit secure use of native REST APIs. Provide two models of operation with
87
+   Magnum.  The first model allows Magnum to manage the lifecycle of Pods,
88
+   ReplicationControllers, and Services.  The second model allows end-users to
89
+   manage the lifecycle of Pods, ReplicationControllers, and Services by
90
+   providing direct secure access to the native ReST APIs in Kubernetes and
91
+   possibly Docker.
92
+
93
+Long Term Use Cases
94
+-------------------
95
+These use cases have been identified by the community as important, but
96
+unlikely to be tackled in short term (especially prior to incubation). We wish
97
+to adapt to these use cases in long term, but this is not a firm project
98
+commitment.
99
+
100
+1. Multi-region/multi-cloud support. End-user wants to deploy applications to
101
+   multiple regions/clouds, and dynamically relocate deployed applications
102
+   across different regions/clouds. In particular, they want a single service
103
+   that could help them (i) provision nodes from multiple regions/clouds, thus
104
+   running containers on top of them, and (ii) dynamically relocate containers
105
+   (e.g. through container migration) between nodes regardless of the
106
+   underlying infrastructure.
107
+
108
+Proposed change
109
+===============
110
+Add a new API service for CRUD and advanced management of containers.
111
+If cloud operators only want to offer basic instance features for their
112
+containers, they may use nova with an alternate virt-driver, such as
113
+libvirt/lxc or nova-docker. For those wanting a full-featured container
114
+experience, they may offer the Containers Service API as well, in combination
115
+with Nova instances that contain an OpenStack agent that connects to the
116
+containers service through a security controlled agent (daemon) that allows
117
+the OpenStack control plane to provision and control containers running on
118
+Compute Hosts.
119
+
120
+The Containers Service will call the Nova API to create one or more Nova
121
+instances inside which containers will be created. The Nova instances may
122
+be of any type, depending on the virt driver(s) chosen by the cloud operator.
123
+This includes bare-metal, virtual machines, containers, and potentially other
124
+instance types.
125
+
126
+This allows the following configurations of containers in OpenStack.
127
+
128
+* Containers in Virtual Machine Instances
129
+* Containers in Bare Metal Instances
130
+* Containers in Container Instances (nested)
131
+
132
+The concept of nesting containers is currently possible if the parent container
133
+runs in privileged mode. Patches to the linux kernel are being developed to
134
+allow nesting of non-privileged containers as well, which provides a higher
135
+level of security.
136
+
137
+The spirit of this plan aims to duplicate as little as possible between Nova
138
+and the Containers Service. Common components like the scheduler are expected
139
+to be abstracted into modules, such as Gantt that can be shared by multiple
140
+projects. Until Gantt is ready for use by the Containers Service, we will
141
+implement only two provisioning schemes for containers:
142
+
143
+1. Create a container on a specified instance by using a nova instance guid.
144
+2. Auto-create instances (applies only until the Gantt scheduler is used)
145
+   2.1. Fill them sequentially until full.
146
+   2.2. Remove them automatically when they become empty.
147
+
148
+The above orchestration will be implemented using Heat. This requires some
149
+kind of hypervisor painting (such as host aggregates) for security reasons.
150
+
151
+The diagram below offers an overview of the system architecture. The OSC box
152
+indicates an OpenStack client, which will communicate with the Containers
153
+Service through a REST API. The containers service may silently create Nova
154
+instances if one with enough capacity to host the requested container is not
155
+already known to the Containers service. The containers service will maintain
156
+a database "Map" of containers, and what Nova instance each belongs to. Nova
157
+creates instances. Instances are created in Nova, and containers belong only
158
+to the Containers Service, and run within a Nova instance. If the instance
159
+includes the agent software "A", then it may be included in the inventory of
160
+the Containers service. Instances that do not contain an agent may not interact
161
+with the Containers Service, and can be controlled only by a Nova virt driver.
162
+
163
+::
164
+
165
+                            +---------+
166
+                            |   OSC   |
167
+                            +----+----+
168
+                                 |
169
+                            +----+----+
170
+ +-------- Nova -------+  +-+  REST   +-- Containers -+
171
+ |                     |  | +---------+    Service    |
172
+ |                     |  |                           |
173
+ |           +-------+ +--+ +-----+                   |
174
+ |           | Gantt | |  | | Map |                   |
175
+ |           +-------+ |  | +-----+                   |
176
+ |                     |  |                           |
177
+ +-----------+---------+  +---------------+-----------+
178
+             |                            |            
179
+ +-----------+----+ Compute Host ---------|-----------+
180
+ |                                    +---+---+       |
181
+ |                               +----+ Relay +---+   |
182
+ |                               |    +-------+   |   |
183
+ |                               |                |   |
184
+ | +-- Instance --+ +-- Instance |-+ +-- Instance |-+ |
185
+ | |              | |            | | |            | | |
186
+ | |              | |        +---+ | |        +---+ | |
187
+ | |              | |        |   | | |        |   | | |
188
+ | |              | |        | A | | |        | A | | |
189
+ | |              | |        |   | | |        |   | | |
190
+ | |              | |        +---+ | |        +---+ | |
191
+ | |              | |              | |              | |
192
+ | |              | | +---+  +---+ | | +---+  +---+ | |
193
+ | |              | | |   |  |   | | | |   |  |   | | |
194
+ | |              | | | C |  | C | | | | C |  | C | | |
195
+ | |              | | |   |  |   | | | |   |  |   | | |
196
+ | |              | | +---+  +---+ | | +---+  +---+ | |
197
+ | |              | |              | |              | |
198
+ | +--------------+ +--------------+ +--------------+ |
199
+ |                                                    |
200
+ +----------------------------------------------------+
201
+ +---+
202
+ |   |
203
+ | A | = Agent
204
+ |   |
205
+ +---+
206
+ +---+
207
+ |   |
208
+ | C | = Container
209
+ |   |
210
+ +---+
211
+
212
+
213
+Design Principles
214
+-----------------
215
+1. Leverage existing OpenStack projects for what they are good at. Do not
216
+   duplicate functionality, or copy code that can be otherwise accessed through
217
+   API calls.
218
+2. Keep modifications to Nova to a minimum.
219
+3. Make the user experience for end users simple and familiar.
220
+4. Allow for implementation of all features containers are intended to offer.
221
+
222
+
223
+Alternatives
224
+------------
225
+
226
+1. Extending Nova's existing feature set to offer container features
227
+1.1. Container features don't fit into Nova's idea of compute (VM/Server)
228
+2. A completely separate containers service forked from Nova.
229
+2.1. Would result in large overlap and duplication in features and code
230
+
231
+
232
+Data model impact
233
+-----------------
234
+For Nova, None. All new data planned will be in the Containers Service.
235
+
236
+
237
+REST API impact
238
+---------------
239
+For Nova, none. All new API calls will be implemented in the Containers
240
+Service. The OpenStack Containers Service API will be a superset of
241
+functionality offered by the, The `Docker Remote API:
242
+<https://docs.docker.com/reference/api/docker_remote_api/>`_
243
+with additionals to make is suitable for general use regardless of the backend
244
+container technology used, and to be compatible with OpenStack multi-tenancy
245
+and Keystone authentication.
246
+
247
+Specific Additions:
248
+
249
+1. Support for the X-Auth-Project-Id HTTP request header to allow for
250
+   multi-tenant use.
251
+2. Support for the X-Auth-Token HTTP request header to allow for authentication
252
+   with keystone.
253
+
254
+If either of the above headers are missing, a 401 Unauthorized response will
255
+be generated.
256
+
257
+Docker CLI clients may communicate with a Swarmd instance that is configured
258
+to use the OpenStack Containers API as the backend for libswarm. This will
259
+allow for tool compatibility with the Docker ecosystem using the officially
260
+supported means for integration of a distributed system.
261
+
262
+The scope of the full API will cause this spec to be too long to review, so
263
+the intent is to deal with the specific API design as a series of Gerrit
264
+reviews that submit API code as Not Implemented stubs with docstrings that
265
+clearly document the design, so allow for approval, and further implementation.
266
+
267
+Security impact
268
+---------------
269
+Because Nova will not be changed, there should be no security impacts to Nova.
270
+The Containers Service implementation, will have the following security related
271
+issues:
272
+
273
+* Need to authenticate against keystone using python-keystoneclient.
274
+* A trust token from Nova will be needed in order for the Containers Service
275
+  to call the Nova API on behalf of a user.
276
+* Limits must be implemented to control resource consumption in accordance with
277
+  quotas.
278
+* Providing STDIO access may generate a considerable amount of network chatter
279
+  between containers and clients through the relay. This could lead to
280
+  bandwidth congestion at the relays, or API nodes. An approach similar to
281
+  how we handle serial console access today will need to be considered to
282
+  mitigate this concern.
283
+
284
+Using containers implies a range of security considerations for cloud
285
+operators. These include:
286
+
287
+* Containers in the same instance share an operating system. If the kernel is
288
+  exploited using a security vulnerability, processes in once container may
289
+  escape the constraints of the container and potentially access other
290
+  resources on the host, including contents of other containers.
291
+* Output of processes may be persisted by the containers service in order to
292
+  allow asynchronous collection of exit status, and terminal output. Such
293
+  content may include sensitive information. Features may be added to mitigate
294
+  the risk of this data being replicated in log messages, including errors.
295
+* Creating containers usually requires root access. This means that the Agent
296
+  may need to be run with special privileges, or be given a method to
297
+  escalate privileges using techniques such as sudo.
298
+* User provided data is passed through the API. This will require sensible
299
+  data input validation.
300
+
301
+
302
+Notifications impact
303
+--------------------
304
+
305
+Contemplated features (in subsequent release cycles):
306
+
307
+* Notify the end user each time a Nova instance is created or deleted by
308
+  the Containers service, if (s)he has registered for such notifications.
309
+* Notify the user each on CRUD of containers containing start and end
310
+  notifications. (compute.container.create/delete/etc)
311
+* Notify user periodically of existence of container service managed
312
+  containers (ex compute.container.exists)
313
+
314
+
315
+Other end user impact
316
+---------------------
317
+
318
+The user interface will be a REST API. On top of that API will be an
319
+implementation of the libswarm API to allow for tools designed to use Docker
320
+to treat OpenStack as an upstream system.
321
+
322
+
323
+Performance Impact
324
+------------------
325
+
326
+The Nova API will be used to create instances as needed. If the Container to
327
+Instance ratio is 10, then the Nova API will be called at least once for every
328
+10 calls to the Containers Service. Instances that are left empty will be
329
+automatically deleted, so in the example of a 10:1 ratio, the Nova API will be
330
+called to perform a delete for every 10 deletes in the Container Service.
331
+Depending on the configuration, the ratio may be as low as 1:1.
332
+The Containers Service will only access Nova through its API, not by accessing
333
+its database.
334
+
335
+
336
+
337
+Other deployer impact
338
+---------------------
339
+
340
+Deployers may want to adjust the default flavor used for Nova Instances created
341
+by the Containers Service.
342
+
343
+There should be no impact on users of prior releases, as this introduces a new
344
+API.
345
+
346
+Developer impact
347
+----------------
348
+
349
+Minimal. There will be minimal changes required in Nova, if any.
350
+
351
+
352
+Implementation
353
+==============
354
+
355
+
356
+Assignee(s)
357
+-----------
358
+
359
+Primary assignee:
360
+aotto
361
+
362
+Other contributors:
363
+andrew-melton
364
+ewindisch
365
+
366
+
367
+Work Items
368
+----------
369
+
370
+1. Agent
371
+2. Relay
372
+3. API Service
373
+4. IO Relays
374
+
375
+
376
+Dependencies
377
+============
378
+
379
+1. <Links to Agent Blueprint and Spec here, once ready>
380
+2. Early implementations may use libswarm, or a python port of libswarm to
381
+   implement Docker API compatibility.
382
+
383
+Testing
384
+=======
385
+
386
+Each commit will be accompanied with unit tests, and Tempest functional tests.
387
+
388
+
389
+Documentation Impact
390
+====================
391
+
392
+A set of documentation for this new service will be required.
393
+
394
+
395
+References
396
+==========
397
+
398
+* Link to high level draft proposal from the Nova Midcycle Meetup for Juno:
399
+  `PDF <https://wiki.openstack.org/w/images/5/51/Containers_Proposal.pdf>`_
400
+* `Libswarm Source <https://github.com/docker/libswarm>`_

+ 186
- 0
specs/implemented/create-trustee-user-for-each-bay.rst View File

@@ -0,0 +1,186 @@
1
+==================================
2
+Create a trustee user for each bay
3
+==================================
4
+
5
+https://blueprints.launchpad.net/magnum/+spec/create-trustee-user-for-each-bay
6
+
7
+Some services which are running in a bay need to access OpenStack services.
8
+For example, Kubernetes load balancer [1]_ needs to access Neutron. Docker
9
+registry [2]_ needs to access Swift. In order to access OpenStack services,
10
+we can create a trustee for each bay and delegate a limited set of rights to
11
+the trustee. [3]_ and [4]_ give a brief introduction to Keystone's trusts
12
+mechanism.
13
+
14
+Problem description
15
+===================
16
+
17
+Some services which are running in a bay need to access OpenStack services,
18
+so we need to pass user credentials into the vms.
19
+
20
+Use Cases
21
+---------
22
+
23
+1. Kubernetes load balancer needs to access Neutron [1]_.
24
+2. For persistent storage, Cloud Provider needs to access Cinder to
25
+   mount/unmount block storage to the node as volume [5]_.
26
+3. TLS cert is generated in the vms and need to be uploaded to Magnum [6]_ and
27
+   [7]_.
28
+4. Docker registry needs to access Swift [2]_.
29
+
30
+Project Priority
31
+----------------
32
+
33
+High
34
+
35
+Proposed change
36
+===============
37
+When a user (the "trustor") wants to create a bay, steps for trust are as
38
+follows.
39
+
40
+1. Create a new service account (the "trustee") without any role in a domain
41
+   which is dedicated for trust. Without any role, the service account can do
42
+   nothing in Openstack.
43
+
44
+2. Define a trust relationship between the trustor and the trustee. The trustor
45
+   can delegate a limited set of roles to the trustee. We can add an option
46
+   named trust_roles in baymodel. Users can add roles which they want to
47
+   delegate into trust_roles. If trust_roles is not provided, we delegate all
48
+   the roles to the trustee.
49
+
50
+3. Services in the bay can access OpenStack services with the trustee
51
+   credentials and the trust.
52
+
53
+The roles which are delegated to the trustee should be limited. If the services
54
+in the bay only need access to Neutron, we should not allow the services to
55
+access to other OpenStack services. But there is a limitation that a trustor
56
+must have the role which is delegated to a trustee [4]_.
57
+
58
+Magnum now only allows the user who create the bay to get the certificate to
59
+avoid the security risk introduced by Docker [8]_. For example, if other users
60
+in the same tenant can get the certificate, then they can use Docker API to
61
+access the host file system of a bay node and get anything they want::
62
+
63
+    docker run --rm -v /:/hostroot ubuntu /bin/bash \
64
+               -c "cat /hostroot/etc/passwd"
65
+
66
+If Keystone doesn't allow to create new service accounts when LDAP is used as
67
+the backend for Keystone, we can use a pre-create service account for all
68
+bays. In this situation, all the bays use the same service account and
69
+different trust. We should add an config option to choose this method.
70
+
71
+Alternatives
72
+------------
73
+
74
+Magnum can create a user for each bay with roles to access OpenStack Services
75
+in a dedicated domain. The method has one disadvantage. The user which is
76
+created by magnum may get the access to OpenStack services which this user can
77
+not access before. For example, a user can not access Swift service and create
78
+a bay. Then Magnum create a service account for this bay with roles to access
79
+Swift. If the user logins into the vms and get the credentials, the user can
80
+use these credentials to access Swift.
81
+
82
+Or Magnum doesn't prepare credentials and the user who create a bay needs to
83
+login into the nodes to manully add credentials in config files for services.
84
+
85
+Data model impact
86
+-----------------
87
+
88
+Trustee id, trustee password and trust id are added to Bay table in Magnum
89
+database.
90
+
91
+REST API impact
92
+---------------
93
+
94
+Only the user who create a bay can get the certificate of this bay. Other
95
+users in the same tenant can not get the certificate now.
96
+
97
+Security impact
98
+---------------
99
+
100
+Trustee id and trustee password are encrypted in magnum database. When Magnum
101
+passes these parameters to heat to create a stack, the transmission is
102
+encrypted by tls, so we don't need to encrypt these credentials. These
103
+credentials are hidden in heat, users can not query them in stack parameters.
104
+
105
+Trustee id, trustee password and trust id can be obtained in the vms. Anyone
106
+who can login into the vms can get them and use these credentials to access
107
+OpenStack services. In a production environment, these vms must be secured
108
+properly to prevent unauthorized access.
109
+
110
+Only the user who create the bay can get the certificate to access the COE
111
+api, so it is not a security risk even if the COE api is not safe.
112
+
113
+Notifications impact
114
+--------------------
115
+
116
+None
117
+
118
+Other end user impact
119
+---------------------
120
+
121
+None
122
+
123
+Performance impact
124
+------------------
125
+
126
+None
127
+
128
+Other deployer impact
129
+---------------------
130
+
131
+None
132
+
133
+Developer impact
134
+----------------
135
+
136
+None
137
+
138
+Implementation
139
+==============
140
+
141
+Assignee(s)
142
+-----------
143
+
144
+Primary assignee:
145
+    humble00 (wanghua.humble@gmail.com)
146
+Other contributors:
147
+    None
148
+
149
+Work Items
150
+----------
151
+
152
+1. Create an trustee for each bay.
153
+2. Change the policy so that only the user who create a bay can get the
154
+   certificate of the bay.
155
+
156
+Dependencies
157
+============
158
+
159
+None
160
+
161
+Testing
162
+=======
163
+
164
+Unit test and functional test for service accounts and the policy change.
165
+
166
+Documentation Impact
167
+====================
168
+
169
+The user guide and troubleshooting guide will be updated with details
170
+regarding the service accounts.
171
+
172
+References
173
+==========
174
+.. [1] http://docs.openstack.org/developer/magnum/dev/dev-kubernetes-load-balancer.html
175
+.. [2] https://blueprints.launchpad.net/magnum/+spec/registryv2-in-master
176
+.. [3] http://blogs.rdoproject.org/5858/role-delegation-in-keystone-trusts
177
+.. [4] https://wiki.openstack.org/wiki/Keystone/Trusts
178
+.. [5] https://github.com/kubernetes/kubernetes/blob/release-1.1/examples/mysql-cinder-pd/README.md
179
+.. [6] https://bugs.launchpad.net/magnum/+bug/1503863
180
+.. [7] https://review.openstack.org/#/c/232152/
181
+.. [8] https://docs.docker.com/engine/articles/security/#docker-daemon-attack-surface
182
+
183
+History
184
+=======
185
+
186
+None

+ 171
- 0
specs/implemented/magnum-horizon-plugin.rst View File

@@ -0,0 +1,171 @@
1
+..
2
+ This work is licensed under a Creative Commons Attribution 3.0 Unported
3
+ License.
4
+
5
+ http://creativecommons.org/licenses/by/3.0/legalcode
6
+
7
+===================================
8
+Web Interface for Magnum in Horizon
9
+===================================
10
+
11
+Launchpad blueprint:
12
+
13
+https://blueprints.launchpad.net/magnum/+spec/magnum-horizon-plugin
14
+
15
+Currently there is no way for a user to interact with Magnum through a web
16
+based user interface, as they are used to doing with other OpenStack
17
+components. This implementation aims to introduce this interface as an
18
+extension of Horizon (the OpenStack Dashboard) and expose all the features of
19
+Magnum in a way familiar to users.
20
+
21
+Problem description
22
+===================
23
+
24
+In order to increase adoption and usability of Magnum we need to introduce a UI
25
+component for users and administrators to interact with Magnum without the need
26
+to use the command line. The UI proposed to be built will model all of the
27
+features currently available in the Magnum REST API and built using the Horizon
28
+plugin architecture to remain in line with other OpenStack UI projects and
29
+minimise the amount of new code that needs to be added.
30
+
31
+Use Cases
32
+----------
33
+1. An end user wanting to use Magnum with OpenStack who is not comfortable in
34
+   issuing commands with the python client will use the web user interface to
35
+   interact with Magnum.
36
+2. An administrator may use the user interface to provide a quick overview of
37
+   what Magnum has deployed in their OpenStack environment.
38
+
39
+Proposed change
40
+===============
41
+
42
+The first step will be to extend the Horizon API to include CRUD operations
43
+that are needed to interact with Magnum. Assuming that there are no issues here
44
+and API changes/additions are not required to Magnum, we can begin to
45
+design/implement the interface. We will aim to minimize the amount of Magnum
46
+specific UI code that will need to be maintained by reusing components from
47
+Horizon. This will also speed up the development significantly.
48
+
49
+It is suggested the initial implementation of Magnum UI will include basic CRUD
50
+operations on BayModel and Bay resources. This will be the starting point for
51
+development and upon completion this will represent version 1.
52
+
53
+Future direction includes adding CRUD operations for other Magnum features
54
+(Pod, Container, Service, ReplicationController) and will be tracked by new
55
+blueprints as they represent significant additional effort. The ultimate goal,
56
+a user should be able to perform all normal interactions with Magnum through
57
+the UI with no need for interaction with the python client.
58
+
59
+Suggestions for further improvement include visualising Magnum resources to
60
+provide a quick overview of how resources are deployed.
61
+
62
+Bugs/Blueprints relating specifically to the Magnum UI will be tracked here:
63
+
64
+https://launchpad.net/magnum-ui
65
+
66
+Mockups/Designs will be shared using the OpenStack Invision account located
67
+here:
68
+
69
+https://openstack.invisionapp.com
70
+
71
+Alternatives
72
+------------
73
+
74
+One alternative to this approach is to develop an entirely separate UI
75
+specifically for Magnum. We will not use this approach as it does not fall in
76
+line with how other projects are managing their user interfaces and this
77
+approach would ultimately result in a significantly larger effort with much
78
+duplication with Horizon.
79
+
80
+Data model impact
81
+-----------------
82
+
83
+None
84
+
85
+REST API impact
86
+---------------
87
+
88
+For Magnum, none. The Horizon API will need to be extended to include Create,
89
+Read, Update, Delete operations for all features available in the Magnum REST
90
+API. However, this extension to the Horizon API will live in the Magnum UI tree
91
+not the upstream Horizon tree.
92
+
93
+Security impact
94
+---------------
95
+
96
+None
97
+
98
+Notifications impact
99
+--------------------
100
+
101
+None
102
+
103
+Other end user impact
104
+---------------------
105
+
106
+None
107
+
108
+Performance Impact
109
+------------------
110
+
111
+The Magnum API will be called from the user interface to return information to
112
+the user about the current state of Magnum objects and perform new interactions
113
+with Magnum. For every action a user performs from the user interface at least
114
+one API call to Magnum will need to be made.
115
+
116
+Other deployer impact
117
+---------------------
118
+
119
+As the Magnum user interface will be managed and stored outside of the Horizon
120
+project deployers will need to pull down the Magnum UI code and add this to
121
+their Horizon install.
122
+
123
+In order to add the Magnum UI to Horizon the deployer will have to copy an
124
+enable file to openstack_dashboard/local/enabled/ in their Horizon directory
125
+and then run Horizon as they would normally.
126
+
127
+Developer impact
128
+----------------
129
+
130
+None
131
+
132
+Implementation
133
+==============
134
+
135
+Assignee(s)
136
+-----------
137
+
138
+Primary assignee:
139
+  bradjones
140
+
141
+Work Items
142
+----------
143
+
144
+1. Extend Horizon API in include Magnum calls
145
+2. CRUD operations on BayModel and Bay resources
146
+3. CRUD operations on other Magnum features (Pod, Container, Service, etc.)
147
+4. Refine the user experience
148
+
149
+Dependencies
150
+============
151
+
152
+None
153
+
154
+Testing
155
+=======
156
+
157
+Each commit will be accompanied with unit tests. There will also be functional
158
+tests which will be used as part of a cross-functional gate test for Magnum.
159
+This additional gate test will be non-voting as failures will not indicate
160
+issues with Magnum but instead serves as advanced warning of any changes that
161
+could potentially break the UI.
162
+
163
+Documentation Impact
164
+====================
165
+
166
+An installation guide will be required.
167
+
168
+References
169
+==========
170
+
171
+None

+ 177
- 0
specs/implemented/open-dcos.rst View File

@@ -0,0 +1,177 @@
1
+..
2
+   This work is licensed under a Creative Commons Attribution 3.0 Unported
3
+ License.
4
+
5
+ http://creativecommons.org/licenses/by/3.0/legalcode
6
+
7
+=================================
8
+Magnum and Open DC/OS Integration
9
+=================================
10
+
11
+Launchpad Blueprint:
12
+
13
+https://blueprints.launchpad.net/magnum/+spec/mesos-dcos
14
+
15
+Open DC/OS [1]_ is a distributed operating system based on the Apache Mesos
16
+distributed systems kernel. It enables the management of multiple machines as
17
+if they were a single computer. It automates resource management, schedules
18
+process placement, facilitates inter-process communication, and simplifies
19
+the installation and management of distributed services. Its included web
20
+interface and available command-line interface (CLI) facilitate remote
21
+management and monitoring of the cluster and its services.
22
+
23
+Open DC/OS now supports both docker containerizer and mesos containerizer.
24
+The mesos containerizer support both docker and AppC image spec, the mesos
25
+containerizer can manage docker containers well even if docker daemon is not
26
+running.
27
+
28
+End user can install Open DC/OS with different ways, such as vagrant, cloud,
29
+local etc. For cloud, the Open DC/OS only supports AWS now, end user can
30
+deploy a DC/OS cluster quickly with a template. For local install, there
31
+are many steps to install a Open DC/OS cluster.
32
+
33
+Problem Description
34
+===================
35
+
36
+COEs (Container Orchestration Engines) are the first class citizen in Magnum,
37
+there are different COEs in Magnum now including Kubernetes, Swarm and Mesos.
38
+All of those COEs are focusing docker container management, the problem is
39
+that the concept of container is not only limited in docker container, but
40
+also others, such as AppC, linux container etc, Open DC/OS is planning to
41
+support different containers by leveraging Mesos unified container feature
42
+and the Open DC/OS has a better management console for container orchestration.
43
+
44
+Currently, Magnum provides limited support for Mesos Bay as there is only one
45
+framework named as Marathon running on top of Mesos. Compared with Open DC/OS,
46
+the current Mesos Bay lack the following features:
47
+
48
+1. App Store for application management. The Open DC/OS has a universe to
49
+   provide app store functions.
50
+
51
+2. Different container technology support. The Open DC/OS support different
52
+   container technologies, such as docker, AppC etc, and may introduce OCI
53
+   support in future. Introducing Open DC/OS Bay can enable Magnum to support
54
+   more container technologies.
55
+
56
+3. Better external storage integration. The Open DC/OS is planning to introduce
57
+   docker volume isolator support in next release, the docker volume isolator
58
+   is leveraging docker volume driver API to integrate with 3rd party
59
+   distributed storage platforms, such as OpenStack Cinder, GlusterFS, Ceph
60
+   etc.
61
+
62
+4. Better network management. The Open DC/OS is planning to introduce CNI
63
+   network isolator in next release, the CNI network isolator is leveraging CNI
64
+   technologies to manage network for containers.
65
+
66
+5. Loosely coupled with docker daemon. The Open DC/OS can work well for docker
67
+   container even if docker daemon is not running. The docker daemon now have
68
+   some issues in large scale cluster, so this approach avoids the limitation
69
+   of the docker daemon but still can enable end user get some docker features
70
+   in large scale cluster.
71
+
72
+
73
+Proposed Changes
74
+================
75
+
76
+We propose extending Magnum as follows.
77
+
78
+1. Leverage bay driver work and structure this new COE as a bay driver.
79
+
80
+2. Leverage mesos-slave-flags [3]_ to customize Open DC/OS.
81
+
82
+   Here is an example of creating an Open DC/OS baymodel that uses
83
+   docker/volume as isolator, linux as launcher and docker as image
84
+   provider: ::
85
+