Browse Source

Provide multiple store backend support

Change-Id: I4b00cf8317adb4318ceb65f336c9fe0f17f4ca9c
Implements: multi-store
Abhishek Kekane 1 year ago
parent
commit
57da144125
1 changed files with 557 additions and 25 deletions
  1. 557
    25
      specs/rocky/approved/glance/multi-store.rst

+ 557
- 25
specs/rocky/approved/glance/multi-store.rst View File

@@ -21,17 +21,17 @@ at once.
21 21
 
22 22
 Consider the following use cases for providing multi-store backend support:
23 23
 
24
- * Deployer might want to provide different level of costing for different
25
-   tiers of storage, i.e. One backend for    SSDs and another for
26
-   spindles. Customer may choose one of those based on his need.
27
- * Old storage is retired and deployer wants to have all new images being
28
-   added to new storage and old storage will be operational until data
29
-   is migrated.
30
- * Operator wants to differentiate the images from images added by user.
31
- * Different hypervisors provided from different backends (For
32
-   example, Ceph, Cinder, VMware etc.).
33
- * Each site with their local backends which nova hosts are accessing
34
-   directly (Ceph) and users can select the site where image will be stored.
24
+* Deployer might want to provide different level of costing for different
25
+  tiers of storage, i.e. One backend for    SSDs and another for
26
+  spindles. Customer may choose one of those based on his need.
27
+* Old storage is retired and deployer wants to have all new images being
28
+  added to new storage and old storage will be operational until data
29
+  is migrated.
30
+* Operator wants to differentiate the images from images added by user.
31
+* Different hypervisors provided from different backends (For
32
+  example, Ceph, Cinder, VMware etc.).
33
+* Each site with their local backends which nova hosts are accessing
34
+  directly (Ceph) and users can select the site where image will be stored.
35 35
 
36 36
 Problem description
37 37
 ===================
@@ -48,14 +48,14 @@ in order to replicate or target image bits on backend glance stores. For
48 48
 example, in order to replicate a existing image's bits to secondary storage
49 49
 of the same type / scheme as the primary:
50 50
 
51
- * It's a manual out-of-band task to copy image bits to secondary storage.
52
- * The operator must manage store locations manually; there is no way to
53
-   query the available stores to back an image's bits in glance.
54
- * The operator must remember to register secondary location URL using
55
-   the glance API.
56
- * Constructing the location URL by hand is error prone as some URLs are
57
-   lengthy and complex. Moreover they require knowledge of the backing store
58
-   in order to construct properly.
51
+* It's a manual out-of-band task to copy image bits to secondary storage.
52
+* The operator must manage store locations manually; there is no way to
53
+  query the available stores to back an image's bits in glance.
54
+* The operator must remember to register secondary location URL using
55
+  the glance API.
56
+* Constructing the location URL by hand is error prone as some URLs are
57
+  lengthy and complex. Moreover they require knowledge of the backing store
58
+  in order to construct properly.
59 59
 
60 60
 Also consider the case when a glance API consumer wants to download the image
61 61
 bits from a secondary backend location which was added out-of-band. Today
@@ -63,11 +63,543 @@ the consumer must use the direct location URL which implies the consumer
63 63
 needs the logic necessary to translate that direct URL into a connection
64 64
 to the backend.
65 65
 
66
+Proposed change
67
+===============
66 68
 
67
-Current state
68
-=============
69
+This spec proposes the following high level features to support multiple
70
+stores of the same scheme/type:
69 71
 
70
-Glance community agrees to address the problem described above during
71
-Rocky/S cycles. The actual detailed specification is still under discussion
72
-and will amend this spec as https://review.openstack.org/#/c/562467 when
73
-the implementation details are agreed on.
72
+* Support in the glance-api.conf and supporting logic to configure, manage
73
+  and use multiple stores of the same or different type/scheme.
74
+* Enhance the image upload API to (optionally) support targeting a backend
75
+  store for the image bits.
76
+* Enhance image download code to download image from any of the enabled
77
+  backends.
78
+* Enhance the image import API to (optionally) support targeting a backend
79
+  store to download the image bits from.
80
+* Additional metadata attribute(s) added to an image locations' metadata
81
+  properties related to backing stores.
82
+* A new REST API to list all configured stores known to the glance service.
83
+
84
+** Config/glance-store changes**
85
+
86
+In order to have multi-store support of same or different type/scheme
87
+we propose to deprecate 'stores', 'default_store' config option and
88
+add a new config option `enabled_backends` under DEFAULT section,
89
+'default_backend' option under 'glance_store' section and
90
+'description' option under each section of desired store of
91
+glance-api.conf. Operator can use `enabled_backends` to specify comma
92
+separated key:value of storage backends and its store type and then
93
+each backends will have a different section to specify related configuration
94
+options. If 'default_backend' is not explicitly set then appropriate
95
+exception will be raised which will prevent the service from starting.
96
+
97
+The reason for deprecating 'default_store' config option is that glance
98
+verifies the default store at the time of service start and if it is not
99
+one of the listed store (file, http, rbd, swift, cinder, vmware or sheepdog)
100
+then it raises the error and prevents the service from start. This validation
101
+has been done using 'choices' attribute of ConfigOption object itself. After
102
+enabling support of multi-store it's difficult to have this kind of
103
+validation check using 'choices' attribute will be difficult as
104
+default_store name will be store identifier and not actual store.
105
+
106
+Consider the following example snippet of glance-api.conf which configures
107
+3 stores for use::
108
+
109
+    [DEFAULT]
110
+    # list of enabled stores identified by their property group name
111
+    enabled_backends = fast:rbd, cheap:rbd, reliable:file
112
+
113
+    # the default store, if not set glance-api service will not start
114
+    default_backend = fast
115
+
116
+    # conf props for file system store instance
117
+    [reliable]
118
+    filesystem_store_datadir = /var/lib/images/data/
119
+    description = Reliable filesystem store
120
+    # etc..
121
+
122
+    # conf props for ceph store instance
123
+    [fast]
124
+    rbd_store_ceph_conf = /opt/ceph1/ceph.conf
125
+    description = Fast access to rbd store
126
+    # etc..
127
+
128
+    # conf props for ceph store instance
129
+    [cheap]
130
+    rbd_store_ceph_conf = /opt/ceph2/ceph.conf
131
+    description = Less expensive rbd store
132
+    # etc..
133
+
134
+The example above is for Ceph, but it could apply to any storage backend. As
135
+shown in the example above the 'enabled_backends' property is a comma delimited
136
+list of store identifiers which are setup for glance. Each of the store
137
+identifiers defines a property group name which itself contains the store
138
+specific properties used to configure the store instance and a store type
139
+which will be used to register configuration options related to that store
140
+under the new configuration section defined using store identifiers. Under
141
+the covers the store identifier (aka id) maps to the store class used for
142
+the instance.
143
+
144
+While 'stores' and 'default_store' are available an operator can leave the
145
+'enabled_backends' property unset in which case multi-store support is not
146
+"enabled" and glance will work as-is. Once these config options are removed
147
+operator at least need to set one store in the 'enabled_backends' or else
148
+glance-api service will fail to start.
149
+
150
+In order to accommodate multiple stores of the same scheme, the scheme_map
151
+used to index stores must also index by store identifier per scheme. The
152
+store identifier is (implicitly) the name of the conf group given in
153
+the glance-api.conf (e.g. reliable, fast and cheap from the example
154
+conf given above).
155
+
156
+Consider the following indexing approach::
157
+
158
+    scheme_map[scheme][store_id]
159
+
160
+Where 'scheme' is the scheme(s) supported by the store implementation and
161
+'store_id' is the identifier for the store.
162
+
163
+To maintain backward compatibility with old store API's while they are not
164
+deprecated we also propose to add new store api's like add, get, delete
165
+etc. to glance_store. The old store api's will be removed when the
166
+deprecation period for 'stores' and 'default_store' config option is over.
167
+
168
+**API to list stores**
169
+
170
+To accommodate the management and discovery of known glance stores, glance
171
+should also provide a REST API which permits users to list all stores
172
+configured for glance. This is a read-only list of glance stores
173
+which includes the store identifier, description for each store and a flag
174
+which will tell whether a store is default or not.
175
+
176
+NOTE: The list of glance stores is based on the glance-api.conf with respect to
177
+store configuration. Operators will not be able to configure new stores using
178
+the API. So in order to show the description in the response as mentioned
179
+earlier we propose to add new config option 'description' to each store
180
+with some default description and operator can set it as per their need.
181
+
182
+As discussed herein, the store identifier can be used to address a particular
183
+instance of a store for applicable operations such as image upload and
184
+image import.
185
+
186
+We propose the new REST resource be located at the following URI::
187
+
188
+    /v2/info/stores
189
+
190
+More details on this API can be found in the REST API section of this spec.
191
+
192
+**Image upload API**
193
+The existing image upload API permits an operator to upload image bits to
194
+the glance service. However today this API does not provide a means for
195
+the operator to target a specific store to back the image bits on
196
+(technically the v1 API allows you to specify a store scheme to target).
197
+This spec proposes the API be enhanced to permit an operator to specify
198
+which store will back the image bits being uploaded.
199
+
200
+We propose a header field be used as a means to transport
201
+the identifier of the store to back the image bits. Again the identifier
202
+under the covers is the property group name of the respective glance
203
+store driver. When the image upload API is used to upload image bits, the
204
+glance logic will determine if the target store is specified, and if so
205
+the image bits will be added to the target store.
206
+
207
+If no store is targeted in the upload request, the 'default_backend'
208
+is used to back the image bits.
209
+
210
+Using this scheme operators will be able to specify which store they want
211
+an image bits to be backed on during an upload operation. We propose the
212
+'X-Image-Meta-Store' header be used as the means to transport a target
213
+store identifier.
214
+
215
+More details on this API can be found in the REST API section of this spec.
216
+
217
+**Image download**
218
+If cloud get upgraded to use multi-store support from the single store
219
+then glance need to deal with the locations from old stores to new
220
+stores. For example, if operator is using rbd (say ceph) backend
221
+and now he has upgraded the environment and introduced two additional
222
+rbd stores as ceph1 and ceph2 with default store as 'ceph1' then
223
+somehow glance must be able to download the images from old stores
224
+(ceph).
225
+
226
+With multiple backends available, Glance needs some enhancements
227
+so it can fulfill the current image download API contract. The
228
+download request will traverse through all the configured backends
229
+to look for the image. As we are adding store information in location
230
+metadata, at first it will look the image in the store which is set
231
+in the location metadata. If location metadata is not set then
232
+as per example from the config section above, if the location
233
+is rbd://<something>, then it will search the image in all available
234
+ceph stores i.e. 'fast' and 'cheap' in this case. This way user will be
235
+able to download his image from the old store even after cloud is
236
+upgraded to use multiple stores.
237
+
238
+**Image import API**
239
+The existing image import API allows end users to import image from the
240
+staging area into glance backend. However today this API does not provide
241
+a means for the end user to target a specific store to import the image.
242
+This spec proposes the API be enhanced to permit an end user to specify
243
+which store will back the image being imported.
244
+
245
+We propose a header field be used as a means to transport the identifier
246
+of the store to import the image. Again the identifier under the covers
247
+is the property group name of the respective glance store driver. When
248
+the image import API is used to import the image, the glance logic will
249
+determine if the target store header is specified, and if so the image
250
+will be imported to the target store.
251
+
252
+If no store is targeted in the import request, the 'default_backend' is
253
+used to import the image.
254
+
255
+Using this scheme end users will be able to specify which store they want
256
+an image to be imported during an import operation. We propose the
257
+'X-Image-Meta-Store' header be used as the means to transport a target
258
+store identifier.
259
+
260
+More details on this API can be found in the REST API section of this spec.
261
+
262
+**Store locations metadata attributes**
263
+In the current glance API, a consumer of the glance API has no way to correlate
264
+an image location with its respective store other than by inspecting the
265
+image's location URL. While this may work fine for many use cases, a more
266
+user friendly relation is needed in a multi-store environment; user's need
267
+an explicit relation between an image location and its respective store
268
+(e.g. what store is this location backed on).
269
+
270
+This spec proposes that when image bits are added with the image upload API,
271
+the core glance logic is responsible for adding a metadata attribute to the
272
+image location URL to reflect the backing store's identifier. For example,
273
+if a user uploads an image to the 'ceph1' store in our example above,
274
+once the image bits are uploaded, the image location URL is added and in the
275
+URL's metadata a property with the store's identifier is added to the location
276
+metadata object.
277
+
278
+With the store identifier in the image location URL metadata, we can expose it
279
+to the end user with a new attribute as 'store' in the image response so that
280
+it can be used for subsequent operations or within the API consumers logic.
281
+
282
+The new image response with store attribute will be something like::
283
+
284
+    "size": 1234,
285
+    "store": ["reliable"],
286
+    "checksum": 1234567890,
287
+    "name": "Import image",
288
+    "status": active
289
+
290
+Alternatives
291
+------------
292
+
293
+Two major alternatives come to mind with respect to a multi-store approach in
294
+glance; multi-store using a service per store and per driver multi-store
295
+support.
296
+
297
+**Per store service**
298
+In the service per store approach, each of the configured store instances would
299
+run as a separate process/service. As a result each service would have its
300
+own AMQP/RPC interface, own PID, etc.. To route requests to store services,
301
+we'd use a scheduler which itself is another process / service. This is the
302
+same approach used in cinder multi-backend support.
303
+
304
+Although the service per store approach may be a longer term goal for glance,
305
+today we haven't seen enough justification from our consumer base to justify
306
+the major refactoring/changes which are required to move glance to this
307
+model.
308
+
309
+Therefore this spec proposes the multi-store approach outlined within this
310
+spec - let's get an initial approximation multi-store working and gage
311
+our next steps based on consumer feedback in the community.
312
+
313
+**Per driver support**
314
+Another potential approach would be to push the multi-store logic down within
315
+each glance store driver's implementation. For example the store driver
316
+itself could contain logic to multiplex with multiple backends.
317
+
318
+While this approach would work, it would require each store driver's
319
+implementation to change and would result in suboptimal reuse of code.
320
+
321
+This spec proposes a multi-store approach which has little or no impact
322
+to each store driver; they can be used as-is in a multi-store implementation.
323
+By pushing the logic to support multiple stores of the same scheme up into
324
+the core of glance, we get maximal reuse and store driver implementations
325
+needn't be concerned with such logic.
326
+
327
+Data model impact
328
+-----------------
329
+
330
+This spec does not propose any changes to the data model. Rather the approach
331
+herein can maintain all new stateful data either in memory or within the
332
+existing schema used by glance. However store identifier will be stored as
333
+a metadata in the location object.
334
+
335
+REST API impact
336
+---------------
337
+
338
+This spec proposes the following API changes:
339
+
340
+**New API**
341
+
342
+* List all stores known to the glance service.
343
+
344
+**Modified APIs**
345
+
346
+* Import an image.
347
+* Upload an image file.
348
+* Get image details.
349
+
350
+**Common Response Codes**
351
+
352
+* Create Success: `201 Created`
353
+* Modify Success: `200 OK`
354
+* Delete Success: `204 No Content`
355
+* Failure: `400 Bad Request` with details.
356
+* Forbidden: `403 Forbidden`
357
+
358
+**API Version**
359
+
360
+All URLS will be under the v2 Glance API.  If it is not explicitly specified
361
+assume /v2/<url>
362
+
363
+**[New API] List stores**
364
+
365
+List all stores known to the glance service::
366
+
367
+    GET /v2/info/stores
368
+
369
+This API takes no query parameters and when authorized returns a listing of all
370
+stores known to the glance service. The stores known to glance are those which
371
+have been configured in the glance-api.conf and have been loaded during glance
372
+startup. The response body payload is JSON and contains a JSON object per
373
+store. Each store JSON object contains the store's identifier (id),
374
+description and if a particular store is a default store the it will
375
+have a flag telling store is default. For example::
376
+
377
+    {
378
+       "stores":[
379
+          {
380
+             "id":"reliable",
381
+             "description": "Reliable filesystem store"
382
+          },
383
+          {
384
+             "id":"fast",
385
+             "description": "Fast access to rbd store",
386
+             "default": true
387
+          },
388
+          {
389
+             "id":"cheap",
390
+             "description": "Less expensive rbd store"
391
+          }
392
+       ]
393
+    }
394
+
395
+Response codes:
396
+
397
+* 200 -- Upon authorization and successful request. The response body
398
+  contains the JSON payload with the known stores.
399
+
400
+**[Modified API] Create image**
401
+We propose to add an 'OpenStack-image-store-ids' header to the image-create
402
+response which would have the available stores. Using this user can decide
403
+which store he needs to upload/import his image and wouldn't have to make
404
+a separate get-stores call.
405
+
406
+New response headers
407
+^^^^^^^^^^^^^^^^^^^^
408
+
409
+``OpenStack-image-store-ids``
410
+
411
+   The value of this header will be a comma-separated list of stores
412
+   available.  For example,
413
+
414
+   OpenStack-image-store-ids: fast, cheap, reliable
415
+
416
+**[Modified API] Upload image binary data**
417
+
418
+Get image binary data::
419
+
420
+    PUT /v2/images/​{image_id}​/file
421
+
422
+This modifies the existing REST API to support a new header field which is
423
+optional and if present specifies the store id to upload the image data to.
424
+For backwards compatibility, if the header is not specified the store
425
+specified as the default (e.g. default_store) is used to upload the image
426
+to.
427
+
428
+New header fields:
429
+
430
+* X-Image-Meta-Store -- If present contains the store id to upload the image
431
+  binary data to.
432
+
433
+New / changed response codes:
434
+
435
+* 400 -- If the X-Image-Meta-Store header is present, but specifies a
436
+  store id for a store that doesn't exist by that id.
437
+
438
+Example curl usage::
439
+
440
+        curl -i -X PUT -H "X-Auth-Token: $token" -H "X-Image-Meta-Store:
441
+            ceph1" -H "Content-Type: application/octet-stream"
442
+            -d @/home/glance/ubuntu-12.10.qcow2
443
+            $image_url/v2/images/{image_id}/file
444
+
445
+
446
+**[Modified API] Get image details**
447
+
448
+Get the details for a specified image::
449
+
450
+    GET /v2/images/​{image_id}​
451
+
452
+Although this spec does not impose any changes on the glance API layer, this
453
+call will now shows the location's store id. In case if there are multiple
454
+locations, then all locations will be displayed as comma separated list. The
455
+new image response with store attribute will be something like::
456
+
457
+    "size": 1234,
458
+    "store": ["reliable"],
459
+    "checksum": 1234567890,
460
+    "name": "Import image",
461
+    "status": active
462
+
463
+
464
+**[Modified API] Import image to the backend**
465
+
466
+Import image to the backend::
467
+
468
+    POST /v2/images/​{image_id}​/import
469
+
470
+This modifies the existing REST API to support a new header field which is
471
+optional and if present specifies the store id to import the image data to.
472
+For backwards compatibility, if the header is not specified the store
473
+specified as the default (e.g. default_store) is used to import the image
474
+to.
475
+
476
+New header fields:
477
+
478
+* X-Image-Meta-Store -- If present contains the store id to upload the image
479
+  binary data to.
480
+
481
+New / changed response codes:
482
+
483
+* 400 -- If the X-Image-Meta-Store header is present, but specifies a
484
+  store id for a store that doesn't exist by that id.
485
+
486
+Example curl usage::
487
+
488
+        curl -i -X PUT -H "X-Auth-Token: $token" -H "X-Image-Meta-Store:
489
+            ceph1" -H "Content-Type: application/json"
490
+            -d '{"method":{"name":"glance-direct"}}'
491
+            $image_url/v2/images/{image_id}/import
492
+
493
+Security impact
494
+---------------
495
+
496
+None
497
+
498
+Notifications impact
499
+--------------------
500
+
501
+Need to add 'stores' field in the notification response as one of the use
502
+case of this proposal is offering different price tiers for storage which
503
+will help systems which consumes notifications to perform the billing.
504
+
505
+Other end user impact
506
+---------------------
507
+
508
+This proposal introduces a few other user impacts worth noting.
509
+
510
+**Glance client**
511
+Ideally the glance client (CLI + REST client) should be updated in accordance
512
+with this spec. Notably:
513
+
514
+* CLI / API support for listing glance stores.
515
+* CLI / API support for specifying a store id on upload/import.
516
+
517
+**Configuration**
518
+Deployers will need to be aware of the configuration aspects for glance
519
+multi-store. From a conf point of view, configuring multi-store for glance
520
+will look very much (from a high level) like configuring cinder for
521
+multi-backend. The conf file specifics will need to be documented.
522
+
523
+Performance Impact
524
+------------------
525
+
526
+A very little hit on the performance of downloading the image from the old
527
+stores. For example, if existing user is using single rbd (say ceph)
528
+backend and now he has upgraded the environment and introduced two
529
+additional rbd stores as ceph1 and ceph2 with default store as 'ceph1'
530
+then if image which needs to be download from old store (ceph) will
531
+take some time as it needs to be looked in all the enabled backends.
532
+
533
+Other deployer impact
534
+---------------------
535
+
536
+Once merged, glance multi-store is not enabled unless the deployer
537
+configures the enabled_backends property in glance-api.conf and thus is backwards
538
+compatible out-of-the-box. When multi-store is disabled, v2 API users can
539
+use the list stores API and will retrieve a list of the current
540
+stores configured (of course only 1 store per scheme).
541
+
542
+Developer impact
543
+----------------
544
+
545
+None
546
+
547
+
548
+Implementation
549
+==============
550
+
551
+Assignee(s)
552
+-----------
553
+
554
+Primary assignee:
555
+  abhishek-kekane
556
+
557
+Other contributors:
558
+  None
559
+
560
+Work Items
561
+----------
562
+
563
+Implementation tasks may consist of:
564
+
565
+* Conf support for multi-store.
566
+* Multi-store loading and indexing.
567
+* List stores API.
568
+* Location URL metadata store id on upload.
569
+* Add new 'store' attribute in image response.
570
+* Image upload with store targeting.
571
+* Image import with store targeting.
572
+* Multi-store delete and other store access codepaths.
573
+* Add python-glanceclient support
574
+
575
+
576
+Dependencies
577
+============
578
+
579
+None
580
+
581
+
582
+Testing
583
+=======
584
+
585
+* Need to add new tempest tests to verify multi-store support
586
+
587
+
588
+Documentation Impact
589
+====================
590
+
591
+As mentioned in the 'work items' section, we'll need to ensure the glance docs
592
+are update for:
593
+
594
+* The new list stores REST API.
595
+* New header field for image upload.
596
+* New header field for image import.
597
+* New store id in image response.
598
+* Overall glance multi-store documentation to educate deployers on the
599
+  feature and how it's used.
600
+
601
+
602
+References
603
+==========
604
+
605
+* https://review.openstack.org/#/c/150967

Loading…
Cancel
Save