Spec for VMware Store to use multiple datastores

This spec includes a proposal to support multiple datastores in
the glance VMware store.

Based on blueprint vmware-store-multiple-datastores

Change-Id: I16229da839ab7f147c36d5857e2269999e8215d7
Sabari Kumar Murugesan 2015-01-12 16:59:40 -08:00 committed by Sabari Kumar Murugesan
parent 224ade269d
commit 21e94ae496
1 changed files with 227 additions and 0 deletions

View File

@ -0,0 +1,227 @@
This work is licensed under a Creative Commons Attribution 3.0 Unported
Glance VMware store to support multiple datastores
Glance, when configured to use VMware store, stores the images in a virtual
storage container called datastore, identified by the configuration option
vmware_datastore_name. The size of the datastore can only be increased to a
certain limit which also depends upon the underlying physical storage.
The capacity of the store is thus bound by the size of a single datastore and
poses serious scalability issues.
Problem description
Currently the number of images stored by glance, when configured to use VMware
store, is limited by the capacity of a single datastore. The size of a
datastore has an upper bound [1].
e.g The maximum datastore size is 64TB when backed by VMFS-5 filesystem. While
the same for a datastore with NFS may depend on the storage array vendor. While
this is the theoretical maximum based on the supported filesystem, datastores
are also sized based on the underlying physical storage, including internal and
external devices and networked storage. With networked storage the choice of
storage protocol (iSCSI, FC, FCoE) also play a vital role in deciding the
optimal size. Thus, the capacity of a datastore used in a datacenter varies by
Apart from the capacity, there may be a performance bottleneck when many
compute nodes concurrently access a single datastore to download images.
Proposed change
To allow adding more capacity and improving performance, we suggest the use of
multiple datastores for storing images.
There is only one major aspect of the proposed change:
- Datastore Selection.
**1) Datastore Selection:**
This change proposes to select a datastore based on the priority given to it by
the operator and the capacity to accommodate the image. A new configuration
option (say vmware_datastores) will be added that would enable the operator to
specify multiple datastores along with their relative weights. In case of equal
priority the selection will be based on the maximum freespace available on the
datastore. This approach is very similar to filesystem_store_datadirs used by
filesystem store.
Example: say vmware_datastores is configured with nfs_datastore,
iscsi_datastore, ssd_datastore with weights 100, 100 and 200 respectively.
Here, ssd_datastore will be the preferred datastore if it
can accommodate the image data being added. Otherwise, nfs_datastore or
iscsi_datastore will be chosen based on maximum freespace available.
1) The datastore selection can be done through round-robin/randomized strategy
instead of priority-based. This would be ideal if all datastores are of same
type and perform similarly. But since datastores are of different types, the
strategy is best driven by operator input. e.g SSD backed datastores may be
faster that networked datastores. While its possible to determine
the type of datastore, implicit assumptions cannot be made.
Data model impact
Does not have any impact on glance data model.
REST API impact
Security impact
This spec only adds a selection logic to the existing functionality and hence
does not have any security impact.
Notifications impact
Other end user impact
Performance Impact
The use of multiple datastores will reduce concurrent operations on a single
Other deployer impact
This proposal will allow administrators to configure multiple datastores to
save the image data in VMware storage backend. If enabled, it will override
the existing configuration option *vmware_datastore_name* that allows to
specify a single datastore. If disabled, the store will fallback to
vmware_datastore_name option. Thus, the change is backwards compatible. We
will deprecate the vmware_datastore_name option going forward.
New Multi Strings configuration option in *glance-api.conf*
Optional. Default: Not set.
A datastore along with its datacenter path and a weight. If the
datastore is a member of a datastore cluster, then the name of the cluster
should also be included. Otherwise, the cluster name can be omitted. This
option can be specified multiple times to specify multiple datastores.
Thus, the required format becomes:
vmware_datastores =
vmware_datastores =
The weights are used to establish the relative priorities of the specified
datastores. Thus, they can be any arbitrary integer values.
vmware_datastores =
vmware_datastores =
vmware_datastores =
In the above example, ssd_datastore will be given highest priority, followed
by nfs_datastore and backup_datastore.
vmware_datastores =
vmware_datastores =
vmware_datastores =
In this example, the nfs datastores will be given equal priority, followed by
the backup_datastore. With equal priority, the contention between nfs
datastores is resolved by the maximum freespace.
Note:- If the datacenter path or datastore name contains a colon (:) symbol,
it must be escaped with a backslash.
Developer impact
Primary assignee:
smurugesan (sabari)
Other contributors:
Core reviewer(s):
Other reviewer(s):
Work Items
1) Implement new config option in VMware store driver.
2) Implement datastore selection logic in the VMware store driver.
3) Implement unit tests.
4) Change glance-api sample conf in glance repository.
5) Add a deprecation warning for vmware_datastore_name.
6) Update the documentation.
* oslo.vmware has introduced a Datastore object which will be used in this
* oslo.vmware has a vim_util module that has some low-level utility methods
to interact with the vmware api's. This is required to parse api responses.
* Tempest tests are not required.
Documentation Impact
* Document new configuration options.
[1] http://www.vmware.com/pdf/vsphere5/r55/vsphere-55-configuration-maximums.pdf