Merge "Add volume replication support"
This commit is contained in:
Jenkins
Gerrit Code Review
commit
59e7c7572f
505
specs/juno/volume-replication.rst
Normal file
505
specs/juno/volume-replication.rst
Normal file
@ -0,0 +1,505 @@
|
|||||||
|
|
||||||
|
..
|
||||||
|
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||||
|
License.
|
||||||
|
|
||||||
|
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||||
|
|
||||||
|
==========================================
|
||||||
|
Volume Replication
|
||||||
|
==========================================
|
||||||
|
|
||||||
|
https://blueprints.launchpad.net/cinder/+spec/volume-replication
|
||||||
|
|
||||||
|
Volume replication is a key storage feature and a requirement for
|
||||||
|
features such as high-availability and disaster recovery of applications
|
||||||
|
running on top of OpenStack clouds.
|
||||||
|
This blueprint is an attempt to add initial support for volume replication
|
||||||
|
in Cinder, and is considered a first take, and will include support for:
|
||||||
|
* Replicate volumes (primary to secondary approach)
|
||||||
|
* Promote a secondary to primary (and stop replication)
|
||||||
|
* Synchronize replication with direction
|
||||||
|
|
||||||
|
This would further be enhanced in the future.
|
||||||
|
|
||||||
|
While this blueprint focuses on volume replication, a related blueprint
|
||||||
|
focuses on consistency groups, and replication would be extended to
|
||||||
|
support it.
|
||||||
|
|
||||||
|
Problem description
|
||||||
|
===================
|
||||||
|
|
||||||
|
The main use of volume replication is resiliency in presence of failures.
|
||||||
|
Examples of possible failures are:
|
||||||
|
|
||||||
|
* Storage system failure
|
||||||
|
* Rack(s) level failure
|
||||||
|
* Datacenter level failure
|
||||||
|
|
||||||
|
Here we specifically exclude failures like media failures, disk failures, etc.
|
||||||
|
Such failures are typically addressed by local resiliency schemes.
|
||||||
|
|
||||||
|
Replication can be implemented in the following ways:
|
||||||
|
|
||||||
|
* Host-based - Requires Nova integration
|
||||||
|
|
||||||
|
* Storage-based
|
||||||
|
|
||||||
|
- Typical block based approach - replication is specified between two
|
||||||
|
existing volumes (or groups of volumes) on the controllers.
|
||||||
|
- Typically file system based approach - a file
|
||||||
|
(in Cinder context, the file representing a block device) placed in a
|
||||||
|
directory (or group or fileset, etc) that is automatically copied to a
|
||||||
|
specified remote location.
|
||||||
|
|
||||||
|
Assumptions:
|
||||||
|
|
||||||
|
* Replication should be transparent to the end-user, failover, failback
|
||||||
|
and test will be executed by the cloud admin.
|
||||||
|
However, to test that the application is working, the end-user may be
|
||||||
|
involved, as he will be required to verify that his application is
|
||||||
|
working with the volume replica.
|
||||||
|
|
||||||
|
* The storage admin will provide the setup and configuration to enable the
|
||||||
|
actual replication between the storage systems. This could be performed
|
||||||
|
at the storage back-end or storage driver level depending on the storage
|
||||||
|
back-end. Specifically, storage drivers are expected to report with whom
|
||||||
|
they can replicate and report this to the scheduler.
|
||||||
|
|
||||||
|
* The cloud admin will enable the replication feature through the use of
|
||||||
|
volume types.
|
||||||
|
|
||||||
|
* The end-user will not be directly exposed to the replication feature.
|
||||||
|
Selecting a volume-type will determine if the volume will be replicated,
|
||||||
|
based on the actual extra-spec definition of the volume type (defined by
|
||||||
|
the cloud admin).
|
||||||
|
|
||||||
|
* Quota management: quota are consumed as 2x as two volumes are
|
||||||
|
created and the consumed space id double.
|
||||||
|
We can re-examine this mechanism after we get comments from deployers.
|
||||||
|
|
||||||
|
Proposed change
|
||||||
|
===============
|
||||||
|
|
||||||
|
Each Cinder host will report replication capabilities:
|
||||||
|
|
||||||
|
* Replication_support: indicate if replication is enabled for this driver
|
||||||
|
instance
|
||||||
|
* Replication_unit_id: device specific id used for replication
|
||||||
|
* Replication_partners: list of device specific ids that this node can
|
||||||
|
replicate with
|
||||||
|
* Replication_rpo_range - supported RPO by this driver instance <min,max>
|
||||||
|
* replication_supported_methods - list of methods supported by the back-end
|
||||||
|
|
||||||
|
Add extra-specs in the volume type to indicate replication:
|
||||||
|
|
||||||
|
* Replication_enabled - if True, volume to be replicated if exists as extra
|
||||||
|
specs. if option is not specified or False, then replication is not
|
||||||
|
enabled. This option is required to enable replication.
|
||||||
|
* replica_same_az - (optional) indicate if replica should be in the same AZ
|
||||||
|
* replica_volume_backend_name - (optional) specify back-end to be used as
|
||||||
|
target
|
||||||
|
* replication_target_rpo - (optional) requested RPO (numeric, minutes) for
|
||||||
|
the volume
|
||||||
|
|
||||||
|
Create volume with replication enabled:
|
||||||
|
|
||||||
|
* Scheduler selects two hosts for volume placement and sets up the replication
|
||||||
|
DB entry
|
||||||
|
* Manager on primary creates the primary volume (as is done today)
|
||||||
|
* Manager on secondary creates the replica volume
|
||||||
|
* Manager on primary sets up the replication
|
||||||
|
|
||||||
|
Re-type volume:
|
||||||
|
|
||||||
|
* Replication_enabled: True->False:
|
||||||
|
drop the replication and continue with the regular retype logic.
|
||||||
|
* Replication_enabled: False->True:
|
||||||
|
after the retype logic selects back-ends (scheduler) and enables
|
||||||
|
replication.
|
||||||
|
|
||||||
|
Promote to primary:
|
||||||
|
|
||||||
|
* Manager on secondary stops the replication.
|
||||||
|
* Switch between volume ids of primary and secondary
|
||||||
|
(user sees no change in volume ids)
|
||||||
|
|
||||||
|
Sync replication:
|
||||||
|
|
||||||
|
* Manager on primary restarts the replication
|
||||||
|
|
||||||
|
Test:
|
||||||
|
|
||||||
|
* Create a clone of the secondary volume.
|
||||||
|
|
||||||
|
Delete volume:
|
||||||
|
|
||||||
|
* Disable the replication
|
||||||
|
* Delete secondary volume
|
||||||
|
* Delete primary volume (as is done today)
|
||||||
|
|
||||||
|
Cloning a volume:
|
||||||
|
|
||||||
|
* Since the replica are added after the primary is created, if we
|
||||||
|
clone a volume and keep the volume-type, it will be replicated.
|
||||||
|
|
||||||
|
Snapshots:
|
||||||
|
|
||||||
|
* Snapshot for the primary volume works as is today, and create
|
||||||
|
a snapshot on the primary. No snapshot is done for the replica.
|
||||||
|
* Snapshot for the replica (secondary) volume will fail.
|
||||||
|
|
||||||
|
Notes:
|
||||||
|
|
||||||
|
* Manager acts via the driver for back-end replication specific functions.
|
||||||
|
* Failover is "promote to primary" as described above.
|
||||||
|
* Failback is "sync replication" + "promote to primary".
|
||||||
|
|
||||||
|
Driver API:
|
||||||
|
|
||||||
|
* create_replica: to be run on secondary to create the volume
|
||||||
|
* enable_replica: to be run on primary to start replication
|
||||||
|
* disable_replica: to be run on primary, stops the replication
|
||||||
|
* delete_replica: to be run on secondary, deletes the replica target volume
|
||||||
|
* replication_status_check: to be run on all hosts, updating the replication
|
||||||
|
status as observed from the back-end perspective
|
||||||
|
* promote_replica: to be run on secondary, make secondary the primary
|
||||||
|
|
||||||
|
Alternatives
|
||||||
|
------------
|
||||||
|
|
||||||
|
Replication can be performed outside of Cinder, and OpenStack can be
|
||||||
|
unaware of it. However, this requires vendor specific scripts, and
|
||||||
|
is not visible to the admin user, as only the storage system admin
|
||||||
|
will see the replica and the state of the replication.
|
||||||
|
Also all recovery actions (failover, failback) will require both the
|
||||||
|
the storage and cloud admins to work together.
|
||||||
|
While replication in Cinder reduces the role of the storage admin to
|
||||||
|
only the setup phase, and the cloud admin is responsible for failover
|
||||||
|
and failback with (typically) not need for intervention from the clouds
|
||||||
|
admin.
|
||||||
|
|
||||||
|
Data model impact
|
||||||
|
-----------------
|
||||||
|
|
||||||
|
* A new replication relationship table will be created.
|
||||||
|
(with its database migration support).
|
||||||
|
|
||||||
|
* On promote to primary, the ids of the primary and secondary volume entries
|
||||||
|
will change (switch).
|
||||||
|
|
||||||
|
Replication relationship db table:
|
||||||
|
|
||||||
|
* id = Column(String(36), primary_key=True)
|
||||||
|
* deleted = Column(Boolean, default=False)
|
||||||
|
* primary_id = Column(String(36), ForeignKey('volumes.id'), nullable=False)
|
||||||
|
* secondary_id = Column(String(36), ForeignKey('volumes.id'), nullable=False)
|
||||||
|
* primary_replication_unit_id = Column(String(255))
|
||||||
|
* secondary_replication_unit_id = Column(String(255))
|
||||||
|
* status = Column(Enum('error', 'creating', 'copying', 'active', 'active-stopped',
|
||||||
|
'stopping', 'deleting', 'deleted', 'inactive',
|
||||||
|
name='replicationrelationship_status'))
|
||||||
|
* extended_status = Column(String(255))
|
||||||
|
* driver_data = Column(String(255))
|
||||||
|
|
||||||
|
State diagram for replication (status)::
|
||||||
|
<start>
|
||||||
|
any error
|
||||||
|
Create replica +----------+ condition +-------+
|
||||||
|
+--------------> | creating | +------------> | error |
|
||||||
|
+----+-----+ +---+---+
|
||||||
|
| | Storage admin to
|
||||||
|
| enable replication | fix, and status
|
||||||
|
| | check will update
|
||||||
|
+----+-----+ |
|
||||||
|
+-------------> | copying | any state <--------+
|
||||||
|
| +----+-----+
|
||||||
|
| |
|
||||||
|
| status |
|
||||||
|
| check | status check
|
||||||
|
| +----++----+ +------> +--+--+-+--------+
|
||||||
|
| | active | | active-stopped |
|
||||||
|
| +----++----+ <------+ +--+--+-+--------+
|
||||||
|
| | status check
|
||||||
|
| |
|
||||||
|
| | promote to primary
|
||||||
|
| |
|
||||||
|
| sync +----+--+--+
|
||||||
|
+------------+ | inactive |
|
||||||
|
+-------+--+
|
||||||
|
<end>
|
||||||
|
|
||||||
|
REST API impact
|
||||||
|
---------------
|
||||||
|
|
||||||
|
* Show replication relationship
|
||||||
|
|
||||||
|
* Show information about a volume replication relationship.
|
||||||
|
* Method type: GET
|
||||||
|
* Normal Response Code: 200
|
||||||
|
* Expected error http response code(s)
|
||||||
|
|
||||||
|
* 404: replication relationship not found
|
||||||
|
|
||||||
|
* /v2/<tenant id>/os-volume-replication/<replication uuid>
|
||||||
|
* JSON schema definition for the response data::
|
||||||
|
|
||||||
|
{
|
||||||
|
'relationship':
|
||||||
|
{
|
||||||
|
'id': 'relationship id'
|
||||||
|
'primary_id': 'primary volume uuid'
|
||||||
|
'status': 'status of relationship'
|
||||||
|
'links': '{ ... }'
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
* Show replication relationship with details
|
||||||
|
|
||||||
|
* Show detailed information about a volume replication relationship.
|
||||||
|
* Method type: GET
|
||||||
|
* Normal Response Code: 200
|
||||||
|
* Expected error http response code(s)
|
||||||
|
|
||||||
|
* 404: replication relationship not found
|
||||||
|
|
||||||
|
* /v2/<tenant id>/os-volume-replication/<replication uuid>/detail
|
||||||
|
* JSON schema definition for the response data::
|
||||||
|
|
||||||
|
{
|
||||||
|
'relationship':
|
||||||
|
{
|
||||||
|
'id': 'relationship id'
|
||||||
|
'primary_id': 'primary volume uuid'
|
||||||
|
'secondary_id': 'secondary volume uuid'
|
||||||
|
'status': 'status of relationship'
|
||||||
|
'extended_status': 'extended status'
|
||||||
|
'links': { ... }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
* List replication relationship with details
|
||||||
|
|
||||||
|
* List detailed information about a volume replication relationship.
|
||||||
|
* Method type: GET
|
||||||
|
* Normal Response Code: 200
|
||||||
|
* Expected error http response code(s)
|
||||||
|
|
||||||
|
* TBD
|
||||||
|
|
||||||
|
* /v2/<tenant id>/os-volume-replication/detail
|
||||||
|
* Parameters:
|
||||||
|
|
||||||
|
*status*
|
||||||
|
filter by replication relationship status
|
||||||
|
*primary_id*
|
||||||
|
Filter by primary volume id
|
||||||
|
*secondary_id*
|
||||||
|
Filter by secondary volume id
|
||||||
|
|
||||||
|
* JSON schema definition for the response data::
|
||||||
|
|
||||||
|
{
|
||||||
|
'relationship':
|
||||||
|
{
|
||||||
|
'id': 'relationship id'
|
||||||
|
'primary_id': 'primary volume uuid'
|
||||||
|
'secondary_id': 'secondary volume uuid'
|
||||||
|
'status': 'status of relationship'
|
||||||
|
'extended_status': 'extended status'
|
||||||
|
'links': { ... }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
* Promote volume to be the primary volume
|
||||||
|
|
||||||
|
* Switch between the uuids of the primary and secondary volumes, and
|
||||||
|
make the secondary volume the primary volume.
|
||||||
|
* Method type: PUT
|
||||||
|
* Normal Response Code: 202
|
||||||
|
* Expected error http response code(s)
|
||||||
|
|
||||||
|
* 404: replication relationship not found
|
||||||
|
|
||||||
|
* /v2/<tenant id>/os-volume-replication/<replication uuid>
|
||||||
|
* JSON schema definition for the body data::
|
||||||
|
|
||||||
|
{
|
||||||
|
'relationship':
|
||||||
|
{
|
||||||
|
'promote': None
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
* Sync between the primary and secondary volume.
|
||||||
|
|
||||||
|
* Resync the replication between the primary and secondary volume.
|
||||||
|
Typically follows a promote operation on the replication.
|
||||||
|
* Method type: PUT
|
||||||
|
* Normal Response Code: 202
|
||||||
|
* Expected error http response code(s)
|
||||||
|
|
||||||
|
* 404: replication relationship not found
|
||||||
|
|
||||||
|
* /v2/<tenant id>/os-volume-replication/<replication uuid>
|
||||||
|
* JSON schema definition for the body data::
|
||||||
|
|
||||||
|
{
|
||||||
|
'relationship':
|
||||||
|
{
|
||||||
|
'sync': None
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
* Test replication by make a copy of the secondary volume available
|
||||||
|
|
||||||
|
* Test the volume replication. Create a clone of the secondary volume
|
||||||
|
and make it accessible, so the promote process can be tested.
|
||||||
|
* Method type: POST
|
||||||
|
* Normal Response Code: 202
|
||||||
|
* Expected error http response code(s)
|
||||||
|
|
||||||
|
* 404: replication relationship not found
|
||||||
|
|
||||||
|
* /v2/<tenant id>/os-volume-replication/<replication uuid>/test
|
||||||
|
* JSON schema definition for the response data::
|
||||||
|
|
||||||
|
{
|
||||||
|
'relationship':
|
||||||
|
{
|
||||||
|
'volume_id': 'volume id of the cloned secondary'
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
Security impact
|
||||||
|
---------------
|
||||||
|
|
||||||
|
* Does this change touch sensitive data such as tokens, keys, or user data?
|
||||||
|
*No*.
|
||||||
|
|
||||||
|
* Does this change alter the API in a way that may impact security, such as
|
||||||
|
a new way to access sensitive information or a new way to login?
|
||||||
|
*No*.
|
||||||
|
|
||||||
|
* Does this change involve cryptography or hashing?
|
||||||
|
*No*.
|
||||||
|
|
||||||
|
* Does this change require the use of sudo or any elevated privileges?
|
||||||
|
*No*.
|
||||||
|
|
||||||
|
* Does this change involve using or parsing user-provided data? This could
|
||||||
|
be directly at the API level or indirectly such as changes to a cache layer.
|
||||||
|
*No*.
|
||||||
|
|
||||||
|
* Can this change enable a resource exhaustion attack, such as allowing a
|
||||||
|
single API interaction to consume significant server resources? Some
|
||||||
|
examples of this include launching subprocesses for each connection, or
|
||||||
|
entity expansion attacks in XML.
|
||||||
|
*Yes*, enabling replication consume cloud and storage resources.
|
||||||
|
|
||||||
|
Notifications impact
|
||||||
|
--------------------
|
||||||
|
|
||||||
|
Will add notification for enabling replication, promoting, syncing and
|
||||||
|
dropping replication.
|
||||||
|
|
||||||
|
Other end user impact
|
||||||
|
---------------------
|
||||||
|
|
||||||
|
* End-user to use volume types to enable/disable replication.
|
||||||
|
|
||||||
|
* Cloud admin to use the *promote*, *sync* and *test* commands
|
||||||
|
in the python-cinderclient to execute failover, failback and test.
|
||||||
|
|
||||||
|
Performance Impact
|
||||||
|
------------------
|
||||||
|
|
||||||
|
* Scheduler now needs to choose two hosts instead of one based on
|
||||||
|
additional input from the driver and volume type.
|
||||||
|
|
||||||
|
* The periodic task will query the driver and back-end for status
|
||||||
|
of all replicated volumes - running on the primary and secondary.
|
||||||
|
|
||||||
|
* Extra db calls identifying if replication exists are added to retype,
|
||||||
|
snapshot operations, etc will add a small latency to these functions.
|
||||||
|
|
||||||
|
Other deployer impact
|
||||||
|
---------------------
|
||||||
|
|
||||||
|
* Added options for volume types (see above)
|
||||||
|
|
||||||
|
* Add new driver capabilities, needs to be supported by the volume drivers,
|
||||||
|
which may imply changes to the driver configuration options.
|
||||||
|
|
||||||
|
* This change will require explicit enablement (to be used by users)
|
||||||
|
from the cloud administrator.
|
||||||
|
|
||||||
|
Developer impact
|
||||||
|
----------------
|
||||||
|
|
||||||
|
* Change to the driver API is noted above. Basically new functions are
|
||||||
|
needed to support using replication.
|
||||||
|
|
||||||
|
* The API will expand to include consistency groups following merging
|
||||||
|
consistency group support to Cinder.
|
||||||
|
|
||||||
|
|
||||||
|
Implementation
|
||||||
|
==============
|
||||||
|
|
||||||
|
Assignee(s)
|
||||||
|
-----------
|
||||||
|
|
||||||
|
Primary assignee:
|
||||||
|
ronenkat
|
||||||
|
|
||||||
|
Other contributors:
|
||||||
|
None
|
||||||
|
|
||||||
|
Work Items
|
||||||
|
----------
|
||||||
|
|
||||||
|
* Cinder public (admin) APIs for replication
|
||||||
|
* DB schema for replication
|
||||||
|
* Cinder scheduler support for replication
|
||||||
|
* Cinder driver API additions for replication
|
||||||
|
* Cinder manager update for replication
|
||||||
|
* Testing
|
||||||
|
|
||||||
|
Note: Code is based on https://review.openstack.org/#/c/64026/ which was
|
||||||
|
submitted in the Icehouse development cycle.
|
||||||
|
|
||||||
|
Dependencies
|
||||||
|
============
|
||||||
|
|
||||||
|
* Related blueprints: Consistency groups
|
||||||
|
https://blueprints.launchpad.net/cinder/+spec/consistency-groups
|
||||||
|
|
||||||
|
* LVM to support replication using DRBD, in a separate contribution.
|
||||||
|
|
||||||
|
Testing
|
||||||
|
=======
|
||||||
|
|
||||||
|
* Testing in gate is not supported due to the following considerations:
|
||||||
|
|
||||||
|
* LVM has no replication support, to be addressed using DRBD in a separate
|
||||||
|
contribution.
|
||||||
|
* requires setting up at least two nodes using DRBD
|
||||||
|
|
||||||
|
* Should be discussed/addressed as support for LVM is added.
|
||||||
|
|
||||||
|
* 3rd party driver CI will be expected to test replication.
|
||||||
|
|
||||||
|
Documentation Impact
|
||||||
|
====================
|
||||||
|
|
||||||
|
* Public (admin) API changes.
|
||||||
|
* Details how replication is used by leveraging volume types.
|
||||||
|
* Driver docs explaining how replication is setup for each driver.
|
||||||
|
|
||||||
|
References
|
||||||
|
==========
|
||||||
|
|
||||||
|
* Volume replication design session
|
||||||
|
https://etherpad.openstack.org/p/juno-cinder-volume-replication
|
||||||
|
|
||||||
Loading…
x
Reference in New Issue
Block a user