RBD instance snapshots
This spec covers using rbd snapshots for instance snapshots rather than copies to local disk and back into rbd. blueprint rbd-instance-snapshots Change-Id: Ia3666d53e663eacf8c65dbffbd4bc847dd948171
This commit is contained in:
Josh Durgin
committed by
Chet Burgess
Chet Burgess
parent
3bf0d79755
commit
b5a895eb81
281
specs/mitaka/approved/rbd-instance-snapshots.rst
Normal file
281
specs/mitaka/approved/rbd-instance-snapshots.rst
Normal file
@@ -0,0 +1,281 @@
|
|||||||
|
..
|
||||||
|
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||||
|
License.
|
||||||
|
|
||||||
|
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||||
|
|
||||||
|
======================
|
||||||
|
RBD Instance Snapshots
|
||||||
|
======================
|
||||||
|
|
||||||
|
https://blueprints.launchpad.net/nova/+spec/rbd-instance-snapshots
|
||||||
|
|
||||||
|
When using RBD as storage for glance and nova, instance snapshots are
|
||||||
|
slow and inefficient, resulting in poor end user experience. Using
|
||||||
|
local disk for the upload increases operator costs for supporting
|
||||||
|
instance snapshots.
|
||||||
|
|
||||||
|
As background reading the follow link provides an overview of the
|
||||||
|
snapshotting capabilities available in ceph.
|
||||||
|
|
||||||
|
http://docs.ceph.com/docs/master/rbd/rbd-snapshot/
|
||||||
|
|
||||||
|
Problem description
|
||||||
|
===================
|
||||||
|
|
||||||
|
RBD is often used to back glance images and nova disks. When using rbd
|
||||||
|
for nova's disks, nova 'snapshots' are slow, since they create full
|
||||||
|
copies by downloading data from rbd to a local file, uploading it to
|
||||||
|
glance, and putting it back into rbd. Since raw images are normally
|
||||||
|
used with rbd to enable copy-on-write clones, this process removes any
|
||||||
|
sparseness in the data uploaded to glance. This is a problem of user
|
||||||
|
experience, since this slow, inefficient process takes much longer
|
||||||
|
than necessary to let users customize images.
|
||||||
|
|
||||||
|
For operators, this is also a problem of efficiency and cost. For
|
||||||
|
rbd-backed nova deployments, this is the last part that uses
|
||||||
|
significant local disk space.
|
||||||
|
|
||||||
|
Use Cases
|
||||||
|
----------
|
||||||
|
|
||||||
|
This allows end users to quickly iterate on images, for example to
|
||||||
|
customize or update them, and start using the snapshots far more
|
||||||
|
quickly.
|
||||||
|
|
||||||
|
For operators, this eliminates any need for large local disks on
|
||||||
|
compute nodes, since instance data in rbd stays in rbd. It also
|
||||||
|
prevents lots of wasted space.
|
||||||
|
|
||||||
|
Project Priority
|
||||||
|
-----------------
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
Proposed change
|
||||||
|
===============
|
||||||
|
|
||||||
|
Instead of copying all the data to local disk, keep it in RBD by
|
||||||
|
taking an RBD snapshot in Nova and cloning it into Glance. Rather
|
||||||
|
than uploading the data, just tell Glance about its location in
|
||||||
|
RBD. This way data stays in the Ceph cluster, and the snapshot is
|
||||||
|
far more rapidly usable by the end user.
|
||||||
|
|
||||||
|
In broad strokes, the workflow is as follows:
|
||||||
|
|
||||||
|
1. Create an RBD snapshot of the ephemeral disk via Nova in
|
||||||
|
the ceph pool Nova is configured to use.
|
||||||
|
|
||||||
|
2. Clone the RBD snapshot into Glance's RBD pool. [7]
|
||||||
|
|
||||||
|
3. To keep from having to manage dependencies between snapshots
|
||||||
|
and clones, deep-flatten the RBD clone in Glance's RBD pool and
|
||||||
|
detach it from the Nova RBD snapshot in ceph. [7]
|
||||||
|
|
||||||
|
5. Remove the RBD snapshot from ceph created in (1) as it is no
|
||||||
|
longer needed.
|
||||||
|
|
||||||
|
6. Update Glance with the location of the RBD clone created and
|
||||||
|
flattend in (2) and (3).
|
||||||
|
|
||||||
|
This is the reverse of how images are cloned into nova instance disks
|
||||||
|
when both are on rbd [0].
|
||||||
|
|
||||||
|
If any of these steps fail, clean up any partial state and fall back
|
||||||
|
to the current full copy method. Failure of the RBD snapshot method
|
||||||
|
will be quick and usually transient in nature. The cloud admin can
|
||||||
|
monitor for these failures and address the underlying CEPH issues
|
||||||
|
causing the RBD snapshot to fail.
|
||||||
|
|
||||||
|
Failures will be reported in the form of stack traces in the nova
|
||||||
|
compute logs.
|
||||||
|
|
||||||
|
There are a few reasons for falling back to full copies instead of
|
||||||
|
bailing out if efficient snapshots fail:
|
||||||
|
|
||||||
|
* It makes upgrades graceful, since nova snapshots still work
|
||||||
|
before glance has enough permissions for efficient snapshots
|
||||||
|
(see Security Impact for glance permission details).
|
||||||
|
|
||||||
|
* Nova snapshots still work when efficient snapshots are not
|
||||||
|
possible due to architecture choices, such as not using rbd as
|
||||||
|
a glance backend, or using different ceph clusters for glance
|
||||||
|
and nova.
|
||||||
|
|
||||||
|
* This is consistent with existing rbd behavior in nova and cinder.
|
||||||
|
If cloning from a glance image fails, both projects fall back
|
||||||
|
to full copies when creating volumes or instance disks.
|
||||||
|
|
||||||
|
Alternatives
|
||||||
|
------------
|
||||||
|
|
||||||
|
The clone flatten step could be handled as a background task in a
|
||||||
|
green thread, or completely asynchronously as a periodic task. This
|
||||||
|
would increase user-facing performance, as the snapshots would be
|
||||||
|
available for use immediately, but it would also introduce
|
||||||
|
race-condition-like issues around deleting dependent images.
|
||||||
|
|
||||||
|
The flatten step could be omitted completely, and glance could be
|
||||||
|
made responsible for tracking the various image dependencies. At
|
||||||
|
the rbd level, an instance snapshot would consist of three things
|
||||||
|
for each disk. This is true of any instance, regardless of whether
|
||||||
|
it was created from a snapshot itself, or is just created from a
|
||||||
|
usual image. In rbd, there would be:
|
||||||
|
|
||||||
|
1. a snapshot of the instance disk
|
||||||
|
|
||||||
|
2. a clone of the instance disk
|
||||||
|
|
||||||
|
3. a snapshot of the clone
|
||||||
|
|
||||||
|
(3) is exposed through glance's backend location.
|
||||||
|
(2) is an internal detail of glance.
|
||||||
|
(1) is an internal detail that nova and glance handle.
|
||||||
|
|
||||||
|
At the rbd level, a disk with snapshots can't be deleted. Hide this
|
||||||
|
from the user if they delete an instance with snapshots by making
|
||||||
|
glance responsible for their eventual deletion, once their dependent
|
||||||
|
snapshots are deleted. Nova does this by renaming instance disks that
|
||||||
|
it deletes in rbd, so glance is aware that they can be deleted.
|
||||||
|
|
||||||
|
When a glance snapshot is deleted, it deletes (3), then (2), and
|
||||||
|
(1). If nova has renamed its parent in rbd with a preset suffix, the
|
||||||
|
instance has been destroyed already, so glance tries to delete the
|
||||||
|
original instance disk. The original instance disk will be
|
||||||
|
successfully deleted when the last snapshot is removed.
|
||||||
|
|
||||||
|
If glance snapshots are created but deleted before the instance is
|
||||||
|
destroyed, nova will delete the instance disks as usual.
|
||||||
|
|
||||||
|
The mechanism nova uses to let glance know it needs to clean up the
|
||||||
|
original disk could be different. It could use an image property with
|
||||||
|
certain restrictions which aren't possible in the current glance api:
|
||||||
|
|
||||||
|
* it must be writeable only once
|
||||||
|
|
||||||
|
* to avoid exposing backend details, it would need to be hidden
|
||||||
|
from end users
|
||||||
|
|
||||||
|
Storing this state in ceph is much easier to keep consistent with
|
||||||
|
ceph, rather than an external database which could become out of sync.
|
||||||
|
It would also be an odd abstraction leak in the glance_store api, when
|
||||||
|
upper layers don't need to be aware of it at all.
|
||||||
|
|
||||||
|
Data model impact
|
||||||
|
-----------------
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
REST API impact
|
||||||
|
---------------
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
Security impact
|
||||||
|
---------------
|
||||||
|
|
||||||
|
Glance will need to be configured with direct_url support enabled
|
||||||
|
in order for Nova to determine what and where to clone the image
|
||||||
|
from, depending on system configurations, this could leak backend
|
||||||
|
credentials [5]. Devstack has already been updated to switch
|
||||||
|
behaviors when Ceph support is requested [6].
|
||||||
|
|
||||||
|
Documentation has typically recommended using different ceph pools
|
||||||
|
for glance and nova, with different access to each. Since nova
|
||||||
|
would need to be able to create the snapshot in the pool used by
|
||||||
|
glance, it would need write access to this pool as well.
|
||||||
|
|
||||||
|
Notifications impact
|
||||||
|
--------------------
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
Performance Impact
|
||||||
|
------------------
|
||||||
|
|
||||||
|
Snapshots of RBD-backed instances would be significantly faster.
|
||||||
|
|
||||||
|
Other end user impact
|
||||||
|
---------------------
|
||||||
|
|
||||||
|
Snapshots of RBD-backed instances would be significantly faster.
|
||||||
|
|
||||||
|
Other deployer impact
|
||||||
|
---------------------
|
||||||
|
|
||||||
|
To use this in an existing installation with authx, adding 'allow
|
||||||
|
rwx pool=images' to nova's ceph user capabilities is necessary. The
|
||||||
|
'ceph auth caps' command can be used for this [1]. If these permissions
|
||||||
|
are not updated, nova will continue using the existing full copy
|
||||||
|
mechanism for instance snapshots because the the fast snapshot will fail
|
||||||
|
and nova compute will fall back to the full copy method.
|
||||||
|
|
||||||
|
Developer impact
|
||||||
|
----------------
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
Implementation
|
||||||
|
==============
|
||||||
|
|
||||||
|
Assignee(s)
|
||||||
|
-----------
|
||||||
|
|
||||||
|
Primary assignee:
|
||||||
|
nic
|
||||||
|
|
||||||
|
Other contributors:
|
||||||
|
jdurgin
|
||||||
|
pbrady
|
||||||
|
nagyz
|
||||||
|
cfb-n/cburgess
|
||||||
|
|
||||||
|
Work Items
|
||||||
|
----------
|
||||||
|
|
||||||
|
Implementation: [4]
|
||||||
|
|
||||||
|
The libvirt imagebackend does not currently recognize AMI images
|
||||||
|
as raw (and therefore cloneable) for whatever reason, so this
|
||||||
|
proposed change is of limited utility with a very popular image
|
||||||
|
format. This should be addressed in a separate change.
|
||||||
|
|
||||||
|
Dependencies
|
||||||
|
============
|
||||||
|
|
||||||
|
You need a Havana or newer version of glance as direct URL was added in
|
||||||
|
Havana.
|
||||||
|
|
||||||
|
Testing
|
||||||
|
=======
|
||||||
|
|
||||||
|
The existing tempest tests with ceph in the gate cover instance
|
||||||
|
snapshots generically. As fast snapshots are enabled automatically, there
|
||||||
|
is no need to change the tempest tests. Additionally, unit tests in nova
|
||||||
|
will verify error handling (falling back to full copies if the process
|
||||||
|
fails), and make sure that when configured correctly rbd snapshots and
|
||||||
|
clones are used rather than full copies.
|
||||||
|
|
||||||
|
Documentation Impact
|
||||||
|
====================
|
||||||
|
|
||||||
|
See the security and other deployer impact sections above.
|
||||||
|
|
||||||
|
References
|
||||||
|
==========
|
||||||
|
|
||||||
|
[0] http://specs.openstack.org/openstack/nova-specs/specs/juno/implemented/rbd-clone-image-handler.html
|
||||||
|
|
||||||
|
[1] Ceph authentication docs: http://ceph.com/docs/master/rados/operations/user-management/#modify-user-capabilities
|
||||||
|
|
||||||
|
[2] Alternative: Glance cleanup patch: https://review.openstack.org/127397
|
||||||
|
|
||||||
|
[3] Alternative: Nova patch: https://review.openstack.org/125963
|
||||||
|
|
||||||
|
[4] Nova patch: https://review.openstack.org/205282
|
||||||
|
|
||||||
|
[5] https://bugs.launchpad.net/glance/+bug/880910
|
||||||
|
|
||||||
|
[6] https://review.openstack.org/206039
|
||||||
|
|
||||||
|
[7] http://docs.ceph.com/docs/master/dev/rbd-layering/
|
||||||
Reference in New Issue
Block a user