Add devref for conditional updates
This patch adds developer's documentation for conditional update functionality used to remove API races. Change-Id: I63ce1c87f7418feba94479b461a4488d4ae6e48c
This commit is contained in:
Gorka Eguileor
parent
4c9a37b4f1
commit
e4fb5aae59
391
doc/source/devref/api_conditional_updates.rst
Normal file
391
doc/source/devref/api_conditional_updates.rst
Normal file
@ -0,0 +1,391 @@
|
||||
API Races - Conditional Updates
|
||||
===============================
|
||||
|
||||
Background
|
||||
----------
|
||||
|
||||
On Cinder API nodes we have to check that requested action can be performed by
|
||||
checking request arguments and involved resources, and only if everything
|
||||
matches required criteria we will proceed with the RPC call to any of the other
|
||||
nodes.
|
||||
|
||||
Checking the conditions must be done in a non racy way to ensure that already
|
||||
checked requirements don't change while we check remaining conditions. This is
|
||||
of utter importance, as Cinder uses resource status as a lock to prevent
|
||||
concurrent operations on a resource.
|
||||
|
||||
An simple example of this would be extending a volume, where we first check the
|
||||
status:
|
||||
|
||||
.. code:: python
|
||||
|
||||
if volume['status'] != 'available':
|
||||
|
||||
Then update the status:
|
||||
|
||||
.. code:: python
|
||||
|
||||
self.update(context, volume, {'status': 'extending'})
|
||||
|
||||
And finally make the RPC call:
|
||||
|
||||
.. code:: python
|
||||
|
||||
self.volume_rpcapi.extend_volume(context, volume, new_size,
|
||||
reservations)
|
||||
|
||||
The problem is that that this code would allow races, as other request could
|
||||
have already changed the volume status between us getting the value and
|
||||
updating the DB.
|
||||
|
||||
There are multiple ways to fix this, such as:
|
||||
|
||||
- Using a Distributed Locking Mechanism
|
||||
- Using DB isolation level
|
||||
- Using SQL SELECT ... FOR UPDATE
|
||||
- USING compare and swap mechanism in SQL query
|
||||
|
||||
Our tests showed that the best alternative was compare and swap and we decided
|
||||
to call this mechanism "Conditional Update" as it seemed more appropriate.
|
||||
|
||||
Conditional Update
|
||||
------------------
|
||||
|
||||
Conditional Update is the mechanism we use in Cinder to prevent races when
|
||||
updating the DB. In essence it is the SQL equivalent of an ``UPDATE ... FROM
|
||||
... WHERE;`` clause
|
||||
|
||||
It is implemented as an abstraction layer on top of SQLAlchemy ORM engine in
|
||||
our DB api layer and exposed for consumption in Cinder's Persistent Versioned
|
||||
Objects through the ``conditional_update`` method so it can be used from any
|
||||
Vesioned Object instance that has persistence (Volume, Snapshot, Backup...).
|
||||
|
||||
Method signature is:
|
||||
|
||||
.. code:: python
|
||||
|
||||
def conditional_update(self, values, expected_values=None, filters=(),
|
||||
save_all=False, session=None, reflect_changes=True):
|
||||
|
||||
:values:
|
||||
Dictionary of key-value pairs with changes that we want to make to the
|
||||
resource in the DB.
|
||||
|
||||
:expected_values:
|
||||
Dictionary with conditions that must be met for the update to be executed.
|
||||
|
||||
Condition ``field.id == resource.id`` is implicit and there is no need to add
|
||||
it to the conditions.
|
||||
|
||||
If no ``expected_values`` argument is provided update will only go through if
|
||||
no field in the DB has changed. Dirty fields from the Versioned Object are
|
||||
excluded as we don't know their original value.
|
||||
|
||||
:filters:
|
||||
Additional SQLAlchemy filters can be provided for more complex conditions.
|
||||
|
||||
:save_all:
|
||||
By default we will only be updating the DB with values provided in the
|
||||
``values`` argument, but we can explicitly say that we also want to save
|
||||
object's current dirty fields.
|
||||
|
||||
:session:
|
||||
A SQLAlchemy session can be provided, although it is unlikely to be needed.
|
||||
|
||||
:reflect_changes:
|
||||
On a successful update we will also update Versioned Object instance to
|
||||
reflect these changes, but we can prevent this instance update passing False
|
||||
on this argument.
|
||||
|
||||
:Return Value:
|
||||
We'll return the number of changed rows. So we'll get a 0 value if the
|
||||
conditional update has not been successful instead of an exception.
|
||||
|
||||
Basic Usage
|
||||
-----------
|
||||
|
||||
- **Simple match**
|
||||
|
||||
The most basic example is doing a simple match, for example for a ``volume``
|
||||
variable that contains a Versioned Object Volume class instance we may want
|
||||
to change the ``status`` to "deleting" and update the ``terminated_at`` field
|
||||
with current UTC time only if current ``status`` is "available" and the
|
||||
volume is not in a consistency group.
|
||||
|
||||
.. code:: python
|
||||
|
||||
values={'status': 'deleting',
|
||||
'terminated_at': timeutils.utcnow()}
|
||||
expected_values = {'status': 'available',
|
||||
'consistencygroup_id': None}
|
||||
|
||||
volume.conditional_update(values, expected_values)
|
||||
|
||||
- **Iterable match**
|
||||
|
||||
Conditions can contain not only single values, but also iterables, and the
|
||||
conditional update mechanism will correctly handle the presence of None
|
||||
values in the range, unlike SQL ``IN`` clause that doesn't support ``NULL``
|
||||
values.
|
||||
|
||||
.. code:: python
|
||||
|
||||
values={'status': 'deleting',
|
||||
'terminated_at': timeutils.utcnow()}
|
||||
expected_values={
|
||||
'status': ('available', 'error', 'error_restoring' 'error_extending'),
|
||||
'migration_status': (None, 'deleting', 'error', 'success'),
|
||||
'consistencygroup_id': None
|
||||
}
|
||||
|
||||
volume.conditional_update(values, expected_values)
|
||||
|
||||
- **Exclusion**
|
||||
|
||||
In some cases we'll need to set conditions on what is *not* in the DB record
|
||||
instead of what is is, for that we will use the exclusion mechanism provided
|
||||
by the ``Not`` class in all persistent objects. This class accepts single
|
||||
values as well as iterables.
|
||||
|
||||
.. code:: python
|
||||
|
||||
values={'status': 'deleting',
|
||||
'terminated_at': timeutils.utcnow()}
|
||||
expected_values={
|
||||
'attach_status': volume.Not('attached'),
|
||||
'status': ('available', 'error', 'error_restoring' 'error_extending'),
|
||||
'migration_status': (None, 'deleting', 'error', 'success'),
|
||||
'consistencygroup_id': None
|
||||
}
|
||||
|
||||
volume.conditional_update(values, expected_values)
|
||||
|
||||
- **Filters**
|
||||
|
||||
We can use complex filters in the conditions, but these must be SQLAlchemy
|
||||
queries/conditions and as the rest of the DB methods must be properly
|
||||
abstracted from the API.
|
||||
|
||||
Therefore we will create the medhot in cinder/db/sqlalchemy/api.py:
|
||||
|
||||
.. code:: python
|
||||
|
||||
def volume_has_snapshots_filter():
|
||||
return sql.exists().where(
|
||||
and_(models.Volume.id == models.Snapshot.volume_id,
|
||||
~models.Snapshot.deleted))
|
||||
|
||||
Then expose this filter through the cinder/db/api.py:
|
||||
|
||||
.. code:: python
|
||||
|
||||
def volume_has_snapshots_filter():volume_has_snapshots_filter
|
||||
return IMPL.volume_has_snapshots_filter()
|
||||
|
||||
And finally used in the API (notice how we are negating the filter at the
|
||||
API):
|
||||
|
||||
.. code:: python
|
||||
|
||||
filters = [~db.volume_has_snapshots_filter()]
|
||||
values={'status': 'deleting',
|
||||
'terminated_at': timeutils.utcnow()}
|
||||
expected_values={
|
||||
'attach_status': volume.Not('attached'),
|
||||
'status': ('available', 'error', 'error_restoring' 'error_extending'),
|
||||
'migration_status': (None, 'deleting', 'error', 'success'),
|
||||
'consistencygroup_id': None
|
||||
}
|
||||
|
||||
volume.conditional_update(values, expected_values, filters)
|
||||
|
||||
Building filters on the API
|
||||
---------------------------
|
||||
|
||||
SQLAlchemy filters created as mentioned above can create very powerful and
|
||||
complex conditions, but sometimes we may require a condition that, while more
|
||||
complex than the basic match and not match on the resource fields, it's still
|
||||
quite simple. For those cases we can create filters directly on the API using
|
||||
the ``model`` field provided in Versioned Objects.
|
||||
|
||||
This ``model`` field is a reference to the ORM model that allows us to
|
||||
reference ORM fields.
|
||||
|
||||
We'll use as an example changing the ``status`` field of a backup to
|
||||
"restoring" if the backup status is "available" and the volume where we are
|
||||
going to restore the backup is also in "available" state.
|
||||
|
||||
Joining of tables is implicit when using a model different from the one used
|
||||
for the Versioned Object instance.
|
||||
|
||||
- **As expected_values**
|
||||
|
||||
Since this is a matching case we can use ``expected_values`` argument to make
|
||||
the condition:
|
||||
|
||||
.. code:: python
|
||||
|
||||
values = {'status': 'restoring'}
|
||||
expected_values={'status': 'available',
|
||||
objects.Volume.model.id: volume.id,
|
||||
objects.Volume.model.status: 'available'}
|
||||
|
||||
- **As filters**
|
||||
|
||||
We can also use the ``filters`` argument to achieve the same results:
|
||||
|
||||
.. code:: python
|
||||
|
||||
filters = [objects.Volume.model.id == volume.id,
|
||||
objects.Volume.model.status == 'available']
|
||||
|
||||
- **Other filters**
|
||||
|
||||
If we are not doing a match for the condition the only available option will
|
||||
be to use ``filters`` argument. For example if we want to do a check on the
|
||||
volume size against the backup size:
|
||||
|
||||
.. code:: python
|
||||
|
||||
filters = [objects.Volume.model.id == volume.id,
|
||||
objects.Volume.model.size >= backup.model.size]
|
||||
|
||||
Using DB fields for assignment
|
||||
------------------------------
|
||||
|
||||
- **Using non modified fields**
|
||||
|
||||
Similar to the way we use the fields to specify conditions, we can also use
|
||||
them to set values in the DB.
|
||||
|
||||
For example when we disable a service we want to keep existing ``updated_at``
|
||||
field value:
|
||||
|
||||
.. code:: python
|
||||
|
||||
values = {'disabled': True,
|
||||
'updated_at': service.model.updated_at}
|
||||
|
||||
- **Using modified field**
|
||||
|
||||
In some cases we may need to use a DB field that we are also updating, for
|
||||
example when we are updating the ``status`` but we also want to keep the old
|
||||
value in the ``previous_status`` field.
|
||||
|
||||
.. code:: python
|
||||
|
||||
values = {'status': 'retyping',
|
||||
'previous_status': volume.model.status}
|
||||
|
||||
Conditional update mechanism takes into account that MySQL does not follow
|
||||
SQL language specs and adjusts the query creation accordingly.
|
||||
|
||||
- **Together with filters**
|
||||
|
||||
Using DB fields for assignment together with using them for values can give
|
||||
us advanced functionality like for example increasing a quota value based on
|
||||
current value and making sure we don't exceed our quota limits.
|
||||
|
||||
.. code:: python
|
||||
|
||||
values = {'in_use': quota.model.in_use + volume.size}
|
||||
filters = [quota.model.in_use <= max_usage - volume.size]
|
||||
|
||||
Conditional value setting
|
||||
-------------------------
|
||||
|
||||
Under certain circumstances you may not know what value should be set in the DB
|
||||
because it depends on another field or on another condition. For those cases
|
||||
we can use the ``Case`` class present in our persistent Versioned Objects which
|
||||
implements the SQL CASE clause.
|
||||
|
||||
The idea is simple, using ``Case`` class we can say which values to set in a
|
||||
field based on conditions and also set a default value if none of the
|
||||
conditions are True.
|
||||
|
||||
Conditions must be SQLAlchemy conditions, so we'll need to use fields from the
|
||||
``model`` attribute.
|
||||
|
||||
For example setting the status to "maintenance" during migration if current
|
||||
status is "available" and leaving it as it was if it's not can be done using
|
||||
the following:
|
||||
|
||||
.. code:: python
|
||||
|
||||
values = {
|
||||
'status': volume.Case(
|
||||
[
|
||||
(volume.model.status == 'available', 'maintenance')
|
||||
],
|
||||
else_=volume.model.status)
|
||||
}
|
||||
|
||||
reflect_changes considerations
|
||||
------------------------------
|
||||
|
||||
As we've already mentioned ``conditional_update`` method will update Versioned
|
||||
Object instance with provided values if the row in the DB has been updated, and
|
||||
in most cases this is OK since we can set the values directly because we are
|
||||
using simple values, but there are cases where we don't know what value we
|
||||
should set in the instance, and is in those cases where the default
|
||||
``reflect_changes`` value of True has performance implications.
|
||||
|
||||
There are 2 cases where Versioned Object ``conditional_update`` method doesn't
|
||||
know the value it has to set on the Versioned Object instance, and they are
|
||||
when we use a field for assignment and when we are using the ``Case`` class,
|
||||
since in both cases the DB is the one deciding the value that will be set.
|
||||
|
||||
In those cases ``conditional_update`` will have to retrieve the value from the
|
||||
DB using ``get_by_id`` method, and this has a performance impact and therefore
|
||||
should be avoided when possible.
|
||||
|
||||
So the recommendation is to set ``reflect_changes`` to False when using
|
||||
``Case`` class or using fields in the ``values`` argument if we don't care
|
||||
about the stored value.
|
||||
|
||||
Limitations
|
||||
-----------
|
||||
|
||||
We can only use functionality that works on **all** supported DBs, and that's
|
||||
why we don't allow multi table updates and will raise DBError exception even
|
||||
when the code is running against a DB engine that supports this functionality.
|
||||
|
||||
This way we make sure that we don't inadvertently add a multi table update that
|
||||
works on MySQL but will surely fail on PostgreSQL.
|
||||
|
||||
Considerations for new ORM & Versioned Objects
|
||||
----------------------------------------------
|
||||
|
||||
Conditional update mechanism works using generic methods for getting an object
|
||||
from the DB as well as determining the model for a specific Versioned Object
|
||||
instance for field binding.
|
||||
|
||||
These generic methods rely on some naming rules for Versioned Object classes,
|
||||
ORM classes, and get methods, so when we are creating a new ORM class and
|
||||
adding the matching Versioned Object and access methods we must be careful to
|
||||
follow these rules or at least specify exceptions if we have a good reason not
|
||||
to follow these conventions.
|
||||
|
||||
Rules:
|
||||
|
||||
- Versioned Object class name must be the same as the ORM class
|
||||
- Get method name must be ORM class converted to snake format with postfix
|
||||
"_get". For example, for ``Volume`` ORM class expected method is
|
||||
``volume_get``, and for an imaginary ``MyORMClass`` it would be
|
||||
``my_orm_class_get``.
|
||||
- Get method must receive the ``context`` as the first argument and the ``id``
|
||||
as the second one, although it may accept more optional arguments.
|
||||
|
||||
We should avoid diverging from these rules whenever is possible, but there are
|
||||
cases where this is not possible, for example ``BackupImport`` Versioned Object
|
||||
that really uses ``Backup`` ORM class. For cases such as this we have a way to
|
||||
set exceptions both for the generic get method and the model for a Versioned
|
||||
Object.
|
||||
|
||||
To add exceptions for the get method we have to add a new entry to
|
||||
``GET_EXCEPTIONS`` dictionary mapping in
|
||||
``cinder.db.sqlalchemy.api._get_get_method``.
|
||||
|
||||
And for determining the model for the Versioned Object we have to add a new
|
||||
entry to ``VO_TO_MODEL_EXCEPTIONS`` dictionary mapping in
|
||||
``cinder.db.sqlalchemy.api.get_model_for_versioned_object``.
|
||||
@ -28,6 +28,7 @@ Programming HowTos and Tutorials
|
||||
|
||||
development.environment
|
||||
api_microversion_dev
|
||||
api_conditional_updates
|
||||
api_microversion_history
|
||||
unit_tests
|
||||
addmethod.openstackapi
|
||||
|
||||
Loading…
Reference in New Issue
Block a user