This specification describes the implementation of ETAG identifiers. This patch is part of a series of patches which propose several discrete units of work, broken out into separate specifications for tracking. These have grown out of the discussions around whether we should have a 'v2' API, and what current pain-points we all experience with the current API. It is my belief, after investigating all of the collected concerns, that we can implement many of the desired changes incrementally and without a full 'v2' rewrite. Change-Id: Id8dc039fbf56f7e74dd79256b7a223f6eb673ad2 Related-bug: #1605728
14 KiB
API Evolution: etag ID
As time has gone on, the API surface has evolved, and the community has given rise to more use-cases and more users. This has resulted in the identification of several shortcomings in the current REST API: it is not as user-friendly as one might expect, and it does not service all of the use-cases we wish that it would.
This specification describes the implementation of etag identifiers on API responses, allowing for better conflict resolution between concurrent API clients.
https://bugs.launchpad.net/ironic/+bug/1605728
Problem description
Multiple clients attempting to update the same resource can overwrite each others' changes accidentally and without notice. This is referred to as the lost update problem, and it is a problem in the Bare Metal service. The lost update problem is commonly addressed through the use of "etag" identifiers, as described in this API Working Group specification on etags. This specification proposes that the Bare Metal service provides etags support to address this problem.
Proposed change
The Bare Metal service shall begin to store, internally, a unique etag identifier for every API modifiable resource (Node, Port, Portgroup, Chassis). This identifier shall be returned in the body and headers of successful responses to GET requests for requested resource beginning with the introduction of a new microversion. There are mainly two cases:
GET one resource(subresource). The ETag will be returned in the response headers and body. Python clients may operate with resource using fresh etag (see client section). Ironic shell users see etag in the response.
GET list of resources(subresources). Etag of each resource will be available in response body, as field of the resource:
- {
-
- 'ports': [
-
- {
-
uuid: '11111111-2222-3333-4444-555555555555',
<all other fields>,
etag: 'W/eeeeeeeeeeeeee'
},
- {
-
uuid: '66666666-7777-8888-9999-222222222222',
<all other fields>,
etag: 'W/tttttttttttttt'
}
]
}
Note that, putting etags to headers does not make sense, because it is impossible to distinguish etags at client side in standard and simple way.
All requests to modify a resource or subresource, whether through a PUT, PATCH, or DELETE request, SHOULD begin to require If-Match header with an appropriate etag identifier to be supplied in the request (for subresource it is etag of that subresource). Note that If-Match will not be required according to RFC standard specification of If-Match Header.
Therefore SHOULD keyword used here which means that
If-Match
header will be optional parameter in client (see
keywords to indicate
requirement levels).
This is important that originally according to rfc entity-tags
using If-Match
MUST be provided by clients to pass
request successfully. When ETags are used for cache validation it is
useful. Specifically to our use case making If-Match
header
necessary will decrease user-friendliness of ironic client. It is up to
users to decide if they want to be aware of what they change.
If-None-Match
or any other 'Precondition Header Fields'
will not be supported.
To save efficiency of the request the If-Match header SHALL be
validated at API and conductor side by comparing the supplied If-Match
header against the current etag string. If the supplied header does not
match the actual value, a 412 Precondition Failed
SHALL be
returned.
On the successful receipt of any request to modify a resource or
subresource, a new etag identifier SHALL be generated by the server (it
MUST NOT depend on which ironic-api-version
came in the
request). Where the modification is synchronous and the response already
includes a representation of the resource, the new etag identifier SHALL
be included in the response body and header.
Etag is a SHA-512 hash generated from JSON string encoded compactly (ordering of dict is not needed) from the dictionary of oslo versioned object fields. Etag will be generated for ALL create and modify requests taking into account the fields except of the ignored ones:
For node ignored fields are:
driver_internal_info, etag, updated_at
For port, portgroup and chassis it is only etag
and
updated_at
.
The generated etag will contain "W/" prefix to specify that weak validator is used. Strong validators are more useful for comparison, but in our use-case, taking into account, that etag is not changed on every update and not when metadata changes (e.g. Content-Type), weak validators are applied. See weak vs strong validators.
Alternatives
If we do not implement etag support, we will not have any means to prevent races between clients' PATCH requests, and we will not be able to implement support for PUT as a means to update resources or subresources.
Data model impact
As on every REST API request the objects are obtained from database and while it is not going to be changed, etag shall be retrieved from object returned from db. This is more efficiently than spending time to generate hash again and again.
For that purpose a new field etag
SHALL be added to each
resource table to store the etag identifier. Etag identifier will be
String Field with data length limited to 130 characters (etag is a
string containing prefix "W/" + SHA-512 hexadecimal number 128 digits
long).
New field etag
SHALL be also added to Object model in
order to be consistent with db layer. Object layer will also be the
place where etag will be regenerated based on current Object fields.
Etag will be also included to notifications payload to make them more flexible and usable.
State Machine Impact
None
REST API impact
A new etag header MUST be sent in response to all GET and POST requests starting from specific API microversion, as well as synchronous PATCH and PUT requests.
The same If-Match etag header SHOULD be accepted in all PUT, PATCH and DELETE requests. That means that each endpoint offering any update capability should have logic to validate etag optioanlly.
A new error status 412 PreconditionFailed
SHALL be
introduced and used to signal the clients that their version of a
resource is stale, when the supplied etag header does not match the
server's version.
Client (CLI) impact
Using new microversion clients get the ability to update resource
being aware of what exactly they change. For that they SHOULD send an
If-Match
header with an etag identifier and supported
Ironic API version in the header of requests to modify any resource.
There are two options: doing this either through CLI or through Python
Client API. The last option is available for any python developer script
used in clouds (useful for production).
The workflow of etags usage in ironicclient shell:
Client does GET request.
Starting from specific API microversion, response SHALL include an etag for requested resource(s) in the headers. ETag SHALL be also included within returned resources in the body, both for GET individual resource and resource collection.
Etag may be specified by users if needed when doing any operation leading to resource changing by adding
--etag
flag to the command. Etag can be obtained from body or headers of response:ironic --ironic-api-version 1.40 node-update \ --etag <etag_string> driver_info/foo=value
This etag string is put as
If-Match
header to the request.Ironic API pops
If-Match
header, checks it with rpc_node's etag and if they match, the entity tag is sent further to RPC where conductor validates it again. If etags are not matching at some point, the412 PreconditionFailed
error will be raised. If requestedX-Openstack-Ironic-Api-Version
does not support etags yet,NotAcceptable
error is raised.
To make Python Client API usable without shell, resources will be
stored as full-featured objects (not just the bag of attributes),
including the etag identifier. To do this the ironicclient API will be
rewritten in the way that the Resource
class is able to
update itself and call manager to send requests available in
NodeManager. The Resource
will be stored in the memory like
all Python objects are stored during process execution.
For the Python API all appropriate actions of Resource
object will accept optional parameter etag. The workflow can be the
following:
In Python Shell or in some script clients do GET request to the resource. The etag returned in the response will be stored in the resource representation. E.g.
node = node_manager.get(node_ident)
Afterwards at any time user scripts can do the action on the resource itself:
new_node = node.update(patch, etag=True)
Afterwards they will have the up-to-date resource representation if the request will be validated on the server.
Note, that for 1 standard deprecation period
If-Match
will not be sent to server by default. Clients will be warned that in the next releaseetag
parameter will beTrue
by default.
If etag is requested it will be retrieved from current resource representation
(as node.etag
or getattr(node, 'etag')
).
Afterwards it will be sent as If-Match
header, it means
that user cares about up-to-date information. If etag is not present at
the resource and clients did not turn off etag
option, they
will fail if using the API version greater or equal than etag API
version.
Depending on the situation, the client may choose to transparently retry, or display a diff to the user of the stored vs. server-side resource. Clients should also begin alerting the user when an update request fails due to a resource conflict.
"ironic" CLI
See above.
"openstack baremetal" CLI
Same workflow described in the Client Impact.
RPC API impact
RPC API version needs to be bumped to accept etag parameter for
actions on the resource. The etag
parameter, default as
None, should be passed to the appropriate methods.
Driver API impact
None
Nova driver impact
Nova ironic driver may use new Ironic API microversion, so ironic api
version used by nova virt driver needs to be bumped. Until etag option
in python ironicclient API is True, in the nova driver we should
explicitly specify etag=False
in appropriate methods
through Node
resource object.
Ramdisk impact
None
Security impact
None
Other end user impact
Sending If-Match
header may fail due to
412 Precondition Failed
error. A client may retry with new
fresh etag or/and display a diff between two resource's
representations.
Scalability impact
None
Performance Impact
New etag's generation may increase the time to respond depending on the resource size.
Other deployer impact
Some services (e.g. Nova) change baremetal resources through API, so they may upgrade Ironic API to use etag. If services do not upgrade, warn the deployer about that in the case skipping these upgrades may violate some strong recommendations and information consistency is not guaranteed on ironic side.
Developer impact
Python developers can work with Resource
object as with
full-featured objects and perform modifying operations on them. Also
they can implement scripts that are using etag option in parallel in
efficient way.
Implementation
Assignee(s)
- Primary assignee:
-
galyna
- Other contributors:
-
vdrok
Work Items
- Implement database migration, adding an internal etag field to all top-level resources.
- Implement generation and validation utility functions in common code.
- Implement changes within the ironic.api.controllers.v1 modules to accept and return etag identifiers when fetching or changing resources as appropriate.
- Implement unit tests and tempest tests.
- Update api-ref documentation.
- Implement changes in the python client library and openstack CLI to begin caching etags on GET requests and sending etags on PUT/PATCH/DELETE requests.
Dependencies
None
Testing
Unit and tempest tests shall be added that ensure etag identifiers are returned, that they are validated by requests to modify resources, and that proper errors are returned when an invalid (or merely not current) etag is supplied.
Upgrades and Backwards Compatibility
Backwards compatibility is retained because etags SHOULD only be returned and required in new microversions.
This change does not include substantive changes to any resource.
Documentation Impact
The proper use of etag identifiers shall be documented in our API reference.