Enable middleware to set metadata on object POST

Adds a new form of system metadata for objects.

Sysmeta cannot be updated by an object POST because
that would cause all existing sysmeta to be deleted.
Crypto middleware will want to add 'system' metadata
to object metadata on PUTs and POSTs, but it is ok
for this metadata to be replaced en-masse on every
POST.

This patch introduces x-object-transient-sysmeta-*
that is persisted by object servers and returned
in GET and HEAD responses, just like user metadata,
without polluting the x-object-meta-* namespace.
All headers in this namespace will be filtered
inbound and outbound by the gatekeeper, so cannot
be set or read by clients.

Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Janie Richling <jrichli@us.ibm.com>

Change-Id: I5075493329935ba6790543fc82ea6e039704811d
This commit is contained in:
Alistair Coles 2016-06-06 18:38:50 +01:00
parent fa7d80029b
commit 3ad003cf51
16 changed files with 450 additions and 80 deletions

View File

@ -200,6 +200,8 @@ core swift features which predate sysmeta have added exceptions for
custom non-user metadata headers (e.g. :ref:`acls`, custom non-user metadata headers (e.g. :ref:`acls`,
:ref:`large-objects`) :ref:`large-objects`)
.. _usermeta:
^^^^^^^^^^^^^ ^^^^^^^^^^^^^
User Metadata User Metadata
^^^^^^^^^^^^^ ^^^^^^^^^^^^^
@ -209,7 +211,7 @@ User metadata takes the form of ``X-<type>-Meta-<key>: <value>``, where
and ``<key>`` and ``<value>`` are set by the client. and ``<key>`` and ``<value>`` are set by the client.
User metadata should generally be reserved for use by the client or User metadata should generally be reserved for use by the client or
client applications. An perfect example use-case for user metadata is client applications. A perfect example use-case for user metadata is
`python-swiftclient`_'s ``X-Object-Meta-Mtime`` which it stores on `python-swiftclient`_'s ``X-Object-Meta-Mtime`` which it stores on
object it uploads to implement its ``--changed`` option which will only object it uploads to implement its ``--changed`` option which will only
upload files that have changed since the last upload. upload files that have changed since the last upload.
@ -223,6 +225,20 @@ borrows the user metadata namespace is :ref:`tempurl`. An example of
middleware which uses custom non-user metadata to avoid the user middleware which uses custom non-user metadata to avoid the user
metadata namespace is :ref:`slo-doc`. metadata namespace is :ref:`slo-doc`.
User metadata that is stored by a PUT or POST request to a container or account
resource persists until it is explicitly removed by a subsequent PUT or POST
request that includes a header ``X-<type>-Meta-<key>`` with no value or a
header ``X-Remove-<type>-Meta-<key>: <ignored-value>``. In the latter case the
``<ignored-value>`` is not stored. All user metadata stored with an account or
container resource is deleted when the account or container is deleted.
User metadata that is stored with an object resource has a different semantic;
object user metadata persists until any subsequent PUT or POST request is made
to the same object, at which point all user metadata stored with that object is
deleted en-masse and replaced with any user metadata included with the PUT or
POST request. As a result, it is not possible to update a subset of the user
metadata items stored with an object while leaving some items unchanged.
.. _sysmeta: .. _sysmeta:
^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^
@ -237,7 +253,7 @@ Swift WSGI Server.
All headers on client requests in the form of ``X-<type>-Sysmeta-<key>`` All headers on client requests in the form of ``X-<type>-Sysmeta-<key>``
will be dropped from the request before being processed by any will be dropped from the request before being processed by any
middleware. All headers on responses from back-end systems in the form middleware. All headers on responses from back-end systems in the form
of ``X-<type>-Sysmeta-<key>`` will be removed after all middleware has of ``X-<type>-Sysmeta-<key>`` will be removed after all middlewares have
processed the response but before the response is sent to the client. processed the response but before the response is sent to the client.
See :ref:`gatekeeper` middleware for more information. See :ref:`gatekeeper` middleware for more information.
@ -249,3 +265,50 @@ modified directly by client requests, and the outgoing filter ensures
that removing middleware that uses a specific system metadata key that removing middleware that uses a specific system metadata key
renders it benign. New middleware should take advantage of system renders it benign. New middleware should take advantage of system
metadata. metadata.
System metadata may be set on accounts and containers by including headers with
a PUT or POST request. Where a header name matches the name of an existing item
of system metadata, the value of the existing item will be updated. Otherwise
existing items are preserved. A system metadata header with an empty value will
cause any existing item with the same name to be deleted.
System metadata may be set on objects using only PUT requests. All items of
existing system metadata will be deleted and replaced en-masse by any system
metadata headers included with the PUT request. System metadata is neither
updated nor deleted by a POST request: updating individual items of system
metadata with a POST request is not yet supported in the same way that updating
individual items of user metadata is not supported. In cases where middleware
needs to store its own metadata with a POST request, it may use Object Transient
Sysmeta.
^^^^^^^^^^^^^^^^^^^^^^^^
Object Transient-Sysmeta
^^^^^^^^^^^^^^^^^^^^^^^^
If middleware needs to store object metadata with a POST request it may do so
using headers of the form ``X-Object-Transient-Sysmeta-<key>: <value>``.
All headers on client requests in the form of
``X-Object-Transient-Sysmeta-<key>`` will be dropped from the request before
being processed by any middleware. All headers on responses from back-end
systems in the form of ``X-Object-Transient-Sysmeta-<key>`` will be removed
after all middlewares have processed the response but before the response is
sent to the client. See :ref:`gatekeeper` middleware for more information.
Transient-sysmeta updates on an object have the same semantic as user
metadata updates on an object (see :ref:`usermeta`) i.e. whenever any PUT or
POST request is made to an object, all existing items of transient-sysmeta are
deleted en-masse and replaced with any transient-sysmeta included with the PUT
or POST request. Transient-sysmeta set by a middleware is therefore prone to
deletion by a subsequent client-generated POST request unless the middleware is
careful to include its transient-sysmeta with every POST. Likewise, user
metadata set by a client is prone to deletion by a subsequent
middleware-generated POST request, and for that reason middleware should avoid
generating POST requests that are independent of any client request.
Transient-sysmeta deliberately uses a different header prefix to user metadata
so that middlewares can avoid potential conflict with user metadata keys.
Transient-sysmeta deliberately uses a different header prefix to system
metadata to emphasize the fact that the data is only persisted until a
subsequent POST.

View File

@ -145,7 +145,7 @@ from swift.common.http import HTTP_MULTIPLE_CHOICES, HTTP_CREATED, \
is_success, HTTP_OK is_success, HTTP_OK
from swift.common.constraints import check_account_format, MAX_FILE_SIZE from swift.common.constraints import check_account_format, MAX_FILE_SIZE
from swift.common.request_helpers import copy_header_subset, remove_items, \ from swift.common.request_helpers import copy_header_subset, remove_items, \
is_sys_meta, is_sys_or_user_meta is_sys_meta, is_sys_or_user_meta, is_object_transient_sysmeta
from swift.common.wsgi import WSGIContext, make_subrequest from swift.common.wsgi import WSGIContext, make_subrequest
@ -206,16 +206,18 @@ def _check_destination_header(req):
'<container name>/<object name>') '<container name>/<object name>')
def _copy_headers_into(from_r, to_r): def _copy_headers(src, dest):
""" """
Will copy desired headers from from_r to to_r Will copy desired headers from src to dest.
:params from_r: a swob Request or Response
:params to_r: a swob Request or Response :params src: an instance of collections.Mapping
:params dest: an instance of collections.Mapping
""" """
pass_headers = ['x-delete-at'] for k, v in src.items():
for k, v in from_r.headers.items(): if (is_sys_or_user_meta('object', k) or
if is_sys_or_user_meta('object', k) or k.lower() in pass_headers: is_object_transient_sysmeta(k) or
to_r.headers[k] = v k.lower() == 'x-delete-at'):
dest[k] = v
class ServerSideCopyWebContext(WSGIContext): class ServerSideCopyWebContext(WSGIContext):
@ -422,9 +424,7 @@ class ServerSideCopyMiddleware(object):
source_resp.headers['last-modified'] source_resp.headers['last-modified']
# Existing sys and user meta of source object is added to response # Existing sys and user meta of source object is added to response
# headers in addition to the new ones. # headers in addition to the new ones.
for k, v in sink_req.headers.items(): _copy_headers(sink_req.headers, resp_headers)
if is_sys_or_user_meta('object', k) or k.lower() == 'x-delete-at':
resp_headers[k] = v
return resp_headers return resp_headers
def handle_PUT(self, req, start_response): def handle_PUT(self, req, start_response):
@ -511,10 +511,10 @@ class ServerSideCopyMiddleware(object):
remove_items(sink_req.headers, condition) remove_items(sink_req.headers, condition)
copy_header_subset(source_resp, sink_req, condition) copy_header_subset(source_resp, sink_req, condition)
else: else:
# Copy/update existing sysmeta and user meta # Copy/update existing sysmeta, transient-sysmeta and user meta
_copy_headers_into(source_resp, sink_req) _copy_headers(source_resp.headers, sink_req.headers)
# Copy/update new metadata provided in request if any # Copy/update new metadata provided in request if any
_copy_headers_into(req, sink_req) _copy_headers(req.headers, sink_req.headers)
# Create response headers for PUT response # Create response headers for PUT response
resp_headers = self._create_response_headers(source_path, resp_headers = self._create_response_headers(source_path,

View File

@ -33,22 +33,25 @@ automatically inserted close to the start of the pipeline by the proxy server.
from swift.common.swob import Request from swift.common.swob import Request
from swift.common.utils import get_logger, config_true_value from swift.common.utils import get_logger, config_true_value
from swift.common.request_helpers import remove_items, get_sys_meta_prefix from swift.common.request_helpers import (
remove_items, get_sys_meta_prefix, OBJECT_TRANSIENT_SYSMETA_PREFIX
)
import re import re
#: A list of python regular expressions that will be used to #: A list of python regular expressions that will be used to
#: match against inbound request headers. Matching headers will #: match against inbound request headers. Matching headers will
#: be removed from the request. #: be removed from the request.
# Exclude headers starting with a sysmeta prefix. # Exclude headers starting with a sysmeta prefix.
# Exclude headers starting with object transient system metadata prefix.
# Exclude headers starting with an internal backend header prefix.
# If adding to this list, note that these are regex patterns, # If adding to this list, note that these are regex patterns,
# so use a trailing $ to constrain to an exact header match # so use a trailing $ to constrain to an exact header match
# rather than prefix match. # rather than prefix match.
inbound_exclusions = [get_sys_meta_prefix('account'), inbound_exclusions = [get_sys_meta_prefix('account'),
get_sys_meta_prefix('container'), get_sys_meta_prefix('container'),
get_sys_meta_prefix('object'), get_sys_meta_prefix('object'),
OBJECT_TRANSIENT_SYSMETA_PREFIX,
'x-backend'] 'x-backend']
# 'x-object-sysmeta' is reserved in anticipation of future support
# for system metadata being applied to objects
#: A list of python regular expressions that will be used to #: A list of python regular expressions that will be used to

View File

@ -44,6 +44,9 @@ from swift.common.utils import split_path, validate_device_partition, \
from swift.common.wsgi import make_subrequest from swift.common.wsgi import make_subrequest
OBJECT_TRANSIENT_SYSMETA_PREFIX = 'x-object-transient-sysmeta-'
def get_param(req, name, default=None): def get_param(req, name, default=None):
""" """
Get parameters from an HTTP request ensuring proper handling UTF-8 Get parameters from an HTTP request ensuring proper handling UTF-8
@ -175,6 +178,19 @@ def is_sys_or_user_meta(server_type, key):
return is_user_meta(server_type, key) or is_sys_meta(server_type, key) return is_user_meta(server_type, key) or is_sys_meta(server_type, key)
def is_object_transient_sysmeta(key):
"""
Tests if a header key starts with and is longer than the prefix for object
transient system metadata.
:param key: header key
:returns: True if the key satisfies the test, False otherwise
"""
if len(key) <= len(OBJECT_TRANSIENT_SYSMETA_PREFIX):
return False
return key.lower().startswith(OBJECT_TRANSIENT_SYSMETA_PREFIX)
def strip_user_meta_prefix(server_type, key): def strip_user_meta_prefix(server_type, key):
""" """
Removes the user metadata prefix for a given server type from the start Removes the user metadata prefix for a given server type from the start
@ -199,6 +215,17 @@ def strip_sys_meta_prefix(server_type, key):
return key[len(get_sys_meta_prefix(server_type)):] return key[len(get_sys_meta_prefix(server_type)):]
def strip_object_transient_sysmeta_prefix(key):
"""
Removes the object transient system metadata prefix from the start of a
header key.
:param key: header key
:returns: stripped header key
"""
return key[len(OBJECT_TRANSIENT_SYSMETA_PREFIX):]
def get_user_meta_prefix(server_type): def get_user_meta_prefix(server_type):
""" """
Returns the prefix for user metadata headers for given server type. Returns the prefix for user metadata headers for given server type.
@ -225,6 +252,20 @@ def get_sys_meta_prefix(server_type):
return 'x-%s-%s-' % (server_type.lower(), 'sysmeta') return 'x-%s-%s-' % (server_type.lower(), 'sysmeta')
def get_object_transient_sysmeta(key):
"""
Returns the Object Transient System Metadata header for key.
The Object Transient System Metadata namespace will be persisted by
backend object servers. These headers are treated in the same way as
object user metadata i.e. all headers in this namespace will be
replaced on every POST request.
:param key: metadata key
:returns: the entire object transient system metadata header for key
"""
return '%s%s' % (OBJECT_TRANSIENT_SYSMETA_PREFIX, key)
def remove_items(headers, condition): def remove_items(headers, condition):
""" """
Removes items from a dict whose keys satisfy Removes items from a dict whose keys satisfy

View File

@ -46,7 +46,8 @@ from swift.common.http import is_success
from swift.common.base_storage_server import BaseStorageServer from swift.common.base_storage_server import BaseStorageServer
from swift.common.header_key_dict import HeaderKeyDict from swift.common.header_key_dict import HeaderKeyDict
from swift.common.request_helpers import get_name_and_placement, \ from swift.common.request_helpers import get_name_and_placement, \
is_user_meta, is_sys_or_user_meta, resolve_etag_is_at_header is_user_meta, is_sys_or_user_meta, is_object_transient_sysmeta, \
resolve_etag_is_at_header
from swift.common.swob import HTTPAccepted, HTTPBadRequest, HTTPCreated, \ from swift.common.swob import HTTPAccepted, HTTPBadRequest, HTTPCreated, \
HTTPInternalServerError, HTTPNoContent, HTTPNotFound, \ HTTPInternalServerError, HTTPNoContent, HTTPNotFound, \
HTTPPreconditionFailed, HTTPRequestTimeout, HTTPUnprocessableEntity, \ HTTPPreconditionFailed, HTTPRequestTimeout, HTTPUnprocessableEntity, \
@ -520,7 +521,8 @@ class ObjectController(BaseStorageServer):
metadata = {'X-Timestamp': req_timestamp.internal} metadata = {'X-Timestamp': req_timestamp.internal}
self._preserve_slo_manifest(metadata, orig_metadata) self._preserve_slo_manifest(metadata, orig_metadata)
metadata.update(val for val in request.headers.items() metadata.update(val for val in request.headers.items()
if is_user_meta('object', val[0])) if (is_user_meta('object', val[0]) or
is_object_transient_sysmeta(val[0])))
headers_to_copy = ( headers_to_copy = (
request.headers.get( request.headers.get(
'X-Backend-Replication-Headers', '').split() + 'X-Backend-Replication-Headers', '').split() +
@ -767,9 +769,11 @@ class ObjectController(BaseStorageServer):
'Content-Length': str(upload_size), 'Content-Length': str(upload_size),
} }
metadata.update(val for val in request.headers.items() metadata.update(val for val in request.headers.items()
if is_sys_or_user_meta('object', val[0])) if (is_sys_or_user_meta('object', val[0]) or
is_object_transient_sysmeta(val[0])))
metadata.update(val for val in footer_meta.items() metadata.update(val for val in footer_meta.items()
if is_sys_or_user_meta('object', val[0])) if (is_sys_or_user_meta('object', val[0]) or
is_object_transient_sysmeta(val[0])))
headers_to_copy = ( headers_to_copy = (
request.headers.get( request.headers.get(
'X-Backend-Replication-Headers', '').split() + 'X-Backend-Replication-Headers', '').split() +
@ -861,8 +865,9 @@ class ObjectController(BaseStorageServer):
response.headers['Content-Type'] = metadata.get( response.headers['Content-Type'] = metadata.get(
'Content-Type', 'application/octet-stream') 'Content-Type', 'application/octet-stream')
for key, value in metadata.items(): for key, value in metadata.items():
if is_sys_or_user_meta('object', key) or \ if (is_sys_or_user_meta('object', key) or
key.lower() in self.allowed_headers: is_object_transient_sysmeta(key) or
key.lower() in self.allowed_headers):
response.headers[key] = value response.headers[key] = value
response.etag = metadata['ETag'] response.etag = metadata['ETag']
response.last_modified = math.ceil(float(file_x_ts)) response.last_modified = math.ceil(float(file_x_ts))
@ -913,8 +918,9 @@ class ObjectController(BaseStorageServer):
response.headers['Content-Type'] = metadata.get( response.headers['Content-Type'] = metadata.get(
'Content-Type', 'application/octet-stream') 'Content-Type', 'application/octet-stream')
for key, value in metadata.items(): for key, value in metadata.items():
if is_sys_or_user_meta('object', key) or \ if (is_sys_or_user_meta('object', key) or
key.lower() in self.allowed_headers: is_object_transient_sysmeta(key) or
key.lower() in self.allowed_headers):
response.headers[key] = value response.headers[key] = value
response.etag = metadata['ETag'] response.etag = metadata['ETag']
ts = Timestamp(metadata['X-Timestamp']) ts = Timestamp(metadata['X-Timestamp'])

View File

@ -58,7 +58,8 @@ from swift.common.swob import Request, Response, Range, \
status_map status_map
from swift.common.request_helpers import strip_sys_meta_prefix, \ from swift.common.request_helpers import strip_sys_meta_prefix, \
strip_user_meta_prefix, is_user_meta, is_sys_meta, is_sys_or_user_meta, \ strip_user_meta_prefix, is_user_meta, is_sys_meta, is_sys_or_user_meta, \
http_response_to_document_iters http_response_to_document_iters, is_object_transient_sysmeta, \
strip_object_transient_sysmeta_prefix
from swift.common.storage_policy import POLICIES from swift.common.storage_policy import POLICIES
@ -180,12 +181,18 @@ def headers_to_object_info(headers, status_int=HTTP_OK):
Construct a cacheable dict of object info based on response headers. Construct a cacheable dict of object info based on response headers.
""" """
headers, meta, sysmeta = _prep_headers_to_info(headers, 'object') headers, meta, sysmeta = _prep_headers_to_info(headers, 'object')
transient_sysmeta = {}
for key, val in headers.iteritems():
if is_object_transient_sysmeta(key):
key = strip_object_transient_sysmeta_prefix(key.lower())
transient_sysmeta[key] = val
info = {'status': status_int, info = {'status': status_int,
'length': headers.get('content-length'), 'length': headers.get('content-length'),
'type': headers.get('content-type'), 'type': headers.get('content-type'),
'etag': headers.get('etag'), 'etag': headers.get('etag'),
'meta': meta, 'meta': meta,
'sysmeta': sysmeta, 'sysmeta': sysmeta,
'transient_sysmeta': transient_sysmeta
} }
return info return info

View File

@ -339,6 +339,8 @@ class Test(ReplProbeTest):
def test_sysmeta_after_replication_with_subsequent_post(self): def test_sysmeta_after_replication_with_subsequent_post(self):
sysmeta = {'x-object-sysmeta-foo': 'sysmeta-foo'} sysmeta = {'x-object-sysmeta-foo': 'sysmeta-foo'}
usermeta = {'x-object-meta-bar': 'meta-bar'} usermeta = {'x-object-meta-bar': 'meta-bar'}
transient_sysmeta = {
'x-object-transient-sysmeta-bar': 'transient-sysmeta-bar'}
self.brain.put_container(policy_index=int(self.policy)) self.brain.put_container(policy_index=int(self.policy))
# put object # put object
self._put_object() self._put_object()
@ -356,11 +358,13 @@ class Test(ReplProbeTest):
# post some user meta to second server subset # post some user meta to second server subset
self.brain.stop_handoff_half() self.brain.stop_handoff_half()
self.container_brain.stop_handoff_half() self.container_brain.stop_handoff_half()
self._post_object(usermeta) user_and_transient_sysmeta = dict(usermeta)
user_and_transient_sysmeta.update(transient_sysmeta)
self._post_object(user_and_transient_sysmeta)
metadata = self._get_object_metadata() metadata = self._get_object_metadata()
for key in usermeta: for key in user_and_transient_sysmeta:
self.assertTrue(key in metadata) self.assertTrue(key in metadata)
self.assertEqual(metadata[key], usermeta[key]) self.assertEqual(metadata[key], user_and_transient_sysmeta[key])
for key in sysmeta: for key in sysmeta:
self.assertFalse(key in metadata) self.assertFalse(key in metadata)
self.brain.start_handoff_half() self.brain.start_handoff_half()
@ -376,6 +380,7 @@ class Test(ReplProbeTest):
metadata = self._get_object_metadata() metadata = self._get_object_metadata()
expected = dict(sysmeta) expected = dict(sysmeta)
expected.update(usermeta) expected.update(usermeta)
expected.update(transient_sysmeta)
for key in expected.keys(): for key in expected.keys():
self.assertTrue(key in metadata, key) self.assertTrue(key in metadata, key)
self.assertEqual(metadata[key], expected[key]) self.assertEqual(metadata[key], expected[key])
@ -399,6 +404,8 @@ class Test(ReplProbeTest):
def test_sysmeta_after_replication_with_prior_post(self): def test_sysmeta_after_replication_with_prior_post(self):
sysmeta = {'x-object-sysmeta-foo': 'sysmeta-foo'} sysmeta = {'x-object-sysmeta-foo': 'sysmeta-foo'}
usermeta = {'x-object-meta-bar': 'meta-bar'} usermeta = {'x-object-meta-bar': 'meta-bar'}
transient_sysmeta = {
'x-object-transient-sysmeta-bar': 'transient-sysmeta-bar'}
self.brain.put_container(policy_index=int(self.policy)) self.brain.put_container(policy_index=int(self.policy))
# put object # put object
self._put_object() self._put_object()
@ -406,11 +413,13 @@ class Test(ReplProbeTest):
# put user meta to first server subset # put user meta to first server subset
self.brain.stop_handoff_half() self.brain.stop_handoff_half()
self.container_brain.stop_handoff_half() self.container_brain.stop_handoff_half()
self._post_object(headers=usermeta) user_and_transient_sysmeta = dict(usermeta)
user_and_transient_sysmeta.update(transient_sysmeta)
self._post_object(user_and_transient_sysmeta)
metadata = self._get_object_metadata() metadata = self._get_object_metadata()
for key in usermeta: for key in user_and_transient_sysmeta:
self.assertTrue(key in metadata) self.assertTrue(key in metadata)
self.assertEqual(metadata[key], usermeta[key]) self.assertEqual(metadata[key], user_and_transient_sysmeta[key])
self.brain.start_handoff_half() self.brain.start_handoff_half()
self.container_brain.start_handoff_half() self.container_brain.start_handoff_half()
@ -436,7 +445,7 @@ class Test(ReplProbeTest):
for key in sysmeta: for key in sysmeta:
self.assertTrue(key in metadata) self.assertTrue(key in metadata)
self.assertEqual(metadata[key], sysmeta[key]) self.assertEqual(metadata[key], sysmeta[key])
for key in usermeta: for key in user_and_transient_sysmeta:
self.assertFalse(key in metadata) self.assertFalse(key in metadata)
self.brain.start_primary_half() self.brain.start_primary_half()
self.container_brain.start_primary_half() self.container_brain.start_primary_half()
@ -449,7 +458,7 @@ class Test(ReplProbeTest):
for key in sysmeta: for key in sysmeta:
self.assertTrue(key in metadata) self.assertTrue(key in metadata)
self.assertEqual(metadata[key], sysmeta[key]) self.assertEqual(metadata[key], sysmeta[key])
for key in usermeta: for key in user_and_transient_sysmeta:
self.assertFalse(key in metadata) self.assertFalse(key in metadata)
self.brain.start_handoff_half() self.brain.start_handoff_half()
self.container_brain.start_handoff_half() self.container_brain.start_handoff_half()

View File

@ -19,6 +19,8 @@ from collections import defaultdict
from hashlib import md5 from hashlib import md5
from swift.common import swob from swift.common import swob
from swift.common.header_key_dict import HeaderKeyDict from swift.common.header_key_dict import HeaderKeyDict
from swift.common.request_helpers import is_user_meta, \
is_object_transient_sysmeta
from swift.common.swob import HTTPNotImplemented from swift.common.swob import HTTPNotImplemented
from swift.common.utils import split_path from swift.common.utils import split_path
@ -87,7 +89,7 @@ class FakeSwift(object):
if resp: if resp:
return resp(env, start_response) return resp(env, start_response)
req_headers = swob.Request(env).headers req = swob.Request(env)
self.swift_sources.append(env.get('swift.source')) self.swift_sources.append(env.get('swift.source'))
self.txn_ids.append(env.get('swift.trans_id')) self.txn_ids.append(env.get('swift.trans_id'))
@ -114,26 +116,41 @@ class FakeSwift(object):
# simulate object PUT # simulate object PUT
if method == 'PUT' and obj: if method == 'PUT' and obj:
input = ''.join(iter(env['wsgi.input'].read, '')) put_body = ''.join(iter(env['wsgi.input'].read, ''))
if 'swift.callback.update_footers' in env: if 'swift.callback.update_footers' in env:
footers = HeaderKeyDict() footers = HeaderKeyDict()
env['swift.callback.update_footers'](footers) env['swift.callback.update_footers'](footers)
req_headers.update(footers) req.headers.update(footers)
etag = md5(input).hexdigest() etag = md5(put_body).hexdigest()
headers.setdefault('Etag', etag) headers.setdefault('Etag', etag)
headers.setdefault('Content-Length', len(input)) headers.setdefault('Content-Length', len(put_body))
# keep it for subsequent GET requests later # keep it for subsequent GET requests later
self.uploaded[path] = (dict(req_headers), input) self.uploaded[path] = (dict(req.headers), put_body)
if "CONTENT_TYPE" in env: if "CONTENT_TYPE" in env:
self.uploaded[path][0]['Content-Type'] = env["CONTENT_TYPE"] self.uploaded[path][0]['Content-Type'] = env["CONTENT_TYPE"]
# simulate object POST
elif method == 'POST' and obj:
metadata, data = self.uploaded.get(path, ({}, None))
# select items to keep from existing...
new_metadata = dict(
(k, v) for k, v in metadata.items()
if (not is_user_meta('object', k) and not
is_object_transient_sysmeta(k)))
# apply from new
new_metadata.update(
dict((k, v) for k, v in req.headers.items()
if (is_user_meta('object', k) or
is_object_transient_sysmeta(k) or
k.lower == 'content-type')))
self.uploaded[path] = new_metadata, data
# note: tests may assume this copy of req_headers is case insensitive # note: tests may assume this copy of req_headers is case insensitive
# so we deliberately use a HeaderKeyDict # so we deliberately use a HeaderKeyDict
self._calls.append((method, path, HeaderKeyDict(req_headers))) self._calls.append((method, path, HeaderKeyDict(req.headers)))
# range requests ought to work, hence conditional_response=True # range requests ought to work, hence conditional_response=True
req = swob.Request(env)
if isinstance(body, list): if isinstance(body, list):
resp = resp_class( resp = resp_class(
req=req, headers=headers, app_iter=body, req=req, headers=headers, app_iter=body,

View File

@ -689,9 +689,11 @@ class TestServerSideCopyMiddleware(unittest.TestCase):
source_headers = { source_headers = {
'x-object-sysmeta-test1': 'copy me', 'x-object-sysmeta-test1': 'copy me',
'x-object-meta-test2': 'copy me too', 'x-object-meta-test2': 'copy me too',
'x-object-transient-sysmeta-test3': 'ditto',
'x-object-sysmeta-container-update-override-etag': 'etag val', 'x-object-sysmeta-container-update-override-etag': 'etag val',
'x-object-sysmeta-container-update-override-size': 'size val', 'x-object-sysmeta-container-update-override-size': 'size val',
'x-object-sysmeta-container-update-override-foo': 'bar'} 'x-object-sysmeta-container-update-override-foo': 'bar',
'x-delete-at': 'delete-at-time'}
get_resp_headers = source_headers.copy() get_resp_headers = source_headers.copy()
get_resp_headers['etag'] = 'source etag' get_resp_headers['etag'] = 'source etag'
@ -713,20 +715,20 @@ class TestServerSideCopyMiddleware(unittest.TestCase):
req = Request.blank('/v1/a/c/o', method='COPY', req = Request.blank('/v1/a/c/o', method='COPY',
headers={'Content-Length': 0, headers={'Content-Length': 0,
'Destination': 'c/o-copy0'}) 'Destination': 'c/o-copy0'})
status, headers, body = self.call_ssc(req) status, resp_headers, body = self.call_ssc(req)
self.assertEqual('201 Created', status) self.assertEqual('201 Created', status)
verify_headers(source_headers.copy(), [], headers) verify_headers(source_headers.copy(), [], resp_headers)
method, path, headers = self.app.calls_with_headers[-1] method, path, put_headers = self.app.calls_with_headers[-1]
self.assertEqual('PUT', method) self.assertEqual('PUT', method)
self.assertEqual('/v1/a/c/o-copy0', path) self.assertEqual('/v1/a/c/o-copy0', path)
verify_headers(source_headers.copy(), [], headers.items()) verify_headers(source_headers.copy(), [], put_headers.items())
self.assertIn('etag', headers) self.assertIn('etag', put_headers)
self.assertEqual(headers['etag'], 'source etag') self.assertEqual(put_headers['etag'], 'source etag')
req = Request.blank('/v1/a/c/o-copy0', method='GET') req = Request.blank('/v1/a/c/o-copy0', method='GET')
status, headers, body = self.call_ssc(req) status, resp_headers, body = self.call_ssc(req)
self.assertEqual('200 OK', status) self.assertEqual('200 OK', status)
verify_headers(source_headers.copy(), [], headers) verify_headers(source_headers.copy(), [], resp_headers)
# use a COPY request with a Range header # use a COPY request with a Range header
self.app.register('PUT', '/v1/a/c/o-copy1', swob.HTTPCreated, {}) self.app.register('PUT', '/v1/a/c/o-copy1', swob.HTTPCreated, {})
@ -734,7 +736,7 @@ class TestServerSideCopyMiddleware(unittest.TestCase):
headers={'Content-Length': 0, headers={'Content-Length': 0,
'Destination': 'c/o-copy1', 'Destination': 'c/o-copy1',
'Range': 'bytes=1-2'}) 'Range': 'bytes=1-2'})
status, headers, body = self.call_ssc(req) status, resp_headers, body = self.call_ssc(req)
expected_headers = source_headers.copy() expected_headers = source_headers.copy()
unexpected_headers = ( unexpected_headers = (
'x-object-sysmeta-container-update-override-etag', 'x-object-sysmeta-container-update-override-etag',
@ -743,38 +745,54 @@ class TestServerSideCopyMiddleware(unittest.TestCase):
for h in unexpected_headers: for h in unexpected_headers:
expected_headers.pop(h) expected_headers.pop(h)
self.assertEqual('201 Created', status) self.assertEqual('201 Created', status)
verify_headers(expected_headers, unexpected_headers, headers) verify_headers(expected_headers, unexpected_headers, resp_headers)
method, path, headers = self.app.calls_with_headers[-1] method, path, put_headers = self.app.calls_with_headers[-1]
self.assertEqual('PUT', method) self.assertEqual('PUT', method)
self.assertEqual('/v1/a/c/o-copy1', path) self.assertEqual('/v1/a/c/o-copy1', path)
verify_headers(expected_headers, unexpected_headers, headers.items()) verify_headers(
expected_headers, unexpected_headers, put_headers.items())
# etag should not be copied with a Range request # etag should not be copied with a Range request
self.assertNotIn('etag', headers) self.assertNotIn('etag', put_headers)
req = Request.blank('/v1/a/c/o-copy1', method='GET') req = Request.blank('/v1/a/c/o-copy1', method='GET')
status, headers, body = self.call_ssc(req) status, resp_headers, body = self.call_ssc(req)
self.assertEqual('200 OK', status) self.assertEqual('200 OK', status)
verify_headers(expected_headers, unexpected_headers, headers) verify_headers(expected_headers, unexpected_headers, resp_headers)
# use a PUT with x-copy-from # use a PUT with x-copy-from
self.app.register('PUT', '/v1/a/c/o-copy2', swob.HTTPCreated, {}) self.app.register('PUT', '/v1/a/c/o-copy2', swob.HTTPCreated, {})
req = Request.blank('/v1/a/c/o-copy2', method='PUT', req = Request.blank('/v1/a/c/o-copy2', method='PUT',
headers={'Content-Length': 0, headers={'Content-Length': 0,
'X-Copy-From': 'c/o'}) 'X-Copy-From': 'c/o'})
status, headers, body = self.call_ssc(req) status, resp_headers, body = self.call_ssc(req)
self.assertEqual('201 Created', status) self.assertEqual('201 Created', status)
verify_headers(source_headers.copy(), [], headers) verify_headers(source_headers.copy(), [], resp_headers)
method, path, headers = self.app.calls_with_headers[-1] method, path, put_headers = self.app.calls_with_headers[-1]
self.assertEqual('PUT', method) self.assertEqual('PUT', method)
self.assertEqual('/v1/a/c/o-copy2', path) self.assertEqual('/v1/a/c/o-copy2', path)
verify_headers(source_headers.copy(), [], headers.items()) verify_headers(source_headers.copy(), [], put_headers.items())
self.assertIn('etag', headers) self.assertIn('etag', put_headers)
self.assertEqual(headers['etag'], 'source etag') self.assertEqual(put_headers['etag'], 'source etag')
req = Request.blank('/v1/a/c/o-copy2', method='GET') req = Request.blank('/v1/a/c/o-copy2', method='GET')
status, headers, body = self.call_ssc(req) status, resp_headers, body = self.call_ssc(req)
self.assertEqual('200 OK', status) self.assertEqual('200 OK', status)
verify_headers(source_headers.copy(), [], headers) verify_headers(source_headers.copy(), [], resp_headers)
# copy to same path as source
self.app.register('PUT', '/v1/a/c/o', swob.HTTPCreated, {})
req = Request.blank('/v1/a/c/o', method='PUT',
headers={'Content-Length': 0,
'X-Copy-From': 'c/o'})
status, resp_headers, body = self.call_ssc(req)
self.assertEqual('201 Created', status)
verify_headers(source_headers.copy(), [], resp_headers)
method, path, put_headers = self.app.calls_with_headers[-1]
self.assertEqual('PUT', method)
self.assertEqual('/v1/a/c/o', path)
verify_headers(source_headers.copy(), [], put_headers.items())
self.assertIn('etag', put_headers)
self.assertEqual(put_headers['etag'], 'source etag')
def test_COPY_no_destination_header(self): def test_COPY_no_destination_header(self):
req = Request.blank( req = Request.blank(

View File

@ -74,12 +74,17 @@ class TestGatekeeper(unittest.TestCase):
x_backend_headers = {'X-Backend-Replication': 'true', x_backend_headers = {'X-Backend-Replication': 'true',
'X-Backend-Replication-Headers': 'stuff'} 'X-Backend-Replication-Headers': 'stuff'}
object_transient_sysmeta_headers = {
'x-object-transient-sysmeta-': 'value',
'x-object-transient-sysmeta-foo': 'value'}
x_timestamp_headers = {'X-Timestamp': '1455952805.719739'} x_timestamp_headers = {'X-Timestamp': '1455952805.719739'}
forbidden_headers_out = dict(sysmeta_headers.items() + forbidden_headers_out = dict(sysmeta_headers.items() +
x_backend_headers.items()) x_backend_headers.items() +
object_transient_sysmeta_headers.items())
forbidden_headers_in = dict(sysmeta_headers.items() + forbidden_headers_in = dict(sysmeta_headers.items() +
x_backend_headers.items()) x_backend_headers.items() +
object_transient_sysmeta_headers.items())
shunted_headers_in = dict(x_timestamp_headers.items()) shunted_headers_in = dict(x_timestamp_headers.items())
def _assertHeadersEqual(self, expected, actual): def _assertHeadersEqual(self, expected, actual):

View File

@ -21,8 +21,8 @@ from swift.common.storage_policy import POLICIES, EC_POLICY, REPL_POLICY
from swift.common.request_helpers import is_sys_meta, is_user_meta, \ from swift.common.request_helpers import is_sys_meta, is_user_meta, \
is_sys_or_user_meta, strip_sys_meta_prefix, strip_user_meta_prefix, \ is_sys_or_user_meta, strip_sys_meta_prefix, strip_user_meta_prefix, \
remove_items, copy_header_subset, get_name_and_placement, \ remove_items, copy_header_subset, get_name_and_placement, \
http_response_to_document_iters, update_etag_is_at_header, \ http_response_to_document_iters, is_object_transient_sysmeta, \
resolve_etag_is_at_header update_etag_is_at_header, resolve_etag_is_at_header
from test.unit import patch_policies from test.unit import patch_policies
from test.unit.common.test_utils import FakeResponse from test.unit.common.test_utils import FakeResponse
@ -69,6 +69,14 @@ class TestRequestHelpers(unittest.TestCase):
self.assertEqual(strip_user_meta_prefix(st, 'x-%s-%s-a' self.assertEqual(strip_user_meta_prefix(st, 'x-%s-%s-a'
% (st, mt)), 'a') % (st, mt)), 'a')
def test_is_object_transient_sysmeta(self):
self.assertTrue(is_object_transient_sysmeta(
'x-object-transient-sysmeta-foo'))
self.assertFalse(is_object_transient_sysmeta(
'x-object-transient-sysmeta-'))
self.assertFalse(is_object_transient_sysmeta(
'x-object-meatmeta-foo'))
def test_remove_items(self): def test_remove_items(self):
src = {'a': 'b', src = {'a': 'b',
'c': 'd'} 'c': 'd'}

View File

@ -2374,6 +2374,7 @@ class DiskFileMixin(BaseDiskFileTestMixin):
def test_disk_file_default_disallowed_metadata(self): def test_disk_file_default_disallowed_metadata(self):
# build an object with some meta (at t0+1s) # build an object with some meta (at t0+1s)
orig_metadata = {'X-Object-Meta-Key1': 'Value1', orig_metadata = {'X-Object-Meta-Key1': 'Value1',
'X-Object-Transient-Sysmeta-KeyA': 'ValueA',
'Content-Type': 'text/garbage'} 'Content-Type': 'text/garbage'}
df = self._get_open_disk_file(ts=self.ts().internal, df = self._get_open_disk_file(ts=self.ts().internal,
extra_metadata=orig_metadata) extra_metadata=orig_metadata)
@ -2382,6 +2383,7 @@ class DiskFileMixin(BaseDiskFileTestMixin):
# write some new metadata (fast POST, don't send orig meta, at t0+1) # write some new metadata (fast POST, don't send orig meta, at t0+1)
df = self._simple_get_diskfile() df = self._simple_get_diskfile()
df.write_metadata({'X-Timestamp': self.ts().internal, df.write_metadata({'X-Timestamp': self.ts().internal,
'X-Object-Transient-Sysmeta-KeyB': 'ValueB',
'X-Object-Meta-Key2': 'Value2'}) 'X-Object-Meta-Key2': 'Value2'})
df = self._simple_get_diskfile() df = self._simple_get_diskfile()
with df.open(): with df.open():
@ -2389,8 +2391,11 @@ class DiskFileMixin(BaseDiskFileTestMixin):
self.assertEqual('text/garbage', df._metadata['Content-Type']) self.assertEqual('text/garbage', df._metadata['Content-Type'])
# original fast-post updateable keys are removed # original fast-post updateable keys are removed
self.assertNotIn('X-Object-Meta-Key1', df._metadata) self.assertNotIn('X-Object-Meta-Key1', df._metadata)
self.assertNotIn('X-Object-Transient-Sysmeta-KeyA', df._metadata)
# new fast-post updateable keys are added # new fast-post updateable keys are added
self.assertEqual('Value2', df._metadata['X-Object-Meta-Key2']) self.assertEqual('Value2', df._metadata['X-Object-Meta-Key2'])
self.assertEqual('ValueB',
df._metadata['X-Object-Transient-Sysmeta-KeyB'])
def test_disk_file_preserves_sysmeta(self): def test_disk_file_preserves_sysmeta(self):
# build an object with some meta (at t0) # build an object with some meta (at t0)

View File

@ -1683,7 +1683,8 @@ class TestObjectController(unittest.TestCase):
'ETag': '1000d172764c9dbc3a5798a67ec5bb76', 'ETag': '1000d172764c9dbc3a5798a67ec5bb76',
'X-Object-Meta-1': 'One', 'X-Object-Meta-1': 'One',
'X-Object-Sysmeta-1': 'One', 'X-Object-Sysmeta-1': 'One',
'X-Object-Sysmeta-Two': 'Two'}) 'X-Object-Sysmeta-Two': 'Two',
'X-Object-Transient-Sysmeta-Foo': 'Bar'})
req.body = 'VERIFY SYSMETA' req.body = 'VERIFY SYSMETA'
resp = req.get_response(self.object_controller) resp = req.get_response(self.object_controller)
self.assertEqual(resp.status_int, 201) self.assertEqual(resp.status_int, 201)
@ -1702,7 +1703,8 @@ class TestObjectController(unittest.TestCase):
'name': '/a/c/o', 'name': '/a/c/o',
'X-Object-Meta-1': 'One', 'X-Object-Meta-1': 'One',
'X-Object-Sysmeta-1': 'One', 'X-Object-Sysmeta-1': 'One',
'X-Object-Sysmeta-Two': 'Two'}) 'X-Object-Sysmeta-Two': 'Two',
'X-Object-Transient-Sysmeta-Foo': 'Bar'})
def test_PUT_succeeds_with_later_POST(self): def test_PUT_succeeds_with_later_POST(self):
ts_iter = make_timestamp_iter() ts_iter = make_timestamp_iter()
@ -1875,6 +1877,62 @@ class TestObjectController(unittest.TestCase):
resp = req.get_response(self.object_controller) resp = req.get_response(self.object_controller)
check_response(resp) check_response(resp)
def test_POST_transient_sysmeta(self):
# check that diskfile transient system meta is changed by a POST
timestamp1 = normalize_timestamp(time())
req = Request.blank(
'/sda1/p/a/c/o', environ={'REQUEST_METHOD': 'PUT'},
headers={'X-Timestamp': timestamp1,
'Content-Type': 'text/plain',
'ETag': '1000d172764c9dbc3a5798a67ec5bb76',
'X-Object-Meta-1': 'One',
'X-Object-Sysmeta-1': 'One',
'X-Object-Transient-Sysmeta-Foo': 'Bar'})
req.body = 'VERIFY SYSMETA'
resp = req.get_response(self.object_controller)
self.assertEqual(resp.status_int, 201)
timestamp2 = normalize_timestamp(time())
req = Request.blank(
'/sda1/p/a/c/o', environ={'REQUEST_METHOD': 'POST'},
headers={'X-Timestamp': timestamp2,
'X-Object-Meta-1': 'Not One',
'X-Object-Sysmeta-1': 'Not One',
'X-Object-Transient-Sysmeta-Foo': 'Not Bar'})
resp = req.get_response(self.object_controller)
self.assertEqual(resp.status_int, 202)
# original .data file metadata should be unchanged
objfile = os.path.join(
self.testdir, 'sda1',
storage_directory(diskfile.get_data_dir(0), 'p',
hash_path('a', 'c', 'o')),
timestamp1 + '.data')
self.assertTrue(os.path.isfile(objfile))
self.assertEqual(open(objfile).read(), 'VERIFY SYSMETA')
self.assertDictEqual(diskfile.read_metadata(objfile),
{'X-Timestamp': timestamp1,
'Content-Length': '14',
'Content-Type': 'text/plain',
'ETag': '1000d172764c9dbc3a5798a67ec5bb76',
'name': '/a/c/o',
'X-Object-Meta-1': 'One',
'X-Object-Sysmeta-1': 'One',
'X-Object-Transient-Sysmeta-Foo': 'Bar'})
# .meta file metadata should have only user meta items
metafile = os.path.join(
self.testdir, 'sda1',
storage_directory(diskfile.get_data_dir(0), 'p',
hash_path('a', 'c', 'o')),
timestamp2 + '.meta')
self.assertTrue(os.path.isfile(metafile))
self.assertDictEqual(diskfile.read_metadata(metafile),
{'X-Timestamp': timestamp2,
'name': '/a/c/o',
'X-Object-Meta-1': 'Not One',
'X-Object-Transient-Sysmeta-Foo': 'Not Bar'})
def test_PUT_then_fetch_system_metadata(self): def test_PUT_then_fetch_system_metadata(self):
timestamp = normalize_timestamp(time()) timestamp = normalize_timestamp(time())
req = Request.blank( req = Request.blank(
@ -1884,7 +1942,8 @@ class TestObjectController(unittest.TestCase):
'ETag': '1000d172764c9dbc3a5798a67ec5bb76', 'ETag': '1000d172764c9dbc3a5798a67ec5bb76',
'X-Object-Meta-1': 'One', 'X-Object-Meta-1': 'One',
'X-Object-Sysmeta-1': 'One', 'X-Object-Sysmeta-1': 'One',
'X-Object-Sysmeta-Two': 'Two'}) 'X-Object-Sysmeta-Two': 'Two',
'X-Object-Transient-Sysmeta-Foo': 'Bar'})
req.body = 'VERIFY SYSMETA' req.body = 'VERIFY SYSMETA'
resp = req.get_response(self.object_controller) resp = req.get_response(self.object_controller)
self.assertEqual(resp.status_int, 201) self.assertEqual(resp.status_int, 201)
@ -1903,6 +1962,8 @@ class TestObjectController(unittest.TestCase):
self.assertEqual(resp.headers['x-object-meta-1'], 'One') self.assertEqual(resp.headers['x-object-meta-1'], 'One')
self.assertEqual(resp.headers['x-object-sysmeta-1'], 'One') self.assertEqual(resp.headers['x-object-sysmeta-1'], 'One')
self.assertEqual(resp.headers['x-object-sysmeta-two'], 'Two') self.assertEqual(resp.headers['x-object-sysmeta-two'], 'Two')
self.assertEqual(resp.headers['x-object-transient-sysmeta-foo'],
'Bar')
req = Request.blank('/sda1/p/a/c/o', req = Request.blank('/sda1/p/a/c/o',
environ={'REQUEST_METHOD': 'HEAD'}) environ={'REQUEST_METHOD': 'HEAD'})
@ -1921,9 +1982,13 @@ class TestObjectController(unittest.TestCase):
headers={'X-Timestamp': timestamp, headers={'X-Timestamp': timestamp,
'Content-Type': 'text/plain', 'Content-Type': 'text/plain',
'ETag': '1000d172764c9dbc3a5798a67ec5bb76', 'ETag': '1000d172764c9dbc3a5798a67ec5bb76',
'X-Object-Meta-0': 'deleted by post',
'X-Object-Sysmeta-0': 'Zero',
'X-Object-Transient-Sysmeta-0': 'deleted by post',
'X-Object-Meta-1': 'One', 'X-Object-Meta-1': 'One',
'X-Object-Sysmeta-1': 'One', 'X-Object-Sysmeta-1': 'One',
'X-Object-Sysmeta-Two': 'Two'}) 'X-Object-Sysmeta-Two': 'Two',
'X-Object-Transient-Sysmeta-Foo': 'Bar'})
req.body = 'VERIFY SYSMETA' req.body = 'VERIFY SYSMETA'
resp = req.get_response(self.object_controller) resp = req.get_response(self.object_controller)
self.assertEqual(resp.status_int, 201) self.assertEqual(resp.status_int, 201)
@ -1934,7 +1999,8 @@ class TestObjectController(unittest.TestCase):
headers={'X-Timestamp': timestamp2, headers={'X-Timestamp': timestamp2,
'X-Object-Meta-1': 'Not One', 'X-Object-Meta-1': 'Not One',
'X-Object-Sysmeta-1': 'Not One', 'X-Object-Sysmeta-1': 'Not One',
'X-Object-Sysmeta-Two': 'Not Two'}) 'X-Object-Sysmeta-Two': 'Not Two',
'X-Object-Transient-Sysmeta-Foo': 'Not Bar'})
resp = req.get_response(self.object_controller) resp = req.get_response(self.object_controller)
self.assertEqual(resp.status_int, 202) self.assertEqual(resp.status_int, 202)
@ -1951,8 +2017,13 @@ class TestObjectController(unittest.TestCase):
self.assertEqual(resp.headers['etag'], self.assertEqual(resp.headers['etag'],
'"1000d172764c9dbc3a5798a67ec5bb76"') '"1000d172764c9dbc3a5798a67ec5bb76"')
self.assertEqual(resp.headers['x-object-meta-1'], 'Not One') self.assertEqual(resp.headers['x-object-meta-1'], 'Not One')
self.assertEqual(resp.headers['x-object-sysmeta-0'], 'Zero')
self.assertEqual(resp.headers['x-object-sysmeta-1'], 'One') self.assertEqual(resp.headers['x-object-sysmeta-1'], 'One')
self.assertEqual(resp.headers['x-object-sysmeta-two'], 'Two') self.assertEqual(resp.headers['x-object-sysmeta-two'], 'Two')
self.assertEqual(resp.headers['x-object-transient-sysmeta-foo'],
'Not Bar')
self.assertNotIn('x-object-meta-0', resp.headers)
self.assertNotIn('x-object-transient-sysmeta-0', resp.headers)
req = Request.blank('/sda1/p/a/c/o', req = Request.blank('/sda1/p/a/c/o',
environ={'REQUEST_METHOD': 'HEAD'}) environ={'REQUEST_METHOD': 'HEAD'})

View File

@ -29,7 +29,9 @@ from swift.common.http import is_success
from swift.common.storage_policy import StoragePolicy from swift.common.storage_policy import StoragePolicy
from test.unit import fake_http_connect, FakeRing, FakeMemcache from test.unit import fake_http_connect, FakeRing, FakeMemcache
from swift.proxy import server as proxy_server from swift.proxy import server as proxy_server
from swift.common.request_helpers import get_sys_meta_prefix from swift.common.request_helpers import (
get_sys_meta_prefix, get_object_transient_sysmeta
)
from test.unit import patch_policies from test.unit import patch_policies
@ -537,6 +539,14 @@ class TestFuncs(unittest.TestCase):
self.assertEqual(resp['sysmeta']['whatevs'], 14) self.assertEqual(resp['sysmeta']['whatevs'], 14)
self.assertEqual(resp['sysmeta']['somethingelse'], 0) self.assertEqual(resp['sysmeta']['somethingelse'], 0)
def test_headers_to_object_info_transient_sysmeta(self):
headers = {get_object_transient_sysmeta('Whatevs'): 14,
get_object_transient_sysmeta('somethingelse'): 0}
resp = headers_to_object_info(headers.items(), 200)
self.assertEqual(len(resp['transient_sysmeta']), 2)
self.assertEqual(resp['transient_sysmeta']['whatevs'], 14)
self.assertEqual(resp['transient_sysmeta']['somethingelse'], 0)
def test_headers_to_object_info_values(self): def test_headers_to_object_info_values(self):
headers = { headers = {
'content-length': '1024', 'content-length': '1024',

View File

@ -53,7 +53,7 @@ from swift.common.utils import hash_path, storage_directory, \
iter_multipart_mime_documents, public iter_multipart_mime_documents, public
from test.unit import ( from test.unit import (
connect_tcp, readuntil2crlfs, FakeLogger, FakeRing, fake_http_connect, connect_tcp, readuntil2crlfs, FakeLogger, fake_http_connect, FakeRing,
FakeMemcache, debug_logger, patch_policies, write_fake_ring, FakeMemcache, debug_logger, patch_policies, write_fake_ring,
mocked_http_conn, DEFAULT_TEST_EC_TYPE) mocked_http_conn, DEFAULT_TEST_EC_TYPE)
from swift.proxy import server as proxy_server from swift.proxy import server as proxy_server

View File

@ -28,6 +28,7 @@ from swift.common.wsgi import monkey_patch_mimetools, WSGIContext
from swift.obj import server as object_server from swift.obj import server as object_server
from swift.proxy import server as proxy from swift.proxy import server as proxy
import swift.proxy.controllers import swift.proxy.controllers
from swift.proxy.controllers.base import get_object_info
from test.unit import FakeMemcache, debug_logger, FakeRing, \ from test.unit import FakeMemcache, debug_logger, FakeRing, \
fake_http_connect, patch_policies fake_http_connect, patch_policies
@ -172,6 +173,17 @@ class TestObjectSysmeta(unittest.TestCase):
'x-object-meta-test1': 'meta1 changed'} 'x-object-meta-test1': 'meta1 changed'}
new_meta_headers = {'x-object-meta-test3': 'meta3'} new_meta_headers = {'x-object-meta-test3': 'meta3'}
bad_headers = {'x-account-sysmeta-test1': 'bad1'} bad_headers = {'x-account-sysmeta-test1': 'bad1'}
# these transient_sysmeta headers get changed...
original_transient_sysmeta_headers_1 = \
{'x-object-transient-sysmeta-testA': 'A'}
# these transient_sysmeta headers get deleted...
original_transient_sysmeta_headers_2 = \
{'x-object-transient-sysmeta-testB': 'B'}
# these are replacement transient_sysmeta headers
changed_transient_sysmeta_headers = \
{'x-object-transient-sysmeta-testA': 'changed_A'}
new_transient_sysmeta_headers_1 = {'x-object-transient-sysmeta-testC': 'C'}
new_transient_sysmeta_headers_2 = {'x-object-transient-sysmeta-testD': 'D'}
def test_PUT_sysmeta_then_GET(self): def test_PUT_sysmeta_then_GET(self):
path = '/v1/a/c/o' path = '/v1/a/c/o'
@ -180,6 +192,7 @@ class TestObjectSysmeta(unittest.TestCase):
hdrs = dict(self.original_sysmeta_headers_1) hdrs = dict(self.original_sysmeta_headers_1)
hdrs.update(self.original_meta_headers_1) hdrs.update(self.original_meta_headers_1)
hdrs.update(self.bad_headers) hdrs.update(self.bad_headers)
hdrs.update(self.original_transient_sysmeta_headers_1)
req = Request.blank(path, environ=env, headers=hdrs, body='x') req = Request.blank(path, environ=env, headers=hdrs, body='x')
resp = req.get_response(self.app) resp = req.get_response(self.app)
self._assertStatus(resp, 201) self._assertStatus(resp, 201)
@ -189,6 +202,7 @@ class TestObjectSysmeta(unittest.TestCase):
self._assertStatus(resp, 200) self._assertStatus(resp, 200)
self._assertInHeaders(resp, self.original_sysmeta_headers_1) self._assertInHeaders(resp, self.original_sysmeta_headers_1)
self._assertInHeaders(resp, self.original_meta_headers_1) self._assertInHeaders(resp, self.original_meta_headers_1)
self._assertInHeaders(resp, self.original_transient_sysmeta_headers_1)
self._assertNotInHeaders(resp, self.bad_headers) self._assertNotInHeaders(resp, self.bad_headers)
def test_PUT_sysmeta_then_HEAD(self): def test_PUT_sysmeta_then_HEAD(self):
@ -198,6 +212,7 @@ class TestObjectSysmeta(unittest.TestCase):
hdrs = dict(self.original_sysmeta_headers_1) hdrs = dict(self.original_sysmeta_headers_1)
hdrs.update(self.original_meta_headers_1) hdrs.update(self.original_meta_headers_1)
hdrs.update(self.bad_headers) hdrs.update(self.bad_headers)
hdrs.update(self.original_transient_sysmeta_headers_1)
req = Request.blank(path, environ=env, headers=hdrs, body='x') req = Request.blank(path, environ=env, headers=hdrs, body='x')
resp = req.get_response(self.app) resp = req.get_response(self.app)
self._assertStatus(resp, 201) self._assertStatus(resp, 201)
@ -208,6 +223,7 @@ class TestObjectSysmeta(unittest.TestCase):
self._assertStatus(resp, 200) self._assertStatus(resp, 200)
self._assertInHeaders(resp, self.original_sysmeta_headers_1) self._assertInHeaders(resp, self.original_sysmeta_headers_1)
self._assertInHeaders(resp, self.original_meta_headers_1) self._assertInHeaders(resp, self.original_meta_headers_1)
self._assertInHeaders(resp, self.original_transient_sysmeta_headers_1)
self._assertNotInHeaders(resp, self.bad_headers) self._assertNotInHeaders(resp, self.bad_headers)
def test_sysmeta_replaced_by_PUT(self): def test_sysmeta_replaced_by_PUT(self):
@ -306,6 +322,8 @@ class TestObjectSysmeta(unittest.TestCase):
hdrs.update(self.original_sysmeta_headers_2) hdrs.update(self.original_sysmeta_headers_2)
hdrs.update(self.original_meta_headers_1) hdrs.update(self.original_meta_headers_1)
hdrs.update(self.original_meta_headers_2) hdrs.update(self.original_meta_headers_2)
hdrs.update(self.original_transient_sysmeta_headers_1)
hdrs.update(self.original_transient_sysmeta_headers_2)
req = Request.blank(path, environ=env, headers=hdrs, body='x') req = Request.blank(path, environ=env, headers=hdrs, body='x')
resp = req.get_response(self.copy_app) resp = req.get_response(self.copy_app)
self._assertStatus(resp, 201) self._assertStatus(resp, 201)
@ -315,6 +333,8 @@ class TestObjectSysmeta(unittest.TestCase):
hdrs.update(self.new_sysmeta_headers) hdrs.update(self.new_sysmeta_headers)
hdrs.update(self.changed_meta_headers) hdrs.update(self.changed_meta_headers)
hdrs.update(self.new_meta_headers) hdrs.update(self.new_meta_headers)
hdrs.update(self.changed_transient_sysmeta_headers)
hdrs.update(self.new_transient_sysmeta_headers_1)
hdrs.update(self.bad_headers) hdrs.update(self.bad_headers)
hdrs.update({'Destination': dest}) hdrs.update({'Destination': dest})
req = Request.blank(path, environ=env, headers=hdrs) req = Request.blank(path, environ=env, headers=hdrs)
@ -326,6 +346,9 @@ class TestObjectSysmeta(unittest.TestCase):
self._assertInHeaders(resp, self.changed_meta_headers) self._assertInHeaders(resp, self.changed_meta_headers)
self._assertInHeaders(resp, self.new_meta_headers) self._assertInHeaders(resp, self.new_meta_headers)
self._assertInHeaders(resp, self.original_meta_headers_2) self._assertInHeaders(resp, self.original_meta_headers_2)
self._assertInHeaders(resp, self.changed_transient_sysmeta_headers)
self._assertInHeaders(resp, self.new_transient_sysmeta_headers_1)
self._assertInHeaders(resp, self.original_transient_sysmeta_headers_2)
self._assertNotInHeaders(resp, self.bad_headers) self._assertNotInHeaders(resp, self.bad_headers)
req = Request.blank('/v1/a/c/o2', environ={}) req = Request.blank('/v1/a/c/o2', environ={})
@ -337,6 +360,9 @@ class TestObjectSysmeta(unittest.TestCase):
self._assertInHeaders(resp, self.changed_meta_headers) self._assertInHeaders(resp, self.changed_meta_headers)
self._assertInHeaders(resp, self.new_meta_headers) self._assertInHeaders(resp, self.new_meta_headers)
self._assertInHeaders(resp, self.original_meta_headers_2) self._assertInHeaders(resp, self.original_meta_headers_2)
self._assertInHeaders(resp, self.changed_transient_sysmeta_headers)
self._assertInHeaders(resp, self.new_transient_sysmeta_headers_1)
self._assertInHeaders(resp, self.original_transient_sysmeta_headers_2)
self._assertNotInHeaders(resp, self.bad_headers) self._assertNotInHeaders(resp, self.bad_headers)
def test_sysmeta_updated_by_COPY_from(self): def test_sysmeta_updated_by_COPY_from(self):
@ -380,3 +406,84 @@ class TestObjectSysmeta(unittest.TestCase):
self._assertInHeaders(resp, self.new_meta_headers) self._assertInHeaders(resp, self.new_meta_headers)
self._assertInHeaders(resp, self.original_meta_headers_2) self._assertInHeaders(resp, self.original_meta_headers_2)
self._assertNotInHeaders(resp, self.bad_headers) self._assertNotInHeaders(resp, self.bad_headers)
def _test_transient_sysmeta_replaced_by_PUT_or_POST(self, app):
# check transient_sysmeta is replaced en-masse by a POST
path = '/v1/a/c/o'
env = {'REQUEST_METHOD': 'PUT'}
hdrs = dict(self.original_transient_sysmeta_headers_1)
hdrs.update(self.original_transient_sysmeta_headers_2)
hdrs.update(self.original_meta_headers_1)
req = Request.blank(path, environ=env, headers=hdrs, body='x')
resp = req.get_response(app)
self._assertStatus(resp, 201)
req = Request.blank(path, environ={})
resp = req.get_response(app)
self._assertStatus(resp, 200)
self._assertInHeaders(resp, self.original_transient_sysmeta_headers_1)
self._assertInHeaders(resp, self.original_transient_sysmeta_headers_2)
self._assertInHeaders(resp, self.original_meta_headers_1)
info = get_object_info(req.environ, app)
self.assertEqual(2, len(info.get('transient_sysmeta', ())))
self.assertEqual({'testa': 'A', 'testb': 'B'},
info['transient_sysmeta'])
# POST will replace all existing transient_sysmeta and usermeta values
env = {'REQUEST_METHOD': 'POST'}
hdrs = dict(self.changed_transient_sysmeta_headers)
hdrs.update(self.new_transient_sysmeta_headers_1)
req = Request.blank(path, environ=env, headers=hdrs)
resp = req.get_response(app)
self._assertStatus(resp, 202)
req = Request.blank(path, environ={})
resp = req.get_response(app)
self._assertStatus(resp, 200)
self._assertInHeaders(resp, self.changed_transient_sysmeta_headers)
self._assertInHeaders(resp, self.new_transient_sysmeta_headers_1)
self._assertNotInHeaders(resp, self.original_meta_headers_1)
self._assertNotInHeaders(resp,
self.original_transient_sysmeta_headers_2)
info = get_object_info(req.environ, app)
self.assertEqual(2, len(info.get('transient_sysmeta', ())))
self.assertEqual({'testa': 'changed_A', 'testc': 'C'},
info['transient_sysmeta'])
# subsequent PUT replaces all transient_sysmeta and usermeta values
env = {'REQUEST_METHOD': 'PUT'}
hdrs = dict(self.new_transient_sysmeta_headers_2)
hdrs.update(self.original_meta_headers_2)
req = Request.blank(path, environ=env, headers=hdrs, body='x')
resp = req.get_response(app)
self._assertStatus(resp, 201)
req = Request.blank(path, environ={})
resp = req.get_response(app)
self._assertStatus(resp, 200)
self._assertInHeaders(resp, self.original_meta_headers_2)
self._assertInHeaders(resp, self.new_transient_sysmeta_headers_2)
# meta from previous POST should have gone away...
self._assertNotInHeaders(resp, self.changed_transient_sysmeta_headers)
self._assertNotInHeaders(resp, self.new_transient_sysmeta_headers_1)
# sanity check that meta from first PUT did not re-appear...
self._assertNotInHeaders(resp, self.original_meta_headers_1)
self._assertNotInHeaders(resp,
self.original_transient_sysmeta_headers_1)
self._assertNotInHeaders(resp,
self.original_transient_sysmeta_headers_2)
info = get_object_info(req.environ, app)
self.assertEqual(1, len(info.get('transient_sysmeta', ())))
self.assertEqual({'testd': 'D'}, info['transient_sysmeta'])
def test_transient_sysmeta_replaced_by_PUT_or_POST(self):
self._test_transient_sysmeta_replaced_by_PUT_or_POST(self.app)
def test_transient_sysmeta_replaced_by_PUT_or_POST_as_copy(self):
# test post-as-copy by issuing requests to the copy middleware app
self.copy_app.object_post_as_copy = True
self._test_transient_sysmeta_replaced_by_PUT_or_POST(self.copy_app)