Symlink implementation.
Add a symbolic link ("symlink") object support to Swift. This
object will reference another object. GET and HEAD
requests for a symlink object will operate on the referenced object.
DELETE and PUT requests for a symlink object will operate on the
symlink object, not the referenced object, and will delete or
overwrite it, respectively.
POST requests are *not* forwarded to the referenced object and should
be sent directly. POST requests sent to a symlink object will
result in a 307 Error.
Historical information on symlink design can be found here:
https://github.com/openstack/swift-specs/blob/master/specs/in_progress/symlinks.rst.
https://etherpad.openstack.org/p/swift_symlinks
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: Janie Richling <jrichli@us.ibm.com>
Co-Authored-By: Kazuhiro MIYAHARA <miyahara.kazuhiro@lab.ntt.co.jp>
Co-Authored-By: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Change-Id: I838ed71bacb3e33916db8dd42c7880d5bb9f8e18
Signed-off-by: Thiago da Silva <thiago@redhat.com>
This commit is contained in:
Robert Francis
Alistair Coles
parent
0e95730eb3
commit
99b89aea10
@ -164,7 +164,7 @@ ETag_obj_req:
|
||||
manifest objects, this value is the MD5 checksum of the
|
||||
concatenated string of ETag values for each of the segments in
|
||||
the manifest. You are strongly recommended to compute
|
||||
the MD5 checksum value and include it in the request. This
|
||||
the MD5 checksum value and include it in the request. This
|
||||
enables the Object Storage API to check the integrity of the
|
||||
upload. The value is not quoted.
|
||||
in: header
|
||||
@ -850,6 +850,44 @@ X-Storage-Policy:
|
||||
in: header
|
||||
required: false
|
||||
type: string
|
||||
X-Symlink-Target:
|
||||
description: |
|
||||
Set to specify that this is a symlink object.
|
||||
The value is the relative path of the target object in the
|
||||
format <container>/<object>. The target object does not need to
|
||||
exist at the time of symlink creation.
|
||||
You must UTF-8-encode and then URL-encode the names of the
|
||||
container and object before you include them in this header.
|
||||
in: header
|
||||
required: false
|
||||
type: string
|
||||
X-Symlink-Target-Account:
|
||||
description: |
|
||||
Set to specify that this is a cross-account symlink to
|
||||
an object in the account specified in the value.
|
||||
The ``X-Symlink-Target`` must also be set for this to
|
||||
be effective.
|
||||
You must UTF-8-encode and then URL-encode the account name
|
||||
before you include it in this header.
|
||||
in: header
|
||||
required: false
|
||||
type: string
|
||||
X-Symlink-Target-Account_resp:
|
||||
description: |
|
||||
If present, and ``X-Symlink-Target`` is present, then
|
||||
this is a cross-account symlink to
|
||||
an object in the account specified in the value.
|
||||
in: header
|
||||
required: false
|
||||
type: string
|
||||
X-Symlink-Target_resp:
|
||||
description: |
|
||||
If present, this is a symlink object.
|
||||
The value is the relative path of the target object in the
|
||||
format <container>/<object>.
|
||||
in: header
|
||||
required: false
|
||||
type: string
|
||||
X-Timestamp:
|
||||
description: |
|
||||
The date and time in `UNIX Epoch time stamp
|
||||
@ -1092,6 +1130,23 @@ swiftinfo_sig:
|
||||
in: query
|
||||
required: false
|
||||
type: string
|
||||
symlink:
|
||||
description: |
|
||||
If you include the ``symlink=get`` query parameter
|
||||
and the object is a symlink, then the response will include
|
||||
data and metadata from the symlink itself rather than from the target.
|
||||
in: query
|
||||
required: false
|
||||
type: string
|
||||
symlink_copy:
|
||||
description: |
|
||||
If you include the ``symlink=get`` query parameter
|
||||
and the object is a symlink, the target object
|
||||
contents are not copied. Instead, the symlink is copied to
|
||||
create a new symlink to the same target.
|
||||
in: query
|
||||
required: false
|
||||
type: string
|
||||
temp_url_expires:
|
||||
description: |
|
||||
The date and time in `UNIX Epoch time stamp
|
||||
@ -1180,5 +1235,12 @@ name_in_container_get:
|
||||
in: body
|
||||
required: true
|
||||
type: string
|
||||
symlink_path:
|
||||
description: |
|
||||
This field exists only when the object is symlink.
|
||||
This is the target path of the symlink object.
|
||||
in: body
|
||||
required: true
|
||||
type: string
|
||||
|
||||
|
||||
|
||||
@ -96,6 +96,7 @@ Response Parameters
|
||||
- content_type: content_type
|
||||
- bytes: bytes_in_container_get
|
||||
- name: name_in_container_get
|
||||
- symlink_path: symlink_path
|
||||
|
||||
|
||||
Response Example format=json
|
||||
|
||||
@ -111,6 +111,7 @@ Request
|
||||
- temp_url_expires: temp_url_expires
|
||||
- filename: filename
|
||||
- multipart-manifest: multipart-manifest_get
|
||||
- symlink: symlink
|
||||
- Range: Range
|
||||
- If-Match: If-Match
|
||||
- If-None-Match: If-None-Match-get-request
|
||||
@ -139,7 +140,8 @@ Response Parameters
|
||||
- X-Openstack-Request-Id: X-Openstack-Request-Id
|
||||
- Date: Date
|
||||
- X-Static-Large-Object: X-Static-Large-Object
|
||||
|
||||
- X-Symlink-Target: X-Symlink-Target_resp
|
||||
- X-Symlink-Target-Account: X-Symlink-Target-Account_resp
|
||||
|
||||
|
||||
Response Example
|
||||
@ -263,6 +265,8 @@ Request
|
||||
- X-Object-Meta-name: X-Object-Meta-name
|
||||
- If-None-Match: If-None-Match-put-request
|
||||
- X-Trans-Id-Extra: X-Trans-Id-Extra
|
||||
- X-Symlink-Target: X-Symlink-Target
|
||||
- X-Symlink-Target-Account: X-Symlink-Target-Account
|
||||
|
||||
|
||||
Response Parameters
|
||||
@ -321,6 +325,12 @@ The new object contains the same manifest as the original.
|
||||
The segment objects are not copied. Instead, both the original
|
||||
and new manifest objects share the same set of segment objects.
|
||||
|
||||
To copy a symlink either with a COPY or a PUT with the
|
||||
``X-Copy-From`` request, include the ``symlink=get`` query string.
|
||||
The new symlink will have the same target as the original.
|
||||
The target object is not copied. Instead, both the original
|
||||
and new symlinks point to the same target object.
|
||||
|
||||
All metadata is
|
||||
preserved during the object copy. If you specify metadata on the
|
||||
request to copy the object, either PUT or COPY , the metadata
|
||||
@ -396,6 +406,7 @@ Request
|
||||
- container: container
|
||||
- object: object
|
||||
- multipart-manifest: multipart-manifest_copy
|
||||
- symlink: symlink_copy
|
||||
- X-Auth-Token: X-Auth-Token
|
||||
- X-Service-Token: X-Service-Token
|
||||
- Destination: Destination
|
||||
@ -445,6 +456,9 @@ manifest=delete`` query parameter. This operation deletes the
|
||||
segment objects and, if all deletions succeed, this operation
|
||||
deletes the manifest object.
|
||||
|
||||
A DELETE request made to a symlink path will delete the symlink
|
||||
rather than the target object.
|
||||
|
||||
An alternative to using the DELETE operation is to use
|
||||
the POST operation with the ``bulk-delete`` query parameter.
|
||||
|
||||
@ -570,6 +584,7 @@ Request
|
||||
- temp_url_expires: temp_url_expires
|
||||
- filename: filename
|
||||
- multipart-manifest: multipart-manifest_head
|
||||
- symlink: symlink
|
||||
- X-Newest: X-Newest
|
||||
- If-Match: If-Match
|
||||
- If-None-Match: If-None-Match-get-request
|
||||
@ -597,7 +612,8 @@ Response Parameters
|
||||
- Date: Date
|
||||
- X-Static-Large-Object: X-Static-Large-Object
|
||||
- Content-Type: Content-Type_obj_resp
|
||||
|
||||
- X-Symlink-Target: X-Symlink-Target_resp
|
||||
- X-Symlink-Target-Account: X-Symlink-Target-Account_resp
|
||||
|
||||
|
||||
Response Example
|
||||
@ -659,6 +675,15 @@ body. There are alternate uses of the POST operation as follows:
|
||||
can be used to upload an archive (tar file). The archive is then extracted
|
||||
to create objects.
|
||||
|
||||
A POST request must not include X-Symlink-Target header. If it does then a
|
||||
400 status code is returned and the object metadata is not modified.
|
||||
|
||||
When a POST request is sent to a symlink, the metadata will be applied to the
|
||||
symlink, but the request will result in a ``307 Temporary Redirect`` response
|
||||
to the client. The POST is never redirected to the target object, thus a
|
||||
GET/HEAD request to the symlink without ``symlink=get`` will not return the
|
||||
metadata that was sent as part of the POST request.
|
||||
|
||||
Example requests and responses:
|
||||
|
||||
- Create object metadata:
|
||||
|
||||
@ -9,7 +9,7 @@ eventlet_debug = true
|
||||
[pipeline:main]
|
||||
# Yes, proxy-logging appears twice. This is so that
|
||||
# middleware-originated requests get logged too.
|
||||
pipeline = catch_errors gatekeeper healthcheck proxy-logging cache listing_formats bulk tempurl ratelimit crossdomain container_sync tempauth staticweb copy container-quotas account-quotas slo dlo versioned_writes proxy-logging proxy-server
|
||||
pipeline = catch_errors gatekeeper healthcheck proxy-logging cache listing_formats bulk tempurl ratelimit crossdomain container_sync tempauth staticweb copy container-quotas account-quotas slo dlo versioned_writes symlink proxy-logging proxy-server
|
||||
|
||||
[filter:catch_errors]
|
||||
use = egg:swift#catch_errors
|
||||
@ -74,6 +74,9 @@ use = egg:swift#copy
|
||||
[filter:listing_formats]
|
||||
use = egg:swift#listing_formats
|
||||
|
||||
[filter:symlink]
|
||||
use = egg:swift#symlink
|
||||
|
||||
[app:proxy-server]
|
||||
use = egg:swift#proxy
|
||||
allow_account_management = true
|
||||
|
||||
@ -87,7 +87,9 @@ The Object Storage system organizes data in a hierarchy, as follows:
|
||||
object.
|
||||
|
||||
- Upload objects directly to the Object Storage system from a
|
||||
browser by using form **POST** middleware
|
||||
browser by using form **POST** middleware.
|
||||
|
||||
- Create symbolic links to other objects.
|
||||
|
||||
The account, container, and object hierarchy affects the way you
|
||||
interact with the Object Storage API.
|
||||
|
||||
@ -49,6 +49,12 @@ returns the following values for this header,
|
||||
``X-Object-Meta-*`` for objects)
|
||||
* headers listed in ``X-Container-Meta-Access-Control-Expose-Headers``
|
||||
|
||||
.. note::
|
||||
An OPTIONS request to a symlink object will respond with the options for
|
||||
the symlink only, the request will not be redirected to the target object.
|
||||
Therefore, if the symlink's target object is in another container with
|
||||
cors settings, the response will not reflect the settings.
|
||||
|
||||
|
||||
-----------------
|
||||
Sample Javascript
|
||||
|
||||
@ -104,6 +104,7 @@ KS :ref:`keystoneauth`
|
||||
RL :ref:`ratelimit`
|
||||
VW :ref:`versioned_writes`
|
||||
SSC :ref:`copy`
|
||||
SYM :ref:`symlink`
|
||||
======================= =============================
|
||||
|
||||
|
||||
|
||||
@ -244,6 +244,15 @@ StaticWeb
|
||||
:members:
|
||||
:show-inheritance:
|
||||
|
||||
.. _symlink:
|
||||
|
||||
Symlink
|
||||
=======
|
||||
|
||||
.. automodule:: swift.common.middleware.symlink
|
||||
:members:
|
||||
:show-inheritance:
|
||||
|
||||
.. _common_tempauth:
|
||||
|
||||
TempAuth
|
||||
|
||||
@ -36,9 +36,15 @@ synchronization key.
|
||||
.. note::
|
||||
|
||||
If you are using encryption middleware in the cluster from which objects
|
||||
are being synced, then you should follow the instructions to configure
|
||||
are being synced, then you should follow the instructions for
|
||||
:ref:`container_sync_client_config` to be compatible with encryption.
|
||||
|
||||
.. note::
|
||||
|
||||
If you are using symlink middleware in the cluster from which objects
|
||||
are being synced, then you should follow the instructions for
|
||||
:ref:`symlink_container_sync_client_config` to be compatible with symlinks.
|
||||
|
||||
--------------------------
|
||||
Configuring Container Sync
|
||||
--------------------------
|
||||
@ -440,7 +446,7 @@ then a symlink to the container database is created in a sync-containers
|
||||
sub-directory on the same device.
|
||||
|
||||
Similarly, when the container sync metadata keys are deleted, the container
|
||||
server and container-replicator would take care of deleting the symlinks
|
||||
server and container-replicator would take care of deleting the symlinks
|
||||
from ``sync-containers``.
|
||||
|
||||
.. note::
|
||||
|
||||
@ -94,12 +94,12 @@ bind_port = 8080
|
||||
[pipeline:main]
|
||||
# This sample pipeline uses tempauth and is used for SAIO dev work and
|
||||
# testing. See below for a pipeline using keystone.
|
||||
pipeline = catch_errors gatekeeper healthcheck proxy-logging cache listing_formats container_sync bulk tempurl ratelimit tempauth copy container-quotas account-quotas slo dlo versioned_writes proxy-logging proxy-server
|
||||
pipeline = catch_errors gatekeeper healthcheck proxy-logging cache listing_formats container_sync bulk tempurl ratelimit tempauth copy container-quotas account-quotas slo dlo versioned_writes symlink proxy-logging proxy-server
|
||||
|
||||
# The following pipeline shows keystone integration. Comment out the one
|
||||
# above and uncomment this one. Additional steps for integrating keystone are
|
||||
# covered further below in the filter sections for authtoken and keystoneauth.
|
||||
#pipeline = catch_errors gatekeeper healthcheck proxy-logging cache container_sync bulk tempurl ratelimit authtoken keystoneauth copy container-quotas account-quotas slo dlo versioned_writes proxy-logging proxy-server
|
||||
#pipeline = catch_errors gatekeeper healthcheck proxy-logging cache container_sync bulk tempurl ratelimit authtoken keystoneauth copy container-quotas account-quotas slo dlo versioned_writes symlink proxy-logging proxy-server
|
||||
|
||||
[app:proxy-server]
|
||||
use = egg:swift#proxy
|
||||
@ -928,3 +928,12 @@ use = egg:swift#encryption
|
||||
# be automatically inserted for you.
|
||||
[filter:listing_formats]
|
||||
use = egg:swift#listing_formats
|
||||
|
||||
# Note: Put after slo, dlo, versioned_writes, but before encryption in the
|
||||
# pipeline.
|
||||
[filter:symlink]
|
||||
use = egg:swift#symlink
|
||||
# Symlinks can point to other symlinks up to the symloop_max value. If the
|
||||
# number of chained symlinks exceeds the limit ``symloop_max`` a 409
|
||||
# (HTTPConflict) error response will be produced.
|
||||
# symloop_max = 2
|
||||
|
||||
@ -106,6 +106,7 @@ paste.filter_factory =
|
||||
encryption = swift.common.middleware.crypto:filter_factory
|
||||
kms_keymaster = swift.common.middleware.crypto.kms_keymaster:filter_factory
|
||||
listing_formats = swift.common.middleware.listing_formats:filter_factory
|
||||
symlink = swift.common.middleware.symlink:filter_factory
|
||||
|
||||
[build_sphinx]
|
||||
all_files = 1
|
||||
|
||||
@ -199,6 +199,10 @@ class SegmentError(SwiftException):
|
||||
pass
|
||||
|
||||
|
||||
class LinkIterError(SwiftException):
|
||||
pass
|
||||
|
||||
|
||||
class ReplicationException(Exception):
|
||||
pass
|
||||
|
||||
|
||||
@ -157,7 +157,8 @@ class InternalClient(object):
|
||||
'auto_create_account_prefix', default='.')
|
||||
|
||||
def make_request(
|
||||
self, method, path, headers, acceptable_statuses, body_file=None):
|
||||
self, method, path, headers, acceptable_statuses, body_file=None,
|
||||
params=None):
|
||||
"""Makes a request to Swift with retries.
|
||||
|
||||
:param method: HTTP method of request.
|
||||
@ -166,6 +167,8 @@ class InternalClient(object):
|
||||
:param acceptable_statuses: List of acceptable statuses for request.
|
||||
:param body_file: Body file to be passed along with request,
|
||||
defaults to None.
|
||||
:param params: A dict of params to be set in request query string,
|
||||
defaults to None.
|
||||
|
||||
:returns: Response object on success.
|
||||
|
||||
@ -185,6 +188,8 @@ class InternalClient(object):
|
||||
if hasattr(body_file, 'seek'):
|
||||
body_file.seek(0)
|
||||
req.body_file = body_file
|
||||
if params:
|
||||
req.params = params
|
||||
try:
|
||||
resp = req.get_response(self.app)
|
||||
if resp.status_int in acceptable_statuses or \
|
||||
@ -606,14 +611,30 @@ class InternalClient(object):
|
||||
headers=headers)
|
||||
|
||||
def get_object(self, account, container, obj, headers,
|
||||
acceptable_statuses=(2,)):
|
||||
acceptable_statuses=(2,), params=None):
|
||||
"""
|
||||
Returns a 3-tuple (status, headers, iterator of object body)
|
||||
Gets an object.
|
||||
|
||||
:param account: The object's account.
|
||||
:param container: The object's container.
|
||||
:param obj: The object name.
|
||||
:param headers: Headers to send with request, defaults to empty dict.
|
||||
:param acceptable_statuses: List of status for valid responses,
|
||||
defaults to (2,).
|
||||
:param params: A dict of params to be set in request query string,
|
||||
defaults to None.
|
||||
|
||||
:raises UnexpectedResponse: Exception raised when requests fail
|
||||
to get a response with an acceptable status
|
||||
:raises Exception: Exception is raised when code fails in an
|
||||
unexpected way.
|
||||
:returns: A 3-tuple (status, headers, iterator of object body)
|
||||
"""
|
||||
|
||||
headers = headers or {}
|
||||
path = self.make_path(account, container, obj)
|
||||
resp = self.make_request('GET', path, headers, acceptable_statuses)
|
||||
resp = self.make_request(
|
||||
'GET', path, headers, acceptable_statuses, params=params)
|
||||
return (resp.status_int, resp.headers, resp.app_iter)
|
||||
|
||||
def iter_object_lines(
|
||||
@ -697,7 +718,7 @@ class InternalClient(object):
|
||||
:param account: The object's account.
|
||||
:param container: The object's container.
|
||||
:param obj: The object.
|
||||
:param headers: Headers to send with request, defaults ot empty dict.
|
||||
:param headers: Headers to send with request, defaults to empty dict.
|
||||
|
||||
:raises UnexpectedResponse: Exception raised when requests fail
|
||||
to get a response with an acceptable status
|
||||
|
||||
@ -114,45 +114,20 @@ greater than 5GB.
|
||||
|
||||
"""
|
||||
|
||||
from six.moves.urllib.parse import quote, unquote
|
||||
from six.moves.urllib.parse import quote
|
||||
|
||||
from swift.common import utils
|
||||
from swift.common.utils import get_logger, \
|
||||
config_true_value, FileLikeIter, close_if_possible
|
||||
from swift.common.utils import get_logger, config_true_value, FileLikeIter, \
|
||||
close_if_possible
|
||||
from swift.common.swob import Request, HTTPPreconditionFailed, \
|
||||
HTTPRequestEntityTooLarge, HTTPBadRequest, HTTPException
|
||||
from swift.common.http import HTTP_MULTIPLE_CHOICES, is_success, HTTP_OK
|
||||
from swift.common.constraints import check_account_format, MAX_FILE_SIZE
|
||||
from swift.common.request_helpers import copy_header_subset, remove_items, \
|
||||
is_sys_meta, is_sys_or_user_meta, is_object_transient_sysmeta
|
||||
is_sys_meta, is_sys_or_user_meta, is_object_transient_sysmeta, \
|
||||
check_path_header
|
||||
from swift.common.wsgi import WSGIContext, make_subrequest, load_app_config
|
||||
|
||||
|
||||
def _check_path_header(req, name, length, error_msg):
|
||||
"""
|
||||
Validate that the value of path-like header is
|
||||
well formatted. We assume the caller ensures that
|
||||
specific header is present in req.headers.
|
||||
|
||||
:param req: HTTP request object
|
||||
:param name: header name
|
||||
:param length: length of path segment check
|
||||
:param error_msg: error message for client
|
||||
:returns: A tuple with path parts according to length
|
||||
:raise HTTPPreconditionFailed: if header value
|
||||
is not well formatted.
|
||||
"""
|
||||
src_header = unquote(req.headers.get(name))
|
||||
if not src_header.startswith('/'):
|
||||
src_header = '/' + src_header
|
||||
try:
|
||||
return utils.split_path(src_header, length, length, True)
|
||||
except ValueError:
|
||||
raise HTTPPreconditionFailed(
|
||||
request=req,
|
||||
body=error_msg)
|
||||
|
||||
|
||||
def _check_copy_from_header(req):
|
||||
"""
|
||||
Validate that the value from x-copy-from header is
|
||||
@ -164,9 +139,9 @@ def _check_copy_from_header(req):
|
||||
:raise HTTPPreconditionFailed: if x-copy-from value
|
||||
is not well formatted.
|
||||
"""
|
||||
return _check_path_header(req, 'X-Copy-From', 2,
|
||||
'X-Copy-From header must be of the form '
|
||||
'<container name>/<object name>')
|
||||
return check_path_header(req, 'X-Copy-From', 2,
|
||||
'X-Copy-From header must be of the form '
|
||||
'<container name>/<object name>')
|
||||
|
||||
|
||||
def _check_destination_header(req):
|
||||
@ -180,9 +155,9 @@ def _check_destination_header(req):
|
||||
:raise HTTPPreconditionFailed: if destination value
|
||||
is not well formatted.
|
||||
"""
|
||||
return _check_path_header(req, 'Destination', 2,
|
||||
'Destination header must be of the form '
|
||||
'<container name>/<object name>')
|
||||
return check_path_header(req, 'Destination', 2,
|
||||
'Destination header must be of the form '
|
||||
'<container name>/<object name>')
|
||||
|
||||
|
||||
def _copy_headers(src, dest):
|
||||
|
||||
@ -103,13 +103,14 @@ class BaseDecrypterContext(CryptoWSGIContext):
|
||||
to be decrypted but crypto meta was not
|
||||
found.
|
||||
"""
|
||||
value, crypto_meta = extract_crypto_meta(value)
|
||||
extracted_value, crypto_meta = extract_crypto_meta(value)
|
||||
if crypto_meta:
|
||||
self.crypto.check_crypto_meta(crypto_meta)
|
||||
value = self.decrypt_value(value, key, crypto_meta)
|
||||
value = self.decrypt_value(extracted_value, key, crypto_meta)
|
||||
elif required:
|
||||
raise EncryptionException(
|
||||
"Missing crypto meta in value %s" % value)
|
||||
|
||||
return value
|
||||
|
||||
def decrypt_value(self, value, key, crypto_meta):
|
||||
|
||||
@ -92,6 +92,11 @@ def container_to_xml(listing, base_name):
|
||||
'last_modified'):
|
||||
SubElement(sub, field).text = six.text_type(
|
||||
record.pop(field))
|
||||
|
||||
if 'symlink_path' in record:
|
||||
SubElement(sub, 'symlink_path').text = six.text_type(
|
||||
record.pop('symlink_path'))
|
||||
|
||||
return tostring(doc, encoding='UTF-8').replace(
|
||||
"<?xml version='1.0' encoding='UTF-8'?>",
|
||||
'<?xml version="1.0" encoding="UTF-8"?>', 1)
|
||||
|
||||
570
swift/common/middleware/symlink.py
Normal file
570
swift/common/middleware/symlink.py
Normal file
@ -0,0 +1,570 @@
|
||||
# Copyright (c) 2010-2017 OpenStack Foundation
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
# implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""
|
||||
Symlink Middleware
|
||||
|
||||
Symlinks are objects stored in Swift that contains a reference to another
|
||||
object (hereinafter, this is called "target object"). They are analogous to
|
||||
symbolic links in Unix-like operating systems. The existence of a symlink
|
||||
object does not affect the target object in any way. An important use case is
|
||||
to use a path in one container to access an object in a different container,
|
||||
with a different policy. This allows policy cost/performance tradeoffs to be
|
||||
made on individual objects.
|
||||
|
||||
Clients create a Swift symlink by performing a zero-length PUT request
|
||||
with the header ``X-Symlink-Target: <container>/<object>``. For a cross-account
|
||||
symlink, the header ``X-Symlink-Target-Account: <account>`` must be included.
|
||||
If omitted, it is inserted automatically with the account of the symlink
|
||||
object in the PUT request process.
|
||||
|
||||
Symlinks must be zero-byte objects. Attempting to PUT a symlink
|
||||
with a non-empty request body will result in a 400-series error. Also, POST
|
||||
with X-Symlink-Target header always results in a 400-series error. The target
|
||||
object need not exist at symlink-creation time. It is suggested to set the
|
||||
``Content-Type`` of symlink objects to a distinct value such as
|
||||
``application/symlink``.
|
||||
|
||||
A GET/HEAD request to a symlink will resolve in a request to the target
|
||||
object referenced by the symlink's ``X-Symlink-Target-Account`` and
|
||||
``X-Symlink-Target`` headers. The response of the GET/HEAD request will contain
|
||||
a ``Content-Location`` header with the path location of the target object. A
|
||||
GET/HEAD request to a symlink with the query parameter ``?symlink=get`` will
|
||||
resolve in the request targeting the symlink itself.
|
||||
|
||||
A symlink can point to another symlink. Chained symlinks will be traversed
|
||||
until target is not a symlink. If the number of chained symlinks exceeds the
|
||||
limit ``symloop_max`` an error response will be produced. The value of
|
||||
``symloop_max`` can be defined in the symlink config section of
|
||||
proxy-server.conf. If not specified, the default ``symloop_max`` value is 2. If
|
||||
a value less than 1 is specified, the default value will be used.
|
||||
|
||||
A HEAD/GET request to a symlink object behaves as a normal HEAD/GET request
|
||||
to the target object. Therefore issuing a HEAD request to the symlink will
|
||||
return the target metadata, and issuing a GET request to the symlink will
|
||||
return the data and metadata of the target object. Only when a GET/HEAD
|
||||
request sent to a symlink object with the ``?symlink=get`` query string
|
||||
will return the symlink metadata with empty body.
|
||||
|
||||
A POST request to a symlink will result in a 307 TemporaryRedirect response.
|
||||
The response will contain a ``Location`` header with the path of the target
|
||||
object as the value. The request is never redirected to the target object by
|
||||
Swift. Nevertheless, the metadata in the POST request will be applied to the
|
||||
symlink because object servers cannot know for sure if the current object is a
|
||||
symlink or not in eventual consistency.
|
||||
|
||||
A DELETE request to a symlink will delete the symlink itself. The target
|
||||
object will not be deleted.
|
||||
|
||||
A COPY request, or a PUT request with a ``X-Copy-From`` header, to a symlink
|
||||
will copy the target object. The same request to a symlink with the query
|
||||
parameter ``?symlink=get`` will copy the symlink itself.
|
||||
|
||||
An OPTIONS request to a symlink will respond with the options for the symlink
|
||||
only, the request will not be redirected to the target object. Please note that
|
||||
if the symlink's target object is in another container with cors settings, the
|
||||
response will not reflect the settings.
|
||||
|
||||
Tempurls can be used to GET/HEAD symlink objects, but PUT is not allowed and
|
||||
will result in a 400-series error. The GET/HEAD tempurls honor the scope of
|
||||
the tempurl key. Container tempurl will only work on symlinks where the target
|
||||
container is the same as the symlink. In case a symlink targets an object
|
||||
in a different container, a GET/HEAD request will result in a 401 Unauthorized
|
||||
error. The account level tempurl will allow cross container symlinks.
|
||||
|
||||
If a symlink object is overwritten while it is in a versioned container, the
|
||||
symlink object itself is versioned, not the referenced object.
|
||||
|
||||
A GET request to a container which contains symlinks will respond with
|
||||
additional information ``symlink_path`` for each symlink objects.
|
||||
``symlink_path`` information are target path strings of the symlinks. Clients
|
||||
can differentiate symlinks and other objects by this function.
|
||||
|
||||
Errors
|
||||
|
||||
* PUT with the header ``X-Symlink-Target`` with non-zero Content-Length
|
||||
will produce a 400 BadRequest error.
|
||||
|
||||
* POST with the header ``X-Symlink-Target`` will produce a
|
||||
400 BadRequest error.
|
||||
|
||||
* GET/HEAD traversing more than ``symloop_max`` chained symlinks will
|
||||
produce a 409 Conflict error.
|
||||
|
||||
* POSTs will produce a 307 TemporaryRedirect error.
|
||||
|
||||
----------
|
||||
Deployment
|
||||
----------
|
||||
|
||||
Symlinks are enabled by adding the `symlink` middleware to the proxy server
|
||||
WSGI pipeline and including a corresponding filter configuration section in the
|
||||
`proxy-server.conf` file. The `symlink` middleware should be placed after
|
||||
`slo`, `dlo` and `versioned_writes` middleware, but before `encryption`
|
||||
middleware in the pipeline. See the `proxy-server.conf-sample` file for further
|
||||
details. :ref:`Additional steps <symlink_container_sync_client_config>` are
|
||||
required if the container sync feature is being used.
|
||||
|
||||
.. note::
|
||||
|
||||
Once you have deployed `symlink` middleware in your pipeline, you should
|
||||
neither remove the `symlink` middleware nor downgrade swift to a version
|
||||
earlier than symlinks being supported. Doing so may result in unexpected
|
||||
container listing results in addition to symlink objects behaving like a
|
||||
normal object.
|
||||
|
||||
.. _symlink_container_sync_client_config:
|
||||
|
||||
Container sync configuration
|
||||
----------------------------
|
||||
|
||||
If container sync is being used then the `symlink` middleware
|
||||
must be added to the container sync internal client pipeline. The following
|
||||
configuration steps are required:
|
||||
|
||||
#. Create a custom internal client configuration file for container sync (if
|
||||
one is not already in use) based on the sample file
|
||||
`internal-client.conf-sample`. For example, copy
|
||||
`internal-client.conf-sample` to `/etc/swift/container-sync-client.conf`.
|
||||
#. Modify this file to include the `symlink` middleware in the pipeline in
|
||||
the same way as described above for the proxy server.
|
||||
#. Modify the container-sync section of all container server config files to
|
||||
point to this internal client config file using the
|
||||
``internal_client_conf_path`` option. For example::
|
||||
|
||||
internal_client_conf_path = /etc/swift/container-sync-client.conf
|
||||
|
||||
.. note::
|
||||
|
||||
These container sync configuration steps will be necessary for container
|
||||
sync probe tests to pass if the `symlink` middleware is included in the
|
||||
proxy pipeline of a test cluster.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
from cgi import parse_header
|
||||
from six.moves.urllib.parse import unquote
|
||||
|
||||
from swift.common.utils import get_logger, register_swift_info, split_path, \
|
||||
MD5_OF_EMPTY_STRING, closing_if_possible
|
||||
from swift.common.constraints import check_account_format
|
||||
from swift.common.wsgi import WSGIContext, make_subrequest
|
||||
from swift.common.request_helpers import get_sys_meta_prefix, \
|
||||
check_path_header
|
||||
from swift.common.swob import Request, HTTPBadRequest, HTTPTemporaryRedirect, \
|
||||
HTTPException, HTTPConflict, HTTPPreconditionFailed
|
||||
from swift.common.http import is_success
|
||||
from swift.common.exceptions import LinkIterError
|
||||
from swift.common.header_key_dict import HeaderKeyDict
|
||||
|
||||
DEFAULT_SYMLOOP_MAX = 2
|
||||
# Header values for symlink target path strings will be quoted values.
|
||||
TGT_OBJ_SYMLINK_HDR = 'x-symlink-target'
|
||||
TGT_ACCT_SYMLINK_HDR = 'x-symlink-target-account'
|
||||
TGT_OBJ_SYSMETA_SYMLINK_HDR = get_sys_meta_prefix('object') + 'symlink-target'
|
||||
TGT_ACCT_SYSMETA_SYMLINK_HDR = \
|
||||
get_sys_meta_prefix('object') + 'symlink-target-account'
|
||||
|
||||
|
||||
def _check_symlink_header(req):
|
||||
"""
|
||||
Validate that the value from x-symlink-target header is
|
||||
well formatted. We assume the caller ensures that
|
||||
x-symlink-target header is present in req.headers.
|
||||
|
||||
:param req: HTTP request object
|
||||
:raise: HTTPPreconditionFailed if x-symlink-target value
|
||||
is not well formatted.
|
||||
"""
|
||||
# N.B. check_path_header doesn't assert the leading slash and
|
||||
# copy middleware may accpet the format. In the symlink, API
|
||||
# says apparently to use "container/object" format so add the
|
||||
# validation fist, here.
|
||||
if unquote(req.headers[TGT_OBJ_SYMLINK_HDR]).startswith('/'):
|
||||
raise HTTPPreconditionFailed(
|
||||
body='X-Symlink-Target header must be of the '
|
||||
'form <container name>/<object name>',
|
||||
request=req, content_type='text/plain')
|
||||
|
||||
# check container and object format
|
||||
container, obj = check_path_header(
|
||||
req, TGT_OBJ_SYMLINK_HDR, 2,
|
||||
'X-Symlink-Target header must be of the '
|
||||
'form <container name>/<object name>')
|
||||
|
||||
# Check account format if it exists
|
||||
account = check_account_format(
|
||||
req, unquote(req.headers[TGT_ACCT_SYMLINK_HDR])) \
|
||||
if TGT_ACCT_SYMLINK_HDR in req.headers else None
|
||||
|
||||
# Extract request path
|
||||
_junk, req_acc, req_cont, req_obj = req.split_path(4, 4, True)
|
||||
|
||||
if not account:
|
||||
account = req_acc
|
||||
|
||||
# Check if symlink targets the symlink itself or not
|
||||
if (account, container, obj) == (req_acc, req_cont, req_obj):
|
||||
raise HTTPBadRequest(
|
||||
body='Symlink cannot target itself',
|
||||
request=req, content_type='text/plain')
|
||||
|
||||
|
||||
def symlink_usermeta_to_sysmeta(headers):
|
||||
"""
|
||||
Helper fucntion to translate from X-Symlink-Target and
|
||||
X-Symlink-Target-Account to X-Object-Sysmeta-Symlink-Target
|
||||
and X-Object-Sysmeta-Symlink-Target-Account.
|
||||
|
||||
:param headers: request headers dict. Note that the headers dict
|
||||
will be updated directly.
|
||||
"""
|
||||
# To preseve url-encoded value in the symlink header, use raw value
|
||||
if TGT_OBJ_SYMLINK_HDR in headers:
|
||||
headers[TGT_OBJ_SYSMETA_SYMLINK_HDR] = headers.pop(
|
||||
TGT_OBJ_SYMLINK_HDR)
|
||||
|
||||
if TGT_ACCT_SYMLINK_HDR in headers:
|
||||
headers[TGT_ACCT_SYSMETA_SYMLINK_HDR] = headers.pop(
|
||||
TGT_ACCT_SYMLINK_HDR)
|
||||
|
||||
|
||||
def symlink_sysmeta_to_usermeta(headers):
|
||||
"""
|
||||
Helper fucntion to translate from X-Object-Sysmeta-Symlink-Target and
|
||||
X-Object-Sysmeta-Symlink-Target-Account to X-Symlink-Target and
|
||||
X-Sysmeta-Symlink-Target-Account
|
||||
|
||||
:param headers: request headers dict. Note that the headers dict
|
||||
will be updated directly.
|
||||
"""
|
||||
if TGT_OBJ_SYSMETA_SYMLINK_HDR in headers:
|
||||
headers[TGT_OBJ_SYMLINK_HDR] = headers.pop(
|
||||
TGT_OBJ_SYSMETA_SYMLINK_HDR)
|
||||
|
||||
if TGT_ACCT_SYSMETA_SYMLINK_HDR in headers:
|
||||
headers[TGT_ACCT_SYMLINK_HDR] = headers.pop(
|
||||
TGT_ACCT_SYSMETA_SYMLINK_HDR)
|
||||
|
||||
|
||||
class SymlinkContainerContext(WSGIContext):
|
||||
def __init__(self, wsgi_app, logger):
|
||||
super(SymlinkContainerContext, self).__init__(wsgi_app)
|
||||
self.app = wsgi_app
|
||||
self.logger = logger
|
||||
|
||||
def handle_container(self, req, start_response):
|
||||
"""
|
||||
Handle container requests.
|
||||
:return: Response Iterator after start_response called.
|
||||
"""
|
||||
app_resp = self._app_call(req.environ)
|
||||
|
||||
if req.method == 'GET' and is_success(self._get_status_int()):
|
||||
app_resp = self._process_json_resp(app_resp, req)
|
||||
|
||||
start_response(self._response_status, self._response_headers,
|
||||
self._response_exc_info)
|
||||
|
||||
return app_resp
|
||||
|
||||
def _process_json_resp(self, resp_iter, req):
|
||||
"""
|
||||
Iterate through json body looking for symlinks and modify its content
|
||||
:return: modified json body
|
||||
"""
|
||||
with closing_if_possible(resp_iter):
|
||||
resp_body = ''.join(resp_iter)
|
||||
body_json = json.loads(resp_body)
|
||||
swift_version, account, _junk = split_path(req.path, 2, 3, True)
|
||||
new_body = json.dumps(
|
||||
[self._extract_symlink_path_json(obj_dict, swift_version, account)
|
||||
for obj_dict in body_json])
|
||||
self.update_content_length(len(new_body))
|
||||
return [new_body]
|
||||
|
||||
def _extract_symlink_path_json(self, obj_dict, swift_version, account):
|
||||
"""
|
||||
Extract the symlink path from the hash value
|
||||
:return: object dictionary with additional key:value pair if object
|
||||
is a symlink. The new key is symlink_path.
|
||||
"""
|
||||
if 'hash' in obj_dict:
|
||||
hash_value, meta = parse_header(obj_dict['hash'])
|
||||
obj_dict['hash'] = hash_value
|
||||
target = None
|
||||
for key in meta:
|
||||
if key == 'symlink_target':
|
||||
target = meta[key]
|
||||
elif key == 'symlink_target_account':
|
||||
account = meta[key]
|
||||
else:
|
||||
# make sure to add all other (key, values) back in place
|
||||
obj_dict['hash'] += '; %s=%s' % (key, meta[key])
|
||||
else:
|
||||
if target:
|
||||
obj_dict['symlink_path'] = os.path.join(
|
||||
'/', swift_version, account, target)
|
||||
|
||||
return obj_dict
|
||||
|
||||
|
||||
class SymlinkObjectContext(WSGIContext):
|
||||
|
||||
def __init__(self, wsgi_app, logger, symloop_max):
|
||||
super(SymlinkObjectContext, self).__init__(wsgi_app)
|
||||
self.app = wsgi_app
|
||||
self.symloop_max = symloop_max
|
||||
self.logger = logger
|
||||
# N.B. _loop_count and _last_target_path are used to keep
|
||||
# the statement in the _recursive_get. Hence it should not be touched
|
||||
# from other resources.
|
||||
self._loop_count = 0
|
||||
self._last_target_path = None
|
||||
|
||||
def handle_get_head_symlink(self, req):
|
||||
"""
|
||||
Handle get/head request when client sent parameter ?symlink=get
|
||||
|
||||
:param req: HTTP GET or HEAD object request with param ?symlink=get
|
||||
:returns: Response Iterator
|
||||
"""
|
||||
|
||||
resp = self._app_call(req.environ)
|
||||
response_header_dict = HeaderKeyDict(self._response_headers)
|
||||
symlink_sysmeta_to_usermeta(response_header_dict)
|
||||
self._response_headers = response_header_dict.items()
|
||||
return resp
|
||||
|
||||
def handle_get_head(self, req):
|
||||
"""
|
||||
Handle get/head request and in case the response is a symlink,
|
||||
redirect request to target object.
|
||||
|
||||
:param req: HTTP GET or HEAD object request
|
||||
:returns: Response Iterator
|
||||
"""
|
||||
try:
|
||||
return self._recursive_get_head(req)
|
||||
except LinkIterError:
|
||||
errmsg = 'Too many levels of symbolic links, ' \
|
||||
'maximum allowed is %d' % self.symloop_max
|
||||
raise HTTPConflict(
|
||||
body=errmsg, request=req, content_type='text/plain')
|
||||
|
||||
def _recursive_get_head(self, req):
|
||||
resp = self._app_call(req.environ)
|
||||
|
||||
def build_traversal_req(symlink_target):
|
||||
"""
|
||||
:returns: new request for target path if it's symlink otherwise
|
||||
None
|
||||
"""
|
||||
version, account, _junk = split_path(req.path, 2, 3, True)
|
||||
account = self._response_header_value(
|
||||
TGT_ACCT_SYSMETA_SYMLINK_HDR) or account
|
||||
target_path = os.path.join(
|
||||
'/', version, account,
|
||||
symlink_target.lstrip('/'))
|
||||
self._last_target_path = target_path
|
||||
new_req = make_subrequest(
|
||||
req.environ, path=target_path, method=req.method,
|
||||
headers=req.headers, swift_source='SYM')
|
||||
new_req.headers.pop('X-Backend-Storage-Policy-Index', None)
|
||||
return new_req
|
||||
|
||||
symlink_target = self._response_header_value(
|
||||
TGT_OBJ_SYSMETA_SYMLINK_HDR)
|
||||
if symlink_target:
|
||||
if self._loop_count >= self.symloop_max:
|
||||
raise LinkIterError()
|
||||
# format: /<account name>/<container name>/<object name>
|
||||
new_req = build_traversal_req(symlink_target)
|
||||
self._loop_count += 1
|
||||
return self._recursive_get_head(new_req)
|
||||
else:
|
||||
if self._last_target_path:
|
||||
# Content-Location will be applied only when one or more
|
||||
# symlink recursion occurred.
|
||||
# In this case, Content-Location is applied to show which
|
||||
# object path caused the error response.
|
||||
# To preserve '%2F'(= quote('/')) in X-Symlink-Target
|
||||
# header value as it is, Content-Location value comes from
|
||||
# TGT_OBJ_SYMLINK_HDR, not req.path
|
||||
self._response_headers.extend(
|
||||
[('Content-Location', self._last_target_path)])
|
||||
|
||||
return resp
|
||||
|
||||
def handle_put(self, req):
|
||||
"""
|
||||
Handle put request when it contains X-Symlink-Target header.
|
||||
|
||||
Symlink headers are validated and moved to sysmeta namespace.
|
||||
:param req: HTTP PUT object request
|
||||
:returns: Response Iterator
|
||||
"""
|
||||
if req.content_length != 0:
|
||||
raise HTTPBadRequest(
|
||||
body='Symlink requests require a zero byte body',
|
||||
request=req,
|
||||
content_type='text/plain')
|
||||
|
||||
_check_symlink_header(req)
|
||||
symlink_usermeta_to_sysmeta(req.headers)
|
||||
# Store info in container update that this object is a symlink.
|
||||
# We have a design decision to use etag space to store symlink info for
|
||||
# object listing because it's immutable unless the object is
|
||||
# overwritten. This may impact the downgrade scenario that the symlink
|
||||
# info can be appreared as the suffix in the hash value of object
|
||||
# listing result for clients.
|
||||
# To create override etag easily, we have a contraint that the symlink
|
||||
# must be 0 byte so we can add etag of the empty string + symlink info
|
||||
# here, simply. Note that this override etag may be encrypted in the
|
||||
# container db by encrypion middleware.
|
||||
etag_override = [
|
||||
MD5_OF_EMPTY_STRING,
|
||||
'symlink_target=%s' % req.headers[TGT_OBJ_SYSMETA_SYMLINK_HDR]
|
||||
]
|
||||
if TGT_ACCT_SYSMETA_SYMLINK_HDR in req.headers:
|
||||
etag_override.append(
|
||||
'symlink_target_account=%s' %
|
||||
req.headers[TGT_OBJ_SYSMETA_SYMLINK_HDR])
|
||||
req.headers['X-Object-Sysmeta-Container-Update-Override-Etag'] = \
|
||||
'; '.join(etag_override)
|
||||
|
||||
return self._app_call(req.environ)
|
||||
|
||||
def handle_post(self, req):
|
||||
"""
|
||||
Handle post request. If POSTing to a symlink, a HTTPTemporaryRedirect
|
||||
error message is returned to client.
|
||||
|
||||
Clients that POST to symlinks should understand that the POST is not
|
||||
redirected to the target object like in a HEAD/GET request. POSTs to a
|
||||
symlink will be handled just like a normal object by the object server.
|
||||
It cannot reject it because it may not have symlink state when the POST
|
||||
lands. The object server has no knowledge of what is a symlink object
|
||||
is. On the other hand, on POST requests, the object server returns all
|
||||
sysmeta of the object. This method uses that sysmeta to determine if
|
||||
the stored object is a symlink or not.
|
||||
|
||||
:param req: HTTP POST object request
|
||||
:returns: HTTPTemporaryRedirect if POSTing to a symlink.
|
||||
:returns: Response Iterator
|
||||
"""
|
||||
if TGT_OBJ_SYMLINK_HDR in req.headers:
|
||||
raise HTTPBadRequest(
|
||||
body='A PUT request is required to set a symlink target',
|
||||
request=req,
|
||||
content_type='text/plain')
|
||||
|
||||
resp = self._app_call(req.environ)
|
||||
if not is_success(self._get_status_int()):
|
||||
return resp
|
||||
|
||||
tgt_co = self._response_header_value(TGT_OBJ_SYSMETA_SYMLINK_HDR)
|
||||
if tgt_co:
|
||||
version, account, _junk = req.split_path(2, 3, True)
|
||||
target_acc = self._response_header_value(
|
||||
TGT_ACCT_SYSMETA_SYMLINK_HDR) or account
|
||||
location_hdr = os.path.join(
|
||||
'/', version, target_acc, tgt_co)
|
||||
req.environ['swift.leave_relative_location'] = True
|
||||
errmsg = 'The requested POST was applied to a symlink. POST ' +\
|
||||
'directly to the target to apply requested metadata.'
|
||||
raise HTTPTemporaryRedirect(
|
||||
body=errmsg, headers={'location': location_hdr})
|
||||
else:
|
||||
return resp
|
||||
|
||||
def handle_object(self, req, start_response):
|
||||
"""
|
||||
Handle object requests
|
||||
"""
|
||||
if req.method in ('GET', 'HEAD'):
|
||||
# if GET request came from versioned writes, then it should get
|
||||
# the symlink only, not the referenced target
|
||||
if req.params.get('symlink') == 'get' or \
|
||||
req.environ.get('swift.source') == 'VW':
|
||||
resp = self.handle_get_head_symlink(req)
|
||||
else:
|
||||
resp = self.handle_get_head(req)
|
||||
elif req.method == 'PUT' and (TGT_OBJ_SYMLINK_HDR in req.headers):
|
||||
resp = self.handle_put(req)
|
||||
elif req.method == 'POST':
|
||||
resp = self.handle_post(req)
|
||||
else:
|
||||
# DELETE and OPTIONS reqs for a symlink and
|
||||
# PUT reqs without X-Symlink-Target behave like any other object
|
||||
resp = self._app_call(req.environ)
|
||||
|
||||
start_response(self._response_status, self._response_headers,
|
||||
self._response_exc_info)
|
||||
|
||||
return resp
|
||||
|
||||
|
||||
class SymlinkMiddleware(object):
|
||||
"""
|
||||
Middleware that implements symlinks.
|
||||
|
||||
Symlinks are objects stored in Swift that contains a reference to another
|
||||
object (i.e., the target object). An important use case is to use a path in
|
||||
one container to access an object in a different container, with a
|
||||
different policy. This allows policy cost/performance tradeoffs to be made
|
||||
on individual objects.
|
||||
"""
|
||||
|
||||
def __init__(self, app, conf, symloop_max):
|
||||
self.app = app
|
||||
self.conf = conf
|
||||
self.logger = get_logger(self.conf, log_route='symlink')
|
||||
self.symloop_max = symloop_max
|
||||
|
||||
def __call__(self, env, start_response):
|
||||
req = Request(env)
|
||||
try:
|
||||
version, acc, cont, obj = req.split_path(3, 4, True)
|
||||
except ValueError:
|
||||
return self.app(env, start_response)
|
||||
|
||||
try:
|
||||
if obj:
|
||||
# object context
|
||||
context = SymlinkObjectContext(self.app, self.logger,
|
||||
self.symloop_max)
|
||||
return context.handle_object(req, start_response)
|
||||
else:
|
||||
# container context
|
||||
context = SymlinkContainerContext(self.app, self.logger)
|
||||
return context.handle_container(req, start_response)
|
||||
except HTTPException as err_resp:
|
||||
return err_resp(env, start_response)
|
||||
|
||||
|
||||
def filter_factory(global_conf, **local_conf):
|
||||
conf = global_conf.copy()
|
||||
conf.update(local_conf)
|
||||
|
||||
symloop_max = int(conf.get('symloop_max', DEFAULT_SYMLOOP_MAX))
|
||||
if symloop_max < 1:
|
||||
symloop_max = int(DEFAULT_SYMLOOP_MAX)
|
||||
register_swift_info('symlink', symloop_max=symloop_max)
|
||||
|
||||
def symlink_mw(app):
|
||||
return SymlinkMiddleware(app, conf, symloop_max)
|
||||
return symlink_mw
|
||||
@ -222,7 +222,7 @@ from swift.common.utils import split_path, get_valid_utf8_str, \
|
||||
register_swift_info, get_hmac, streq_const_time, quote
|
||||
|
||||
|
||||
DISALLOWED_INCOMING_HEADERS = 'x-object-manifest'
|
||||
DISALLOWED_INCOMING_HEADERS = 'x-object-manifest x-symlink-target'
|
||||
|
||||
#: Default headers to remove from incoming requests. Simply a whitespace
|
||||
#: delimited list of header names and names can optionally end with '*' to
|
||||
|
||||
@ -34,7 +34,8 @@ from swift.common.storage_policy import POLICIES
|
||||
from swift.common.exceptions import ListingIterError, SegmentError
|
||||
from swift.common.http import is_success
|
||||
from swift.common.swob import HTTPBadRequest, \
|
||||
HTTPServiceUnavailable, Range, is_chunked, multi_range_iterator
|
||||
HTTPServiceUnavailable, Range, is_chunked, multi_range_iterator, \
|
||||
HTTPPreconditionFailed
|
||||
from swift.common.utils import split_path, validate_device_partition, \
|
||||
close_if_possible, maybe_multipart_byteranges_to_document_iters, \
|
||||
multipart_byteranges_to_document_iters, parse_content_type, \
|
||||
@ -281,6 +282,31 @@ def copy_header_subset(from_r, to_r, condition):
|
||||
to_r.headers[k] = v
|
||||
|
||||
|
||||
def check_path_header(req, name, length, error_msg):
|
||||
"""
|
||||
Validate that the value of path-like header is
|
||||
well formatted. We assume the caller ensures that
|
||||
specific header is present in req.headers.
|
||||
|
||||
:param req: HTTP request object
|
||||
:param name: header name
|
||||
:param length: length of path segment check
|
||||
:param error_msg: error message for client
|
||||
:returns: A tuple with path parts according to length
|
||||
:raise: HTTPPreconditionFailed if header value
|
||||
is not well formatted.
|
||||
"""
|
||||
hdr = unquote(req.headers.get(name))
|
||||
if not hdr.startswith('/'):
|
||||
hdr = '/' + hdr
|
||||
try:
|
||||
return split_path(hdr, length, length, True)
|
||||
except ValueError:
|
||||
raise HTTPPreconditionFailed(
|
||||
request=req,
|
||||
body=error_msg)
|
||||
|
||||
|
||||
class SegmentedIterable(object):
|
||||
"""
|
||||
Iterable that returns the object contents for a large object.
|
||||
|
||||
@ -568,7 +568,10 @@ class ContainerSync(Daemon):
|
||||
realm_key, ts_meta):
|
||||
return True
|
||||
exc = None
|
||||
# look up for the newest one
|
||||
# look up for the newest one; the symlink=get query-string has
|
||||
# no effect unless symlinks are enabled in the internal client
|
||||
# in which case it ensures that symlink objects retain their
|
||||
# symlink property when sync'd.
|
||||
headers_out = {'X-Newest': True,
|
||||
'X-Backend-Storage-Policy-Index':
|
||||
str(info['storage_policy_index'])}
|
||||
@ -577,7 +580,8 @@ class ContainerSync(Daemon):
|
||||
self.swift.get_object(info['account'],
|
||||
info['container'], row['name'],
|
||||
headers=headers_out,
|
||||
acceptable_statuses=(2, 4))
|
||||
acceptable_statuses=(2, 4),
|
||||
params={'symlink': 'get'})
|
||||
|
||||
except (Exception, UnexpectedResponse, Timeout) as err:
|
||||
headers = {}
|
||||
|
||||
@ -794,7 +794,8 @@ class File(Base):
|
||||
['content_type', 'content-type'],
|
||||
['last_modified', 'last-modified'],
|
||||
['etag', 'etag']]
|
||||
optional_fields = [['x_object_manifest', 'x-object-manifest']]
|
||||
optional_fields = [['x_object_manifest', 'x-object-manifest'],
|
||||
['x_symlink_target', 'x-symlink-target']]
|
||||
|
||||
header_fields = self.header_fields(fields,
|
||||
optional_fields=optional_fields)
|
||||
|
||||
1839
test/functional/test_symlink.py
Executable file
1839
test/functional/test_symlink.py
Executable file
File diff suppressed because it is too large
Load Diff
@ -25,14 +25,17 @@ from test.probe.brain import BrainSplitter
|
||||
from test.probe.common import ReplProbeTest, ENABLED_POLICIES
|
||||
|
||||
|
||||
def get_current_realm_cluster(url):
|
||||
def get_info(url):
|
||||
parts = urlparse(url)
|
||||
url = parts.scheme + '://' + parts.netloc + '/info'
|
||||
http_conn = client.http_connection(url)
|
||||
try:
|
||||
info = client.get_capabilities(http_conn)
|
||||
return client.get_capabilities(http_conn)
|
||||
except client.ClientException:
|
||||
raise unittest.SkipTest('Unable to retrieve cluster info')
|
||||
|
||||
|
||||
def get_current_realm_cluster(info):
|
||||
try:
|
||||
realms = info['container_sync']['realms']
|
||||
except KeyError:
|
||||
@ -44,11 +47,12 @@ def get_current_realm_cluster(url):
|
||||
raise unittest.SkipTest('Unable find current realm cluster')
|
||||
|
||||
|
||||
class TestContainerSync(ReplProbeTest):
|
||||
class BaseTestContainerSync(ReplProbeTest):
|
||||
|
||||
def setUp(self):
|
||||
super(TestContainerSync, self).setUp()
|
||||
self.realm, self.cluster = get_current_realm_cluster(self.url)
|
||||
super(BaseTestContainerSync, self).setUp()
|
||||
self.info = get_info(self.url)
|
||||
self.realm, self.cluster = get_current_realm_cluster(self.info)
|
||||
|
||||
def _setup_synced_containers(
|
||||
self, source_overrides=None, dest_overrides=None):
|
||||
@ -92,6 +96,9 @@ class TestContainerSync(ReplProbeTest):
|
||||
|
||||
return source['name'], dest['name']
|
||||
|
||||
|
||||
class TestContainerSync(BaseTestContainerSync):
|
||||
|
||||
def test_sync(self):
|
||||
source_container, dest_container = self._setup_synced_containers()
|
||||
|
||||
@ -377,5 +384,184 @@ class TestContainerSync(ReplProbeTest):
|
||||
self.assertEqual(body, 'new-test-body')
|
||||
|
||||
|
||||
class TestContainerSyncAndSymlink(BaseTestContainerSync):
|
||||
|
||||
def setUp(self):
|
||||
super(TestContainerSyncAndSymlink, self).setUp()
|
||||
symlinks_enabled = self.info.get('symlink') or False
|
||||
if not symlinks_enabled:
|
||||
raise unittest.SkipTest("Symlinks not enabled")
|
||||
|
||||
def test_sync_symlink(self):
|
||||
# Verify that symlinks are sync'd as symlinks.
|
||||
dest_account = self.account_2
|
||||
source_container, dest_container = self._setup_synced_containers(
|
||||
dest_overrides=dest_account
|
||||
)
|
||||
|
||||
# Create source and dest containers for target objects in separate
|
||||
# accounts.
|
||||
# These containers must have same name for the destination symlink
|
||||
# to use the same target object. Initially the destination has no sync
|
||||
# key so target will not sync.
|
||||
tgt_container = 'targets-%s' % uuid.uuid4()
|
||||
dest_tgt_info = dict(dest_account)
|
||||
dest_tgt_info.update({'name': tgt_container, 'sync_key': None})
|
||||
self._setup_synced_containers(
|
||||
source_overrides={'name': tgt_container, 'sync_key': 'tgt_key'},
|
||||
dest_overrides=dest_tgt_info)
|
||||
|
||||
# upload a target to source
|
||||
target_name = 'target-%s' % uuid.uuid4()
|
||||
target_body = 'target body'
|
||||
client.put_object(
|
||||
self.url, self.token, tgt_container, target_name,
|
||||
target_body)
|
||||
|
||||
# Note that this tests when the target object is in the same account
|
||||
target_path = '%s/%s' % (tgt_container, target_name)
|
||||
symlink_name = 'symlink-%s' % uuid.uuid4()
|
||||
put_headers = {'X-Symlink-Target': target_path}
|
||||
|
||||
# upload the symlink
|
||||
client.put_object(
|
||||
self.url, self.token, source_container, symlink_name,
|
||||
'', headers=put_headers)
|
||||
|
||||
# verify object is a symlink
|
||||
resp_headers, symlink_body = client.get_object(
|
||||
self.url, self.token, source_container, symlink_name,
|
||||
query_string='symlink=get')
|
||||
self.assertEqual('', symlink_body)
|
||||
self.assertIn('x-symlink-target', resp_headers)
|
||||
|
||||
# verify symlink behavior
|
||||
resp_headers, actual_target_body = client.get_object(
|
||||
self.url, self.token, source_container, symlink_name)
|
||||
self.assertEqual(target_body, actual_target_body)
|
||||
|
||||
# cycle container-sync
|
||||
Manager(['container-sync']).once()
|
||||
|
||||
# verify symlink was sync'd
|
||||
resp_headers, dest_listing = client.get_container(
|
||||
dest_account['url'], dest_account['token'], dest_container)
|
||||
self.assertFalse(dest_listing[1:])
|
||||
self.assertEqual(symlink_name, dest_listing[0]['name'])
|
||||
|
||||
# verify symlink remained only a symlink
|
||||
resp_headers, symlink_body = client.get_object(
|
||||
dest_account['url'], dest_account['token'], dest_container,
|
||||
symlink_name, query_string='symlink=get')
|
||||
self.assertEqual('', symlink_body)
|
||||
self.assertIn('x-symlink-target', resp_headers)
|
||||
|
||||
# attempt to GET the target object via symlink will fail because
|
||||
# the target wasn't sync'd
|
||||
with self.assertRaises(ClientException) as cm:
|
||||
client.get_object(dest_account['url'], dest_account['token'],
|
||||
dest_container, symlink_name)
|
||||
self.assertEqual(404, cm.exception.http_status)
|
||||
|
||||
# now set sync key on destination target container
|
||||
client.put_container(
|
||||
dest_account['url'], dest_account['token'], tgt_container,
|
||||
headers={'X-Container-Sync-Key': 'tgt_key'})
|
||||
|
||||
# cycle container-sync
|
||||
Manager(['container-sync']).once()
|
||||
|
||||
# sanity:
|
||||
resp_headers, body = client.get_object(
|
||||
dest_account['url'], dest_account['token'],
|
||||
tgt_container, target_name)
|
||||
|
||||
# sanity check - verify symlink remained only a symlink
|
||||
resp_headers, symlink_body = client.get_object(
|
||||
dest_account['url'], dest_account['token'], dest_container,
|
||||
symlink_name, query_string='symlink=get')
|
||||
self.assertEqual('', symlink_body)
|
||||
self.assertIn('x-symlink-target', resp_headers)
|
||||
|
||||
# verify GET of target object via symlink now succeeds
|
||||
resp_headers, actual_target_body = client.get_object(
|
||||
dest_account['url'], dest_account['token'], dest_container,
|
||||
symlink_name)
|
||||
self.assertEqual(target_body, actual_target_body)
|
||||
|
||||
def test_sync_cross_acc_symlink(self):
|
||||
# Verify that cross-account symlinks are sync'd as cross-account
|
||||
# symlinks.
|
||||
source_container, dest_container = self._setup_synced_containers()
|
||||
|
||||
# Sync'd symlinks will have the same target path "/a/c/o".
|
||||
# So if we want to execute probe test with syncing targets,
|
||||
# two swift clusters will be required.
|
||||
# Therefore, for probe test in single cluster, target object is not
|
||||
# sync'd in this test.
|
||||
tgt_account = self.account_2
|
||||
tgt_container = 'targets-%s' % uuid.uuid4()
|
||||
|
||||
tgt_container_headers = {'X-Container-Read': 'test:tester'}
|
||||
if len(ENABLED_POLICIES) > 1:
|
||||
tgt_policy = random.choice(ENABLED_POLICIES)
|
||||
tgt_container_headers['X-Storage-Policy'] = tgt_policy.name
|
||||
client.put_container(tgt_account['url'], tgt_account['token'],
|
||||
tgt_container, headers=tgt_container_headers)
|
||||
|
||||
# upload a target to source
|
||||
target_name = 'target-%s' % uuid.uuid4()
|
||||
target_body = 'target body'
|
||||
client.put_object(tgt_account['url'], tgt_account['token'],
|
||||
tgt_container, target_name, target_body)
|
||||
|
||||
# Note that this tests when the target object is in a different account
|
||||
target_path = '%s/%s' % (tgt_container, target_name)
|
||||
symlink_name = 'symlink-%s' % uuid.uuid4()
|
||||
put_headers = {
|
||||
'X-Symlink-Target': target_path,
|
||||
'X-Symlink-Target-Account': tgt_account['account']}
|
||||
|
||||
# upload the symlink
|
||||
client.put_object(
|
||||
self.url, self.token, source_container, symlink_name,
|
||||
'', headers=put_headers)
|
||||
|
||||
# verify object is a cross-account symlink
|
||||
resp_headers, symlink_body = client.get_object(
|
||||
self.url, self.token, source_container, symlink_name,
|
||||
query_string='symlink=get')
|
||||
self.assertEqual('', symlink_body)
|
||||
self.assertIn('x-symlink-target', resp_headers)
|
||||
self.assertIn('x-symlink-target-account', resp_headers)
|
||||
|
||||
# verify symlink behavior
|
||||
resp_headers, actual_target_body = client.get_object(
|
||||
self.url, self.token, source_container, symlink_name)
|
||||
self.assertEqual(target_body, actual_target_body)
|
||||
|
||||
# cycle container-sync
|
||||
Manager(['container-sync']).once()
|
||||
|
||||
# verify symlink was sync'd
|
||||
resp_headers, dest_listing = client.get_container(
|
||||
self.url, self.token, dest_container)
|
||||
self.assertFalse(dest_listing[1:])
|
||||
self.assertEqual(symlink_name, dest_listing[0]['name'])
|
||||
|
||||
# verify symlink remained only a symlink
|
||||
resp_headers, symlink_body = client.get_object(
|
||||
self.url, self.token, dest_container,
|
||||
symlink_name, query_string='symlink=get')
|
||||
self.assertEqual('', symlink_body)
|
||||
self.assertIn('x-symlink-target', resp_headers)
|
||||
self.assertIn('x-symlink-target-account', resp_headers)
|
||||
|
||||
# verify GET of target object via symlink now succeeds
|
||||
resp_headers, actual_target_body = client.get_object(
|
||||
self.url, self.token, dest_container, symlink_name)
|
||||
self.assertEqual(target_body, actual_target_body)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
@ -19,6 +19,7 @@ import unittest
|
||||
|
||||
import mock
|
||||
|
||||
from swift.common.utils import MD5_OF_EMPTY_STRING
|
||||
from swift.common.header_key_dict import HeaderKeyDict
|
||||
from swift.common.middleware.crypto import decrypter
|
||||
from swift.common.middleware.crypto.crypto_utils import CRYPTO_KEY_CALLBACK, \
|
||||
@ -960,6 +961,27 @@ class TestDecrypterContainerRequests(unittest.TestCase):
|
||||
self.assertIn("Cipher must be AES_CTR_256",
|
||||
self.decrypter.logger.get_lines_for_level('error')[0])
|
||||
|
||||
def test_GET_container_json_not_encrypted_obj(self):
|
||||
pt_etag = '%s; symlink_path=/a/c/o' % MD5_OF_EMPTY_STRING
|
||||
|
||||
obj_dict = {"bytes": 0,
|
||||
"last_modified": "2015-04-14T23:33:06.439040",
|
||||
"hash": pt_etag,
|
||||
"name": "symlink",
|
||||
"content_type": 'application/symlink'}
|
||||
|
||||
listing = [obj_dict]
|
||||
fake_body = json.dumps(listing)
|
||||
|
||||
resp = self._make_cont_get_req(fake_body, 'json')
|
||||
|
||||
self.assertEqual('200 OK', resp.status)
|
||||
body = resp.body
|
||||
self.assertEqual(len(body), int(resp.headers['Content-Length']))
|
||||
body_json = json.loads(body)
|
||||
self.assertEqual(1, len(body_json))
|
||||
self.assertEqual(pt_etag, body_json[0]['hash'])
|
||||
|
||||
|
||||
class TestModuleMethods(unittest.TestCase):
|
||||
def test_purge_crypto_sysmeta_headers(self):
|
||||
|
||||
835
test/unit/common/middleware/test_symlink.py
Normal file
835
test/unit/common/middleware/test_symlink.py
Normal file
@ -0,0 +1,835 @@
|
||||
#!/usr/bin/env python
|
||||
# Copyright (c) 2016 OpenStack Foundation
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
# implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import unittest
|
||||
import json
|
||||
import mock
|
||||
|
||||
from six.moves.urllib.parse import quote
|
||||
from swift.common import swob
|
||||
from swift.common.middleware import symlink, copy, versioned_writes, \
|
||||
listing_formats
|
||||
from swift.common.swob import Request
|
||||
from swift.common.utils import MD5_OF_EMPTY_STRING
|
||||
from test.unit.common.middleware.helpers import FakeSwift
|
||||
from test.unit.common.middleware.test_versioned_writes import FakeCache
|
||||
|
||||
|
||||
class TestSymlinkMiddlewareBase(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.app = FakeSwift()
|
||||
self.sym = symlink.filter_factory({
|
||||
'symloop_max': '2',
|
||||
})(self.app)
|
||||
self.sym.logger = self.app.logger
|
||||
|
||||
def call_app(self, req, app=None, expect_exception=False):
|
||||
if app is None:
|
||||
app = self.app
|
||||
|
||||
self.authorized = []
|
||||
|
||||
def authorize(req):
|
||||
self.authorized.append(req)
|
||||
|
||||
if 'swift.authorize' not in req.environ:
|
||||
req.environ['swift.authorize'] = authorize
|
||||
|
||||
status = [None]
|
||||
headers = [None]
|
||||
|
||||
def start_response(s, h, ei=None):
|
||||
status[0] = s
|
||||
headers[0] = h
|
||||
|
||||
body_iter = app(req.environ, start_response)
|
||||
body = ''
|
||||
caught_exc = None
|
||||
try:
|
||||
for chunk in body_iter:
|
||||
body += chunk
|
||||
except Exception as exc:
|
||||
if expect_exception:
|
||||
caught_exc = exc
|
||||
else:
|
||||
raise
|
||||
|
||||
if expect_exception:
|
||||
return status[0], headers[0], body, caught_exc
|
||||
else:
|
||||
return status[0], headers[0], body
|
||||
|
||||
def call_sym(self, req, **kwargs):
|
||||
return self.call_app(req, app=self.sym, **kwargs)
|
||||
|
||||
|
||||
class TestSymlinkMiddleware(TestSymlinkMiddlewareBase):
|
||||
def test_symlink_simple_put(self):
|
||||
self.app.register('PUT', '/v1/a/c/symlink', swob.HTTPCreated, {})
|
||||
req = Request.blank('/v1/a/c/symlink', method='PUT',
|
||||
headers={'X-Symlink-Target': 'c1/o'},
|
||||
body='')
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, '201 Created')
|
||||
method, path, hdrs = self.app.calls_with_headers[0]
|
||||
val = hdrs.get('X-Object-Sysmeta-Symlink-Target')
|
||||
self.assertEqual(val, 'c1/o')
|
||||
self.assertNotIn('X-Object-Sysmeta-Symlink-Target-Account', hdrs)
|
||||
val = hdrs.get('X-Object-Sysmeta-Container-Update-Override-Etag')
|
||||
self.assertEqual(val, '%s; symlink_target=c1/o' % MD5_OF_EMPTY_STRING)
|
||||
|
||||
def test_symlink_put_different_account(self):
|
||||
self.app.register('PUT', '/v1/a/c/symlink', swob.HTTPCreated, {})
|
||||
req = Request.blank('/v1/a/c/symlink', method='PUT',
|
||||
headers={'X-Symlink-Target': 'c1/o',
|
||||
'X-Symlink-Target-Account': 'a1'},
|
||||
body='')
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, '201 Created')
|
||||
method, path, hdrs = self.app.calls_with_headers[0]
|
||||
val = hdrs.get('X-Object-Sysmeta-Symlink-Target')
|
||||
self.assertEqual(val, 'c1/o')
|
||||
self.assertEqual(hdrs.get('X-Object-Sysmeta-Symlink-Target-Account'),
|
||||
'a1')
|
||||
|
||||
def test_symlink_put_leading_slash(self):
|
||||
self.app.register('PUT', '/v1/a/c/symlink', swob.HTTPCreated, {})
|
||||
req = Request.blank('/v1/a/c/symlink', method='PUT',
|
||||
headers={'X-Symlink-Target': '/c1/o'},
|
||||
body='')
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, '412 Precondition Failed')
|
||||
self.assertEqual(body, "X-Symlink-Target header must be of "
|
||||
"the form <container name>/<object name>")
|
||||
|
||||
def test_symlink_put_non_zero_length(self):
|
||||
req = Request.blank('/v1/a/c/symlink', method='PUT', body='req_body',
|
||||
headers={'X-Symlink-Target': 'c1/o'})
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, '400 Bad Request')
|
||||
self.assertEqual(body, 'Symlink requests require a zero byte body')
|
||||
|
||||
def test_symlink_put_bad_object_header(self):
|
||||
req = Request.blank('/v1/a/c/symlink', method='PUT',
|
||||
headers={'X-Symlink-Target': 'o'},
|
||||
body='')
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, "412 Precondition Failed")
|
||||
self.assertEqual(body, "X-Symlink-Target header must be of "
|
||||
"the form <container name>/<object name>")
|
||||
|
||||
def test_symlink_put_bad_account_header(self):
|
||||
req = Request.blank('/v1/a/c/symlink', method='PUT',
|
||||
headers={'X-Symlink-Target': 'c1/o',
|
||||
'X-Symlink-Target-Account': 'a1/c1'},
|
||||
body='')
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, "412 Precondition Failed")
|
||||
self.assertEqual(body, "Account name cannot contain slashes")
|
||||
|
||||
def test_get_symlink(self):
|
||||
self.app.register('GET', '/v1/a/c/symlink', swob.HTTPOk,
|
||||
{'X-Object-Sysmeta-Symlink-Target': 'c1/o'})
|
||||
req = Request.blank('/v1/a/c/symlink?symlink=get', method='GET')
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, '200 OK')
|
||||
self.assertIn(('X-Symlink-Target', 'c1/o'), headers)
|
||||
self.assertNotIn('X-Symlink-Target-Account', dict(headers))
|
||||
|
||||
def test_get_symlink_with_account(self):
|
||||
self.app.register('GET', '/v1/a/c/symlink', swob.HTTPOk,
|
||||
{'X-Object-Sysmeta-Symlink-Target': 'c1/o',
|
||||
'X-Object-Sysmeta-Symlink-Target-Account': 'a2'})
|
||||
req = Request.blank('/v1/a/c/symlink?symlink=get', method='GET')
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, '200 OK')
|
||||
self.assertIn(('X-Symlink-Target', 'c1/o'), headers)
|
||||
self.assertIn(('X-Symlink-Target-Account', 'a2'), headers)
|
||||
|
||||
def test_get_symlink_not_found(self):
|
||||
self.app.register('GET', '/v1/a/c/symlink', swob.HTTPNotFound, {})
|
||||
req = Request.blank('/v1/a/c/symlink', method='GET')
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, '404 Not Found')
|
||||
self.assertNotIn('Content-Location', dict(headers))
|
||||
|
||||
def test_get_target_object(self):
|
||||
self.app.register('GET', '/v1/a/c/symlink', swob.HTTPOk,
|
||||
{'X-Object-Sysmeta-Symlink-Target': 'c1/o',
|
||||
'X-Object-Sysmeta-Symlink-Target-Account': 'a2'})
|
||||
self.app.register('GET', '/v1/a2/c1/o', swob.HTTPOk, {}, 'resp_body')
|
||||
req = Request.blank('/v1/a/c/symlink', method='GET')
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, '200 OK')
|
||||
self.assertEqual(body, 'resp_body')
|
||||
self.assertNotIn('X-Symlink-Target', dict(headers))
|
||||
self.assertNotIn('X-Symlink-Target-Account', dict(headers))
|
||||
self.assertIn(('Content-Location', '/v1/a2/c1/o'), headers)
|
||||
|
||||
def test_get_target_object_not_found(self):
|
||||
self.app.register('GET', '/v1/a/c/symlink', swob.HTTPOk,
|
||||
{'X-Object-Sysmeta-Symlink-Target': 'c1/o',
|
||||
'X-Object-Sysmeta-Symlink-Target-account': 'a2'})
|
||||
self.app.register('GET', '/v1/a2/c1/o', swob.HTTPNotFound, {}, '')
|
||||
req = Request.blank('/v1/a/c/symlink', method='GET')
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, '404 Not Found')
|
||||
self.assertEqual(body, '')
|
||||
self.assertNotIn('X-Symlink-Target', dict(headers))
|
||||
self.assertNotIn('X-Symlink-Target-Account', dict(headers))
|
||||
self.assertIn(('Content-Location', '/v1/a2/c1/o'), headers)
|
||||
|
||||
def test_get_target_object_range_not_satisfiable(self):
|
||||
self.app.register('GET', '/v1/a/c/symlink', swob.HTTPOk,
|
||||
{'X-Object-Sysmeta-Symlink-Target': 'c1/o',
|
||||
'X-Object-Sysmeta-Symlink-Target-Account': 'a2'})
|
||||
self.app.register('GET', '/v1/a2/c1/o',
|
||||
swob.HTTPRequestedRangeNotSatisfiable, {}, '')
|
||||
req = Request.blank('/v1/a/c/symlink', method='GET',
|
||||
headers={'Range': 'bytes=1-2'})
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, '416 Requested Range Not Satisfiable')
|
||||
self.assertEqual(
|
||||
body, '<html><h1>Requested Range Not Satisfiable</h1>'
|
||||
'<p>The Range requested is not available.</p></html>')
|
||||
self.assertNotIn('X-Symlink-Target', dict(headers))
|
||||
self.assertNotIn('X-Symlink-Target-Account', dict(headers))
|
||||
self.assertIn(('Content-Location', '/v1/a2/c1/o'), headers)
|
||||
|
||||
def test_get_ec_symlink_range_unsatisfiable_can_redirect_to_target(self):
|
||||
self.app.register('GET', '/v1/a/c/symlink',
|
||||
swob.HTTPRequestedRangeNotSatisfiable,
|
||||
{'X-Object-Sysmeta-Symlink-Target': 'c1/o',
|
||||
'X-Object-Sysmeta-Symlink-Target-Account': 'a2'})
|
||||
self.app.register('GET', '/v1/a2/c1/o', swob.HTTPOk,
|
||||
{'Content-Range': 'bytes 1-2/10'}, 'es')
|
||||
req = Request.blank('/v1/a/c/symlink', method='GET',
|
||||
headers={'Range': 'bytes=1-2'})
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, '200 OK')
|
||||
self.assertEqual(body, 'es')
|
||||
self.assertNotIn('X-Symlink-Target', dict(headers))
|
||||
self.assertNotIn('X-Symlink-Target-Account', dict(headers))
|
||||
self.assertIn(('Content-Location', '/v1/a2/c1/o'), headers)
|
||||
self.assertIn(('Content-Range', 'bytes 1-2/10'), headers)
|
||||
|
||||
def test_get_non_symlink(self):
|
||||
# this is not symlink object
|
||||
self.app.register('GET', '/v1/a/c/obj', swob.HTTPOk, {}, 'resp_body')
|
||||
req = Request.blank('/v1/a/c/obj', method='GET')
|
||||
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, '200 OK')
|
||||
self.assertEqual(body, 'resp_body')
|
||||
|
||||
# Assert special headers for symlink are not in response
|
||||
self.assertNotIn('X-Symlink-Target', dict(headers))
|
||||
self.assertNotIn('X-Symlink-Target-Account', dict(headers))
|
||||
self.assertNotIn('Content-Location', dict(headers))
|
||||
|
||||
def test_head_symlink(self):
|
||||
self.app.register('HEAD', '/v1/a/c/symlink', swob.HTTPOk,
|
||||
{'X-Object-Sysmeta-Symlink-Target': 'c1/o',
|
||||
'X-Object-Meta-Color': 'Red'})
|
||||
req = Request.blank('/v1/a/c/symlink?symlink=get', method='HEAD')
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, '200 OK')
|
||||
self.assertIn(('X-Symlink-Target', 'c1/o'), headers)
|
||||
self.assertNotIn('X-Symlink-Target-Account', dict(headers))
|
||||
self.assertIn(('X-Object-Meta-Color', 'Red'), headers)
|
||||
|
||||
def test_head_symlink_with_account(self):
|
||||
self.app.register('HEAD', '/v1/a/c/symlink', swob.HTTPOk,
|
||||
{'X-Object-Sysmeta-Symlink-Target': 'c1/o',
|
||||
'X-Object-Sysmeta-Symlink-Target-Account': 'a2',
|
||||
'X-Object-Meta-Color': 'Red'})
|
||||
req = Request.blank('/v1/a/c/symlink?symlink=get', method='HEAD')
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, '200 OK')
|
||||
self.assertIn(('X-Symlink-Target', 'c1/o'), headers)
|
||||
self.assertIn(('X-Symlink-Target-Account', 'a2'), headers)
|
||||
self.assertIn(('X-Object-Meta-Color', 'Red'), headers)
|
||||
|
||||
def test_head_target_object(self):
|
||||
# this test is also validating that the symlink metadata is not
|
||||
# returned, but the target object metadata does return
|
||||
self.app.register('HEAD', '/v1/a/c/symlink', swob.HTTPOk,
|
||||
{'X-Object-Sysmeta-Symlink-Target': 'c1/o',
|
||||
'X-Object-Sysmeta-Symlink-Target-Account': 'a2',
|
||||
'X-Object-Meta-Color': 'Red'})
|
||||
self.app.register('HEAD', '/v1/a2/c1/o', swob.HTTPOk,
|
||||
{'X-Object-Meta-Color': 'Green'})
|
||||
req = Request.blank('/v1/a/c/symlink', method='HEAD')
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, '200 OK')
|
||||
self.assertNotIn('X-Symlink-Target', dict(headers))
|
||||
self.assertNotIn('X-Symlink-Target-Account', dict(headers))
|
||||
self.assertNotIn(('X-Object-Meta-Color', 'Red'), headers)
|
||||
self.assertIn(('X-Object-Meta-Color', 'Green'), headers)
|
||||
self.assertIn(('Content-Location', '/v1/a2/c1/o'), headers)
|
||||
|
||||
def test_symlink_too_deep(self):
|
||||
self.app.register('HEAD', '/v1/a/c/symlink', swob.HTTPOk,
|
||||
{'X-Object-Sysmeta-Symlink-Target': 'c/sym1'})
|
||||
self.app.register('HEAD', '/v1/a/c/sym1', swob.HTTPOk,
|
||||
{'X-Object-Sysmeta-Symlink-Target': 'c/sym2'})
|
||||
self.app.register('HEAD', '/v1/a/c/sym2', swob.HTTPOk,
|
||||
{'X-Object-Sysmeta-Symlink-Target': 'c/o'})
|
||||
req = Request.blank('/v1/a/c/symlink', method='HEAD')
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, '409 Conflict')
|
||||
|
||||
def test_symlink_change_symloopmax(self):
|
||||
# similar test to test_symlink_too_deep, but now changed the limit to 3
|
||||
self.sym = symlink.filter_factory({
|
||||
'symloop_max': '3',
|
||||
})(self.app)
|
||||
self.sym.logger = self.app.logger
|
||||
self.app.register('HEAD', '/v1/a/c/symlink', swob.HTTPOk,
|
||||
{'X-Object-Sysmeta-Symlink-Target': 'c/sym1'})
|
||||
self.app.register('HEAD', '/v1/a/c/sym1', swob.HTTPOk,
|
||||
{'X-Object-Sysmeta-Symlink-Target': 'c/sym2'})
|
||||
self.app.register('HEAD', '/v1/a/c/sym2', swob.HTTPOk,
|
||||
{'X-Object-Sysmeta-Symlink-Target': 'c/o',
|
||||
'X-Object-Meta-Color': 'Red'})
|
||||
self.app.register('HEAD', '/v1/a/c/o', swob.HTTPOk,
|
||||
{'X-Object-Meta-Color': 'Green'})
|
||||
req = Request.blank('/v1/a/c/symlink', method='HEAD')
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, '200 OK')
|
||||
|
||||
# assert that the correct metadata was returned
|
||||
self.assertNotIn(('X-Object-Meta-Color', 'Red'), headers)
|
||||
self.assertIn(('X-Object-Meta-Color', 'Green'), headers)
|
||||
|
||||
def test_sym_to_sym_to_target(self):
|
||||
# this test is also validating that the symlink metadata is not
|
||||
# returned, but the target object metadata does return
|
||||
self.app.register('HEAD', '/v1/a/c/symlink', swob.HTTPOk,
|
||||
{'X-Object-Sysmeta-Symlink-Target': 'c/sym1',
|
||||
'X-Object-Meta-Color': 'Red'})
|
||||
self.app.register('HEAD', '/v1/a/c/sym1', swob.HTTPOk,
|
||||
{'X-Object-Sysmeta-Symlink-Target': 'c1/o',
|
||||
'X-Object-Meta-Color': 'Yellow'})
|
||||
self.app.register('HEAD', '/v1/a/c1/o', swob.HTTPOk,
|
||||
{'X-Object-Meta-Color': 'Green'})
|
||||
req = Request.blank('/v1/a/c/symlink', method='HEAD')
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, '200 OK')
|
||||
self.assertNotIn(('X-Symlink-Target', 'c1/o'), headers)
|
||||
self.assertNotIn(('X-Symlink-Target-Account', 'a2'), headers)
|
||||
self.assertNotIn(('X-Object-Meta-Color', 'Red'), headers)
|
||||
self.assertNotIn(('X-Object-Meta-Color', 'Yellow'), headers)
|
||||
self.assertIn(('X-Object-Meta-Color', 'Green'), headers)
|
||||
self.assertIn(('Content-Location', '/v1/a/c1/o'), headers)
|
||||
|
||||
def test_symlink_post(self):
|
||||
self.app.register('POST', '/v1/a/c/symlink', swob.HTTPAccepted,
|
||||
{'X-Object-Sysmeta-Symlink-Target': 'c1/o'})
|
||||
req = Request.blank('/v1/a/c/symlink', method='POST',
|
||||
headers={'X-Object-Meta-Color': 'Red'})
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, '307 Temporary Redirect')
|
||||
self.assertEqual(body,
|
||||
'The requested POST was applied to a symlink. POST '
|
||||
'directly to the target to apply requested metadata.')
|
||||
method, path, hdrs = self.app.calls_with_headers[0]
|
||||
val = hdrs.get('X-Object-Meta-Color')
|
||||
self.assertEqual(val, 'Red')
|
||||
|
||||
def test_non_symlink_post(self):
|
||||
self.app.register('POST', '/v1/a/c/o', swob.HTTPAccepted, {})
|
||||
req = Request.blank('/v1/a/c/o', method='POST',
|
||||
headers={'X-Object-Meta-Color': 'Red'})
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, '202 Accepted')
|
||||
|
||||
def test_set_symlink_POST_fail(self):
|
||||
# Setting a link with a POST request is not allowed
|
||||
req = Request.blank('/v1/a/c/o', method='POST',
|
||||
headers={'X-Symlink-Target': 'c1/regular_obj'})
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, '400 Bad Request')
|
||||
self.assertEqual(body, "A PUT request is required to set a symlink "
|
||||
"target")
|
||||
|
||||
def test_symlink_post_but_fail_at_server(self):
|
||||
self.app.register('POST', '/v1/a/c/o', swob.HTTPNotFound, {})
|
||||
req = Request.blank('/v1/a/c/o', method='POST',
|
||||
headers={'X-Object-Meta-Color': 'Red'})
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, '404 Not Found')
|
||||
|
||||
def test_check_symlink_header(self):
|
||||
def do_test(headers):
|
||||
req = Request.blank('/v1/a/c/o', method='PUT',
|
||||
headers=headers)
|
||||
symlink._check_symlink_header(req)
|
||||
|
||||
# normal cases
|
||||
do_test({'X-Symlink-Target': 'c1/o1'})
|
||||
do_test({'X-Symlink-Target': 'c1/sub/o1'})
|
||||
do_test({'X-Symlink-Target': 'c1%2Fo1'})
|
||||
# specify account
|
||||
do_test({'X-Symlink-Target': 'c1/o1',
|
||||
'X-Symlink-Target-Account': 'another'})
|
||||
# URL encoded is safe
|
||||
do_test({'X-Symlink-Target': 'c1%2Fo1'})
|
||||
# URL encoded + multibytes is also safe
|
||||
do_test(
|
||||
{'X-Symlink-Target':
|
||||
u'\u30b0\u30e9\u30d6\u30eb/\u30a2\u30ba\u30ec\u30f3'})
|
||||
target = u'\u30b0\u30e9\u30d6\u30eb/\u30a2\u30ba\u30ec\u30f3'
|
||||
encoded_target = quote(target.encode('utf-8'), '')
|
||||
do_test({'X-Symlink-Target': encoded_target})
|
||||
|
||||
do_test(
|
||||
{'X-Symlink-Target': 'cont/obj',
|
||||
'X-Symlink-Target-Account': u'\u30b0\u30e9\u30d6\u30eb'})
|
||||
|
||||
def test_check_symlink_header_invalid_format(self):
|
||||
def do_test(headers, status, err_msg):
|
||||
req = Request.blank('/v1/a/c/o', method='PUT',
|
||||
headers=headers)
|
||||
with self.assertRaises(swob.HTTPException) as cm:
|
||||
symlink._check_symlink_header(req)
|
||||
|
||||
self.assertEqual(cm.exception.status, status)
|
||||
self.assertEqual(cm.exception.body, err_msg)
|
||||
|
||||
do_test({'X-Symlink-Target': '/c1/o1'},
|
||||
'412 Precondition Failed',
|
||||
'X-Symlink-Target header must be of the '
|
||||
'form <container name>/<object name>')
|
||||
do_test({'X-Symlink-Target': 'c1o1'},
|
||||
'412 Precondition Failed',
|
||||
'X-Symlink-Target header must be of the '
|
||||
'form <container name>/<object name>')
|
||||
do_test({'X-Symlink-Target': 'c1/o1',
|
||||
'X-Symlink-Target-Account': '/another'},
|
||||
'412 Precondition Failed',
|
||||
'Account name cannot contain slashes')
|
||||
do_test({'X-Symlink-Target': 'c1/o1',
|
||||
'X-Symlink-Target-Account': 'an/other'},
|
||||
'412 Precondition Failed',
|
||||
'Account name cannot contain slashes')
|
||||
# url encoded case
|
||||
do_test({'X-Symlink-Target': '%2Fc1%2Fo1'},
|
||||
'412 Precondition Failed',
|
||||
'X-Symlink-Target header must be of the '
|
||||
'form <container name>/<object name>')
|
||||
do_test({'X-Symlink-Target': 'c1/o1',
|
||||
'X-Symlink-Target-Account': '%2Fanother'},
|
||||
'412 Precondition Failed',
|
||||
'Account name cannot contain slashes')
|
||||
do_test({'X-Symlink-Target': 'c1/o1',
|
||||
'X-Symlink-Target-Account': 'an%2Fother'},
|
||||
'412 Precondition Failed',
|
||||
'Account name cannot contain slashes')
|
||||
# with multi-bytes
|
||||
do_test(
|
||||
{'X-Symlink-Target':
|
||||
u'/\u30b0\u30e9\u30d6\u30eb/\u30a2\u30ba\u30ec\u30f3'},
|
||||
'412 Precondition Failed',
|
||||
'X-Symlink-Target header must be of the '
|
||||
'form <container name>/<object name>')
|
||||
target = u'/\u30b0\u30e9\u30d6\u30eb/\u30a2\u30ba\u30ec\u30f3'
|
||||
encoded_target = quote(target.encode('utf-8'), '')
|
||||
do_test(
|
||||
{'X-Symlink-Target': encoded_target},
|
||||
'412 Precondition Failed',
|
||||
'X-Symlink-Target header must be of the '
|
||||
'form <container name>/<object name>')
|
||||
account = u'\u30b0\u30e9\u30d6\u30eb/\u30a2\u30ba\u30ec\u30f3'
|
||||
encoded_account = quote(account.encode('utf-8'), '')
|
||||
do_test(
|
||||
{'X-Symlink-Target': 'c/o',
|
||||
'X-Symlink-Target-Account': encoded_account},
|
||||
'412 Precondition Failed',
|
||||
'Account name cannot contain slashes')
|
||||
|
||||
def test_check_symlink_header_points_to_itself(self):
|
||||
req = Request.blank('/v1/a/c/o', method='PUT',
|
||||
headers={'X-Symlink-Target': 'c/o'})
|
||||
with self.assertRaises(swob.HTTPException) as cm:
|
||||
symlink._check_symlink_header(req)
|
||||
self.assertEqual(cm.exception.status, '400 Bad Request')
|
||||
self.assertEqual(cm.exception.body, 'Symlink cannot target itself')
|
||||
|
||||
# Even if set account to itself, it will fail as well
|
||||
req = Request.blank('/v1/a/c/o', method='PUT',
|
||||
headers={'X-Symlink-Target': 'c/o',
|
||||
'X-Symlink-Target-Account': 'a'})
|
||||
with self.assertRaises(swob.HTTPException) as cm:
|
||||
symlink._check_symlink_header(req)
|
||||
self.assertEqual(cm.exception.status, '400 Bad Request')
|
||||
self.assertEqual(cm.exception.body, 'Symlink cannot target itself')
|
||||
|
||||
# sanity, the case to another account is safe
|
||||
req = Request.blank('/v1/a/c/o', method='PUT',
|
||||
headers={'X-Symlink-Target': 'c/o',
|
||||
'X-Symlink-Target-Account': 'a1'})
|
||||
symlink._check_symlink_header(req)
|
||||
|
||||
def test_symloop_max_config(self):
|
||||
self.app = FakeSwift()
|
||||
# sanity
|
||||
self.sym = symlink.filter_factory({
|
||||
'symloop_max': '1',
|
||||
})(self.app)
|
||||
self.assertEqual(self.sym.symloop_max, 1)
|
||||
# < 1 case will result in default
|
||||
self.sym = symlink.filter_factory({
|
||||
'symloop_max': '-1',
|
||||
})(self.app)
|
||||
self.assertEqual(self.sym.symloop_max, symlink.DEFAULT_SYMLOOP_MAX)
|
||||
|
||||
|
||||
class SymlinkCopyingTestCase(TestSymlinkMiddlewareBase):
|
||||
# verify interaction of copy and symlink middlewares
|
||||
|
||||
def setUp(self):
|
||||
self.app = FakeSwift()
|
||||
conf = {'symloop_max': '2'}
|
||||
self.sym = symlink.filter_factory(conf)(self.app)
|
||||
self.sym.logger = self.app.logger
|
||||
self.copy = copy.filter_factory({})(self.sym)
|
||||
|
||||
def call_copy(self, req, **kwargs):
|
||||
return self.call_app(req, app=self.copy, **kwargs)
|
||||
|
||||
def test_copy_symlink_target(self):
|
||||
self.app.register('GET', '/v1/a/src_cont/symlink', swob.HTTPOk,
|
||||
{'X-Object-Sysmeta-Symlink-Target': 'c1/o',
|
||||
'X-Object-Sysmeta-Symlink-Target-Account': 'a2'})
|
||||
self.app.register('GET', '/v1/a2/c1/o', swob.HTTPOk, {}, 'resp_body')
|
||||
self.app.register('PUT', '/v1/a/tgt_cont/tgt_obj', swob.HTTPCreated,
|
||||
{}, 'resp_body')
|
||||
req = Request.blank('/v1/a/src_cont/symlink', method='COPY',
|
||||
headers={'Destination': 'tgt_cont/tgt_obj'})
|
||||
status, headers, body = self.call_copy(req)
|
||||
self.assertEqual(status, '201 Created')
|
||||
|
||||
def test_copy_symlink(self):
|
||||
self.app.register('GET', '/v1/a/src_cont/symlink', swob.HTTPOk,
|
||||
{'X-Object-Sysmeta-Symlink-Target': 'c1/o',
|
||||
'X-Object-Sysmeta-Symlink-Target-Account': 'a2'})
|
||||
self.app.register('PUT', '/v1/a/tgt_cont/tgt_obj', swob.HTTPCreated,
|
||||
{'X-Symlink-Target': 'c1/o',
|
||||
'X-Symlink-Target-Account': 'a2'})
|
||||
req = Request.blank('/v1/a/src_cont/symlink?symlink=get',
|
||||
method='COPY',
|
||||
headers={'Destination': 'tgt_cont/tgt_obj'})
|
||||
status, headers, body = self.call_copy(req)
|
||||
self.assertEqual(status, '201 Created')
|
||||
method, path, hdrs = self.app.calls_with_headers[1]
|
||||
val = hdrs.get('X-Object-Sysmeta-Symlink-Target')
|
||||
self.assertEqual(val, 'c1/o')
|
||||
self.assertEqual(
|
||||
hdrs.get('X-Object-Sysmeta-Symlink-Target-Account'), 'a2')
|
||||
|
||||
def test_copy_symlink_new_target(self):
|
||||
self.app.register('GET', '/v1/a/src_cont/symlink', swob.HTTPOk,
|
||||
{'X-Object-Sysmeta-Symlink-Target': 'c1/o',
|
||||
'X-Object-Sysmeta-Symlink-Target-Account': 'a2'})
|
||||
self.app.register('PUT', '/v1/a/tgt_cont/tgt_obj', swob.HTTPCreated,
|
||||
{'X-Symlink-Target': 'c1/o',
|
||||
'X-Symlink-Target-Account': 'a2'})
|
||||
req = Request.blank('/v1/a/src_cont/symlink?symlink=get',
|
||||
method='COPY',
|
||||
headers={'Destination': 'tgt_cont/tgt_obj',
|
||||
'X-Symlink-Target': 'new_cont/new_obj',
|
||||
'X-Symlink-Target-Account': 'new_acct'})
|
||||
status, headers, body = self.call_copy(req)
|
||||
self.assertEqual(status, '201 Created')
|
||||
method, path, hdrs = self.app.calls_with_headers[1]
|
||||
self.assertEqual(method, 'PUT')
|
||||
self.assertEqual(path, '/v1/a/tgt_cont/tgt_obj?symlink=get')
|
||||
val = hdrs.get('X-Object-Sysmeta-Symlink-Target')
|
||||
self.assertEqual(val, 'new_cont/new_obj')
|
||||
self.assertEqual(hdrs.get('X-Object-Sysmeta-Symlink-Target-Account'),
|
||||
'new_acct')
|
||||
|
||||
|
||||
class SymlinkVersioningTestCase(TestSymlinkMiddlewareBase):
|
||||
# verify interaction of versioned_writes and symlink middlewares
|
||||
|
||||
def setUp(self):
|
||||
self.app = FakeSwift()
|
||||
conf = {'symloop_max': '2'}
|
||||
self.sym = symlink.filter_factory(conf)(self.app)
|
||||
self.sym.logger = self.app.logger
|
||||
vw_conf = {'allow_versioned_writes': 'true'}
|
||||
self.vw = versioned_writes.filter_factory(vw_conf)(self.sym)
|
||||
|
||||
def call_vw(self, req, **kwargs):
|
||||
return self.call_app(req, app=self.vw, **kwargs)
|
||||
|
||||
def assertRequestEqual(self, req, other):
|
||||
self.assertEqual(req.method, other.method)
|
||||
self.assertEqual(req.path, other.path)
|
||||
|
||||
def test_new_symlink_version_success(self):
|
||||
self.app.register(
|
||||
'PUT', '/v1/a/c/symlink', swob.HTTPCreated,
|
||||
{'X-Symlink-Target': 'new_cont/new_tgt',
|
||||
'X-Symlink-Target-Account': 'a'}, None)
|
||||
self.app.register(
|
||||
'GET', '/v1/a/c/symlink', swob.HTTPOk,
|
||||
{'last-modified': 'Thu, 1 Jan 1970 00:00:01 GMT',
|
||||
'X-Object-Sysmeta-Symlink-Target': 'old_cont/old_tgt',
|
||||
'X-Object-Sysmeta-Symlink-Target-Account': 'a'},
|
||||
'')
|
||||
self.app.register(
|
||||
'PUT', '/v1/a/ver_cont/007symlink/0000000001.00000',
|
||||
swob.HTTPCreated,
|
||||
{'X-Symlink-Target': 'old_cont/old_tgt',
|
||||
'X-Symlink-Target-Account': 'a'}, None)
|
||||
cache = FakeCache({'sysmeta': {'versions-location': 'ver_cont'}})
|
||||
req = Request.blank(
|
||||
'/v1/a/c/symlink',
|
||||
headers={'X-Symlink-Target': 'new_cont/new_tgt'},
|
||||
environ={'REQUEST_METHOD': 'PUT', 'swift.cache': cache,
|
||||
'CONTENT_LENGTH': '0',
|
||||
'swift.trans_id': 'fake_trans_id'})
|
||||
status, headers, body = self.call_vw(req)
|
||||
self.assertEqual(status, '201 Created')
|
||||
# authorized twice now because versioned_writes now makes a check on
|
||||
# PUT
|
||||
self.assertEqual(len(self.authorized), 2)
|
||||
self.assertRequestEqual(req, self.authorized[0])
|
||||
self.assertEqual(['VW', 'VW', None], self.app.swift_sources)
|
||||
self.assertEqual({'fake_trans_id'}, set(self.app.txn_ids))
|
||||
calls = self.app.calls_with_headers
|
||||
method, path, req_headers = calls[2]
|
||||
self.assertEqual('PUT', method)
|
||||
self.assertEqual('/v1/a/c/symlink', path)
|
||||
self.assertEqual(
|
||||
'new_cont/new_tgt',
|
||||
req_headers['X-Object-Sysmeta-Symlink-Target'])
|
||||
|
||||
def test_delete_latest_version_no_marker_success(self):
|
||||
self.app.register(
|
||||
'GET',
|
||||
'/v1/a/ver_cont?prefix=003sym/&marker=&reverse=on',
|
||||
swob.HTTPOk, {},
|
||||
'[{"hash": "y", '
|
||||
'"last_modified": "2014-11-21T14:23:02.206740", '
|
||||
'"bytes": 0, '
|
||||
'"name": "003sym/2", '
|
||||
'"content_type": "text/plain"}, '
|
||||
'{"hash": "x", '
|
||||
'"last_modified": "2014-11-21T14:14:27.409100", '
|
||||
'"bytes": 0, '
|
||||
'"name": "003sym/1", '
|
||||
'"content_type": "text/plain"}]')
|
||||
self.app.register(
|
||||
'GET', '/v1/a/ver_cont/003sym/2', swob.HTTPCreated,
|
||||
{'content-length': '0',
|
||||
'X-Object-Sysmeta-Symlink-Target': 'c/tgt'}, None)
|
||||
self.app.register(
|
||||
'PUT', '/v1/a/c/sym', swob.HTTPCreated,
|
||||
{'X-Symlink-Target': 'c/tgt', 'X-Symlink-Target-Account': 'a'},
|
||||
None)
|
||||
self.app.register(
|
||||
'DELETE', '/v1/a/ver_cont/003sym/2', swob.HTTPOk,
|
||||
{}, None)
|
||||
|
||||
cache = FakeCache({'sysmeta': {'versions-location': 'ver_cont'}})
|
||||
req = Request.blank(
|
||||
'/v1/a/c/sym',
|
||||
headers={'X-If-Delete-At': 1},
|
||||
environ={'REQUEST_METHOD': 'DELETE', 'swift.cache': cache,
|
||||
'CONTENT_LENGTH': '0', 'swift.trans_id': 'fake_trans_id'})
|
||||
status, headers, body = self.call_vw(req)
|
||||
self.assertEqual(status, '200 OK')
|
||||
self.assertEqual(len(self.authorized), 1)
|
||||
self.assertRequestEqual(req, self.authorized[0])
|
||||
self.assertEqual(4, self.app.call_count)
|
||||
self.assertEqual(['VW', 'VW', 'VW', 'VW'], self.app.swift_sources)
|
||||
self.assertEqual({'fake_trans_id'}, set(self.app.txn_ids))
|
||||
calls = self.app.calls_with_headers
|
||||
method, path, req_headers = calls[2]
|
||||
self.assertEqual('PUT', method)
|
||||
self.assertEqual('/v1/a/c/sym', path)
|
||||
self.assertEqual(
|
||||
'c/tgt',
|
||||
req_headers['X-Object-Sysmeta-Symlink-Target'])
|
||||
|
||||
|
||||
class TestSymlinkContainerContext(TestSymlinkMiddlewareBase):
|
||||
|
||||
def setUp(self):
|
||||
super(TestSymlinkContainerContext, self).setUp()
|
||||
self.context = symlink.SymlinkContainerContext(
|
||||
self.sym.app, self.sym.logger)
|
||||
|
||||
def test_extract_symlink_path_json_simple_etag(self):
|
||||
obj_dict = {"bytes": 6,
|
||||
"last_modified": "1",
|
||||
"hash": "etag",
|
||||
"name": "obj",
|
||||
"content_type": "application/octet-stream"}
|
||||
obj_dict = self.context._extract_symlink_path_json(
|
||||
obj_dict, 'v1', 'AUTH_a')
|
||||
self.assertEqual(obj_dict['hash'], 'etag')
|
||||
self.assertNotIn('symlink_path', obj_dict)
|
||||
|
||||
def test_extract_symlink_path_json_symlink_path(self):
|
||||
obj_dict = {"bytes": 6,
|
||||
"last_modified": "1",
|
||||
"hash": "etag; symlink_target=c/o",
|
||||
"name": "obj",
|
||||
"content_type": "application/octet-stream"}
|
||||
obj_dict = self.context._extract_symlink_path_json(
|
||||
obj_dict, 'v1', 'AUTH_a')
|
||||
self.assertEqual(obj_dict['hash'], 'etag')
|
||||
self.assertEqual(obj_dict['symlink_path'], '/v1/AUTH_a/c/o')
|
||||
|
||||
def test_extract_symlink_path_json_symlink_path_and_account(self):
|
||||
obj_dict = {
|
||||
"bytes": 6,
|
||||
"last_modified": "1",
|
||||
"hash": "etag; symlink_target=c/o; symlink_target_account=AUTH_a2",
|
||||
"name": "obj",
|
||||
"content_type": "application/octet-stream"}
|
||||
obj_dict = self.context._extract_symlink_path_json(
|
||||
obj_dict, 'v1', 'AUTH_a')
|
||||
self.assertEqual(obj_dict['hash'], 'etag')
|
||||
self.assertEqual(obj_dict['symlink_path'], '/v1/AUTH_a2/c/o')
|
||||
|
||||
def test_extract_symlink_path_json_extra_key(self):
|
||||
obj_dict = {"bytes": 6,
|
||||
"last_modified": "1",
|
||||
"hash": "etag; symlink_target=c/o; extra_key=value",
|
||||
"name": "obj",
|
||||
"content_type": "application/octet-stream"}
|
||||
obj_dict = self.context._extract_symlink_path_json(
|
||||
obj_dict, 'v1', 'AUTH_a')
|
||||
self.assertEqual(obj_dict['hash'], 'etag; extra_key=value')
|
||||
self.assertEqual(obj_dict['symlink_path'], '/v1/AUTH_a/c/o')
|
||||
|
||||
def test_get_container_simple(self):
|
||||
self.app.register(
|
||||
'GET',
|
||||
'/v1/a/c',
|
||||
swob.HTTPOk, {},
|
||||
json.dumps(
|
||||
[{"hash": "etag; symlink_target=c/o;",
|
||||
"last_modified": "2014-11-21T14:23:02.206740",
|
||||
"bytes": 0,
|
||||
"name": "sym_obj",
|
||||
"content_type": "text/plain"},
|
||||
{"hash": "etag2",
|
||||
"last_modified": "2014-11-21T14:14:27.409100",
|
||||
"bytes": 32,
|
||||
"name": "normal_obj",
|
||||
"content_type": "text/plain"}]))
|
||||
req = Request.blank(path='/v1/a/c')
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, '200 OK')
|
||||
obj_list = json.loads(body)
|
||||
self.assertIn('symlink_path', obj_list[0])
|
||||
self.assertIn(obj_list[0]['symlink_path'], '/v1/a/c/o')
|
||||
self.assertNotIn('symlink_path', obj_list[1])
|
||||
|
||||
def test_get_container_with_subdir(self):
|
||||
self.app.register(
|
||||
'GET',
|
||||
'/v1/a/c?delimiter=/',
|
||||
swob.HTTPOk, {},
|
||||
json.dumps([{"subdir": "photos/"}]))
|
||||
req = Request.blank(path='/v1/a/c?delimiter=/')
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, '200 OK')
|
||||
obj_list = json.loads(body)
|
||||
self.assertEqual(len(obj_list), 1)
|
||||
self.assertEqual(obj_list[0]['subdir'], 'photos/')
|
||||
|
||||
def test_get_container_error_cases(self):
|
||||
# No affect for error cases
|
||||
for error in (swob.HTTPNotFound, swob.HTTPUnauthorized,
|
||||
swob.HTTPServiceUnavailable,
|
||||
swob.HTTPInternalServerError):
|
||||
self.app.register('GET', '/v1/a/c', error, {}, '')
|
||||
req = Request.blank(path='/v1/a/c')
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, error().status)
|
||||
|
||||
def test_no_affect_for_account_request(self):
|
||||
with mock.patch.object(self.sym, 'app') as mock_app:
|
||||
mock_app.return_value = 'ok'
|
||||
req = Request.blank(path='/v1/a')
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(body, 'ok')
|
||||
|
||||
def test_get_container_simple_with_listing_format(self):
|
||||
self.app.register(
|
||||
'GET',
|
||||
'/v1/a/c?format=json',
|
||||
swob.HTTPOk, {},
|
||||
json.dumps(
|
||||
[{"hash": "etag; symlink_target=c/o;",
|
||||
"last_modified": "2014-11-21T14:23:02.206740",
|
||||
"bytes": 0,
|
||||
"name": "sym_obj",
|
||||
"content_type": "text/plain"},
|
||||
{"hash": "etag2",
|
||||
"last_modified": "2014-11-21T14:14:27.409100",
|
||||
"bytes": 32,
|
||||
"name": "normal_obj",
|
||||
"content_type": "text/plain"}]))
|
||||
self.lf = listing_formats.filter_factory({})(self.sym)
|
||||
req = Request.blank(path='/v1/a/c?format=json')
|
||||
status, headers, body = self.call_app(req, app=self.lf)
|
||||
self.assertEqual(status, '200 OK')
|
||||
obj_list = json.loads(body)
|
||||
self.assertIn('symlink_path', obj_list[0])
|
||||
self.assertIn(obj_list[0]['symlink_path'], '/v1/a/c/o')
|
||||
self.assertNotIn('symlink_path', obj_list[1])
|
||||
|
||||
def test_get_container_simple_with_listing_format_xml(self):
|
||||
self.app.register(
|
||||
'GET',
|
||||
'/v1/a/c?format=json',
|
||||
swob.HTTPOk, {'Content-Type': 'application/json'},
|
||||
json.dumps(
|
||||
[{"hash": "etag; symlink_target=c/o;",
|
||||
"last_modified": "2014-11-21T14:23:02.206740",
|
||||
"bytes": 0,
|
||||
"name": "sym_obj",
|
||||
"content_type": "text/plain"},
|
||||
{"hash": "etag2",
|
||||
"last_modified": "2014-11-21T14:14:27.409100",
|
||||
"bytes": 32,
|
||||
"name": "normal_obj",
|
||||
"content_type": "text/plain"}]))
|
||||
self.lf = listing_formats.filter_factory({})(self.sym)
|
||||
req = Request.blank(path='/v1/a/c?format=xml')
|
||||
status, headers, body = self.call_app(req, app=self.lf)
|
||||
self.assertEqual(status, '200 OK')
|
||||
self.assertEqual(body.split('\n'), [
|
||||
'<?xml version="1.0" encoding="UTF-8"?>',
|
||||
'<container name="c"><object><name>sym_obj</name>'
|
||||
'<hash>etag</hash><bytes>0</bytes>'
|
||||
'<content_type>text/plain</content_type>'
|
||||
'<last_modified>2014-11-21T14:23:02.206740</last_modified>'
|
||||
'<symlink_path>/v1/a/c/o</symlink_path>'
|
||||
'</object>'
|
||||
'<object><name>normal_obj</name><hash>etag2</hash>'
|
||||
'<bytes>32</bytes><content_type>text/plain</content_type>'
|
||||
'<last_modified>2014-11-21T14:14:27.409100</last_modified>'
|
||||
'</object></container>'])
|
||||
@ -857,18 +857,21 @@ class TestTempURL(unittest.TestCase):
|
||||
path = '/v1/a/c/o'
|
||||
key = 'abc'
|
||||
for method in ('PUT', 'POST'):
|
||||
hmac_body = '%s\n%s\n%s' % (method, expires, path)
|
||||
sig = hmac.new(key, hmac_body, sha1).hexdigest()
|
||||
req = self._make_request(
|
||||
path, method=method, keys=[key],
|
||||
headers={'x-object-manifest': 'private/secret'},
|
||||
environ={'QUERY_STRING': 'temp_url_sig=%s&temp_url_expires=%s'
|
||||
% (sig, expires)})
|
||||
resp = req.get_response(self.tempurl)
|
||||
self.assertEqual(resp.status_int, 400)
|
||||
self.assertTrue('header' in resp.body)
|
||||
self.assertTrue('not allowed' in resp.body)
|
||||
self.assertTrue('X-Object-Manifest' in resp.body)
|
||||
for hdr, value in [('X-Object-Manifest', 'private/secret'),
|
||||
('X-Symlink-Target', 'cont/symlink')]:
|
||||
hmac_body = '%s\n%s\n%s' % (method, expires, path)
|
||||
sig = hmac.new(key, hmac_body, sha1).hexdigest()
|
||||
req = self._make_request(
|
||||
path, method=method, keys=[key],
|
||||
headers={hdr: value},
|
||||
environ={'QUERY_STRING':
|
||||
'temp_url_sig=%s&temp_url_expires=%s'
|
||||
% (sig, expires)})
|
||||
resp = req.get_response(self.tempurl)
|
||||
self.assertEqual(resp.status_int, 400)
|
||||
self.assertTrue('header' in resp.body)
|
||||
self.assertTrue('not allowed' in resp.body)
|
||||
self.assertTrue(hdr in resp.body)
|
||||
|
||||
def test_removed_incoming_header(self):
|
||||
self.tempurl = tempurl.filter_factory({
|
||||
|
||||
@ -23,7 +23,7 @@ import os
|
||||
import six
|
||||
from six import StringIO
|
||||
from six.moves import range
|
||||
from six.moves.urllib.parse import quote
|
||||
from six.moves.urllib.parse import quote, parse_qsl
|
||||
from test.unit import FakeLogger
|
||||
from swift.common import exceptions, internal_client, swob
|
||||
from swift.common.header_key_dict import HeaderKeyDict
|
||||
@ -316,6 +316,29 @@ class TestInternalClient(unittest.TestCase):
|
||||
client = InternalClient(self)
|
||||
client.make_request('GET', '/', {}, (200,))
|
||||
|
||||
def test_make_request_sets_query_string(self):
|
||||
captured_envs = []
|
||||
|
||||
class InternalClient(internal_client.InternalClient):
|
||||
def __init__(self, test):
|
||||
self.test = test
|
||||
self.app = self.fake_app
|
||||
self.user_agent = 'some_agent'
|
||||
self.request_tries = 1
|
||||
|
||||
def fake_app(self, env, start_response):
|
||||
captured_envs.append(env)
|
||||
start_response('200 Ok', [('Content-Length', '0')])
|
||||
return []
|
||||
|
||||
client = InternalClient(self)
|
||||
params = {'param1': 'p1', 'tasty': 'soup'}
|
||||
client.make_request('GET', '/', {}, (200,), params=params)
|
||||
actual_params = dict(parse_qsl(captured_envs[0]['QUERY_STRING'],
|
||||
keep_blank_values=True,
|
||||
strict_parsing=True))
|
||||
self.assertEqual(params, actual_params)
|
||||
|
||||
def test_make_request_retries(self):
|
||||
class InternalClient(internal_client.InternalClient):
|
||||
def __init__(self, test):
|
||||
@ -1047,10 +1070,11 @@ class TestInternalClient(unittest.TestCase):
|
||||
client, app = get_client_app()
|
||||
headers = {'foo': 'bar'}
|
||||
body = 'some_object_body'
|
||||
params = {'symlink': 'get'}
|
||||
app.register('GET', path_info, swob.HTTPOk, headers, body)
|
||||
req_headers = {'x-important-header': 'some_important_value'}
|
||||
status_int, resp_headers, obj_iter = client.get_object(
|
||||
account, container, obj, req_headers)
|
||||
account, container, obj, req_headers, params=params)
|
||||
self.assertEqual(status_int // 100, 2)
|
||||
for k, v in headers.items():
|
||||
self.assertEqual(v, resp_headers[k])
|
||||
@ -1062,7 +1086,7 @@ class TestInternalClient(unittest.TestCase):
|
||||
'user-agent': 'test', # from InternalClient.make_request
|
||||
})
|
||||
self.assertEqual(app.calls_with_headers, [(
|
||||
'GET', path_info, HeaderKeyDict(req_headers))])
|
||||
'GET', path_info + '?symlink=get', HeaderKeyDict(req_headers))])
|
||||
|
||||
def test_iter_object_lines(self):
|
||||
class InternalClient(internal_client.InternalClient):
|
||||
|
||||
@ -968,7 +968,9 @@ class TestContainerSync(unittest.TestCase):
|
||||
logger=self.logger)
|
||||
cs.http_proxies = ['http://proxy']
|
||||
|
||||
def fake_get_object(acct, con, obj, headers, acceptable_statuses):
|
||||
def fake_get_object(acct, con, obj, headers, acceptable_statuses,
|
||||
params=None):
|
||||
self.assertEqual({'symlink': 'get'}, params)
|
||||
self.assertEqual(headers['X-Backend-Storage-Policy-Index'],
|
||||
'0')
|
||||
return (200,
|
||||
@ -1004,7 +1006,9 @@ class TestContainerSync(unittest.TestCase):
|
||||
expected_put_count += 1
|
||||
self.assertEqual(cs.container_puts, expected_put_count)
|
||||
|
||||
def fake_get_object(acct, con, obj, headers, acceptable_statuses):
|
||||
def fake_get_object(acct, con, obj, headers, acceptable_statuses,
|
||||
params=None):
|
||||
self.assertEqual({'symlink': 'get'}, params)
|
||||
self.assertEqual(headers['X-Newest'], True)
|
||||
self.assertEqual(headers['X-Backend-Storage-Policy-Index'],
|
||||
'0')
|
||||
@ -1055,7 +1059,9 @@ class TestContainerSync(unittest.TestCase):
|
||||
expected_put_count += 1
|
||||
self.assertEqual(cs.container_puts, expected_put_count)
|
||||
|
||||
def fake_get_object(acct, con, obj, headers, acceptable_statuses):
|
||||
def fake_get_object(acct, con, obj, headers, acceptable_statuses,
|
||||
params=None):
|
||||
self.assertEqual({'symlink': 'get'}, params)
|
||||
self.assertEqual(headers['X-Newest'], True)
|
||||
self.assertEqual(headers['X-Backend-Storage-Policy-Index'],
|
||||
'0')
|
||||
@ -1090,7 +1096,9 @@ class TestContainerSync(unittest.TestCase):
|
||||
|
||||
exc = []
|
||||
|
||||
def fake_get_object(acct, con, obj, headers, acceptable_statuses):
|
||||
def fake_get_object(acct, con, obj, headers, acceptable_statuses,
|
||||
params=None):
|
||||
self.assertEqual({'symlink': 'get'}, params)
|
||||
self.assertEqual(headers['X-Newest'], True)
|
||||
self.assertEqual(headers['X-Backend-Storage-Policy-Index'],
|
||||
'0')
|
||||
@ -1114,7 +1122,9 @@ class TestContainerSync(unittest.TestCase):
|
||||
|
||||
exc = []
|
||||
|
||||
def fake_get_object(acct, con, obj, headers, acceptable_statuses):
|
||||
def fake_get_object(acct, con, obj, headers, acceptable_statuses,
|
||||
params=None):
|
||||
self.assertEqual({'symlink': 'get'}, params)
|
||||
self.assertEqual(headers['X-Newest'], True)
|
||||
self.assertEqual(headers['X-Backend-Storage-Policy-Index'],
|
||||
'0')
|
||||
@ -1137,7 +1147,9 @@ class TestContainerSync(unittest.TestCase):
|
||||
self.assertEqual(len(exc), 1)
|
||||
self.assertEqual(str(exc[-1]), 'test client exception')
|
||||
|
||||
def fake_get_object(acct, con, obj, headers, acceptable_statuses):
|
||||
def fake_get_object(acct, con, obj, headers, acceptable_statuses,
|
||||
params=None):
|
||||
self.assertEqual({'symlink': 'get'}, params)
|
||||
self.assertEqual(headers['X-Newest'], True)
|
||||
self.assertEqual(headers['X-Backend-Storage-Policy-Index'],
|
||||
'0')
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user