Merge "New container sync configuration option"
This commit is contained in:
commit
d698c21ab3
|
@ -217,6 +217,20 @@ List Endpoints
|
||||||
:members:
|
:members:
|
||||||
:show-inheritance:
|
:show-inheritance:
|
||||||
|
|
||||||
|
Container Sync Realms
|
||||||
|
=====================
|
||||||
|
|
||||||
|
.. automodule:: swift.common.container_sync_realms
|
||||||
|
:members:
|
||||||
|
:show-inheritance:
|
||||||
|
|
||||||
|
Container Sync Middleware
|
||||||
|
=========================
|
||||||
|
|
||||||
|
.. automodule:: swift.common.middleware.container_sync
|
||||||
|
:members:
|
||||||
|
:show-inheritance:
|
||||||
|
|
||||||
Discoverability
|
Discoverability
|
||||||
===============
|
===============
|
||||||
|
|
||||||
|
|
|
@ -25,13 +25,76 @@ synchronization key.
|
||||||
your manifest file and your segment files are synced if they happen to be
|
your manifest file and your segment files are synced if they happen to be
|
||||||
in different containers.
|
in different containers.
|
||||||
|
|
||||||
--------------------------------------------
|
--------------------------
|
||||||
Configuring a Cluster's Allowable Sync Hosts
|
Configuring Container Sync
|
||||||
--------------------------------------------
|
--------------------------
|
||||||
|
|
||||||
The Swift cluster operator must allow synchronization with a set of hosts
|
Create a container-sync-realms.conf file specifying the allowable clusters
|
||||||
before the user can enable container synchronization. First, the backend
|
and their information::
|
||||||
container server needs to be given this list of hosts in the
|
|
||||||
|
[realm1]
|
||||||
|
key = realm1key
|
||||||
|
key2 = realm1key2
|
||||||
|
cluster_name1 = https://host1/v1/
|
||||||
|
cluster_name2 = https://host2/v1/
|
||||||
|
|
||||||
|
[realm2]
|
||||||
|
key = realm2key
|
||||||
|
key2 = realm2key2
|
||||||
|
cluster_name3 = https://host3/v1/
|
||||||
|
cluster_name4 = https://host4/v1/
|
||||||
|
|
||||||
|
|
||||||
|
Each section name is the name of a sync realm. A sync realm is a set of
|
||||||
|
clusters that have agreed to allow container syncing with each other. Realm
|
||||||
|
names will be considered case insensitive.
|
||||||
|
|
||||||
|
The key is the overall cluster-to-cluster key used in combination with the
|
||||||
|
external users' key that they set on their containers' X-Container-Sync-Key
|
||||||
|
metadata header values. These keys will be used to sign each request the
|
||||||
|
container sync daemon makes and used to validate each incoming container sync
|
||||||
|
request.
|
||||||
|
|
||||||
|
The key2 is optional and is an additional key incoming requests will be checked
|
||||||
|
against. This is so you can rotate keys if you wish; you move the existing key
|
||||||
|
to key2 and make a new key value.
|
||||||
|
|
||||||
|
Any values in the realm section whose names begin with cluster\_ will indicate
|
||||||
|
the name and endpoint of a cluster and will be used by external users in
|
||||||
|
their containers' X-Container-Sync-To metadata header values with the format
|
||||||
|
"//realm_name/cluster_name/account_name/container_name". Realm and cluster
|
||||||
|
names are considered case insensitive.
|
||||||
|
|
||||||
|
The endpoint is what the container sync daemon will use when sending out
|
||||||
|
requests to that cluster. Keep in mind this endpoint must be reachable by all
|
||||||
|
container servers, since that is where the container sync daemon runs. Note
|
||||||
|
that the endpoint ends with /v1/ and that the container sync daemon will then
|
||||||
|
add the account/container/obj name after that.
|
||||||
|
|
||||||
|
Distribute this container-sync-realms.conf file to all your proxy servers
|
||||||
|
and container servers.
|
||||||
|
|
||||||
|
You also need to add the container_sync middleware to your proxy pipeline. It
|
||||||
|
needs to be after any memcache middleware and before any auth middleware. The
|
||||||
|
container_sync section only needs the "use" item. For example::
|
||||||
|
|
||||||
|
[pipeline:main]
|
||||||
|
pipeline = healthcheck proxy-logging cache container_sync tempauth proxy-logging proxy-server
|
||||||
|
|
||||||
|
[filter:container_sync]
|
||||||
|
use = egg:swift#container_sync
|
||||||
|
|
||||||
|
|
||||||
|
-------------------------------------------------------
|
||||||
|
Old-Style: Configuring a Cluster's Allowable Sync Hosts
|
||||||
|
-------------------------------------------------------
|
||||||
|
|
||||||
|
This section is for the old-style of using container sync. See the previous
|
||||||
|
section, Configuring Container Sync, for the new-style.
|
||||||
|
|
||||||
|
With the old-style, the Swift cluster operator must allow synchronization with
|
||||||
|
a set of hosts before the user can enable container synchronization. First, the
|
||||||
|
backend container server needs to be given this list of hosts in the
|
||||||
container-server.conf file::
|
container-server.conf file::
|
||||||
|
|
||||||
[DEFAULT]
|
[DEFAULT]
|
||||||
|
@ -52,13 +115,18 @@ container-server.conf file::
|
||||||
# Maximum amount of time to spend syncing each container
|
# Maximum amount of time to spend syncing each container
|
||||||
# container_time = 60
|
# container_time = 60
|
||||||
|
|
||||||
|
|
||||||
|
----------------------
|
||||||
|
Logging Container Sync
|
||||||
|
----------------------
|
||||||
|
|
||||||
Tracking sync progress, problems, and just general activity can only be
|
Tracking sync progress, problems, and just general activity can only be
|
||||||
achieved with log processing for this first release of container
|
achieved with log processing currently for container synchronization. In that
|
||||||
synchronization. In that light, you may wish to set the above `log_` options to
|
light, you may wish to set the above `log_` options to direct the
|
||||||
direct the container-sync logs to a different file for easier monitoring.
|
container-sync logs to a different file for easier monitoring. Additionally, it
|
||||||
Additionally, it should be noted there is no way for an end user to detect sync
|
should be noted there is no way for an end user to detect sync progress or
|
||||||
progress or problems other than HEADing both containers and comparing the
|
problems other than HEADing both containers and comparing the overall
|
||||||
overall information.
|
information.
|
||||||
|
|
||||||
----------------------------------------------------------
|
----------------------------------------------------------
|
||||||
Using the ``swift`` tool to set up synchronized containers
|
Using the ``swift`` tool to set up synchronized containers
|
||||||
|
@ -73,6 +141,112 @@ Using the ``swift`` tool to set up synchronized containers
|
||||||
You must be the account admin on the account to set synchronization targets
|
You must be the account admin on the account to set synchronization targets
|
||||||
and keys.
|
and keys.
|
||||||
|
|
||||||
|
You simply tell each container where to sync to and give it a secret
|
||||||
|
synchronization key. First, let's get the account details for our two cluster
|
||||||
|
accounts::
|
||||||
|
|
||||||
|
$ swift -A http://cluster1/auth/v1.0 -U test:tester -K testing stat -v
|
||||||
|
StorageURL: http://cluster1/v1/AUTH_208d1854-e475-4500-b315-81de645d060e
|
||||||
|
Auth Token: AUTH_tkd5359e46ff9e419fa193dbd367f3cd19
|
||||||
|
Account: AUTH_208d1854-e475-4500-b315-81de645d060e
|
||||||
|
Containers: 0
|
||||||
|
Objects: 0
|
||||||
|
Bytes: 0
|
||||||
|
|
||||||
|
$ swift -A http://cluster2/auth/v1.0 -U test2:tester2 -K testing2 stat -v
|
||||||
|
StorageURL: http://cluster2/v1/AUTH_33cdcad8-09fb-4940-90da-0f00cbf21c7c
|
||||||
|
Auth Token: AUTH_tk816a1aaf403c49adb92ecfca2f88e430
|
||||||
|
Account: AUTH_33cdcad8-09fb-4940-90da-0f00cbf21c7c
|
||||||
|
Containers: 0
|
||||||
|
Objects: 0
|
||||||
|
Bytes: 0
|
||||||
|
|
||||||
|
Now, let's make our first container and tell it to synchronize to a second
|
||||||
|
we'll make next::
|
||||||
|
|
||||||
|
$ swift -A http://cluster1/auth/v1.0 -U test:tester -K testing post \
|
||||||
|
-t '//realm_name/cluster2_name/AUTH_33cdcad8-09fb-4940-90da-0f00cbf21c7c/container2' \
|
||||||
|
-k 'secret' container1
|
||||||
|
|
||||||
|
The ``-t`` indicates the cluster to sync to, which is the realm name of the
|
||||||
|
section from container-sync-realms.conf, followed by the cluster name from
|
||||||
|
that section, followed by the account and container names we want to sync to.
|
||||||
|
The ``-k`` specifies the secret key the two containers will share for
|
||||||
|
synchronization; this is the user key, the cluster key in
|
||||||
|
container-sync-realms.conf will also be used behind the scenes.
|
||||||
|
|
||||||
|
Now, we'll do something similar for the second cluster's container::
|
||||||
|
|
||||||
|
$ swift -A http://cluster2/auth/v1.0 -U test2:tester2 -K testing2 post \
|
||||||
|
-t '//realm_name/cluster1_name/AUTH_208d1854-e475-4500-b315-81de645d060e/container1' \
|
||||||
|
-k 'secret' container2
|
||||||
|
|
||||||
|
That's it. Now we can upload a bunch of stuff to the first container and watch
|
||||||
|
as it gets synchronized over to the second::
|
||||||
|
|
||||||
|
$ swift -A http://cluster1/auth/v1.0 -U test:tester -K testing \
|
||||||
|
upload container1 .
|
||||||
|
photo002.png
|
||||||
|
photo004.png
|
||||||
|
photo001.png
|
||||||
|
photo003.png
|
||||||
|
|
||||||
|
$ swift -A http://cluster2/auth/v1.0 -U test2:tester2 -K testing2 \
|
||||||
|
list container2
|
||||||
|
|
||||||
|
[Nothing there yet, so we wait a bit...]
|
||||||
|
[If you're an operator running SAIO and just testing, you may need to
|
||||||
|
run 'swift-init container-sync once' to perform a sync scan.]
|
||||||
|
|
||||||
|
$ swift -A http://cluster2/auth/v1.0 -U test2:tester2 -K testing2 \
|
||||||
|
list container2
|
||||||
|
photo001.png
|
||||||
|
photo002.png
|
||||||
|
photo003.png
|
||||||
|
photo004.png
|
||||||
|
|
||||||
|
You can also set up a chain of synced containers if you want more than two.
|
||||||
|
You'd point 1 -> 2, then 2 -> 3, and finally 3 -> 1 for three containers.
|
||||||
|
They'd all need to share the same secret synchronization key.
|
||||||
|
|
||||||
|
.. _`python-swiftclient`: http://github.com/openstack/python-swiftclient
|
||||||
|
|
||||||
|
-----------------------------------
|
||||||
|
Using curl (or other tools) instead
|
||||||
|
-----------------------------------
|
||||||
|
|
||||||
|
So what's ``swift`` doing behind the scenes? Nothing overly complicated. It
|
||||||
|
translates the ``-t <value>`` option into an ``X-Container-Sync-To: <value>``
|
||||||
|
header and the ``-k <value>`` option into an ``X-Container-Sync-Key: <value>``
|
||||||
|
header.
|
||||||
|
|
||||||
|
For instance, when we created the first container above and told it to
|
||||||
|
synchronize to the second, we could have used this curl command::
|
||||||
|
|
||||||
|
$ curl -i -X POST -H 'X-Auth-Token: AUTH_tkd5359e46ff9e419fa193dbd367f3cd19' \
|
||||||
|
-H 'X-Container-Sync-To: //realm_name/cluster2_name/AUTH_33cdcad8-09fb-4940-90da-0f00cbf21c7c/container2' \
|
||||||
|
-H 'X-Container-Sync-Key: secret' \
|
||||||
|
'http://cluster1/v1/AUTH_208d1854-e475-4500-b315-81de645d060e/container1'
|
||||||
|
HTTP/1.1 204 No Content
|
||||||
|
Content-Length: 0
|
||||||
|
Content-Type: text/plain; charset=UTF-8
|
||||||
|
Date: Thu, 24 Feb 2011 22:39:14 GMT
|
||||||
|
|
||||||
|
---------------------------------------------------------------------
|
||||||
|
Old-Style: Using the ``swift`` tool to set up synchronized containers
|
||||||
|
---------------------------------------------------------------------
|
||||||
|
|
||||||
|
.. note::
|
||||||
|
|
||||||
|
The ``swift`` tool is available from the `python-swiftclient`_ library.
|
||||||
|
|
||||||
|
.. note::
|
||||||
|
|
||||||
|
You must be the account admin on the account to set synchronization targets
|
||||||
|
and keys.
|
||||||
|
|
||||||
|
This is for the old-style of container syncing using allowed_sync_hosts.
|
||||||
|
|
||||||
You simply tell each container where to sync to and give it a secret
|
You simply tell each container where to sync to and give it a secret
|
||||||
synchronization key. First, let's get the account details for our two cluster
|
synchronization key. First, let's get the account details for our two cluster
|
||||||
accounts::
|
accounts::
|
||||||
|
@ -139,9 +313,11 @@ They'd all need to share the same secret synchronization key.
|
||||||
|
|
||||||
.. _`python-swiftclient`: http://github.com/openstack/python-swiftclient
|
.. _`python-swiftclient`: http://github.com/openstack/python-swiftclient
|
||||||
|
|
||||||
-----------------------------------
|
----------------------------------------------
|
||||||
Using curl (or other tools) instead
|
Old-Style: Using curl (or other tools) instead
|
||||||
-----------------------------------
|
----------------------------------------------
|
||||||
|
|
||||||
|
This is for the old-style of container syncing using allowed_sync_hosts.
|
||||||
|
|
||||||
So what's ``swift`` doing behind the scenes? Nothing overly complicated. It
|
So what's ``swift`` doing behind the scenes? Nothing overly complicated. It
|
||||||
translates the ``-t <value>`` option into an ``X-Container-Sync-To: <value>``
|
translates the ``-t <value>`` option into an ``X-Container-Sync-To: <value>``
|
||||||
|
@ -174,10 +350,10 @@ to the other container.
|
||||||
|
|
||||||
.. note::
|
.. note::
|
||||||
|
|
||||||
The swift-container-sync process runs on each container server in
|
The swift-container-sync process runs on each container server in the
|
||||||
the cluster and talks to the proxy servers in the remote cluster.
|
cluster and talks to the proxy servers (or load balancers) in the remote
|
||||||
Therefore, the container servers must be permitted to initiate
|
cluster. Therefore, the container servers must be permitted to initiate
|
||||||
outbound connections to the remote proxy servers.
|
outbound connections to the remote proxy servers (or load balancers).
|
||||||
|
|
||||||
.. note::
|
.. note::
|
||||||
|
|
||||||
|
|
|
@ -17,7 +17,9 @@
|
||||||
# max_clients = 1024
|
# max_clients = 1024
|
||||||
#
|
#
|
||||||
# This is a comma separated list of hosts allowed in the X-Container-Sync-To
|
# This is a comma separated list of hosts allowed in the X-Container-Sync-To
|
||||||
# field for containers.
|
# field for containers. This is the old-style of using container sync. It is
|
||||||
|
# strongly recommended to use the new style of a separate
|
||||||
|
# container-sync-realms.conf -- see container-sync-realms.conf-sample
|
||||||
# allowed_sync_hosts = 127.0.0.1
|
# allowed_sync_hosts = 127.0.0.1
|
||||||
#
|
#
|
||||||
# You can specify default log routing here if you want:
|
# You can specify default log routing here if you want:
|
||||||
|
|
|
@ -0,0 +1,47 @@
|
||||||
|
# [DEFAULT]
|
||||||
|
# The number of seconds between checking the modified time of this config file
|
||||||
|
# for changes and therefore reloading it.
|
||||||
|
# mtime_check_interval = 300
|
||||||
|
|
||||||
|
|
||||||
|
# [realm1]
|
||||||
|
# key = realm1key
|
||||||
|
# key2 = realm1key2
|
||||||
|
# cluster_name1 = https://host1/v1/
|
||||||
|
# cluster_name2 = https://host2/v1/
|
||||||
|
#
|
||||||
|
# [realm2]
|
||||||
|
# key = realm2key
|
||||||
|
# key2 = realm2key2
|
||||||
|
# cluster_name3 = https://host3/v1/
|
||||||
|
# cluster_name4 = https://host4/v1/
|
||||||
|
|
||||||
|
|
||||||
|
# Each section name is the name of a sync realm. A sync realm is a set of
|
||||||
|
# clusters that have agreed to allow container syncing with each other. Realm
|
||||||
|
# names will be considered case insensitive.
|
||||||
|
#
|
||||||
|
# The key is the overall cluster-to-cluster key used in combination with the
|
||||||
|
# external users' key that they set on their containers' X-Container-Sync-Key
|
||||||
|
# metadata header values. These keys will be used to sign each request the
|
||||||
|
# container sync daemon makes and used to validate each incoming container sync
|
||||||
|
# request.
|
||||||
|
#
|
||||||
|
# The key2 is optional and is an additional key incoming requests will be
|
||||||
|
# checked against. This is so you can rotate keys if you wish; you move the
|
||||||
|
# existing key to key2 and make a new key value.
|
||||||
|
#
|
||||||
|
# Any values in the realm section whose names begin with cluster_ will indicate
|
||||||
|
# the name and endpoint of a cluster and will be used by external users in
|
||||||
|
# their containers' X-Container-Sync-To metadata header values with the format
|
||||||
|
# "realm_name/cluster_name/container_name". Realm and cluster names are
|
||||||
|
# considered case insensitive.
|
||||||
|
#
|
||||||
|
# The endpoint is what the container sync daemon will use when sending out
|
||||||
|
# requests to that cluster. Keep in mind this endpoint must be reachable by all
|
||||||
|
# container servers, since that is where the container sync daemon runs. Note
|
||||||
|
# the the endpoint ends with /v1/ and that the container sync daemon will then
|
||||||
|
# add the account/container/obj name after that.
|
||||||
|
#
|
||||||
|
# Distribute this container-sync-realms.conf file to all your proxy servers
|
||||||
|
# and container servers.
|
|
@ -69,7 +69,7 @@
|
||||||
# eventlet_debug = false
|
# eventlet_debug = false
|
||||||
|
|
||||||
[pipeline:main]
|
[pipeline:main]
|
||||||
pipeline = catch_errors gatekeeper healthcheck proxy-logging cache bulk slo ratelimit tempauth container-quotas account-quotas proxy-logging proxy-server
|
pipeline = catch_errors gatekeeper healthcheck proxy-logging cache container_sync bulk slo ratelimit tempauth container-quotas account-quotas proxy-logging proxy-server
|
||||||
|
|
||||||
[app:proxy-server]
|
[app:proxy-server]
|
||||||
use = egg:swift#proxy
|
use = egg:swift#proxy
|
||||||
|
@ -529,3 +529,12 @@ use = egg:swift#gatekeeper
|
||||||
# set log_level = INFO
|
# set log_level = INFO
|
||||||
# set log_headers = false
|
# set log_headers = false
|
||||||
# set log_address = /dev/log
|
# set log_address = /dev/log
|
||||||
|
|
||||||
|
[filter:container_sync]
|
||||||
|
use = egg:swift#container_sync
|
||||||
|
# Set this to false if you want to disallow any full url values to be set for
|
||||||
|
# any new X-Container-Sync-To headers. This will keep any new full urls from
|
||||||
|
# coming in, but won't change any existing values already in the cluster.
|
||||||
|
# Updating those will have to be done manually, as knowing what the true realm
|
||||||
|
# endpoint should be cannot always be guessed.
|
||||||
|
# allow_full_urls = true
|
||||||
|
|
|
@ -87,6 +87,7 @@ paste.filter_factory =
|
||||||
slo = swift.common.middleware.slo:filter_factory
|
slo = swift.common.middleware.slo:filter_factory
|
||||||
list_endpoints = swift.common.middleware.list_endpoints:filter_factory
|
list_endpoints = swift.common.middleware.list_endpoints:filter_factory
|
||||||
gatekeeper = swift.common.middleware.gatekeeper:filter_factory
|
gatekeeper = swift.common.middleware.gatekeeper:filter_factory
|
||||||
|
container_sync = swift.common.middleware.container_sync:filter_factory
|
||||||
|
|
||||||
[build_sphinx]
|
[build_sphinx]
|
||||||
all_files = 1
|
all_files = 1
|
||||||
|
|
|
@ -0,0 +1,159 @@
|
||||||
|
# Copyright (c) 2013 OpenStack Foundation
|
||||||
|
#
|
||||||
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
# you may not use this file except in compliance with the License.
|
||||||
|
# You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||||
|
# implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
import ConfigParser
|
||||||
|
import errno
|
||||||
|
import hashlib
|
||||||
|
import hmac
|
||||||
|
import os
|
||||||
|
import time
|
||||||
|
|
||||||
|
from swift import gettext_ as _
|
||||||
|
from swift.common.utils import get_valid_utf8_str
|
||||||
|
|
||||||
|
|
||||||
|
class ContainerSyncRealms(object):
|
||||||
|
"""
|
||||||
|
Loads and parses the container-sync-realms.conf, occasionally
|
||||||
|
checking the file's mtime to see if it needs to be reloaded.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, conf_path, logger):
|
||||||
|
self.conf_path = conf_path
|
||||||
|
self.logger = logger
|
||||||
|
self.next_mtime_check = 0
|
||||||
|
self.mtime_check_interval = 300
|
||||||
|
self.conf_path_mtime = 0
|
||||||
|
self.data = {}
|
||||||
|
self.reload()
|
||||||
|
|
||||||
|
def reload(self):
|
||||||
|
"""Forces a reload of the conf file."""
|
||||||
|
self.next_mtime_check = 0
|
||||||
|
self.conf_path_mtime = 0
|
||||||
|
self._reload()
|
||||||
|
|
||||||
|
def _reload(self):
|
||||||
|
now = time.time()
|
||||||
|
if now >= self.next_mtime_check:
|
||||||
|
self.next_mtime_check = now + self.mtime_check_interval
|
||||||
|
try:
|
||||||
|
mtime = os.path.getmtime(self.conf_path)
|
||||||
|
except OSError as err:
|
||||||
|
if err.errno == errno.ENOENT:
|
||||||
|
log_func = self.logger.debug
|
||||||
|
else:
|
||||||
|
log_func = self.logger.error
|
||||||
|
log_func(_('Could not load %r: %s'), self.conf_path, err)
|
||||||
|
else:
|
||||||
|
if mtime != self.conf_path_mtime:
|
||||||
|
self.conf_path_mtime = mtime
|
||||||
|
try:
|
||||||
|
conf = ConfigParser.SafeConfigParser()
|
||||||
|
conf.read(self.conf_path)
|
||||||
|
except ConfigParser.ParsingError as err:
|
||||||
|
self.logger.error(
|
||||||
|
_('Could not load %r: %s'), self.conf_path, err)
|
||||||
|
else:
|
||||||
|
try:
|
||||||
|
self.mtime_check_interval = conf.getint(
|
||||||
|
'DEFAULT', 'mtime_check_interval')
|
||||||
|
self.next_mtime_check = \
|
||||||
|
now + self.mtime_check_interval
|
||||||
|
except ConfigParser.NoOptionError:
|
||||||
|
self.mtime_check_interval = 300
|
||||||
|
self.next_mtime_check = \
|
||||||
|
now + self.mtime_check_interval
|
||||||
|
except (ConfigParser.ParsingError, ValueError) as err:
|
||||||
|
self.logger.error(
|
||||||
|
_('Error in %r with mtime_check_interval: %s'),
|
||||||
|
self.conf_path, err)
|
||||||
|
realms = {}
|
||||||
|
for section in conf.sections():
|
||||||
|
realm = {}
|
||||||
|
clusters = {}
|
||||||
|
for option, value in conf.items(section):
|
||||||
|
if option in ('key', 'key2'):
|
||||||
|
realm[option] = value
|
||||||
|
elif option.startswith('cluster_'):
|
||||||
|
clusters[option[8:].upper()] = value
|
||||||
|
realm['clusters'] = clusters
|
||||||
|
realms[section.upper()] = realm
|
||||||
|
self.data = realms
|
||||||
|
|
||||||
|
def realms(self):
|
||||||
|
"""Returns a list of realms."""
|
||||||
|
self._reload()
|
||||||
|
return self.data.keys()
|
||||||
|
|
||||||
|
def key(self, realm):
|
||||||
|
"""Returns the key for the realm."""
|
||||||
|
self._reload()
|
||||||
|
result = self.data.get(realm.upper())
|
||||||
|
if result:
|
||||||
|
result = result.get('key')
|
||||||
|
return result
|
||||||
|
|
||||||
|
def key2(self, realm):
|
||||||
|
"""Returns the key2 for the realm."""
|
||||||
|
self._reload()
|
||||||
|
result = self.data.get(realm.upper())
|
||||||
|
if result:
|
||||||
|
result = result.get('key2')
|
||||||
|
return result
|
||||||
|
|
||||||
|
def clusters(self, realm):
|
||||||
|
"""Returns a list of clusters for the realm."""
|
||||||
|
self._reload()
|
||||||
|
result = self.data.get(realm.upper())
|
||||||
|
if result:
|
||||||
|
result = result.get('clusters')
|
||||||
|
if result:
|
||||||
|
result = result.keys()
|
||||||
|
return result or []
|
||||||
|
|
||||||
|
def endpoint(self, realm, cluster):
|
||||||
|
"""Returns the endpoint for the cluster in the realm."""
|
||||||
|
self._reload()
|
||||||
|
result = None
|
||||||
|
realm_data = self.data.get(realm.upper())
|
||||||
|
if realm_data:
|
||||||
|
cluster_data = realm_data.get('clusters')
|
||||||
|
if cluster_data:
|
||||||
|
result = cluster_data.get(cluster.upper())
|
||||||
|
return result
|
||||||
|
|
||||||
|
def get_sig(self, request_method, path, x_timestamp, nonce, realm_key,
|
||||||
|
user_key):
|
||||||
|
"""
|
||||||
|
Returns the hexdigest string of the HMAC-SHA1 (RFC 2104) for
|
||||||
|
the information given.
|
||||||
|
|
||||||
|
:param request_method: HTTP method of the request.
|
||||||
|
:param path: The path to the resource.
|
||||||
|
:param x_timestamp: The X-Timestamp header value for the request.
|
||||||
|
:param nonce: A unique value for the request.
|
||||||
|
:param realm_key: Shared secret at the cluster operator level.
|
||||||
|
:param user_key: Shared secret at the user's container level.
|
||||||
|
:returns: hexdigest str of the HMAC-SHA1 for the request.
|
||||||
|
"""
|
||||||
|
nonce = get_valid_utf8_str(nonce)
|
||||||
|
realm_key = get_valid_utf8_str(realm_key)
|
||||||
|
user_key = get_valid_utf8_str(user_key)
|
||||||
|
return hmac.new(
|
||||||
|
realm_key,
|
||||||
|
'%s\n%s\n%s\n%s\n%s' % (
|
||||||
|
request_method, path, x_timestamp, nonce, user_key),
|
||||||
|
hashlib.sha1).hexdigest()
|
|
@ -0,0 +1,122 @@
|
||||||
|
# Copyright (c) 2013 OpenStack Foundation
|
||||||
|
#
|
||||||
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
# you may not use this file except in compliance with the License.
|
||||||
|
# You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||||
|
# implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
import os
|
||||||
|
|
||||||
|
from swift.common.container_sync_realms import ContainerSyncRealms
|
||||||
|
from swift.common.swob import HTTPBadRequest, HTTPUnauthorized, wsgify
|
||||||
|
from swift.common.utils import (
|
||||||
|
config_true_value, get_logger, register_swift_info, streq_const_time)
|
||||||
|
from swift.proxy.controllers.base import get_container_info
|
||||||
|
|
||||||
|
|
||||||
|
class ContainerSync(object):
|
||||||
|
"""
|
||||||
|
WSGI middleware that validates an incoming container sync request
|
||||||
|
using the container-sync-realms.conf style of container sync.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, app, conf):
|
||||||
|
self.app = app
|
||||||
|
self.conf = conf
|
||||||
|
self.logger = get_logger(conf, log_route='container_sync')
|
||||||
|
self.realms_conf = ContainerSyncRealms(
|
||||||
|
os.path.join(
|
||||||
|
conf.get('swift_dir', '/etc/swift'),
|
||||||
|
'container-sync-realms.conf'),
|
||||||
|
self.logger)
|
||||||
|
self.allow_full_urls = config_true_value(
|
||||||
|
conf.get('allow_full_urls', 'true'))
|
||||||
|
|
||||||
|
@wsgify
|
||||||
|
def __call__(self, req):
|
||||||
|
if not self.allow_full_urls:
|
||||||
|
sync_to = req.headers.get('x-container-sync-to')
|
||||||
|
if sync_to and not sync_to.startswith('//'):
|
||||||
|
raise HTTPBadRequest(
|
||||||
|
body='Full URLs are not allowed for X-Container-Sync-To '
|
||||||
|
'values. Only realm values of the format '
|
||||||
|
'//realm/cluster/account/container are allowed.\n',
|
||||||
|
request=req)
|
||||||
|
auth = req.headers.get('x-container-sync-auth')
|
||||||
|
if auth:
|
||||||
|
valid = False
|
||||||
|
auth = auth.split()
|
||||||
|
if len(auth) != 3:
|
||||||
|
req.environ.setdefault('swift.log_info', []).append(
|
||||||
|
'cs:not-3-args')
|
||||||
|
else:
|
||||||
|
realm, nonce, sig = auth
|
||||||
|
realm_key = self.realms_conf.key(realm)
|
||||||
|
realm_key2 = self.realms_conf.key2(realm)
|
||||||
|
if not realm_key:
|
||||||
|
req.environ.setdefault('swift.log_info', []).append(
|
||||||
|
'cs:no-local-realm-key')
|
||||||
|
else:
|
||||||
|
info = get_container_info(
|
||||||
|
req.environ, self.app, swift_source='CS')
|
||||||
|
user_key = info.get('sync_key')
|
||||||
|
if not user_key:
|
||||||
|
req.environ.setdefault('swift.log_info', []).append(
|
||||||
|
'cs:no-local-user-key')
|
||||||
|
else:
|
||||||
|
expected = self.realms_conf.get_sig(
|
||||||
|
req.method, req.path,
|
||||||
|
req.headers.get('x-timestamp', '0'), nonce,
|
||||||
|
realm_key, user_key)
|
||||||
|
expected2 = self.realms_conf.get_sig(
|
||||||
|
req.method, req.path,
|
||||||
|
req.headers.get('x-timestamp', '0'), nonce,
|
||||||
|
realm_key2, user_key) if realm_key2 else expected
|
||||||
|
if not streq_const_time(sig, expected) and \
|
||||||
|
not streq_const_time(sig, expected2):
|
||||||
|
req.environ.setdefault(
|
||||||
|
'swift.log_info', []).append('cs:invalid-sig')
|
||||||
|
else:
|
||||||
|
req.environ.setdefault(
|
||||||
|
'swift.log_info', []).append('cs:valid')
|
||||||
|
valid = True
|
||||||
|
if not valid:
|
||||||
|
exc = HTTPUnauthorized(
|
||||||
|
body='X-Container-Sync-Auth header not valid; '
|
||||||
|
'contact cluster operator for support.',
|
||||||
|
headers={'content-type': 'text/plain'},
|
||||||
|
request=req)
|
||||||
|
exc.headers['www-authenticate'] = ' '.join([
|
||||||
|
'SwiftContainerSync',
|
||||||
|
exc.www_authenticate().split(None, 1)[1]])
|
||||||
|
raise exc
|
||||||
|
else:
|
||||||
|
req.environ['swift.authorize_override'] = True
|
||||||
|
if req.path == '/info':
|
||||||
|
# Ensure /info requests get the freshest results
|
||||||
|
dct = {}
|
||||||
|
for realm in self.realms_conf.realms():
|
||||||
|
clusters = self.realms_conf.clusters(realm)
|
||||||
|
if clusters:
|
||||||
|
dct[realm] = {'clusters': dict((c, {}) for c in clusters)}
|
||||||
|
register_swift_info('container_sync', realms=dct)
|
||||||
|
return self.app
|
||||||
|
|
||||||
|
|
||||||
|
def filter_factory(global_conf, **local_conf):
|
||||||
|
conf = global_conf.copy()
|
||||||
|
conf.update(local_conf)
|
||||||
|
register_swift_info('container_sync')
|
||||||
|
|
||||||
|
def cache_filter(app):
|
||||||
|
return ContainerSync(app, conf)
|
||||||
|
|
||||||
|
return cache_filter
|
|
@ -1817,21 +1817,66 @@ def urlparse(url):
|
||||||
return ModifiedParseResult(*stdlib_urlparse(url))
|
return ModifiedParseResult(*stdlib_urlparse(url))
|
||||||
|
|
||||||
|
|
||||||
def validate_sync_to(value, allowed_sync_hosts):
|
def validate_sync_to(value, allowed_sync_hosts, realms_conf):
|
||||||
|
"""
|
||||||
|
Validates an X-Container-Sync-To header value, returning the
|
||||||
|
validated endpoint, realm, and realm_key, or an error string.
|
||||||
|
|
||||||
|
:param value: The X-Container-Sync-To header value to validate.
|
||||||
|
:param allowed_sync_hosts: A list of allowed hosts in endpoints,
|
||||||
|
if realms_conf does not apply.
|
||||||
|
:param realms_conf: A instance of
|
||||||
|
swift.common.container_sync_realms.ContainerSyncRealms to
|
||||||
|
validate against.
|
||||||
|
:returns: A tuple of (error_string, validated_endpoint, realm,
|
||||||
|
realm_key). The error_string will None if the rest of the
|
||||||
|
values have been validated. The validated_endpoint will be
|
||||||
|
the validated endpoint to sync to. The realm and realm_key
|
||||||
|
will be set if validation was done through realms_conf.
|
||||||
|
"""
|
||||||
|
orig_value = value
|
||||||
|
value = value.rstrip('/')
|
||||||
if not value:
|
if not value:
|
||||||
return None
|
return (None, None, None, None)
|
||||||
|
if value.startswith('//'):
|
||||||
|
if not realms_conf:
|
||||||
|
return (None, None, None, None)
|
||||||
|
data = value[2:].split('/')
|
||||||
|
if len(data) != 4:
|
||||||
|
return (
|
||||||
|
_('Invalid X-Container-Sync-To format %r') % orig_value,
|
||||||
|
None, None, None)
|
||||||
|
realm, cluster, account, container = data
|
||||||
|
realm_key = realms_conf.key(realm)
|
||||||
|
if not realm_key:
|
||||||
|
return (_('No realm key for %r') % realm, None, None, None)
|
||||||
|
endpoint = realms_conf.endpoint(realm, cluster)
|
||||||
|
if not endpoint:
|
||||||
|
return (
|
||||||
|
_('No cluster endpoint for %r %r') % (realm, cluster),
|
||||||
|
None, None, None)
|
||||||
|
return (
|
||||||
|
None,
|
||||||
|
'%s/%s/%s' % (endpoint.rstrip('/'), account, container),
|
||||||
|
realm.upper(), realm_key)
|
||||||
p = urlparse(value)
|
p = urlparse(value)
|
||||||
if p.scheme not in ('http', 'https'):
|
if p.scheme not in ('http', 'https'):
|
||||||
return _('Invalid scheme %r in X-Container-Sync-To, must be "http" '
|
return (
|
||||||
'or "https".') % p.scheme
|
_('Invalid scheme %r in X-Container-Sync-To, must be "//", '
|
||||||
|
'"http", or "https".') % p.scheme,
|
||||||
|
None, None, None)
|
||||||
if not p.path:
|
if not p.path:
|
||||||
return _('Path required in X-Container-Sync-To')
|
return (_('Path required in X-Container-Sync-To'), None, None, None)
|
||||||
if p.params or p.query or p.fragment:
|
if p.params or p.query or p.fragment:
|
||||||
return _('Params, queries, and fragments not allowed in '
|
return (
|
||||||
'X-Container-Sync-To')
|
_('Params, queries, and fragments not allowed in '
|
||||||
|
'X-Container-Sync-To'),
|
||||||
|
None, None, None)
|
||||||
if p.hostname not in allowed_sync_hosts:
|
if p.hostname not in allowed_sync_hosts:
|
||||||
return _('Invalid host %r in X-Container-Sync-To') % p.hostname
|
return (
|
||||||
return None
|
_('Invalid host %r in X-Container-Sync-To') % p.hostname,
|
||||||
|
None, None, None)
|
||||||
|
return (None, value, None, None)
|
||||||
|
|
||||||
|
|
||||||
def affinity_key_function(affinity_str):
|
def affinity_key_function(affinity_str):
|
||||||
|
|
|
@ -25,6 +25,7 @@ from eventlet import Timeout
|
||||||
import swift.common.db
|
import swift.common.db
|
||||||
from swift.container.backend import ContainerBroker
|
from swift.container.backend import ContainerBroker
|
||||||
from swift.common.db import DatabaseAlreadyExists
|
from swift.common.db import DatabaseAlreadyExists
|
||||||
|
from swift.common.container_sync_realms import ContainerSyncRealms
|
||||||
from swift.common.request_helpers import get_param, get_listing_content_type, \
|
from swift.common.request_helpers import get_param, get_listing_content_type, \
|
||||||
split_and_validate_path, is_sys_or_user_meta
|
split_and_validate_path, is_sys_or_user_meta
|
||||||
from swift.common.utils import get_logger, hash_path, public, \
|
from swift.common.utils import get_logger, hash_path, public, \
|
||||||
|
@ -62,6 +63,14 @@ class ContainerController(object):
|
||||||
if replication_server is not None:
|
if replication_server is not None:
|
||||||
replication_server = config_true_value(replication_server)
|
replication_server = config_true_value(replication_server)
|
||||||
self.replication_server = replication_server
|
self.replication_server = replication_server
|
||||||
|
#: ContainerSyncCluster instance for validating sync-to values.
|
||||||
|
self.realms_conf = ContainerSyncRealms(
|
||||||
|
os.path.join(
|
||||||
|
conf.get('swift_dir', '/etc/swift'),
|
||||||
|
'container-sync-realms.conf'),
|
||||||
|
self.logger)
|
||||||
|
#: The list of hosts we're allowed to send syncs to. This can be
|
||||||
|
#: overridden by data in self.realms_conf
|
||||||
self.allowed_sync_hosts = [
|
self.allowed_sync_hosts = [
|
||||||
h.strip()
|
h.strip()
|
||||||
for h in conf.get('allowed_sync_hosts', '127.0.0.1').split(',')
|
for h in conf.get('allowed_sync_hosts', '127.0.0.1').split(',')
|
||||||
|
@ -228,8 +237,9 @@ class ContainerController(object):
|
||||||
return HTTPBadRequest(body='Missing timestamp', request=req,
|
return HTTPBadRequest(body='Missing timestamp', request=req,
|
||||||
content_type='text/plain')
|
content_type='text/plain')
|
||||||
if 'x-container-sync-to' in req.headers:
|
if 'x-container-sync-to' in req.headers:
|
||||||
err = validate_sync_to(req.headers['x-container-sync-to'],
|
err, sync_to, realm, realm_key = validate_sync_to(
|
||||||
self.allowed_sync_hosts)
|
req.headers['x-container-sync-to'], self.allowed_sync_hosts,
|
||||||
|
self.realms_conf)
|
||||||
if err:
|
if err:
|
||||||
return HTTPBadRequest(err)
|
return HTTPBadRequest(err)
|
||||||
if self.mount_check and not check_mount(self.root, drive):
|
if self.mount_check and not check_mount(self.root, drive):
|
||||||
|
@ -438,8 +448,9 @@ class ContainerController(object):
|
||||||
return HTTPBadRequest(body='Missing or bad timestamp',
|
return HTTPBadRequest(body='Missing or bad timestamp',
|
||||||
request=req, content_type='text/plain')
|
request=req, content_type='text/plain')
|
||||||
if 'x-container-sync-to' in req.headers:
|
if 'x-container-sync-to' in req.headers:
|
||||||
err = validate_sync_to(req.headers['x-container-sync-to'],
|
err, sync_to, realm, realm_key = validate_sync_to(
|
||||||
self.allowed_sync_hosts)
|
req.headers['x-container-sync-to'], self.allowed_sync_hosts,
|
||||||
|
self.realms_conf)
|
||||||
if err:
|
if err:
|
||||||
return HTTPBadRequest(err)
|
return HTTPBadRequest(err)
|
||||||
if self.mount_check and not check_mount(self.root, drive):
|
if self.mount_check and not check_mount(self.root, drive):
|
||||||
|
|
|
@ -13,6 +13,8 @@
|
||||||
# See the License for the specific language governing permissions and
|
# See the License for the specific language governing permissions and
|
||||||
# limitations under the License.
|
# limitations under the License.
|
||||||
|
|
||||||
|
import os
|
||||||
|
import uuid
|
||||||
from swift import gettext_ as _
|
from swift import gettext_ as _
|
||||||
from time import ctime, time
|
from time import ctime, time
|
||||||
from random import random, shuffle
|
from random import random, shuffle
|
||||||
|
@ -24,11 +26,13 @@ import swift.common.db
|
||||||
from swift.container import server as container_server
|
from swift.container import server as container_server
|
||||||
from swiftclient import delete_object, put_object, quote
|
from swiftclient import delete_object, put_object, quote
|
||||||
from swift.container.backend import ContainerBroker
|
from swift.container.backend import ContainerBroker
|
||||||
|
from swift.common.container_sync_realms import ContainerSyncRealms
|
||||||
from swift.common.direct_client import direct_get_object
|
from swift.common.direct_client import direct_get_object
|
||||||
from swift.common.exceptions import ClientException
|
from swift.common.exceptions import ClientException
|
||||||
from swift.common.ring import Ring
|
from swift.common.ring import Ring
|
||||||
from swift.common.utils import audit_location_generator, get_logger, \
|
from swift.common.utils import audit_location_generator, get_logger, \
|
||||||
hash_path, config_true_value, validate_sync_to, whataremyips, FileLikeIter
|
hash_path, config_true_value, validate_sync_to, whataremyips, \
|
||||||
|
FileLikeIter, urlparse
|
||||||
from swift.common.daemon import Daemon
|
from swift.common.daemon import Daemon
|
||||||
from swift.common.http import HTTP_UNAUTHORIZED, HTTP_NOT_FOUND
|
from swift.common.http import HTTP_UNAUTHORIZED, HTTP_NOT_FOUND
|
||||||
|
|
||||||
|
@ -117,7 +121,14 @@ class ContainerSync(Daemon):
|
||||||
#: to the next one. If a conatiner sync hasn't finished in this time,
|
#: to the next one. If a conatiner sync hasn't finished in this time,
|
||||||
#: it'll just be resumed next scan.
|
#: it'll just be resumed next scan.
|
||||||
self.container_time = int(conf.get('container_time', 60))
|
self.container_time = int(conf.get('container_time', 60))
|
||||||
#: The list of hosts we're allowed to send syncs to.
|
#: ContainerSyncCluster instance for validating sync-to values.
|
||||||
|
self.realms_conf = ContainerSyncRealms(
|
||||||
|
os.path.join(
|
||||||
|
conf.get('swift_dir', '/etc/swift'),
|
||||||
|
'container-sync-realms.conf'),
|
||||||
|
self.logger)
|
||||||
|
#: The list of hosts we're allowed to send syncs to. This can be
|
||||||
|
#: overridden by data in self.realms_conf
|
||||||
self.allowed_sync_hosts = [
|
self.allowed_sync_hosts = [
|
||||||
h.strip()
|
h.strip()
|
||||||
for h in conf.get('allowed_sync_hosts', '127.0.0.1').split(',')
|
for h in conf.get('allowed_sync_hosts', '127.0.0.1').split(',')
|
||||||
|
@ -228,20 +239,20 @@ class ContainerSync(Daemon):
|
||||||
return
|
return
|
||||||
if not broker.is_deleted():
|
if not broker.is_deleted():
|
||||||
sync_to = None
|
sync_to = None
|
||||||
sync_key = None
|
user_key = None
|
||||||
sync_point1 = info['x_container_sync_point1']
|
sync_point1 = info['x_container_sync_point1']
|
||||||
sync_point2 = info['x_container_sync_point2']
|
sync_point2 = info['x_container_sync_point2']
|
||||||
for key, (value, timestamp) in broker.metadata.iteritems():
|
for key, (value, timestamp) in broker.metadata.iteritems():
|
||||||
if key.lower() == 'x-container-sync-to':
|
if key.lower() == 'x-container-sync-to':
|
||||||
sync_to = value
|
sync_to = value
|
||||||
elif key.lower() == 'x-container-sync-key':
|
elif key.lower() == 'x-container-sync-key':
|
||||||
sync_key = value
|
user_key = value
|
||||||
if not sync_to or not sync_key:
|
if not sync_to or not user_key:
|
||||||
self.container_skips += 1
|
self.container_skips += 1
|
||||||
self.logger.increment('skips')
|
self.logger.increment('skips')
|
||||||
return
|
return
|
||||||
sync_to = sync_to.rstrip('/')
|
err, sync_to, realm, realm_key = validate_sync_to(
|
||||||
err = validate_sync_to(sync_to, self.allowed_sync_hosts)
|
sync_to, self.allowed_sync_hosts, self.realms_conf)
|
||||||
if err:
|
if err:
|
||||||
self.logger.info(
|
self.logger.info(
|
||||||
_('ERROR %(db_file)s: %(validate_sync_to_err)s'),
|
_('ERROR %(db_file)s: %(validate_sync_to_err)s'),
|
||||||
|
@ -267,8 +278,9 @@ class ContainerSync(Daemon):
|
||||||
# This section will attempt to sync previously skipped
|
# This section will attempt to sync previously skipped
|
||||||
# rows in case the previous attempts by any of the nodes
|
# rows in case the previous attempts by any of the nodes
|
||||||
# didn't succeed.
|
# didn't succeed.
|
||||||
if not self.container_sync_row(row, sync_to, sync_key,
|
if not self.container_sync_row(
|
||||||
broker, info):
|
row, sync_to, user_key, broker, info, realm,
|
||||||
|
realm_key):
|
||||||
if not next_sync_point:
|
if not next_sync_point:
|
||||||
next_sync_point = sync_point2
|
next_sync_point = sync_point2
|
||||||
sync_point2 = row['ROWID']
|
sync_point2 = row['ROWID']
|
||||||
|
@ -289,8 +301,9 @@ class ContainerSync(Daemon):
|
||||||
# succeed or in case it failed to do so the first time.
|
# succeed or in case it failed to do so the first time.
|
||||||
if unpack_from('>I', key)[0] % \
|
if unpack_from('>I', key)[0] % \
|
||||||
len(nodes) == ordinal:
|
len(nodes) == ordinal:
|
||||||
self.container_sync_row(row, sync_to, sync_key,
|
self.container_sync_row(
|
||||||
broker, info)
|
row, sync_to, user_key, broker, info, realm,
|
||||||
|
realm_key)
|
||||||
sync_point1 = row['ROWID']
|
sync_point1 = row['ROWID']
|
||||||
broker.set_x_container_sync_points(sync_point1, None)
|
broker.set_x_container_sync_points(sync_point1, None)
|
||||||
self.container_syncs += 1
|
self.container_syncs += 1
|
||||||
|
@ -301,27 +314,44 @@ class ContainerSync(Daemon):
|
||||||
self.logger.exception(_('ERROR Syncing %s'),
|
self.logger.exception(_('ERROR Syncing %s'),
|
||||||
broker if broker else path)
|
broker if broker else path)
|
||||||
|
|
||||||
def container_sync_row(self, row, sync_to, sync_key, broker, info):
|
def container_sync_row(self, row, sync_to, user_key, broker, info,
|
||||||
|
realm, realm_key):
|
||||||
"""
|
"""
|
||||||
Sends the update the row indicates to the sync_to container.
|
Sends the update the row indicates to the sync_to container.
|
||||||
|
|
||||||
:param row: The updated row in the local database triggering the sync
|
:param row: The updated row in the local database triggering the sync
|
||||||
update.
|
update.
|
||||||
:param sync_to: The URL to the remote container.
|
:param sync_to: The URL to the remote container.
|
||||||
:param sync_key: The X-Container-Sync-Key to use when sending requests
|
:param user_key: The X-Container-Sync-Key to use when sending requests
|
||||||
to the other container.
|
to the other container.
|
||||||
:param broker: The local container database broker.
|
:param broker: The local container database broker.
|
||||||
:param info: The get_info result from the local container database
|
:param info: The get_info result from the local container database
|
||||||
broker.
|
broker.
|
||||||
|
:param realm: The realm from self.realms_conf, if there is one.
|
||||||
|
If None, fallback to using the older allowed_sync_hosts
|
||||||
|
way of syncing.
|
||||||
|
:param realm_key: The realm key from self.realms_conf, if there
|
||||||
|
is one. If None, fallback to using the older
|
||||||
|
allowed_sync_hosts way of syncing.
|
||||||
:returns: True on success
|
:returns: True on success
|
||||||
"""
|
"""
|
||||||
try:
|
try:
|
||||||
start_time = time()
|
start_time = time()
|
||||||
if row['deleted']:
|
if row['deleted']:
|
||||||
try:
|
try:
|
||||||
delete_object(sync_to, name=row['name'],
|
headers = {'x-timestamp': row['created_at']}
|
||||||
headers={'x-timestamp': row['created_at'],
|
if realm and realm_key:
|
||||||
'x-container-sync-key': sync_key},
|
nonce = uuid.uuid4().hex
|
||||||
|
path = urlparse(sync_to).path + '/' + quote(
|
||||||
|
row['name'])
|
||||||
|
sig = self.realms_conf.get_sig(
|
||||||
|
'DELETE', path, headers['x-timestamp'], nonce,
|
||||||
|
realm_key, user_key)
|
||||||
|
headers['x-container-sync-auth'] = '%s %s %s' % (
|
||||||
|
realm, nonce, sig)
|
||||||
|
else:
|
||||||
|
headers['x-container-sync-key'] = user_key
|
||||||
|
delete_object(sync_to, name=row['name'], headers=headers,
|
||||||
proxy=self.proxy)
|
proxy=self.proxy)
|
||||||
except ClientException as err:
|
except ClientException as err:
|
||||||
if err.http_status != HTTP_NOT_FOUND:
|
if err.http_status != HTTP_NOT_FOUND:
|
||||||
|
@ -373,7 +403,16 @@ class ContainerSync(Daemon):
|
||||||
if 'etag' in headers:
|
if 'etag' in headers:
|
||||||
headers['etag'] = headers['etag'].strip('"')
|
headers['etag'] = headers['etag'].strip('"')
|
||||||
headers['x-timestamp'] = row['created_at']
|
headers['x-timestamp'] = row['created_at']
|
||||||
headers['x-container-sync-key'] = sync_key
|
if realm and realm_key:
|
||||||
|
nonce = uuid.uuid4().hex
|
||||||
|
path = urlparse(sync_to).path + '/' + quote(row['name'])
|
||||||
|
sig = self.realms_conf.get_sig(
|
||||||
|
'PUT', path, headers['x-timestamp'], nonce, realm_key,
|
||||||
|
user_key)
|
||||||
|
headers['x-container-sync-auth'] = '%s %s %s' % (
|
||||||
|
realm, nonce, sig)
|
||||||
|
else:
|
||||||
|
headers['x-container-sync-key'] = user_key
|
||||||
put_object(sync_to, name=row['name'], headers=headers,
|
put_object(sync_to, name=row['name'], headers=headers,
|
||||||
contents=FileLikeIter(body),
|
contents=FileLikeIter(body),
|
||||||
proxy=self.proxy)
|
proxy=self.proxy)
|
||||||
|
|
|
@ -0,0 +1,233 @@
|
||||||
|
# Copyright (c) 2013 OpenStack Foundation
|
||||||
|
#
|
||||||
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
# you may not use this file except in compliance with the License.
|
||||||
|
# You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||||
|
# implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import shutil
|
||||||
|
import tempfile
|
||||||
|
import unittest
|
||||||
|
import uuid
|
||||||
|
|
||||||
|
from swift.common import swob
|
||||||
|
from swift.common.middleware import container_sync
|
||||||
|
from swift.proxy.controllers.base import _get_cache_key
|
||||||
|
from swift.proxy.controllers.info import InfoController
|
||||||
|
|
||||||
|
|
||||||
|
class FakeApp(object):
|
||||||
|
|
||||||
|
def __call__(self, env, start_response):
|
||||||
|
if env.get('PATH_INFO') == '/info':
|
||||||
|
controller = InfoController(
|
||||||
|
app=None, version=None, expose_info=True,
|
||||||
|
disallowed_sections=[], admin_key=None)
|
||||||
|
handler = getattr(controller, env.get('REQUEST_METHOD'))
|
||||||
|
return handler(swob.Request(env))(env, start_response)
|
||||||
|
if env.get('swift.authorize_override'):
|
||||||
|
body = 'Response to Authorized Request'
|
||||||
|
else:
|
||||||
|
body = 'Pass-Through Response'
|
||||||
|
start_response('200 OK', [('Content-Length', str(len(body)))])
|
||||||
|
return body
|
||||||
|
|
||||||
|
|
||||||
|
class TestContainerSync(unittest.TestCase):
|
||||||
|
|
||||||
|
def setUp(self):
|
||||||
|
self.tempdir = tempfile.mkdtemp()
|
||||||
|
with open(
|
||||||
|
os.path.join(self.tempdir, 'container-sync-realms.conf'),
|
||||||
|
'w') as fp:
|
||||||
|
fp.write('''
|
||||||
|
[US]
|
||||||
|
key = 9ff3b71c849749dbaec4ccdd3cbab62b
|
||||||
|
key2 = 1a0a5a0cbd66448084089304442d6776
|
||||||
|
cluster_dfw1 = http://dfw1.host/v1/
|
||||||
|
''')
|
||||||
|
self.app = FakeApp()
|
||||||
|
self.conf = {'swift_dir': self.tempdir}
|
||||||
|
self.sync = container_sync.ContainerSync(self.app, self.conf)
|
||||||
|
|
||||||
|
def tearDown(self):
|
||||||
|
shutil.rmtree(self.tempdir, ignore_errors=1)
|
||||||
|
|
||||||
|
def test_pass_through(self):
|
||||||
|
req = swob.Request.blank('/v1/a/c')
|
||||||
|
resp = req.get_response(self.sync)
|
||||||
|
self.assertEqual(resp.status, '200 OK')
|
||||||
|
self.assertEqual(resp.body, 'Pass-Through Response')
|
||||||
|
|
||||||
|
def test_not_enough_args(self):
|
||||||
|
req = swob.Request.blank(
|
||||||
|
'/v1/a/c', headers={'x-container-sync-auth': 'a'})
|
||||||
|
resp = req.get_response(self.sync)
|
||||||
|
self.assertEqual(resp.status, '401 Unauthorized')
|
||||||
|
self.assertEqual(
|
||||||
|
resp.body,
|
||||||
|
'X-Container-Sync-Auth header not valid; contact cluster operator '
|
||||||
|
'for support.')
|
||||||
|
self.assertTrue(
|
||||||
|
'cs:not-3-args' in req.environ.get('swift.log_info'),
|
||||||
|
req.environ.get('swift.log_info'))
|
||||||
|
|
||||||
|
def test_realm_miss(self):
|
||||||
|
req = swob.Request.blank(
|
||||||
|
'/v1/a/c', headers={'x-container-sync-auth': 'invalid nonce sig'})
|
||||||
|
resp = req.get_response(self.sync)
|
||||||
|
self.assertEqual(resp.status, '401 Unauthorized')
|
||||||
|
self.assertEqual(
|
||||||
|
resp.body,
|
||||||
|
'X-Container-Sync-Auth header not valid; contact cluster operator '
|
||||||
|
'for support.')
|
||||||
|
self.assertTrue(
|
||||||
|
'cs:no-local-realm-key' in req.environ.get('swift.log_info'),
|
||||||
|
req.environ.get('swift.log_info'))
|
||||||
|
|
||||||
|
def test_user_key_miss(self):
|
||||||
|
req = swob.Request.blank(
|
||||||
|
'/v1/a/c', headers={'x-container-sync-auth': 'US nonce sig'})
|
||||||
|
resp = req.get_response(self.sync)
|
||||||
|
self.assertEqual(resp.status, '401 Unauthorized')
|
||||||
|
self.assertEqual(
|
||||||
|
resp.body,
|
||||||
|
'X-Container-Sync-Auth header not valid; contact cluster operator '
|
||||||
|
'for support.')
|
||||||
|
self.assertTrue(
|
||||||
|
'cs:no-local-user-key' in req.environ.get('swift.log_info'),
|
||||||
|
req.environ.get('swift.log_info'))
|
||||||
|
|
||||||
|
def test_invalid_sig(self):
|
||||||
|
req = swob.Request.blank(
|
||||||
|
'/v1/a/c', headers={'x-container-sync-auth': 'US nonce sig'})
|
||||||
|
req.environ[_get_cache_key('a', 'c')[1]] = {'sync_key': 'abc'}
|
||||||
|
resp = req.get_response(self.sync)
|
||||||
|
self.assertEqual(resp.status, '401 Unauthorized')
|
||||||
|
self.assertEqual(
|
||||||
|
resp.body,
|
||||||
|
'X-Container-Sync-Auth header not valid; contact cluster operator '
|
||||||
|
'for support.')
|
||||||
|
self.assertTrue(
|
||||||
|
'cs:invalid-sig' in req.environ.get('swift.log_info'),
|
||||||
|
req.environ.get('swift.log_info'))
|
||||||
|
|
||||||
|
def test_valid_sig(self):
|
||||||
|
sig = self.sync.realms_conf.get_sig(
|
||||||
|
'GET', '/v1/a/c', '0', 'nonce',
|
||||||
|
self.sync.realms_conf.key('US'), 'abc')
|
||||||
|
req = swob.Request.blank(
|
||||||
|
'/v1/a/c', headers={'x-container-sync-auth': 'US nonce ' + sig})
|
||||||
|
req.environ[_get_cache_key('a', 'c')[1]] = {'sync_key': 'abc'}
|
||||||
|
resp = req.get_response(self.sync)
|
||||||
|
self.assertEqual(resp.status, '200 OK')
|
||||||
|
self.assertEqual(resp.body, 'Response to Authorized Request')
|
||||||
|
self.assertTrue(
|
||||||
|
'cs:valid' in req.environ.get('swift.log_info'),
|
||||||
|
req.environ.get('swift.log_info'))
|
||||||
|
|
||||||
|
def test_valid_sig2(self):
|
||||||
|
sig = self.sync.realms_conf.get_sig(
|
||||||
|
'GET', '/v1/a/c', '0', 'nonce',
|
||||||
|
self.sync.realms_conf.key2('US'), 'abc')
|
||||||
|
req = swob.Request.blank(
|
||||||
|
'/v1/a/c', headers={'x-container-sync-auth': 'US nonce ' + sig})
|
||||||
|
req.environ[_get_cache_key('a', 'c')[1]] = {'sync_key': 'abc'}
|
||||||
|
resp = req.get_response(self.sync)
|
||||||
|
self.assertEqual(resp.status, '200 OK')
|
||||||
|
self.assertEqual(resp.body, 'Response to Authorized Request')
|
||||||
|
self.assertTrue(
|
||||||
|
'cs:valid' in req.environ.get('swift.log_info'),
|
||||||
|
req.environ.get('swift.log_info'))
|
||||||
|
|
||||||
|
def test_info(self):
|
||||||
|
req = swob.Request.blank('/info')
|
||||||
|
resp = req.get_response(self.sync)
|
||||||
|
self.assertEqual(resp.status, '200 OK')
|
||||||
|
result = json.loads(resp.body)
|
||||||
|
self.assertEqual(
|
||||||
|
result.get('container_sync'),
|
||||||
|
{'realms': {'US': {'clusters': {'DFW1': {}}}}})
|
||||||
|
|
||||||
|
def test_info_always_fresh(self):
|
||||||
|
req = swob.Request.blank('/info')
|
||||||
|
resp = req.get_response(self.sync)
|
||||||
|
self.assertEqual(resp.status, '200 OK')
|
||||||
|
result = json.loads(resp.body)
|
||||||
|
self.assertEqual(
|
||||||
|
result.get('container_sync'),
|
||||||
|
{'realms': {'US': {'clusters': {'DFW1': {}}}}})
|
||||||
|
with open(
|
||||||
|
os.path.join(self.tempdir, 'container-sync-realms.conf'),
|
||||||
|
'w') as fp:
|
||||||
|
fp.write('''
|
||||||
|
[US]
|
||||||
|
key = 9ff3b71c849749dbaec4ccdd3cbab62b
|
||||||
|
key2 = 1a0a5a0cbd66448084089304442d6776
|
||||||
|
cluster_dfw1 = http://dfw1.host/v1/
|
||||||
|
|
||||||
|
[UK]
|
||||||
|
key = 400b3b357a80413f9d956badff1d9dfe
|
||||||
|
cluster_lon3 = http://lon3.host/v1/
|
||||||
|
''')
|
||||||
|
self.sync.realms_conf.reload()
|
||||||
|
req = swob.Request.blank('/info')
|
||||||
|
resp = req.get_response(self.sync)
|
||||||
|
self.assertEqual(resp.status, '200 OK')
|
||||||
|
result = json.loads(resp.body)
|
||||||
|
self.assertEqual(
|
||||||
|
result.get('container_sync'),
|
||||||
|
{'realms': {
|
||||||
|
'US': {'clusters': {'DFW1': {}}},
|
||||||
|
'UK': {'clusters': {'LON3': {}}}}})
|
||||||
|
|
||||||
|
def test_allow_full_urls_setting(self):
|
||||||
|
req = swob.Request.blank(
|
||||||
|
'/v1/a/c',
|
||||||
|
environ={'REQUEST_METHOD': 'PUT'},
|
||||||
|
headers={'x-container-sync-to': 'http://host/v1/a/c'})
|
||||||
|
resp = req.get_response(self.sync)
|
||||||
|
self.assertEqual(resp.status, '200 OK')
|
||||||
|
self.conf = {'swift_dir': self.tempdir, 'allow_full_urls': 'false'}
|
||||||
|
self.sync = container_sync.ContainerSync(self.app, self.conf)
|
||||||
|
req = swob.Request.blank(
|
||||||
|
'/v1/a/c',
|
||||||
|
environ={'REQUEST_METHOD': 'PUT'},
|
||||||
|
headers={'x-container-sync-to': 'http://host/v1/a/c'})
|
||||||
|
resp = req.get_response(self.sync)
|
||||||
|
self.assertEqual(resp.status, '400 Bad Request')
|
||||||
|
self.assertEqual(
|
||||||
|
resp.body,
|
||||||
|
'Full URLs are not allowed for X-Container-Sync-To values. Only '
|
||||||
|
'realm values of the format //realm/cluster/account/container are '
|
||||||
|
'allowed.\n')
|
||||||
|
|
||||||
|
def test_filter(self):
|
||||||
|
app = FakeApp()
|
||||||
|
unique = uuid.uuid4().hex
|
||||||
|
sync = container_sync.filter_factory(
|
||||||
|
{'global': 'global_value', 'swift_dir': unique},
|
||||||
|
**{'local': 'local_value'})(app)
|
||||||
|
self.assertEqual(sync.app, app)
|
||||||
|
self.assertEqual(sync.conf, {
|
||||||
|
'global': 'global_value', 'swift_dir': unique,
|
||||||
|
'local': 'local_value'})
|
||||||
|
req = swob.Request.blank('/info')
|
||||||
|
resp = req.get_response(sync)
|
||||||
|
self.assertEqual(resp.status, '200 OK')
|
||||||
|
result = json.loads(resp.body)
|
||||||
|
self.assertEqual(result.get('container_sync'), {'realms': {}})
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
unittest.main()
|
|
@ -0,0 +1,187 @@
|
||||||
|
# Copyright (c) 2013 OpenStack Foundation
|
||||||
|
#
|
||||||
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
# you may not use this file except in compliance with the License.
|
||||||
|
# You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||||
|
# implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
import os
|
||||||
|
import unittest
|
||||||
|
import uuid
|
||||||
|
|
||||||
|
from swift.common.container_sync_realms import ContainerSyncRealms
|
||||||
|
from test.unit import FakeLogger, temptree
|
||||||
|
|
||||||
|
|
||||||
|
class TestUtils(unittest.TestCase):
|
||||||
|
|
||||||
|
def test_no_file_there(self):
|
||||||
|
unique = uuid.uuid4().hex
|
||||||
|
logger = FakeLogger()
|
||||||
|
csr = ContainerSyncRealms(unique, logger)
|
||||||
|
self.assertEqual(
|
||||||
|
logger.lines_dict,
|
||||||
|
{'debug': [
|
||||||
|
"Could not load '%s': [Errno 2] No such file or directory: "
|
||||||
|
"'%s'" % (unique, unique)]})
|
||||||
|
self.assertEqual(csr.mtime_check_interval, 300)
|
||||||
|
self.assertEqual(csr.realms(), [])
|
||||||
|
|
||||||
|
def test_os_error(self):
|
||||||
|
fname = 'container-sync-realms.conf'
|
||||||
|
fcontents = ''
|
||||||
|
with temptree([fname], [fcontents]) as tempdir:
|
||||||
|
logger = FakeLogger()
|
||||||
|
fpath = os.path.join(tempdir, fname)
|
||||||
|
os.chmod(tempdir, 0)
|
||||||
|
csr = ContainerSyncRealms(fpath, logger)
|
||||||
|
try:
|
||||||
|
self.assertEqual(
|
||||||
|
logger.lines_dict,
|
||||||
|
{'error': [
|
||||||
|
"Could not load '%s': [Errno 13] Permission denied: "
|
||||||
|
"'%s'" % (fpath, fpath)]})
|
||||||
|
self.assertEqual(csr.mtime_check_interval, 300)
|
||||||
|
self.assertEqual(csr.realms(), [])
|
||||||
|
finally:
|
||||||
|
os.chmod(tempdir, 0700)
|
||||||
|
|
||||||
|
def test_empty(self):
|
||||||
|
fname = 'container-sync-realms.conf'
|
||||||
|
fcontents = ''
|
||||||
|
with temptree([fname], [fcontents]) as tempdir:
|
||||||
|
logger = FakeLogger()
|
||||||
|
fpath = os.path.join(tempdir, fname)
|
||||||
|
csr = ContainerSyncRealms(fpath, logger)
|
||||||
|
self.assertEqual(logger.lines_dict, {})
|
||||||
|
self.assertEqual(csr.mtime_check_interval, 300)
|
||||||
|
self.assertEqual(csr.realms(), [])
|
||||||
|
|
||||||
|
def test_error_parsing(self):
|
||||||
|
fname = 'container-sync-realms.conf'
|
||||||
|
fcontents = 'invalid'
|
||||||
|
with temptree([fname], [fcontents]) as tempdir:
|
||||||
|
logger = FakeLogger()
|
||||||
|
fpath = os.path.join(tempdir, fname)
|
||||||
|
csr = ContainerSyncRealms(fpath, logger)
|
||||||
|
self.assertEqual(
|
||||||
|
logger.lines_dict,
|
||||||
|
{'error': [
|
||||||
|
"Could not load '%s': File contains no section headers.\n"
|
||||||
|
"file: %s, line: 1\n"
|
||||||
|
"'invalid'" % (fpath, fpath)]})
|
||||||
|
self.assertEqual(csr.mtime_check_interval, 300)
|
||||||
|
self.assertEqual(csr.realms(), [])
|
||||||
|
|
||||||
|
def test_one_realm(self):
|
||||||
|
fname = 'container-sync-realms.conf'
|
||||||
|
fcontents = '''
|
||||||
|
[US]
|
||||||
|
key = 9ff3b71c849749dbaec4ccdd3cbab62b
|
||||||
|
cluster_dfw1 = http://dfw1.host/v1/
|
||||||
|
'''
|
||||||
|
with temptree([fname], [fcontents]) as tempdir:
|
||||||
|
logger = FakeLogger()
|
||||||
|
fpath = os.path.join(tempdir, fname)
|
||||||
|
csr = ContainerSyncRealms(fpath, logger)
|
||||||
|
self.assertEqual(logger.lines_dict, {})
|
||||||
|
self.assertEqual(csr.mtime_check_interval, 300)
|
||||||
|
self.assertEqual(csr.realms(), ['US'])
|
||||||
|
self.assertEqual(csr.key('US'), '9ff3b71c849749dbaec4ccdd3cbab62b')
|
||||||
|
self.assertEqual(csr.key2('US'), None)
|
||||||
|
self.assertEqual(csr.clusters('US'), ['DFW1'])
|
||||||
|
self.assertEqual(
|
||||||
|
csr.endpoint('US', 'DFW1'), 'http://dfw1.host/v1/')
|
||||||
|
|
||||||
|
def test_two_realms_and_change_a_default(self):
|
||||||
|
fname = 'container-sync-realms.conf'
|
||||||
|
fcontents = '''
|
||||||
|
[DEFAULT]
|
||||||
|
mtime_check_interval = 60
|
||||||
|
|
||||||
|
[US]
|
||||||
|
key = 9ff3b71c849749dbaec4ccdd3cbab62b
|
||||||
|
cluster_dfw1 = http://dfw1.host/v1/
|
||||||
|
|
||||||
|
[UK]
|
||||||
|
key = e9569809dc8b4951accc1487aa788012
|
||||||
|
key2 = f6351bd1cc36413baa43f7ba1b45e51d
|
||||||
|
cluster_lon3 = http://lon3.host/v1/
|
||||||
|
'''
|
||||||
|
with temptree([fname], [fcontents]) as tempdir:
|
||||||
|
logger = FakeLogger()
|
||||||
|
fpath = os.path.join(tempdir, fname)
|
||||||
|
csr = ContainerSyncRealms(fpath, logger)
|
||||||
|
self.assertEqual(logger.lines_dict, {})
|
||||||
|
self.assertEqual(csr.mtime_check_interval, 60)
|
||||||
|
self.assertEqual(sorted(csr.realms()), ['UK', 'US'])
|
||||||
|
self.assertEqual(csr.key('US'), '9ff3b71c849749dbaec4ccdd3cbab62b')
|
||||||
|
self.assertEqual(csr.key2('US'), None)
|
||||||
|
self.assertEqual(csr.clusters('US'), ['DFW1'])
|
||||||
|
self.assertEqual(
|
||||||
|
csr.endpoint('US', 'DFW1'), 'http://dfw1.host/v1/')
|
||||||
|
self.assertEqual(csr.key('UK'), 'e9569809dc8b4951accc1487aa788012')
|
||||||
|
self.assertEqual(
|
||||||
|
csr.key2('UK'), 'f6351bd1cc36413baa43f7ba1b45e51d')
|
||||||
|
self.assertEqual(csr.clusters('UK'), ['LON3'])
|
||||||
|
self.assertEqual(
|
||||||
|
csr.endpoint('UK', 'LON3'), 'http://lon3.host/v1/')
|
||||||
|
|
||||||
|
def test_empty_realm(self):
|
||||||
|
fname = 'container-sync-realms.conf'
|
||||||
|
fcontents = '''
|
||||||
|
[US]
|
||||||
|
'''
|
||||||
|
with temptree([fname], [fcontents]) as tempdir:
|
||||||
|
logger = FakeLogger()
|
||||||
|
fpath = os.path.join(tempdir, fname)
|
||||||
|
csr = ContainerSyncRealms(fpath, logger)
|
||||||
|
self.assertEqual(logger.lines_dict, {})
|
||||||
|
self.assertEqual(csr.mtime_check_interval, 300)
|
||||||
|
self.assertEqual(csr.realms(), ['US'])
|
||||||
|
self.assertEqual(csr.key('US'), None)
|
||||||
|
self.assertEqual(csr.key2('US'), None)
|
||||||
|
self.assertEqual(csr.clusters('US'), [])
|
||||||
|
self.assertEqual(csr.endpoint('US', 'JUST_TESTING'), None)
|
||||||
|
|
||||||
|
def test_bad_mtime_check_interval(self):
|
||||||
|
fname = 'container-sync-realms.conf'
|
||||||
|
fcontents = '''
|
||||||
|
[DEFAULT]
|
||||||
|
mtime_check_interval = invalid
|
||||||
|
'''
|
||||||
|
with temptree([fname], [fcontents]) as tempdir:
|
||||||
|
logger = FakeLogger()
|
||||||
|
fpath = os.path.join(tempdir, fname)
|
||||||
|
csr = ContainerSyncRealms(fpath, logger)
|
||||||
|
self.assertEqual(
|
||||||
|
logger.lines_dict,
|
||||||
|
{'error': [
|
||||||
|
"Error in '%s' with mtime_check_interval: invalid literal "
|
||||||
|
"for int() with base 10: 'invalid'" % fpath]})
|
||||||
|
self.assertEqual(csr.mtime_check_interval, 300)
|
||||||
|
|
||||||
|
def test_get_sig(self):
|
||||||
|
fname = 'container-sync-realms.conf'
|
||||||
|
fcontents = ''
|
||||||
|
with temptree([fname], [fcontents]) as tempdir:
|
||||||
|
logger = FakeLogger()
|
||||||
|
fpath = os.path.join(tempdir, fname)
|
||||||
|
csr = ContainerSyncRealms(fpath, logger)
|
||||||
|
self.assertEqual(
|
||||||
|
csr.get_sig(
|
||||||
|
'GET', '/some/path', '1387212345.67890', 'my_nonce',
|
||||||
|
'realm_key', 'user_key'),
|
||||||
|
'5a6eb486eb7b44ae1b1f014187a94529c3f9c8f9')
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
unittest.main()
|
|
@ -52,6 +52,7 @@ from swift.common.exceptions import (Timeout, MessageTimeout,
|
||||||
ConnectionTimeout, LockTimeout,
|
ConnectionTimeout, LockTimeout,
|
||||||
ReplicationLockTimeout)
|
ReplicationLockTimeout)
|
||||||
from swift.common import utils
|
from swift.common import utils
|
||||||
|
from swift.common.container_sync_realms import ContainerSyncRealms
|
||||||
from swift.common.swob import Response
|
from swift.common.swob import Response
|
||||||
from test.unit import FakeLogger
|
from test.unit import FakeLogger
|
||||||
|
|
||||||
|
@ -1184,25 +1185,108 @@ log_name = %(yarr)s'''
|
||||||
'1024Yi')
|
'1024Yi')
|
||||||
|
|
||||||
def test_validate_sync_to(self):
|
def test_validate_sync_to(self):
|
||||||
for goodurl in ('http://1.1.1.1/v1/a/c/o',
|
fname = 'container-sync-realms.conf'
|
||||||
'http://1.1.1.1:8080/a/c/o',
|
fcontents = '''
|
||||||
'http://2.2.2.2/a/c/o',
|
[US]
|
||||||
'https://1.1.1.1/v1/a/c/o',
|
key = 9ff3b71c849749dbaec4ccdd3cbab62b
|
||||||
''):
|
cluster_dfw1 = http://dfw1.host/v1/
|
||||||
self.assertEquals(utils.validate_sync_to(goodurl,
|
'''
|
||||||
['1.1.1.1', '2.2.2.2']),
|
with temptree([fname], [fcontents]) as tempdir:
|
||||||
None)
|
logger = FakeLogger()
|
||||||
for badurl in ('http://1.1.1.1',
|
fpath = os.path.join(tempdir, fname)
|
||||||
'httpq://1.1.1.1/v1/a/c/o',
|
csr = ContainerSyncRealms(fpath, logger)
|
||||||
'http://1.1.1.1/v1/a/c/o?query',
|
for realms_conf in (None, csr):
|
||||||
'http://1.1.1.1/v1/a/c/o#frag',
|
for goodurl, result in (
|
||||||
'http://1.1.1.1/v1/a/c/o?query#frag',
|
('http://1.1.1.1/v1/a/c',
|
||||||
'http://1.1.1.1/v1/a/c/o?query=param',
|
(None, 'http://1.1.1.1/v1/a/c', None, None)),
|
||||||
'http://1.1.1.1/v1/a/c/o?query=param#frag',
|
('http://1.1.1.1:8080/a/c',
|
||||||
'http://1.1.1.2/v1/a/c/o'):
|
(None, 'http://1.1.1.1:8080/a/c', None, None)),
|
||||||
self.assertNotEquals(
|
('http://2.2.2.2/a/c',
|
||||||
utils.validate_sync_to(badurl, ['1.1.1.1', '2.2.2.2']),
|
(None, 'http://2.2.2.2/a/c', None, None)),
|
||||||
None)
|
('https://1.1.1.1/v1/a/c',
|
||||||
|
(None, 'https://1.1.1.1/v1/a/c', None, None)),
|
||||||
|
('//US/DFW1/a/c',
|
||||||
|
(None, 'http://dfw1.host/v1/a/c', 'US',
|
||||||
|
'9ff3b71c849749dbaec4ccdd3cbab62b')),
|
||||||
|
('//us/DFW1/a/c',
|
||||||
|
(None, 'http://dfw1.host/v1/a/c', 'US',
|
||||||
|
'9ff3b71c849749dbaec4ccdd3cbab62b')),
|
||||||
|
('//us/dfw1/a/c',
|
||||||
|
(None, 'http://dfw1.host/v1/a/c', 'US',
|
||||||
|
'9ff3b71c849749dbaec4ccdd3cbab62b')),
|
||||||
|
('//',
|
||||||
|
(None, None, None, None)),
|
||||||
|
('',
|
||||||
|
(None, None, None, None))):
|
||||||
|
if goodurl.startswith('//') and not realms_conf:
|
||||||
|
self.assertEquals(
|
||||||
|
utils.validate_sync_to(
|
||||||
|
goodurl, ['1.1.1.1', '2.2.2.2'], realms_conf),
|
||||||
|
(None, None, None, None))
|
||||||
|
else:
|
||||||
|
self.assertEquals(
|
||||||
|
utils.validate_sync_to(
|
||||||
|
goodurl, ['1.1.1.1', '2.2.2.2'], realms_conf),
|
||||||
|
result)
|
||||||
|
for badurl, result in (
|
||||||
|
('http://1.1.1.1',
|
||||||
|
('Path required in X-Container-Sync-To', None, None,
|
||||||
|
None)),
|
||||||
|
('httpq://1.1.1.1/v1/a/c',
|
||||||
|
('Invalid scheme \'httpq\' in X-Container-Sync-To, '
|
||||||
|
'must be "//", "http", or "https".', None, None,
|
||||||
|
None)),
|
||||||
|
('http://1.1.1.1/v1/a/c?query',
|
||||||
|
('Params, queries, and fragments not allowed in '
|
||||||
|
'X-Container-Sync-To', None, None, None)),
|
||||||
|
('http://1.1.1.1/v1/a/c#frag',
|
||||||
|
('Params, queries, and fragments not allowed in '
|
||||||
|
'X-Container-Sync-To', None, None, None)),
|
||||||
|
('http://1.1.1.1/v1/a/c?query#frag',
|
||||||
|
('Params, queries, and fragments not allowed in '
|
||||||
|
'X-Container-Sync-To', None, None, None)),
|
||||||
|
('http://1.1.1.1/v1/a/c?query=param',
|
||||||
|
('Params, queries, and fragments not allowed in '
|
||||||
|
'X-Container-Sync-To', None, None, None)),
|
||||||
|
('http://1.1.1.1/v1/a/c?query=param#frag',
|
||||||
|
('Params, queries, and fragments not allowed in '
|
||||||
|
'X-Container-Sync-To', None, None, None)),
|
||||||
|
('http://1.1.1.2/v1/a/c',
|
||||||
|
("Invalid host '1.1.1.2' in X-Container-Sync-To",
|
||||||
|
None, None, None)),
|
||||||
|
('//us/invalid/a/c',
|
||||||
|
("No cluster endpoint for 'us' 'invalid'", None,
|
||||||
|
None, None)),
|
||||||
|
('//invalid/dfw1/a/c',
|
||||||
|
("No realm key for 'invalid'", None, None, None)),
|
||||||
|
('//us/invalid1/a/',
|
||||||
|
("Invalid X-Container-Sync-To format "
|
||||||
|
"'//us/invalid1/a/'", None, None, None)),
|
||||||
|
('//us/invalid1/a',
|
||||||
|
("Invalid X-Container-Sync-To format "
|
||||||
|
"'//us/invalid1/a'", None, None, None)),
|
||||||
|
('//us/invalid1/',
|
||||||
|
("Invalid X-Container-Sync-To format "
|
||||||
|
"'//us/invalid1/'", None, None, None)),
|
||||||
|
('//us/invalid1',
|
||||||
|
("Invalid X-Container-Sync-To format "
|
||||||
|
"'//us/invalid1'", None, None, None)),
|
||||||
|
('//us/',
|
||||||
|
("Invalid X-Container-Sync-To format "
|
||||||
|
"'//us/'", None, None, None)),
|
||||||
|
('//us',
|
||||||
|
("Invalid X-Container-Sync-To format "
|
||||||
|
"'//us'", None, None, None))):
|
||||||
|
if badurl.startswith('//') and not realms_conf:
|
||||||
|
self.assertEquals(
|
||||||
|
utils.validate_sync_to(
|
||||||
|
badurl, ['1.1.1.1', '2.2.2.2'], realms_conf),
|
||||||
|
(None, None, None, None))
|
||||||
|
else:
|
||||||
|
self.assertEquals(
|
||||||
|
utils.validate_sync_to(
|
||||||
|
badurl, ['1.1.1.1', '2.2.2.2'], realms_conf),
|
||||||
|
result)
|
||||||
|
|
||||||
def test_TRUE_VALUES(self):
|
def test_TRUE_VALUES(self):
|
||||||
for v in utils.TRUE_VALUES:
|
for v in utils.TRUE_VALUES:
|
||||||
|
|
|
@ -626,15 +626,33 @@ class TestContainerSync(unittest.TestCase):
|
||||||
sync.delete_object = orig_delete_object
|
sync.delete_object = orig_delete_object
|
||||||
|
|
||||||
def test_container_sync_row_delete(self):
|
def test_container_sync_row_delete(self):
|
||||||
|
self._test_container_sync_row_delete(None, None)
|
||||||
|
|
||||||
|
def test_container_sync_row_delete_using_realms(self):
|
||||||
|
self._test_container_sync_row_delete('US', 'realm_key')
|
||||||
|
|
||||||
|
def _test_container_sync_row_delete(self, realm, realm_key):
|
||||||
|
orig_uuid = sync.uuid
|
||||||
orig_delete_object = sync.delete_object
|
orig_delete_object = sync.delete_object
|
||||||
try:
|
try:
|
||||||
|
class FakeUUID(object):
|
||||||
|
class uuid4(object):
|
||||||
|
hex = 'abcdef'
|
||||||
|
|
||||||
|
sync.uuid = FakeUUID
|
||||||
|
|
||||||
def fake_delete_object(path, name=None, headers=None, proxy=None):
|
def fake_delete_object(path, name=None, headers=None, proxy=None):
|
||||||
self.assertEquals(path, 'http://sync/to/path')
|
self.assertEquals(path, 'http://sync/to/path')
|
||||||
self.assertEquals(name, 'object')
|
self.assertEquals(name, 'object')
|
||||||
self.assertEquals(
|
if realm:
|
||||||
headers,
|
self.assertEquals(headers, {
|
||||||
{'x-container-sync-key': 'key', 'x-timestamp': '1.2'})
|
'x-container-sync-auth':
|
||||||
|
'US abcdef 90e95aabb45a6cdc0892a3db5535e7f918428c90',
|
||||||
|
'x-timestamp': '1.2'})
|
||||||
|
else:
|
||||||
|
self.assertEquals(
|
||||||
|
headers,
|
||||||
|
{'x-container-sync-key': 'key', 'x-timestamp': '1.2'})
|
||||||
self.assertEquals(proxy, 'http://proxy')
|
self.assertEquals(proxy, 'http://proxy')
|
||||||
|
|
||||||
sync.delete_object = fake_delete_object
|
sync.delete_object = fake_delete_object
|
||||||
|
@ -646,7 +664,8 @@ class TestContainerSync(unittest.TestCase):
|
||||||
{'deleted': True,
|
{'deleted': True,
|
||||||
'name': 'object',
|
'name': 'object',
|
||||||
'created_at': '1.2'}, 'http://sync/to/path',
|
'created_at': '1.2'}, 'http://sync/to/path',
|
||||||
'key', FakeContainerBroker('broker'), 'info'))
|
'key', FakeContainerBroker('broker'), 'info', realm,
|
||||||
|
realm_key))
|
||||||
self.assertEquals(cs.container_deletes, 1)
|
self.assertEquals(cs.container_deletes, 1)
|
||||||
|
|
||||||
exc = []
|
exc = []
|
||||||
|
@ -661,7 +680,8 @@ class TestContainerSync(unittest.TestCase):
|
||||||
{'deleted': True,
|
{'deleted': True,
|
||||||
'name': 'object',
|
'name': 'object',
|
||||||
'created_at': '1.2'}, 'http://sync/to/path',
|
'created_at': '1.2'}, 'http://sync/to/path',
|
||||||
'key', FakeContainerBroker('broker'), 'info'))
|
'key', FakeContainerBroker('broker'), 'info', realm,
|
||||||
|
realm_key))
|
||||||
self.assertEquals(cs.container_deletes, 1)
|
self.assertEquals(cs.container_deletes, 1)
|
||||||
self.assertEquals(len(exc), 1)
|
self.assertEquals(len(exc), 1)
|
||||||
self.assertEquals(str(exc[-1]), 'test exception')
|
self.assertEquals(str(exc[-1]), 'test exception')
|
||||||
|
@ -676,7 +696,8 @@ class TestContainerSync(unittest.TestCase):
|
||||||
{'deleted': True,
|
{'deleted': True,
|
||||||
'name': 'object',
|
'name': 'object',
|
||||||
'created_at': '1.2'}, 'http://sync/to/path',
|
'created_at': '1.2'}, 'http://sync/to/path',
|
||||||
'key', FakeContainerBroker('broker'), 'info'))
|
'key', FakeContainerBroker('broker'), 'info', realm,
|
||||||
|
realm_key))
|
||||||
self.assertEquals(cs.container_deletes, 1)
|
self.assertEquals(cs.container_deletes, 1)
|
||||||
self.assertEquals(len(exc), 2)
|
self.assertEquals(len(exc), 2)
|
||||||
self.assertEquals(str(exc[-1]), 'test client exception')
|
self.assertEquals(str(exc[-1]), 'test client exception')
|
||||||
|
@ -692,29 +713,51 @@ class TestContainerSync(unittest.TestCase):
|
||||||
{'deleted': True,
|
{'deleted': True,
|
||||||
'name': 'object',
|
'name': 'object',
|
||||||
'created_at': '1.2'}, 'http://sync/to/path',
|
'created_at': '1.2'}, 'http://sync/to/path',
|
||||||
'key', FakeContainerBroker('broker'), 'info'))
|
'key', FakeContainerBroker('broker'), 'info', realm,
|
||||||
|
realm_key))
|
||||||
self.assertEquals(cs.container_deletes, 2)
|
self.assertEquals(cs.container_deletes, 2)
|
||||||
self.assertEquals(len(exc), 3)
|
self.assertEquals(len(exc), 3)
|
||||||
self.assertEquals(str(exc[-1]), 'test client exception: 404')
|
self.assertEquals(str(exc[-1]), 'test client exception: 404')
|
||||||
finally:
|
finally:
|
||||||
|
sync.uuid = orig_uuid
|
||||||
sync.delete_object = orig_delete_object
|
sync.delete_object = orig_delete_object
|
||||||
|
|
||||||
def test_container_sync_row_put(self):
|
def test_container_sync_row_put(self):
|
||||||
|
self._test_container_sync_row_put(None, None)
|
||||||
|
|
||||||
|
def test_container_sync_row_put_using_realms(self):
|
||||||
|
self._test_container_sync_row_put('US', 'realm_key')
|
||||||
|
|
||||||
|
def _test_container_sync_row_put(self, realm, realm_key):
|
||||||
|
orig_uuid = sync.uuid
|
||||||
orig_shuffle = sync.shuffle
|
orig_shuffle = sync.shuffle
|
||||||
orig_put_object = sync.put_object
|
orig_put_object = sync.put_object
|
||||||
orig_direct_get_object = sync.direct_get_object
|
orig_direct_get_object = sync.direct_get_object
|
||||||
try:
|
try:
|
||||||
|
class FakeUUID(object):
|
||||||
|
class uuid4(object):
|
||||||
|
hex = 'abcdef'
|
||||||
|
|
||||||
|
sync.uuid = FakeUUID
|
||||||
sync.shuffle = lambda x: x
|
sync.shuffle = lambda x: x
|
||||||
|
|
||||||
def fake_put_object(sync_to, name=None, headers=None,
|
def fake_put_object(sync_to, name=None, headers=None,
|
||||||
contents=None, proxy=None):
|
contents=None, proxy=None):
|
||||||
self.assertEquals(sync_to, 'http://sync/to/path')
|
self.assertEquals(sync_to, 'http://sync/to/path')
|
||||||
self.assertEquals(name, 'object')
|
self.assertEquals(name, 'object')
|
||||||
self.assertEquals(headers, {
|
if realm:
|
||||||
'x-container-sync-key': 'key',
|
self.assertEqual(headers, {
|
||||||
'x-timestamp': '1.2',
|
'x-container-sync-auth':
|
||||||
'other-header': 'other header value',
|
'US abcdef ef62c64bb88a33fa00722daa23d5d43253164962',
|
||||||
'etag': 'etagvalue'})
|
'x-timestamp': '1.2',
|
||||||
|
'etag': 'etagvalue',
|
||||||
|
'other-header': 'other header value'})
|
||||||
|
else:
|
||||||
|
self.assertEquals(headers, {
|
||||||
|
'x-container-sync-key': 'key',
|
||||||
|
'x-timestamp': '1.2',
|
||||||
|
'other-header': 'other header value',
|
||||||
|
'etag': 'etagvalue'})
|
||||||
self.assertEquals(contents.read(), 'contents')
|
self.assertEquals(contents.read(), 'contents')
|
||||||
self.assertEquals(proxy, 'http://proxy')
|
self.assertEquals(proxy, 'http://proxy')
|
||||||
|
|
||||||
|
@ -738,7 +781,7 @@ class TestContainerSync(unittest.TestCase):
|
||||||
'created_at': '1.2'}, 'http://sync/to/path',
|
'created_at': '1.2'}, 'http://sync/to/path',
|
||||||
'key', FakeContainerBroker('broker'), {
|
'key', FakeContainerBroker('broker'), {
|
||||||
'account': 'a',
|
'account': 'a',
|
||||||
'container': 'c'}))
|
'container': 'c'}, realm, realm_key))
|
||||||
self.assertEquals(cs.container_puts, 1)
|
self.assertEquals(cs.container_puts, 1)
|
||||||
|
|
||||||
def fake_direct_get_object(node, part, account, container, obj,
|
def fake_direct_get_object(node, part, account, container, obj,
|
||||||
|
@ -760,7 +803,7 @@ class TestContainerSync(unittest.TestCase):
|
||||||
'created_at': '1.2'}, 'http://sync/to/path',
|
'created_at': '1.2'}, 'http://sync/to/path',
|
||||||
'key', FakeContainerBroker('broker'), {
|
'key', FakeContainerBroker('broker'), {
|
||||||
'account': 'a',
|
'account': 'a',
|
||||||
'container': 'c'}))
|
'container': 'c'}, realm, realm_key))
|
||||||
self.assertEquals(cs.container_puts, 2)
|
self.assertEquals(cs.container_puts, 2)
|
||||||
|
|
||||||
exc = []
|
exc = []
|
||||||
|
@ -778,7 +821,7 @@ class TestContainerSync(unittest.TestCase):
|
||||||
'created_at': '1.2'}, 'http://sync/to/path',
|
'created_at': '1.2'}, 'http://sync/to/path',
|
||||||
'key', FakeContainerBroker('broker'), {
|
'key', FakeContainerBroker('broker'), {
|
||||||
'account': 'a',
|
'account': 'a',
|
||||||
'container': 'c'}))
|
'container': 'c'}, realm, realm_key))
|
||||||
self.assertEquals(cs.container_puts, 2)
|
self.assertEquals(cs.container_puts, 2)
|
||||||
self.assertEquals(len(exc), 3)
|
self.assertEquals(len(exc), 3)
|
||||||
self.assertEquals(str(exc[-1]), 'test exception')
|
self.assertEquals(str(exc[-1]), 'test exception')
|
||||||
|
@ -798,7 +841,7 @@ class TestContainerSync(unittest.TestCase):
|
||||||
'created_at': '1.2'}, 'http://sync/to/path',
|
'created_at': '1.2'}, 'http://sync/to/path',
|
||||||
'key', FakeContainerBroker('broker'), {
|
'key', FakeContainerBroker('broker'), {
|
||||||
'account': 'a',
|
'account': 'a',
|
||||||
'container': 'c'}))
|
'container': 'c'}, realm, realm_key))
|
||||||
self.assertEquals(cs.container_puts, 2)
|
self.assertEquals(cs.container_puts, 2)
|
||||||
self.assertEquals(len(exc), 3)
|
self.assertEquals(len(exc), 3)
|
||||||
self.assertEquals(str(exc[-1]), 'test client exception')
|
self.assertEquals(str(exc[-1]), 'test client exception')
|
||||||
|
@ -823,7 +866,7 @@ class TestContainerSync(unittest.TestCase):
|
||||||
'created_at': '1.2'}, 'http://sync/to/path',
|
'created_at': '1.2'}, 'http://sync/to/path',
|
||||||
'key', FakeContainerBroker('broker'), {
|
'key', FakeContainerBroker('broker'), {
|
||||||
'account': 'a',
|
'account': 'a',
|
||||||
'container': 'c'}))
|
'container': 'c'}, realm, realm_key))
|
||||||
self.assertEquals(cs.container_puts, 2)
|
self.assertEquals(cs.container_puts, 2)
|
||||||
self.assert_(re.match('Unauth ',
|
self.assert_(re.match('Unauth ',
|
||||||
cs.logger.log_dict['info'][0][0][0]))
|
cs.logger.log_dict['info'][0][0][0]))
|
||||||
|
@ -841,7 +884,7 @@ class TestContainerSync(unittest.TestCase):
|
||||||
'created_at': '1.2'}, 'http://sync/to/path',
|
'created_at': '1.2'}, 'http://sync/to/path',
|
||||||
'key', FakeContainerBroker('broker'), {
|
'key', FakeContainerBroker('broker'), {
|
||||||
'account': 'a',
|
'account': 'a',
|
||||||
'container': 'c'}))
|
'container': 'c'}, realm, realm_key))
|
||||||
self.assertEquals(cs.container_puts, 2)
|
self.assertEquals(cs.container_puts, 2)
|
||||||
self.assert_(re.match('Not found ',
|
self.assert_(re.match('Not found ',
|
||||||
cs.logger.log_dict['info'][0][0][0]))
|
cs.logger.log_dict['info'][0][0][0]))
|
||||||
|
@ -858,12 +901,13 @@ class TestContainerSync(unittest.TestCase):
|
||||||
'created_at': '1.2'}, 'http://sync/to/path',
|
'created_at': '1.2'}, 'http://sync/to/path',
|
||||||
'key', FakeContainerBroker('broker'), {
|
'key', FakeContainerBroker('broker'), {
|
||||||
'account': 'a',
|
'account': 'a',
|
||||||
'container': 'c'}))
|
'container': 'c'}, realm, realm_key))
|
||||||
self.assertEquals(cs.container_puts, 2)
|
self.assertEquals(cs.container_puts, 2)
|
||||||
self.assertTrue(
|
self.assertTrue(
|
||||||
cs.logger.log_dict['exception'][0][0][0].startswith(
|
cs.logger.log_dict['exception'][0][0][0].startswith(
|
||||||
'ERROR Syncing '))
|
'ERROR Syncing '))
|
||||||
finally:
|
finally:
|
||||||
|
sync.uuid = orig_uuid
|
||||||
sync.shuffle = orig_shuffle
|
sync.shuffle = orig_shuffle
|
||||||
sync.put_object = orig_put_object
|
sync.put_object = orig_put_object
|
||||||
sync.direct_get_object = orig_direct_get_object
|
sync.direct_get_object = orig_direct_get_object
|
||||||
|
|
Loading…
Reference in New Issue