openstack/python-swiftclient
Code Issues Proposed changes

ISO 8601 timestamps for tempurl

Browse Source 
Client-side implementation for ISO 8601 timestamp
support of tempurl middleware. Please see

https://review.openstack.org/#/c/422679/

Change-Id: I76da28b48948475ec1bae5258e0b39a316553fb7
This commit is contained in:
Christopher Bartz 2017-01-20 18:04:31 +01:00 committed by Thiago da Silva
parent 2710ff255b
commit 8e08931b9f
6 changed files with 310 additions and 62 deletions
doc
manpages
swift.1
source
cli.rst
swiftclient
shell.pyutils.py
tests/unit
test_shell.pytest_utils.py

31
doc/manpages/swift.1

@ -130,14 +130,33 @@ programs, such as jq.
capabilities \-\-json
.RE
\fBtempurl\fR [\fIcommand-option\fR] \fImethod\fR \fIseconds\fR \fIpath\fR \fIkey\fR
\fBtempurl\fR [\fIcommand-option\fR] \fImethod\fR \fItime\fR \fIpath\fR \fIkey\fR
.RS 4
Generates a temporary URL allowing unauthenticated access to the Swift object
at the given path, using the given HTTP method, for the given number of
seconds, using the given TempURL key. With the optional \-\-prefix\-based option a
prefix-based URL is generated. If optional \-\-absolute argument is
provided, seconds is instead interpreted as a Unix timestamp at which the URL
should expire. \fBExample\fR: tempurl GET $(date \-d "Jan 1 2016" +%s)
at the given path, using the given HTTP method, for the given time,
using the given TempURL key.
The time can be specified either as an integer
denoting the amount of seconds the temporary URL is valid, or as an ISO 8601
timestamp in one of following formats: Complete date: YYYY\-MM\-DD (eg 1997\-07\-16),
complete date plus hours, minutes and seconds: YYYY\-MM\-DDThh:mm:ss
(eg 1997\-07\-16T19:20:30) or complete date plus hours, minutes and seconds with
UTC designator: YYYY\-MM\-DDThh:mm:ssZ (eg 1997\-07\-16T19:20:30Z). Be aware that
if you do not use the latter format, the timestamp is generated using your locale
timezone. If the first format is used, the time part used will equal to 00:00:00.
With the \-\-prefix\-based option a
prefix-based URL is generated.
The option \-\-iso8601 provides ISO 8601 UTC timestamps
instead of Unix timestamps inside the generated URL.
If optional \-\-absolute argument is
provided and the time argument is specified in seconds, the seconds are
interpreted as a Unix timestamp at which the URL
should expire.
\fBExample\fR: tempurl GET $(date \-d "Jan 1 2016" +%s)
/v1/AUTH_foo/bar_container/quux.md my_secret_tempurl_key \-\-absolute
.RE

33
doc/source/cli.rst

@ -228,18 +228,39 @@ Capabilities
Tempurl
-------
``tempurl [command-options] [method] [seconds] [path] [key]``
``tempurl [command-options] [method] [time] [path] [key]``
Generates a temporary URL for a Swift object. ``method`` option sets an HTTP method to
allow for this temporary URL that is usually 'GET' or 'PUT'. ``seconds`` option sets
the amount of time in seconds the temporary URL will be valid for; or, if ``--absolute``
is passed, the Unix timestamp when the temporary URL will expire. ``path`` option sets
the full path to the Swift object. Example: ``/v1/AUTH_account/c/o``. ``key`` option is
allow for this temporary URL that is usually ``GET` or ``PUT``. ``time`` option sets
the amount of time the temporary URL will be valid for.
``time`` can be specified as an integer, denoting the number of seconds
from now on until the URL shall be valid; or, if ``--absolute``
is passed, the Unix timestamp when the temporary URL will expire.
But beyond that, ``time`` can also be specified as an ISO 8601 timestamp
in one of following formats:
i) Complete date: YYYY-MM-DD (eg 1997-07-16)
ii) Complete date plus hours, minutes and seconds:
YYYY-MM-DDThh:mm:ss
(eg 1997-07-16T19:20:30)
iii) Complete date plus hours, minutes and seconds with UTC designator:
YYYY-MM-DDThh:mm:ssZ
(eg 1997-07-16T19:20:30Z)
Please be aware that if you don't provide the UTC designator (i.e., Z)
the timestamp is generated using your local timezone. If only a date is
specified, the time part used will equal to ``00:00:00``.
``path`` option sets the full path to the Swift object.
Example: ``/v1/AUTH_account/c/o``. ``key`` option is
the secret temporary URL key set on the Swift cluster. To set a key, run
``swift post -m "Temp-URL-Key: <your secret key>"``. To generate a prefix-based temporary
URL use the ``--prefix-based`` option. This URL will contain the path to the prefix. Do not
forget to append the desired objectname at the end of the path portion (and before the
query portion) before sharing the URL.
query portion) before sharing the URL. It is possible to use ISO 8601 UTC timestamps within the
URL by using the ``--iso8601`` option.
Auth
----

68
swiftclient/shell.py

@ -1224,8 +1224,8 @@ def st_auth(parser, args, thread_manager):
print('export OS_AUTH_TOKEN=%s' % sh_quote(token))
st_tempurl_options = '''[--absolute] [--prefix-based]
<method> <seconds> <path> <key>'''
st_tempurl_options = '''[--absolute] [--prefix-based] [--iso8601]
<method> <time> <path> <key>'''
st_tempurl_help = '''
@ -1234,9 +1234,35 @@ Generates a temporary URL for a Swift object.
Positional arguments:
<method> An HTTP method to allow for this temporary URL.
Usually 'GET' or 'PUT'.
<seconds> The amount of time in seconds the temporary URL will be
valid for; or, if --absolute is passed, the Unix
timestamp when the temporary URL will expire.
<time> The amount of time the temporary URL will be
valid. The time can be specified in two ways:
an integer representing the time in seconds or an
ISO 8601 timestamp in a specific format.
If --absolute is passed and time
is an integer, the seconds are intepreted as the Unix
timestamp when the temporary URL will expire. The ISO
8601 timestamp can be specified in one of following
formats:
i) Complete date: YYYY-MM-DD (eg 1997-07-16)
ii) Complete date plus hours, minutes and seconds:
YYYY-MM-DDThh:mm:ss
(eg 1997-07-16T19:20:30)
iii) Complete date plus hours, minutes and seconds with
UTC designator:
YYYY-MM-DDThh:mm:ssZ
(eg 1997-07-16T19:20:30Z)
Please be aware that if you don't provide the UTC
designator (i.e., Z) the timestamp is generated using
your local timezone. If only a date is specified,
the time part used will equal to 00:00:00.
<path> The full path or storage URL to the Swift object.
Example: /v1/AUTH_account/c/o
or: http://saio:8080/v1/AUTH_account/c/o
@ -1245,10 +1271,14 @@ Positional arguments:
"Temp-URL-Key:b3968d0207b54ece87cccc06515a89d4"\'
Optional arguments:
--absolute Interpret the <seconds> positional argument as a Unix
--absolute Interpret the <time> positional argument as a Unix
timestamp rather than a number of seconds in the
future.
--prefix-based If present, a prefix-based tempURL will be generated.
future. If an ISO 8601 timestamp is passed for <time>,
this argument is ignored.
--prefix-based If present, a prefix-based temporary URL will be
generated.
--iso8601 If present, the generated temporary URL will contain an
ISO 8601 UTC timestamp instead of a Unix timestamp.
'''.strip('\n')
@ -1256,14 +1286,21 @@ def st_tempurl(parser, args, thread_manager):
parser.add_argument(
'--absolute', action='store_true',
dest='absolute_expiry', default=False,
help=("If present, seconds argument will be interpreted as a Unix "
"timestamp representing when the tempURL should expire, rather "
"than an offset from the current time"),
help=("If present, and time argument is an integer, "
"time argument will be interpreted as a Unix "
"timestamp representing when the temporary URL should expire, "
"rather than an offset from the current time."),
)
parser.add_argument(
'--prefix-based', action='store_true',
default=False,
help=("If present, a prefix-based tempURL will be generated."),
help=("If present, a prefix-based temporary URL will be generated."),
)
parser.add_argument(
'--iso8601', action='store_true',
default=False,
help=("If present, the temporary URL will contain an ISO 8601 UTC "
"timestamp instead of a Unix timestamp."),
)
(options, args) = parse_args(parser, args)
@ -1272,7 +1309,7 @@ def st_tempurl(parser, args, thread_manager):
thread_manager.error('Usage: %s tempurl %s\n%s', BASENAME,
st_tempurl_options, st_tempurl_help)
return
method, seconds, path, key = args[:4]
method, timestamp, path, key = args[:4]
parsed = urlparse(path)
@ -1281,9 +1318,10 @@ def st_tempurl(parser, args, thread_manager):
'tempurl specified, possibly an error' %
method.upper())
try:
path = generate_temp_url(parsed.path, seconds, key, method,
path = generate_temp_url(parsed.path, timestamp, key, method,
absolute=options['absolute_expiry'],
prefix=options['prefix_based'],)
iso8601=options['iso8601'],
prefix=options['prefix_based'])
except ValueError as err:
thread_manager.error(err)
return

73
swiftclient/utils.py

@ -13,6 +13,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
"""Miscellaneous utility functions for use with Swift."""
from calendar import timegm
import collections
import gzip
import hashlib
@ -25,6 +26,10 @@ import traceback
TRUE_VALUES = set(('true', '1', 'yes', 'on', 't', 'y'))
EMPTY_ETAG = 'd41d8cd98f00b204e9800998ecf8427e'
EXPIRES_ISO8601_FORMAT = '%Y-%m-%dT%H:%M:%SZ'
SHORT_EXPIRES_ISO8601_FORMAT = '%Y-%m-%d'
TIME_ERRMSG = ('time must either be a whole number or in specific '
'ISO 8601 format.')
def config_true_value(value):
@ -64,39 +69,65 @@ def prt_bytes(num_bytes, human_flag):
def generate_temp_url(path, seconds, key, method, absolute=False,
prefix=False):
prefix=False, iso8601=False):
"""Generates a temporary URL that gives unauthenticated access to the
Swift object.
:param path: The full path to the Swift object or prefix if
a prefix-based temporary URL should be generated. Example:
/v1/AUTH_account/c/o or /v1/AUTH_account/c/prefix.
:param seconds: If absolute is False then this specifies the amount of time
in seconds for which the temporary URL will be valid. If absolute is
True then this specifies an absolute time at which the temporary URL
will expire.
:param seconds: time in seconds or ISO 8601 timestamp.
If absolute is False and this is the string representation of an
integer, then this specifies the amount of time in seconds for which
the temporary URL will be valid.
If absolute is True then this specifies an absolute time at which the
temporary URL will expire.
:param key: The secret temporary URL key set on the Swift
cluster. To set a key, run 'swift post -m
"Temp-URL-Key: <substitute tempurl key here>"'
:param method: A HTTP method, typically either GET or PUT, to allow
for this temporary URL.
:param absolute: if True then the seconds parameter is interpreted as an
absolute Unix time, otherwise seconds is interpreted as a relative time
offset from current time.
:param absolute: if True then the seconds parameter is interpreted as a
Unix timestamp, if seconds represents an integer.
:param prefix: if True then a prefix-based temporary URL will be generated.
:raises ValueError: if seconds is not a whole number or path is not to
an object.
:param iso8601: if True, a URL containing an ISO 8601 UTC timestamp
instead of a UNIX timestamp will be created.
:raises ValueError: if timestamp or path is not in valid format.
:return: the path portion of a temporary URL
"""
try:
seconds = float(seconds)
if not seconds.is_integer():
raise ValueError()
seconds = int(seconds)
if seconds < 0:
raise ValueError()
try:
timestamp = float(seconds)
except ValueError:
formats = (
EXPIRES_ISO8601_FORMAT,
EXPIRES_ISO8601_FORMAT[:-1],
SHORT_EXPIRES_ISO8601_FORMAT)
for f in formats:
try:
t = time.strptime(seconds, f)
except ValueError:
t = None
else:
if f == EXPIRES_ISO8601_FORMAT:
timestamp = timegm(t)
else:
# Use local time if UTC designator is missing.
timestamp = int(time.mktime(t))
absolute = True
break
if t is None:
raise ValueError()
else:
if not timestamp.is_integer():
raise ValueError()
timestamp = int(timestamp)
if timestamp < 0:
raise ValueError()
except ValueError:
raise ValueError('seconds must be a whole number')
raise ValueError(TIME_ERRMSG)
if isinstance(path, six.binary_type):
try:
@ -121,9 +152,9 @@ def generate_temp_url(path, seconds, key, method, absolute=False,
'possibly an error', method.upper())
if not absolute:
expiration = int(time.time() + seconds)
expiration = int(time.time() + timestamp)
else:
expiration = seconds
expiration = timestamp
hmac_body = u'\n'.join([method.upper(), str(expiration),
('prefix:' if prefix else '') + path_for_body])
@ -132,6 +163,10 @@ def generate_temp_url(path, seconds, key, method, absolute=False,
key = key.encode('utf-8')
sig = hmac.new(key, hmac_body.encode('utf-8'), hashlib.sha1).hexdigest()
if iso8601:
expiration = time.strftime(
EXPIRES_ISO8601_FORMAT, time.gmtime(expiration))
temp_url = u'{path}?temp_url_sig={sig}&temp_url_expires={exp}'.format(
path=path_for_body, sig=sig, exp=expiration)
if prefix:

80
tests/unit/test_shell.py

@ -23,6 +23,7 @@ import os
import tempfile
import unittest
import textwrap
from time import localtime, mktime, strftime, strptime
from requests.packages.urllib3.exceptions import InsecureRequestWarning
import six
@ -36,7 +37,9 @@ from os.path import basename, dirname
from .utils import (
CaptureOutput, fake_get_auth_keystone, _make_fake_import_keystone_client,
FakeKeystone, StubResponse, MockHttpTest)
from swiftclient.utils import EMPTY_ETAG
from swiftclient.utils import (
EMPTY_ETAG, EXPIRES_ISO8601_FORMAT,
SHORT_EXPIRES_ISO8601_FORMAT, TIME_ERRMSG)
if six.PY2:
@ -1550,7 +1553,7 @@ class TestShell(unittest.TestCase):
swiftclient.shell.main(argv)
temp_url.assert_called_with(
'/v1/AUTH_account/c/o', "60", 'secret_key', 'GET', absolute=False,
prefix=False)
iso8601=False, prefix=False)
@mock.patch('swiftclient.shell.generate_temp_url', return_value='')
def test_temp_url_prefix_based(self, temp_url):
@ -1559,7 +1562,28 @@ class TestShell(unittest.TestCase):
swiftclient.shell.main(argv)
temp_url.assert_called_with(
'/v1/AUTH_account/c/', "60", 'secret_key', 'GET', absolute=False,
prefix=True)
iso8601=False, prefix=True)
@mock.patch('swiftclient.shell.generate_temp_url', return_value='')
def test_temp_url_iso8601_in(self, temp_url):
dates = ('1970-01-01T00:01:00Z', '1970-01-01T00:01:00',
'1970-01-01')
for d in dates:
argv = ["", "tempurl", "GET", d, "/v1/AUTH_account/c/",
"secret_key"]
swiftclient.shell.main(argv)
temp_url.assert_called_with(
'/v1/AUTH_account/c/', d, 'secret_key', 'GET', absolute=False,
iso8601=False, prefix=False)
@mock.patch('swiftclient.shell.generate_temp_url', return_value='')
def test_temp_url_iso8601_out(self, temp_url):
argv = ["", "tempurl", "GET", "60", "/v1/AUTH_account/c/",
"secret_key", "--iso8601"]
swiftclient.shell.main(argv)
temp_url.assert_called_with(
'/v1/AUTH_account/c/', "60", 'secret_key', 'GET', absolute=False,
iso8601=True, prefix=False)
@mock.patch('swiftclient.shell.generate_temp_url', return_value='')
def test_absolute_expiry_temp_url(self, temp_url):
@ -1568,7 +1592,7 @@ class TestShell(unittest.TestCase):
swiftclient.shell.main(argv)
temp_url.assert_called_with(
'/v1/AUTH_account/c/o', "60", 'secret_key', 'GET', absolute=True,
prefix=False)
iso8601=False, prefix=False)
def test_temp_url_output(self):
argv = ["", "tempurl", "GET", "60", "/v1/a/c/o",
@ -1595,6 +1619,42 @@ class TestShell(unittest.TestCase):
"&temp_url_prefix=\n" % sig)
self.assertEqual(expected, output.out)
argv = ["", "tempurl", "GET", "60", "/v1/a/c/",
"secret_key", "--absolute", "--prefix", '--iso8601']
with CaptureOutput(suppress_systemexit=True) as output:
swiftclient.shell.main(argv)
sig = '00008c4be1573ba74fc2ab9bce02e3a93d04b349'
expires = '1970-01-01T00:01:00Z'
expected = ("/v1/a/c/?temp_url_sig=%s&temp_url_expires=%s"
"&temp_url_prefix=\n" % (sig, expires))
self.assertEqual(expected, output.out)
dates = ("1970-01-01T00:01:00Z",
strftime(EXPIRES_ISO8601_FORMAT[:-1], localtime(60)))
for d in dates:
argv = ["", "tempurl", "GET", d, "/v1/a/c/o",
"secret_key"]
with CaptureOutput(suppress_systemexit=True) as output:
swiftclient.shell.main(argv)
sig = "63bc77a473a1c2ce956548cacf916f292eb9eac3"
expected = "/v1/a/c/o?temp_url_sig=%s&temp_url_expires=60\n" % sig
self.assertEqual(expected, output.out)
ts = str(int(
mktime(strptime('2005-05-01', SHORT_EXPIRES_ISO8601_FORMAT))))
argv = ["", "tempurl", "GET", ts, "/v1/a/c/",
"secret_key", "--absolute"]
with CaptureOutput(suppress_systemexit=True) as output:
swiftclient.shell.main(argv)
expected = output.out
argv = ["", "tempurl", "GET", '2005-05-01', "/v1/a/c/",
"secret_key", "--absolute"]
with CaptureOutput(suppress_systemexit=True) as output:
swiftclient.shell.main(argv)
self.assertEqual(expected, output.out)
def test_temp_url_error_output(self):
expected = 'path must be full path to an object e.g. /v1/a/c/o\n'
for bad_path in ('/v1/a/c', 'v1/a/c/o', '/v1/a/c/', '/v1/a//o',
@ -1614,7 +1674,17 @@ class TestShell(unittest.TestCase):
swiftclient.shell.main(argv)
self.assertEqual(expected, output.err,
'Expected %r but got %r for path %r' %
(expected, output.err, bad_path))
(expected, output.err, '/v1/a/c'))
expected = TIME_ERRMSG + '\n'
for bad_time in ('not_an_int', '-1', '2015-05', '2015-05-01T01:00'):
argv = ["", "tempurl", "GET", bad_time, '/v1/a/c/o',
"secret_key", "--absolute"]
with CaptureOutput(suppress_systemexit=True) as output:
swiftclient.shell.main(argv)
self.assertEqual(expected, output.err,
'Expected %r but got %r for time %r' %
(expected, output.err, bad_time))
@mock.patch('swiftclient.service.Connection')
def test_capabilities(self, connection):

87
tests/unit/test_utils.py

@ -18,6 +18,7 @@ import unittest
import mock
import six
import tempfile
from time import gmtime, localtime, mktime, strftime, strptime
from hashlib import md5, sha1
from swiftclient import utils as u
@ -150,6 +151,67 @@ class TestTempURL(unittest.TestCase):
])
self.assertIsInstance(url, type(self.url))
@mock.patch('hmac.HMAC')
def test_generate_temp_url_iso8601_argument(self, hmac_mock):
hmac_mock().hexdigest.return_value = 'temp_url_signature'
url = u.generate_temp_url(self.url, '2014-05-13T17:53:20Z',
self.key, self.method)
self.assertEqual(url, self.expected_url)
# Don't care about absolute arg.
url = u.generate_temp_url(self.url, '2014-05-13T17:53:20Z',
self.key, self.method, absolute=True)
self.assertEqual(url, self.expected_url)
lt = localtime()
expires = strftime(u.EXPIRES_ISO8601_FORMAT[:-1], lt)
if not isinstance(self.expected_url, six.string_types):
expected_url = self.expected_url.replace(
b'1400003600', bytes(str(int(mktime(lt))), encoding='ascii'))
else:
expected_url = self.expected_url.replace(
'1400003600', str(int(mktime(lt))))
url = u.generate_temp_url(self.url, expires,
self.key, self.method)
self.assertEqual(url, expected_url)
expires = strftime(u.SHORT_EXPIRES_ISO8601_FORMAT, lt)
lt = strptime(expires, u.SHORT_EXPIRES_ISO8601_FORMAT)
if not isinstance(self.expected_url, six.string_types):
expected_url = self.expected_url.replace(
b'1400003600', bytes(str(int(mktime(lt))), encoding='ascii'))
else:
expected_url = self.expected_url.replace(
'1400003600', str(int(mktime(lt))))
url = u.generate_temp_url(self.url, expires,
self.key, self.method)
self.assertEqual(url, expected_url)
@mock.patch('hmac.HMAC')
@mock.patch('time.time', return_value=1400000000)
def test_generate_temp_url_iso8601_output(self, time_mock, hmac_mock):
hmac_mock().hexdigest.return_value = 'temp_url_signature'
url = u.generate_temp_url(self.url, self.seconds,
self.key, self.method,
iso8601=True)
key = self.key
if not isinstance(key, six.binary_type):
key = key.encode('utf-8')
expires = strftime(u.EXPIRES_ISO8601_FORMAT, gmtime(1400003600))
if not isinstance(self.url, six.string_types):
self.assertTrue(url.endswith(bytes(expires, 'utf-8')))
else:
self.assertTrue(url.endswith(expires))
self.assertEqual(hmac_mock.mock_calls, [
mock.call(),
mock.call(key, self.expected_body, sha1),
mock.call().hexdigest(),
])
self.assertIsInstance(url, type(self.url))
@mock.patch('hmac.HMAC')
@mock.patch('time.time', return_value=1400000000)
def test_generate_temp_url_prefix(self, time_mock, hmac_mock):
@ -198,31 +260,34 @@ class TestTempURL(unittest.TestCase):
absolute=True)
self.assertEqual(url, expected_url)
def test_generate_temp_url_bad_seconds(self):
def test_generate_temp_url_bad_time(self):
with self.assertRaises(ValueError) as exc_manager:
u.generate_temp_url(self.url, 'not_an_int', self.key, self.method)
self.assertEqual(exc_manager.exception.args[0],
'seconds must be a whole number')
self.assertEqual(exc_manager.exception.args[0], u.TIME_ERRMSG)
with self.assertRaises(ValueError) as exc_manager:
u.generate_temp_url(self.url, -1, self.key, self.method)
self.assertEqual(exc_manager.exception.args[0],
'seconds must be a whole number')
self.assertEqual(exc_manager.exception.args[0], u.TIME_ERRMSG)
with self.assertRaises(ValueError) as exc_manager:
u.generate_temp_url(self.url, 1.1, self.key, self.method)
self.assertEqual(exc_manager.exception.args[0],
'seconds must be a whole number')
self.assertEqual(exc_manager.exception.args[0], u.TIME_ERRMSG)
with self.assertRaises(ValueError) as exc_manager:
u.generate_temp_url(self.url, '-1', self.key, self.method)
self.assertEqual(exc_manager.exception.args[0],
'seconds must be a whole number')
self.assertEqual(exc_manager.exception.args[0], u.TIME_ERRMSG)
with self.assertRaises(ValueError) as exc_manager:
u.generate_temp_url(self.url, '1.1', self.key, self.method)
self.assertEqual(exc_manager.exception.args[0],
'seconds must be a whole number')
self.assertEqual(exc_manager.exception.args[0], u.TIME_ERRMSG)
with self.assertRaises(ValueError) as exc_manager:
u.generate_temp_url(self.url, '2015-05', self.key, self.method)
self.assertEqual(exc_manager.exception.args[0], u.TIME_ERRMSG)
with self.assertRaises(ValueError) as exc_manager:
u.generate_temp_url(
self.url, '2015-05-01T01:00', self.key, self.method)
self.assertEqual(exc_manager.exception.args[0], u.TIME_ERRMSG)
def test_generate_temp_url_bad_path(self):
with self.assertRaises(ValueError) as exc_manager: