Loading payload from remote URI
POC on loading payloads using remote URI. This is part of a larger effort in packaging syntribos to ensure that the project would work without much configuration post install from pypi. Change-Id: Id61e840d4f49d5b6deb72bce2e8bcc0e1096fa52
This commit is contained in:
Rahul Nair
parent
244e8c48bc
commit
b7b925cf4d
53
README.rst
53
README.rst
|
|
@ -198,14 +198,16 @@ pip <https://pypi.python.org/pypi/pip>`__ from the git repository.
|
|||
Configuration
|
||||
=============
|
||||
|
||||
This is the basic structure of a syntribos configuration file.
|
||||
All configuration files should have at least the section
|
||||
``[syntribos]``. Depending upon what extensions you are using
|
||||
and what you are testing, you can add other sections as well,
|
||||
for example, if you are using the built-in identity extension
|
||||
and what you are testing, you can add other sections as well.
|
||||
For example, if you are using the built-in identity extension
|
||||
you would also need the ``[user]`` section. The sections
|
||||
``[logging]`` and ``[remote]`` are optional.
|
||||
|
||||
Given below is the basic structure of a syntribos configuration
|
||||
file.
|
||||
|
||||
::
|
||||
|
||||
[syntribos]
|
||||
|
|
@ -226,6 +228,14 @@ you would also need the ``[user]`` section. The sections
|
|||
username=<yourusername>
|
||||
password=<yourpassword>
|
||||
|
||||
[remote]
|
||||
#
|
||||
# Optional, to define remote URI and cache_dir explictly
|
||||
#
|
||||
templates_uri=<URI to a tar file of set of templates>
|
||||
payloads_uri=<URI to a tar file of set of payloads>
|
||||
cache_dir=<a local path to save the downloaded files>
|
||||
|
||||
[logging]
|
||||
log_dir=<location_to_save_debug_logs>
|
||||
|
||||
|
|
@ -236,6 +246,31 @@ credentials if needed. The endpoint URI in the ``[syntribos]``
|
|||
section is the one being tested by syntribos and the endpoint URI in
|
||||
``[user]`` section is just used to get an AUTH_TOKEN.
|
||||
|
||||
Downloading templates and payloads remotely
|
||||
-------------------------------------------
|
||||
|
||||
Payload and template files can be downloaded remotely in syntribos.
|
||||
In the config file under ``[syntribos]`` section, if ``templates``
|
||||
and ``payloads_dir`` options are not set then by default syntribos will
|
||||
download templates for a few OpenStack projects and all the
|
||||
latest payloads. As a user you can specify a URI to download custom
|
||||
templates and payloads from as well; this is done by using
|
||||
``[remotes]`` section in the config file. Available options under
|
||||
``[remotes]`` are ``cache_dir``, ``templates_uri``, ``payloads_uri`` and
|
||||
``enable_cache``. The ``enable_cache`` option is ``on`` by default
|
||||
and can be set to ``off`` to disable caching of remote content while
|
||||
syntribos is running. ``cache_dir`` if set to a path, syntribos will
|
||||
attempt to use that as a base directory to save downloaded template
|
||||
and payload files.
|
||||
|
||||
The advantage of using these options are that you will be able to get
|
||||
the latest payloads from the official repository and if you are
|
||||
using syntribos to test OpenStack projects, then in most cases
|
||||
you would already have well defined templates availble to work with.
|
||||
|
||||
This option also helps to easily manage different versions of
|
||||
templates remotely, without the need to maintain a set of
|
||||
different versions offline.
|
||||
|
||||
Testing keystone API
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
|
@ -297,6 +332,16 @@ necessary fields like user credentials, log, template directory etc.
|
|||
# For Keystone V2 API
|
||||
#tenant_name=<name_of_the_project>
|
||||
|
||||
[remote]
|
||||
#
|
||||
# Optional, Used to specify URLs of templates and payloads
|
||||
#
|
||||
#cache_dir=<a local path to save the downloaded files>
|
||||
#templates_uri=https://github.com/your_project/templates.tar
|
||||
#payloads_uri=https://github.com/your_project/payloads.tar
|
||||
# To disable caching of these remote contents, set the following variable to False
|
||||
#enable_caching=True
|
||||
|
||||
[logging]
|
||||
#
|
||||
# Logger options go here
|
||||
|
|
@ -356,7 +401,7 @@ Running syntribos
|
|||
=================
|
||||
|
||||
To run syntribos against all the available tests, just specify the
|
||||
command :command:`syntribos run` with the configuration file without specifying
|
||||
command ``syntribos`` with the configuration file without specifying
|
||||
any test type.
|
||||
|
||||
::
|
||||
|
|
|
|||
|
|
@ -2,14 +2,16 @@
|
|||
Configuration
|
||||
=============
|
||||
|
||||
This is the basic structure of a syntribos configuration file.
|
||||
All configuration files should have at least the section
|
||||
``[syntribos]``. Depending upon what extensions you are using
|
||||
and what you are testing, you can add other sections as well,
|
||||
for example, if you are using the built-in identity extension
|
||||
and what you are testing, you can add other sections as well.
|
||||
For example, if you are using the built-in identity extension
|
||||
you would also need the ``[user]`` section. The sections
|
||||
``[logging]`` and ``[remote]`` are optional.
|
||||
|
||||
Given below is the basic structure of a syntribos configuration
|
||||
file.
|
||||
|
||||
::
|
||||
|
||||
[syntribos]
|
||||
|
|
@ -30,6 +32,14 @@ you would also need the ``[user]`` section. The sections
|
|||
username=<yourusername>
|
||||
password=<yourpassword>
|
||||
|
||||
[remote]
|
||||
#
|
||||
# Optional, to define remote URI and cache_dir explictly
|
||||
#
|
||||
templates_uri=<URI to a tar file of set of templates>
|
||||
payloads_uri=<URI to a tar file of set of payloads>
|
||||
cache_dir=<a local path to save the downloaded files>
|
||||
|
||||
[logging]
|
||||
log_dir=<location_to_save_debug_logs>
|
||||
|
||||
|
|
@ -40,6 +50,31 @@ credentials if needed. The endpoint URI in the ``[syntribos]``
|
|||
section is the one being tested by syntribos and the endpoint URI in
|
||||
``[user]`` section is just used to get an AUTH_TOKEN.
|
||||
|
||||
Downloading templates and payloads remotely
|
||||
-------------------------------------------
|
||||
|
||||
Payload and template files can be downloaded remotely in syntribos.
|
||||
In the config file under ``[syntribos]`` section, if ``templates``
|
||||
and ``payloads_dir`` options are not set then by default syntribos will
|
||||
download templates for a few OpenStack projects and all the
|
||||
latest payloads. As a user you can specify a URI to download custom
|
||||
templates and payloads from as well; this is done by using
|
||||
``[remotes]`` section in the config file. Available options under
|
||||
``[remotes]`` are ``cache_dir``, ``templates_uri``, ``payloads_uri`` and
|
||||
``enable_cache``. The ``enable_cache`` option is ``on`` by default
|
||||
and can be set to ``off`` to disable caching of remote content while
|
||||
syntribos is running. ``cache_dir`` if set to a path, syntribos will
|
||||
attempt to use that as a base directory to save downloaded template
|
||||
and payload files.
|
||||
|
||||
The advantage of using these options are that you will be able to get
|
||||
the latest payloads from the official repository and if you are
|
||||
using syntribos to test OpenStack projects, then in most cases
|
||||
you would already have well defined templates availble to work with.
|
||||
|
||||
This option also helps to easily manage different versions of
|
||||
templates remotely, without the need to maintain a set of
|
||||
different versions offline.
|
||||
|
||||
Testing keystone API
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
|
@ -101,6 +136,16 @@ necessary fields like user credentials, log, template directory etc.
|
|||
# For Keystone V2 API
|
||||
#tenant_name=<name_of_the_project>
|
||||
|
||||
[remote]
|
||||
#
|
||||
# Optional, Used to specify URLs of templates and payloads
|
||||
#
|
||||
#cache_dir=<a local path to save the downloaded files>
|
||||
#templates_uri=https://github.com/your_project/templates.tar
|
||||
#payloads_uri=https://github.com/your_project/payloads.tar
|
||||
# To disable caching of these remote contents, set the following variable to False
|
||||
#enable_caching=True
|
||||
|
||||
[logging]
|
||||
#
|
||||
# Logger options go here
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@ Running syntribos
|
|||
=================
|
||||
|
||||
To run syntribos against all the available tests, just specify the
|
||||
command :command:`syntribos run` with the configuration file without specifying
|
||||
command ``syntribos`` with the configuration file without specifying
|
||||
any test type.
|
||||
|
||||
::
|
||||
|
|
|
|||
|
|
@ -118,6 +118,8 @@ class TemplateType(ExistingPathType):
|
|||
:rtype: tuple
|
||||
:returns: (file name, file contents)
|
||||
"""
|
||||
if not string:
|
||||
return
|
||||
super(TemplateType, self).__call__(string)
|
||||
|
||||
if os.path.isdir(string):
|
||||
|
|
@ -131,6 +133,7 @@ syntribos_group = cfg.OptGroup(name="syntribos", title="Main syntribos Config")
|
|||
user_group = cfg.OptGroup(name="user", title="Identity Config")
|
||||
test_group = cfg.OptGroup(name="test", title="Test Config")
|
||||
logger_group = cfg.OptGroup(name="logging", title="Logger config")
|
||||
remote_group = cfg.OptGroup(name="remote", title="Remote config")
|
||||
|
||||
|
||||
def sub_commands(sub_parser):
|
||||
|
|
@ -149,6 +152,7 @@ def list_opts():
|
|||
results.append((user_group, list_user_opts()))
|
||||
results.append((test_group, list_test_opts()))
|
||||
results.append((logger_group, list_logger_opts()))
|
||||
results.append((remote_group, list_remote_opts()))
|
||||
return results
|
||||
|
||||
|
||||
|
|
@ -167,6 +171,9 @@ def register_opts():
|
|||
# Logger options
|
||||
CONF.register_group(logger_group)
|
||||
CONF.register_opts(list_logger_opts(), group=logger_group)
|
||||
# Remote options
|
||||
CONF.register_group(remote_group)
|
||||
CONF.register_opts(list_remote_opts(), group=remote_group)
|
||||
|
||||
|
||||
def list_cli_opts():
|
||||
|
|
@ -203,11 +210,11 @@ def list_syntribos_opts():
|
|||
cfg.StrOpt("endpoint", default="",
|
||||
sample_default="http://localhost/app", required=True,
|
||||
help="The target host to be tested"),
|
||||
cfg.Opt("templates", type=TemplateType('r', 0), required=True,
|
||||
cfg.Opt("templates", type=TemplateType('r', 0), default="",
|
||||
sample_default="~/.syntribos/templates",
|
||||
help="A directory of template files, or a single template "
|
||||
"file, to test on the target API"),
|
||||
cfg.StrOpt("payload_dir", default="", required=True,
|
||||
cfg.StrOpt("payloads_dir", default="",
|
||||
sample_default="~/.syntribos/data",
|
||||
help="The location where we can find syntribos' payloads"),
|
||||
cfg.MultiStrOpt("exclude_results",
|
||||
|
|
@ -273,3 +280,27 @@ def list_logger_opts():
|
|||
sample_default="~/.syntribos/logs",
|
||||
help="Where to save debug log files for a Syntribos run")
|
||||
]
|
||||
|
||||
|
||||
def list_remote_opts():
|
||||
"""Method defining remote URIs for payloads and templates."""
|
||||
return [
|
||||
cfg.StrOpt(
|
||||
"cache_dir",
|
||||
default="",
|
||||
help="Base directory where cached files can be saved"),
|
||||
cfg.StrOpt(
|
||||
"payloads_uri",
|
||||
default=("https://github.com/rahulunair/"
|
||||
"syntribos-payloads/"
|
||||
"raw/master/syntribos-payloads.tar"),
|
||||
help="Remote URI to download payloads."),
|
||||
cfg.StrOpt(
|
||||
"templates_uri",
|
||||
default=("https://github.com/rahulunair/"
|
||||
"syntribos-openstack-templates/"
|
||||
"raw/master/syntribos-openstack-templates.tar"),
|
||||
help="Remote URI to download templates."),
|
||||
cfg.BoolOpt("enable_cache", default=True,
|
||||
help="Cache remote template & payload resources locally"),
|
||||
]
|
||||
|
|
|
|||
|
|
@ -19,6 +19,7 @@ from oslo_config import cfg
|
|||
import syntribos
|
||||
from syntribos.formatters.json_formatter import JSONFormatter
|
||||
from syntribos.runner import Runner
|
||||
import syntribos.utils.remotes
|
||||
|
||||
CONF = cfg.CONF
|
||||
|
||||
|
|
|
|||
|
|
@ -23,13 +23,17 @@ from oslo_config import cfg
|
|||
import six
|
||||
|
||||
import syntribos.config
|
||||
from syntribos.config import TemplateType
|
||||
from syntribos.formatters.json_formatter import JSONFormatter
|
||||
import syntribos.result
|
||||
import syntribos.tests as tests
|
||||
import syntribos.tests.base
|
||||
from syntribos.utils import cleanup
|
||||
from syntribos.utils import cli as cli
|
||||
from syntribos.utils import remotes
|
||||
|
||||
result = None
|
||||
user_base_dir = None
|
||||
CONF = cfg.CONF
|
||||
LOG = logging.getLogger(__name__)
|
||||
|
||||
|
|
@ -165,7 +169,13 @@ class Runner(object):
|
|||
dry_run_output = {"failures": [], "successes": []}
|
||||
list_of_tests = list(cls.get_tests(dry_run=True))
|
||||
print("\nRunning Tests...:")
|
||||
for file_path, req_str in CONF.syntribos.templates:
|
||||
templates_dir = CONF.syntribos.templates
|
||||
if templates_dir is None:
|
||||
print("Attempting to download templates from {}".format(
|
||||
CONF.remote.templates_uri))
|
||||
templates_path = remotes.get(CONF.remote.templates_uri)
|
||||
templates_dir = TemplateType('r', 0)(templates_path)
|
||||
for file_path, req_str in templates_dir:
|
||||
LOG = cls.get_logger(file_path)
|
||||
CONF.log_opt_values(LOG, logging.DEBUG)
|
||||
if not file_path.endswith(".template"):
|
||||
|
|
@ -191,6 +201,7 @@ class Runner(object):
|
|||
|
||||
if CONF.sub_command.name == "run":
|
||||
result.print_result(cls.start_time)
|
||||
cleanup.delete_temps()
|
||||
elif CONF.sub_command.name == "dry_run":
|
||||
cls.dry_run_report(dry_run_output)
|
||||
|
||||
|
|
@ -325,6 +336,7 @@ class Runner(object):
|
|||
|
||||
except KeyboardInterrupt:
|
||||
result.print_result(cls.start_time)
|
||||
cleanup.delete_temps()
|
||||
print("Keyboard interrupt, exiting...")
|
||||
exit(0)
|
||||
|
||||
|
|
|
|||
|
|
@ -20,9 +20,9 @@ import syntribos
|
|||
from syntribos.checks import length_diff as length_diff
|
||||
from syntribos.tests import base
|
||||
import syntribos.tests.fuzz.datagen
|
||||
from syntribos.utils import remotes
|
||||
|
||||
CONF = cfg.CONF
|
||||
payload_dir = CONF.syntribos.payload_dir
|
||||
|
||||
|
||||
class BaseFuzzTestCase(base.BaseTestCase):
|
||||
|
|
@ -31,8 +31,10 @@ class BaseFuzzTestCase(base.BaseTestCase):
|
|||
|
||||
@classmethod
|
||||
def _get_strings(cls, file_name=None):
|
||||
path = os.path.join(payload_dir, file_name or cls.data_key)
|
||||
|
||||
payloads_dir = CONF.syntribos.payloads_dir
|
||||
if not payloads_dir:
|
||||
payloads_dir = remotes.get(CONF.remote.payloads_uri)
|
||||
path = os.path.join(payloads_dir, file_name or cls.data_key)
|
||||
with open(path, "rb") as fp:
|
||||
return fp.read().splitlines()
|
||||
|
||||
|
|
@ -45,8 +47,10 @@ class BaseFuzzTestCase(base.BaseTestCase):
|
|||
"""being used as a setup test not."""
|
||||
super(BaseFuzzTestCase, cls).setUpClass()
|
||||
cls.test_resp, cls.test_signals = cls.client.request(
|
||||
method=cls.request.method, url=cls.request.url,
|
||||
headers=cls.request.headers, params=cls.request.params,
|
||||
method=cls.request.method,
|
||||
url=cls.request.url,
|
||||
headers=cls.request.headers,
|
||||
params=cls.request.params,
|
||||
data=cls.request.data)
|
||||
cls.test_req = cls.request
|
||||
|
||||
|
|
@ -89,9 +93,10 @@ class BaseFuzzTestCase(base.BaseTestCase):
|
|||
"vulnerability to injection attacks"
|
||||
).format(CONF.test.length_diff_percent)
|
||||
self.register_issue(
|
||||
defect_type="length_diff", severity=syntribos.LOW,
|
||||
confidence=syntribos.LOW, description=description
|
||||
)
|
||||
defect_type="length_diff",
|
||||
severity=syntribos.LOW,
|
||||
confidence=syntribos.LOW,
|
||||
description=description)
|
||||
|
||||
def test_case(self):
|
||||
"""Performs the test
|
||||
|
|
@ -116,8 +121,9 @@ class BaseFuzzTestCase(base.BaseTestCase):
|
|||
cls.failures = []
|
||||
if hasattr(cls, 'data_key'):
|
||||
prefix_name = "{filename}_{test_name}_{fuzz_file}_".format(
|
||||
filename=filename, test_name=cls.test_name, fuzz_file=cls.
|
||||
data_key)
|
||||
filename=filename,
|
||||
test_name=cls.test_name,
|
||||
fuzz_file=cls.data_key)
|
||||
else:
|
||||
prefix_name = "{filename}_{test_name}_".format(
|
||||
filename=filename, test_name=cls.test_name)
|
||||
|
|
@ -125,8 +131,9 @@ class BaseFuzzTestCase(base.BaseTestCase):
|
|||
fr = syntribos.tests.fuzz.datagen.fuzz_request(
|
||||
cls.init_req, cls._get_strings(), cls.test_type, prefix_name)
|
||||
for fuzz_name, request, fuzz_string, param_path in fr:
|
||||
yield cls.extend_class(fuzz_name, fuzz_string, param_path,
|
||||
{"request": request})
|
||||
yield cls.extend_class(fuzz_name, fuzz_string, param_path, {
|
||||
"request": request
|
||||
})
|
||||
|
||||
@classmethod
|
||||
def extend_class(cls, new_name, fuzz_string, param_path, kwargs):
|
||||
|
|
@ -167,10 +174,11 @@ class BaseFuzzTestCase(base.BaseTestCase):
|
|||
:rtype: :class:`syntribos.issue.Issue`
|
||||
"""
|
||||
|
||||
issue = syntribos.Issue(defect_type=defect_type,
|
||||
severity=severity,
|
||||
confidence=confidence,
|
||||
description=description)
|
||||
issue = syntribos.Issue(
|
||||
defect_type=defect_type,
|
||||
severity=severity,
|
||||
confidence=confidence,
|
||||
description=description)
|
||||
|
||||
# Still associating request and response objects with issue in event of
|
||||
# debug log
|
||||
|
|
@ -190,8 +198,10 @@ class BaseFuzzTestCase(base.BaseTestCase):
|
|||
issue.content_type = None
|
||||
|
||||
issue.impacted_parameter = ImpactedParameter(
|
||||
method=issue.request.method, location=self.test_type,
|
||||
name=self.param_path, value=self.fuzz_string)
|
||||
method=issue.request.method,
|
||||
location=self.test_type,
|
||||
name=self.param_path,
|
||||
value=self.fuzz_string)
|
||||
|
||||
self.failures.append(issue)
|
||||
|
||||
|
|
@ -199,7 +209,6 @@ class BaseFuzzTestCase(base.BaseTestCase):
|
|||
|
||||
|
||||
class ImpactedParameter(object):
|
||||
|
||||
"""Object that encapsulates the details about what caused the defect
|
||||
|
||||
:ivar method: The HTTP method used in the test
|
||||
|
|
@ -215,8 +224,7 @@ class ImpactedParameter(object):
|
|||
self.location = location
|
||||
if len(value) >= 128:
|
||||
self.trunc_fuzz_string = "{0}...({1} chars)...{2}".format(
|
||||
value[:64], len(value),
|
||||
value[-64:])
|
||||
value[:64], len(value), value[-64:])
|
||||
else:
|
||||
self.trunc_fuzz_string = value
|
||||
self.fuzz_string = value
|
||||
|
|
|
|||
|
|
@ -0,0 +1,35 @@
|
|||
# Copyright 2016 Intel
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
import os
|
||||
import shutil
|
||||
|
||||
import syntribos.utils.remotes
|
||||
|
||||
|
||||
def delete_temps():
|
||||
"""Deletes all temporary dirs used for saving cached files."""
|
||||
remote_dirs = set(syntribos.utils.remotes.remote_dirs)
|
||||
temp_dirs = set(syntribos.utils.remotes.temp_dirs)
|
||||
[delete_dir(temp_dir) for temp_dir in temp_dirs]
|
||||
if remote_dirs - temp_dirs:
|
||||
print("All downloaded files have been saved to: {}".format(
|
||||
",".join([ele for ele in (remote_dirs - temp_dirs)])))
|
||||
|
||||
|
||||
def delete_file(path):
|
||||
os.remove(path)
|
||||
|
||||
|
||||
def delete_dir(dir_path):
|
||||
return shutil.rmtree(dir_path)
|
||||
|
|
@ -33,6 +33,8 @@ class ConfFixture(config_fixture.Config):
|
|||
self.conf.set_default("password", "pass", group="user")
|
||||
self.conf.set_default("serialize_format", "json", group="user")
|
||||
self.conf.set_default("deserialize_format", "json", group="user")
|
||||
self.conf.set_default("enable_cache", True, group="remote")
|
||||
self.conf.set_default("cache_dir", "", group="remote")
|
||||
|
||||
def v2_identity_fixture(self):
|
||||
"""config values only applicable to keystone v2."""
|
||||
|
|
|
|||
|
|
@ -0,0 +1,134 @@
|
|||
# Copyright 2016 Intel
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
from functools import wraps
|
||||
import logging
|
||||
import os
|
||||
import tarfile
|
||||
import tempfile
|
||||
|
||||
from oslo_config import cfg
|
||||
|
||||
from syntribos.clients.http.client import SynHTTPClient
|
||||
|
||||
CONF = cfg.CONF
|
||||
LOG = logging.getLogger(__name__)
|
||||
temp_dirs = []
|
||||
remote_dirs = []
|
||||
|
||||
|
||||
def cache(func):
|
||||
"""A method to cache return values of any method."""
|
||||
cached_content = {}
|
||||
|
||||
@wraps(func)
|
||||
def cached_func(*args, **kwargs):
|
||||
if CONF.remote.enable_cache:
|
||||
try:
|
||||
return cached_content[args]
|
||||
except KeyError:
|
||||
return cached_content.setdefault(args, func(*args, **kwargs))
|
||||
return func(*args, **kwargs)
|
||||
return cached_func
|
||||
|
||||
|
||||
def download(uri, cache_dir=None):
|
||||
"""A simple file downloader.
|
||||
|
||||
A simple file downloader which returns the absolute
|
||||
path to where the file has been saved. In case of tar
|
||||
files the absolute patch excluding .tar extension is
|
||||
passed.
|
||||
|
||||
:param str uri: The remote uri of the file
|
||||
:param str cache_dir: The directory name/handle
|
||||
:returns str: Absolute path to the downloaded file
|
||||
"""
|
||||
global temp_dirs
|
||||
global remote_dirs
|
||||
if not cache_dir:
|
||||
cache_dir = tempfile.mkdtemp()
|
||||
temp_dirs.append(cache_dir)
|
||||
remote_dirs.append(cache_dir)
|
||||
log_string = "Remote file location: {}".format(remote_dirs)
|
||||
LOG.debug(log_string)
|
||||
resp, signals = SynHTTPClient().request("GET", uri)
|
||||
os.chdir(cache_dir)
|
||||
saved_umask = os.umask(0o77)
|
||||
fname = uri.split("/")[-1]
|
||||
try:
|
||||
with open(fname, 'w') as fh:
|
||||
fh.write(resp.text)
|
||||
return os.path.abspath(fname)
|
||||
except IOError:
|
||||
LOG.error("IOError in writing the downloaded file to disk.")
|
||||
finally:
|
||||
os.umask(saved_umask)
|
||||
|
||||
|
||||
def extract_tar(abs_path):
|
||||
"""Extract tar file from the given absolute_path
|
||||
|
||||
:param str abs_path: The absolute path to the tar file
|
||||
:returns str untar_dir: The absolute path to untarred file
|
||||
"""
|
||||
try:
|
||||
tarfile.TarFile(abs_path)
|
||||
except tarfile.TarError as e:
|
||||
msg = "Not a tar file, returning abs_path, exception is: {}".format(e)
|
||||
LOG.debug(msg)
|
||||
return abs_path
|
||||
work_dir, tar_file = os.path.split(abs_path)
|
||||
os.chdir(work_dir)
|
||||
|
||||
def safe_paths(tar_meta):
|
||||
"""Makes sure all tar file paths are relative to the base path
|
||||
|
||||
Orignal from https://stackoverflow.com/questions/
|
||||
10060069/safely-extract-zip-or-tar-using-python
|
||||
|
||||
:param tarfile.TarFile tar_meta: TarFile object
|
||||
:returns tarfile:TarFile fh: TarFile object
|
||||
"""
|
||||
for fh in tar_meta:
|
||||
each_f = os.path.abspath(os.path.join(work_dir, fh.name))
|
||||
if os.path.realpath(each_f).startswith(work_dir):
|
||||
yield fh
|
||||
|
||||
with tarfile.open(tar_file) as tarf:
|
||||
tarf.extractall(members=safe_paths(tarf))
|
||||
untar_dir = os.path.splitext(abs_path)[0]
|
||||
os.remove(abs_path)
|
||||
return untar_dir
|
||||
|
||||
|
||||
@cache
|
||||
def get(uri):
|
||||
"""Entry method for download method
|
||||
|
||||
:param str uri: A formatted remote URL of a file
|
||||
:param str: Absolute path to the downloaded content
|
||||
"""
|
||||
user_base_dir = CONF.remote.cache_dir
|
||||
if user_base_dir:
|
||||
try:
|
||||
temp = tempfile.TemporaryFile(dir=os.path.abspath(user_base_dir))
|
||||
temp.close()
|
||||
except OSError:
|
||||
LOG.error("Failed to write remote files to: {}".format(
|
||||
os.path.abspath(user_base_dir)))
|
||||
exit(1)
|
||||
abs_path = download(uri, os.path.abspath(user_base_dir))
|
||||
else:
|
||||
abs_path = download(uri)
|
||||
return extract_tar(abs_path)
|
||||
|
|
@ -0,0 +1,40 @@
|
|||
# Copyright 2016 Intel
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
import os
|
||||
import tempfile
|
||||
|
||||
import testtools
|
||||
|
||||
from syntribos.utils.config_fixture import ConfFixture
|
||||
from syntribos.utils import remotes
|
||||
|
||||
|
||||
@remotes.cache
|
||||
def fake_method_taking_long_time(name):
|
||||
"""Fake method to check caching."""
|
||||
return 3
|
||||
|
||||
|
||||
class TestRemotes(testtools.TestCase):
|
||||
"""Basic unit test for testing remote methods."""
|
||||
def test_cache(self):
|
||||
self.useFixture(ConfFixture())
|
||||
self.assertEqual(3, fake_method_taking_long_time("fake"))
|
||||
|
||||
def test_extract_tar(self):
|
||||
temp_fh, temp_fn = tempfile.mkstemp()
|
||||
abs_path = os.path.abspath(temp_fn)
|
||||
path = remotes.extract_tar(abs_path)
|
||||
self.assertEqual(abs_path, path)
|
||||
os.remove(path)
|
||||
Loading…
Reference in New Issue