Bug fixes and error message updates

Fixes:
1) Crashes in runner and file_utils
2) Binary strings being read in as payloads

Updates:
1) Clarified error messages in parser
2) Confusing variable names in test cases vs issues

Adds:
1) A `syntribos root` CLI sub command to display the current syntribos root dir

Change-Id: I22edf7a1f3d39724522aee88d08b00d299b67248

.mailmap

@@ -1,6 +0,0 @@
-# Format is:
-# <preferred e-mail> <other e-mail 1>
-# <preferred e-mail> <other e-mail 2>
-<michael.xin@rackspace.com> <jqxin2006@gmail.com>
-Nathan Buckner <nathan.buckner@rackspace.com> bucknerns <nathan.buckner@rackspace.com>
-

README.rst

@@ -27,29 +27,6 @@ Team and repository tags
 Syntribos, An Automated API Security Testing Tool
 =================================================
 
-::
-
-                      syntribos
-                       xxxxxxx
-                  x xxxxxxxxxxxxx x
-               x     xxxxxxxxxxx     x
-                      xxxxxxxxx
-            x          xxxxxxx          x
-                        xxxxx
-           x             xxx             x
-                          x
-          xxxxxxxxxxxxxxx   xxxxxxxxxxxxxxx
-           xxxxxxxxxxxxx     xxxxxxxxxxxxx
-            xxxxxxxxxxx       xxxxxxxxxxx
-             xxxxxxxxx         xxxxxxxxx
-               xxxxxx           xxxxxx
-                 xxx             xxx
-                     x         x
-                          x
-             === Automated API Scanning  ===
-
-
-
 Syntribos is an open source automated API security testing tool that is
 maintained by members of the `OpenStack Security Project <https://wiki.openstack.org/wiki/Security>`_.
 
@@ -182,7 +159,7 @@ User defined Test
 
 This test gives users the ability to fuzz using user defined fuzz data and
 provides an option to look for failure strings provided by the user. The fuzz
-data needs to be provided using the config option :option:`[user_defined]`.
+data needs to be provided using the config option `[user_defined]`.
 
 Example::
 
@@ -281,6 +258,9 @@ environment, you can specify the ``--force`` flag to overwrite existing files.
 The ``--custom_install_root`` and ``--force`` flags can be combined to
 overwrite files in a custom install root.
 
+Note: if you install syntribos to a custom install root, you must supply the
+``--custom_install_root`` flag when running syntribos.
+
 **Example:**
 
 ::
@@ -516,8 +496,14 @@ using syntribos:
 Running syntribos
 =================
 
+By default, syntribos looks in the syntribos home directory (the directory
+specified when running the ``syntribos init`` command on install) for config
+files, payloads, and templates. This can all be overridden through command
+line options. For a full list of command line options available, run
+``syntribos --help`` from the command line.
+
 To run syntribos against all the available tests, specify the
-command ``syntribos`` with the configuration file without
+command ``syntribos``, with the configuration file (if needed), without
 specifying any test type.
 
 ::
@@ -563,6 +549,9 @@ There are two types of logs generated by syntribos:
 Results Log
 ~~~~~~~~~~~
 
+The results log is displayed at the end of every syntribos run, it can be
+written to a file by using the ``-o`` flag on the command line.
+
 The results log includes failures and errors. The ``"failures"`` key represents
 tests that have failed, indicating a possible security vulnerability. The
 ``"errors"`` key gives us information on any unhandled exceptions, such as
@@ -726,6 +715,25 @@ This section describes how to write templates and how to run specific tests.
 Templates are input files which have raw HTTP requests and may be
 supplemented with variable data using extensions.
 
+In general, a request template is a marked-up raw HTTP request. It's possible
+for you to test your application by using raw HTTP requests as your request
+templates, but syntribos allows you to mark-up your request templates for
+further functionality.
+
+A request template looks something like this:
+
+::
+
+    POST /users/{user1} HTTP/1.1
+    Content-Type: application/json
+    X-Auth-Token: CALL_EXTERNAL|syntribos.extensions.vAPI.client:get_token:[]|
+
+    {"newpassword": "qwerty123"}
+
+For fuzz tests, syntribos will automatically detect URL parameters, headers,
+and body content as fields to fuzz. It will not automatically detect URL path
+elements as fuzz fields, but they can be specified with curly braces ``{}``.
+
 Note: The name of a template file must end with the extension ``.template``
 Otherwise, syntribos will skip the file and will not attempt to parse any files
 that do not adhere to this naming scheme.
@@ -963,7 +971,7 @@ XML external entity, reflected cross-site scripting,
 Cross Origin Resource Sharing (CORS), SSL, Regex Denial of Service,
 JSON Parser Depth Limit, and User defined.
 
-In order to run a specific test, use the :option:`-t, --test-types`
+In order to run a specific test, use the `-t, --test-types`
 option and provide ``syntribos`` with a keyword, or keywords, to match from
 the test files located in ``syntribos/tests/``.
 

doc/source/conf.py

@@ -33,8 +33,8 @@ source_suffix = ".rst"
 master_doc = "index"
 
 # General information about the project.
-project = u"syntribos"
+project = "syntribos"
-copyright = u"2015-present, OpenStack Foundation"
+copyright = "2015-present, OpenStack Foundation"
 
 # If true, "()" will be appended to :func: etc. cross-reference text.
 add_function_parentheses = True
@@ -52,8 +52,8 @@ pygments_style = "sphinx"
 # List of tuples "sourcefile", "target", u"title", u"Authors name", "manual"
 
 man_pages = [("man/syntribos", "syntribos",
-              u"Automated API security testing tool",
+              "Automated API security testing tool",
-              [u"OpenStack Security Group"], 1)]
+              ["OpenStack Security Group"], 1)]
 
 # -- Options for HTML output --------------------------------------------------
 
@@ -70,8 +70,8 @@ htmlhelp_basename = "%sdoc" % project
 # Grouping the document tree into LaTeX files. List of tuples
 # (source start file, target name, title, author, documentclass
 # [howto/manual]).
-latex_documents = [("index", "%s.tex" % project, u"%s Documentation" % project,
+latex_documents = [("index", "%s.tex" % project, "%s Documentation" % project,
-                    u"OpenStack Foundation", "manual"), ]
+                    "OpenStack Foundation", "manual"), ]
 
 # Example configuration for intersphinx: refer to the Python standard library.
 # intersphinx_mapping = {"http://docs.python.org/": None}

examples/templates/example_get.template

@@ -0,0 +1,2 @@
+GET /examples?query=yes HTTP/1.1
+Accept: application/json

examples/templates/example_post.template

@@ -0,0 +1,13 @@
+POST /examples HTTP/1.1
+Accept: application/json
+Content-type: application/json
+
+{   
+    "id": 24601,
+    "name": "myname",
+    "password": "letmein",
+    "params": {
+        "string": "aaa",
+        "array": [1,2,3,4,5]
+    }
+}

requirements.txt

@@ -10,3 +10,4 @@ python-cinderclient>=3.3.0 # Apache-2.0
 python-glanceclient>=2.8.0 # Apache-2.0
 python-neutronclient>=6.7.0 # Apache-2.0
 python-novaclient>=9.1.0 # Apache-2.0
+PyYAML>=3.12 # MIT

syntribos/__init__.py

@@ -12,6 +12,6 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 # pylint: skip-file
-from syntribos.issue import Issue  # flake8: noqa
+from syntribos.issue import Issue  # noqa
-from syntribos.constants import *
+from syntribos.constants import *  # noqa
-from syntribos.result import IssueTestResult  # flake8: noqa
+from syntribos.result import IssueTestResult  # noqa

syntribos/checks/http.py

@@ -59,7 +59,8 @@ def check_fail(exception):
             desc="An unknown exception was raised. Please report this.")
 
     # CONNECTION FAILURES
-    if isinstance(exception, (rex.ProxyError, rex.SSLError)):
+    if isinstance(exception, (rex.ProxyError, rex.SSLError,
+                              rex.ChunkedEncodingError, rex.ConnectionError)):
         tags.update(["CONNECTION_FAIL"])
     # TIMEOUTS
     elif isinstance(exception, (rex.ConnectTimeout, rex.ReadTimeout)):

syntribos/clients/http/parser.py

@@ -85,9 +85,9 @@ class RequestCreator(object):
         """
         if not cls.meta_vars:
             msg = ("Template contains reference to meta variable of the form "
-                   "'|{}|', but no meta.json file is found in the"
+                   "'|{}|', but no valid meta.json file was found in the "
-                   "templates directory. Check your templates and the "
+                   "templates directory. Check that your templates reference "
-                   "documentation on how to resolve this".format(var))
+                   "a meta.json file that is correctly formatted.".format(var))
             raise TemplateParseException(msg)
 
         if var not in cls.meta_vars:
@@ -431,7 +431,6 @@ class RequestHelperMixin(object):
         self.params = ""
         self.data = ""
         self.url = ""
-        self.url = ""
 
     @classmethod
     def _run_iters(cls, data, action_field):
@@ -506,6 +505,8 @@ class RequestHelperMixin(object):
         if data_type == 'json':
             return json.dumps(data)
         elif data_type == 'xml':
+            if isinstance(data, str):
+                return data
             str_data = ElementTree.tostring(data)
             # No way to stop tostring from HTML escaping even if we wanted
             h = html_parser.HTMLParser()

syntribos/config.py

@@ -13,8 +13,6 @@
 # limitations under the License.
 # pylint: skip-file
 import logging
-import sys
-
 from oslo_config import cfg
 
 import syntribos
@@ -51,7 +49,6 @@ def handle_config_exception(exc):
     if msg:
         LOG.warning(msg)
         print(syntribos.SEP)
-        sys.exit(0)
     else:
         LOG.exception(exc)
 
@@ -108,6 +105,8 @@ def sub_commands(sub_parser):
     sub_parser.add_parser("dry_run",
                           help=_("Dry run syntribos with given config"
                                  "options"))
+    sub_parser.add_parser("root",
+                          help=_("Print syntribos root directory"))
 
 
 def list_opts():
@@ -144,6 +143,13 @@ def register_opts():
         OPTS_REGISTERED = True
 
 
+def list_payment_system_opts():
+    return [
+        cfg.StrOpt('ran', default='', help='Rackspace Account Number'),
+        cfg.StrOpt('alt_ran', default='', help='Alternate RAN')
+    ]
+
+
 def list_cli_opts():
     return [
         cfg.SubCommandOpt(name="sub_command",
@@ -158,8 +164,8 @@ def list_cli_opts():
                         default=[""], sample_default=["SQL", "XSS"],
                         help=_("Test types to be excluded from "
                                "current run against the target API")),
-        cfg.BoolOpt("no_colorize", dest="no_colorize", short="ncl",
+        cfg.BoolOpt("colorize", dest="colorize", short="cl",
-                    default=False,
+                    default=True,
                     help=_("Enable color in syntribos terminal output")),
         cfg.StrOpt("outfile", short="o",
                    sample_default="out.json", help=_("File to print "
@@ -174,7 +180,10 @@ def list_cli_opts():
         cfg.StrOpt("min-confidence", dest="min_confidence", short="C",
                    default="LOW", choices=syntribos.RANKING,
                    help=_("Select a minimum confidence for reported "
-                          "defects"))
+                          "defects")),
+        cfg.BoolOpt("stacktrace", dest="stacktrace", default=True,
+                    help=_("Select if Syntribos outputs a stacktrace "
+                           " if an exception is raised")),
     ]
 
 

syntribos/extensions/basic_http/client.py

@@ -22,5 +22,6 @@ CONF = cfg.CONF
 def basic_auth(user_section='user'):
     password = CONF.get(user_section).password or CONF.user.password
     username = CONF.get(user_section).username or CONF.user.username
-    encoded_creds = base64.b64encode("{}:{}".format(username, password))
+    encoded_creds = base64.b64encode(
-    return "Basic %s" % encoded_creds
+        "{}:{}".format(username, password).encode())
+    return "Basic %s" % encoded_creds.decode()

syntribos/extensions/identity/models/base.py

@@ -19,8 +19,6 @@ import xml.etree.ElementTree as ET
 class Namespaces(object):
     XMLNS_XSI = "http://www.w3.org/2001/XMLSchema-instance"
     XMLNS = "http://docs.openstack.org/identity/api/v2.0"
-    XMLNS_KSKEY = "http://docs.rackspace.com/identity/api/ext/RAX-KSKEY/v1.0"
-    XMLNS_RAX_AUTH = "http://docs.rackspace.com/identity/api/ext/RAX-AUTH/v1.0"
 
 
 class BaseIdentityModel(object):

syntribos/result.py

@@ -13,6 +13,7 @@
 # limitations under the License.
 import threading
 import time
+import traceback
 import unittest
 
 from oslo_config import cfg
@@ -20,7 +21,6 @@ from oslo_config import cfg
 import syntribos
 from syntribos._i18n import _
 from syntribos.formatters.json_formatter import JSONFormatter
-from syntribos.runner import Runner
 import syntribos.utils.remotes
 
 CONF = cfg.CONF
@@ -33,6 +33,7 @@ class IssueTestResult(unittest.TextTestResult):
     This class aggregates :class:`syntribos.issue.Issue` objects from all the
     tests as they run
     """
+    raw_issues = []
     output = {"failures": {}, "errors": [], "stats": {}}
     output["stats"]["severity"] = {
         "UNDEFINED": 0,
@@ -88,6 +89,7 @@ class IssueTestResult(unittest.TextTestResult):
         """
         lock.acquire()
         for issue in test.failures:
+            self.raw_issues.append(issue)
             defect_type = issue.defect_type
             if any([
                     True for x in CONF.syntribos.exclude_results
@@ -212,17 +214,22 @@ class IssueTestResult(unittest.TextTestResult):
         :type tuple: Tuple of format ``(type, value, traceback)``
         """
         with lock:
+            err_str = "{}: {}".format(err[0].__name__, str(err[1]))
             for e in self.errors:
-                if e['error'] == self._exc_info_to_string(err, test):
+                if e['error'] == err_str:
                     if self.getDescription(test) in e['test']:
                         return
                     e['test'].append(self.getDescription(test))
                     self.stats["errors"] += 1
                     return
-            self.errors.append({
+            stacktrace = traceback.format_exception(*err, limit=0)
+            _e = {
                 "test": [self.getDescription(test)],
-                "error": self._exc_info_to_string(err, test)
+                "error": err_str
-            })
+            }
+            if CONF.stacktrace:
+                _e["stacktrace"] = [x.strip() for x in stacktrace]
+            self.errors.append(_e)
             self.stats["errors"] += 1
 
     def addSuccess(self, test):
@@ -237,7 +244,7 @@ class IssueTestResult(unittest.TextTestResult):
     def printErrors(self, output_format):
         """Print out each :class:`syntribos.issue.Issue` that was encountered
 
-        :param str output_format: Either "json" or "xml"
+        :param str output_format: "json"
         """
         self.output["errors"] = self.errors
         self.output["failures"] = self.failures
@@ -250,9 +257,8 @@ class IssueTestResult(unittest.TextTestResult):
         self.printErrors(CONF.output_format)
         self.print_log_path_and_stats(start_time)
 
-    def print_log_path_and_stats(self, start_time):
+    def print_log_path_and_stats(self, start_time, log_path):
         """Print the path to the log folder for this run."""
-        test_log = Runner.log_path
         run_time = time.time() - start_time
         num_fail = self.stats["unique_failures"]
         num_err = self.stats["errors"]
@@ -267,7 +273,7 @@ class IssueTestResult(unittest.TextTestResult):
                   e=num_err,
                   fsuff="s" * bool(num_fail - 1),
                   esuff="s" * bool(num_err - 1)))
-        if test_log:
+        if log_path:
             print(syntribos.SEP)
-            print(_("LOG PATH...: %s") % test_log)
+            print(_("LOG PATH...: %s") % log_path)
             print(syntribos.SEP)

syntribos/runner.py

@@ -13,7 +13,6 @@
 # limitations under the License.
 import json
 import logging
-from multiprocessing.dummy import Pool as ThreadPool
 import os
 import pkgutil
 import sys
@@ -21,21 +20,22 @@ import threading
 import time
 import traceback
 import unittest
+from multiprocessing.dummy import Pool as ThreadPool
 
 from oslo_config import cfg
 from six.moves import input
 
-from syntribos._i18n import _
 import syntribos.config
-from syntribos.formatters.json_formatter import JSONFormatter
 import syntribos.result
 import syntribos.tests as tests
 import syntribos.tests.base
+from syntribos._i18n import _
+from syntribos.formatters.json_formatter import JSONFormatter
 from syntribos.utils import cleanup
 from syntribos.utils import cli as cli
 from syntribos.utils import env as ENV
-from syntribos.utils.file_utils import ContentType
 from syntribos.utils import remotes
+from syntribos.utils.file_utils import ContentType
 
 result = None
 user_base_dir = None
@@ -144,6 +144,10 @@ class Runner(object):
                 CONF(argv, default_config_files=[])
         except Exception as exc:
             syntribos.config.handle_config_exception(exc)
+            if cls.worker:
+                raise exc
+            else:
+                sys.exit(1)
 
     @classmethod
     def setup_runtime_env(cls):
@@ -191,7 +195,7 @@ class Runner(object):
         return meta_vars
 
     @classmethod
-    def run(cls):
+    def run(cls, argv=sys.argv[1:], worker=False):
         """Method sets up logger and decides on Syntribos control flow
 
         This is the method where control flow of Syntribos is decided
@@ -199,26 +203,32 @@ class Runner(object):
         as ```list_tests``` or ```run``` the respective method is called.
         """
         global result
-
+        cls.worker = worker
-        cli.print_symbol()
-
         # If we are initializing, don't look for a default config file
         if "init" in sys.argv:
             cls.setup_config()
         else:
-            cls.setup_config(use_file=True)
+            cls.setup_config(use_file=True, argv=argv)
         try:
             if CONF.sub_command.name == "init":
+                cli.print_symbol()
                 ENV.initialize_syntribos_env()
                 exit(0)
 
             elif CONF.sub_command.name == "list_tests":
+                cli.print_symbol()
                 cls.list_tests()
                 exit(0)
 
             elif CONF.sub_command.name == "download":
+                cli.print_symbol()
                 ENV.download_wrapper()
                 exit(0)
+
+            elif CONF.sub_command.name == "root":
+                print(ENV.get_syntribos_root())
+                exit(0)
+
         except AttributeError:
             print(
                 _(
@@ -248,15 +258,19 @@ class Runner(object):
         print(_("\nRunning Tests...:"))
         templates_dir = CONF.syntribos.templates
         if templates_dir is None:
-            print(_("Attempting to download templates from {}").format(
+            if cls.worker:
-                CONF.remote.templates_uri))
+                raise Exception("No templates directory was found in the "
-            templates_path = remotes.get(CONF.remote.templates_uri)
+                                "config file.")
-            try:
+            else:
-                templates_dir = ContentType("r", 0)(templates_path)
+                print(_("Attempting to download templates from {}").format(
-            except IOError:
+                    CONF.remote.templates_uri))
-                print(_("Not able to open `%s`; please verify path, "
+                templates_path = remotes.get(CONF.remote.templates_uri)
-                        "exiting...") % templates_path)
+                try:
-                exit(1)
+                    templates_dir = ContentType("r")(templates_path)
+                except IOError:
+                    print(_("Not able to open `%s`; please verify path, "
+                            "exiting...") % templates_path)
+                    exit(1)
 
         print(_("\nPress Ctrl-C to pause or exit...\n"))
         meta_vars = None
@@ -267,8 +281,15 @@ class Runner(object):
                 meta_path = os.path.dirname(file_path)
                 try:
                     cls.meta_dir_dict[meta_path] = json.loads(file_content)
-                except Exception:
+                except json.decoder.JSONDecodeError:
-                    print("Unable to parse %s, skipping..." % file_path)
+                    _full_path = os.path.abspath(file_path)
+                    print(syntribos.SEP)
+                    print(
+                        "\n"
+                        "*** The JSON parser raised an exception when parsing "
+                        "{}. Check that the file contains correctly formatted "
+                        "JSON data. *** \n".format(_full_path)
+                    )
         for file_path, req_str in templates_dir:
             if "meta.json" in file_path:
                 continue
@@ -299,7 +320,8 @@ class Runner(object):
                             req_str, dry_run_output, meta_vars)
 
         if CONF.sub_command.name == "run":
-            result.print_result(cls.start_time)
+            result.print_result(cls.start_time, cls.log_path)
+            cls.result = result
             cleanup.delete_temps()
         elif CONF.sub_command.name == "dry_run":
             cls.dry_run_report(dry_run_output)
@@ -337,7 +359,9 @@ class Runner(object):
                 print(_("\nRequest sucessfully generated!\n"))
                 output["successes"].append(file_path)
 
-            test_cases = list(test_class.get_test_cases(file_path, req_str))
+            test_cases = list(
+                test_class.get_test_cases(file_path, req_str, meta_vars)
+            )
             if len(test_cases) > 0:
                 for test in test_cases:
                     if test:
@@ -346,7 +370,9 @@ class Runner(object):
     @classmethod
     def dry_run_report(cls, output):
         """Reports the dry run through a formatter."""
-        formatter_types = {"json": JSONFormatter(result)}
+        formatter_types = {
+            "json": JSONFormatter(result),
+        }
         formatter = formatter_types[CONF.output_format]
         formatter.report(output)
 
@@ -383,7 +409,7 @@ class Runner(object):
                     test_id=cli.colorize(
                         test_class.test_id, color="green"),
                     name=test_name.replace("_", " ").capitalize())
-                if CONF.no_colorize:
+                if not CONF.colorize:
                     result_string = result_string.ljust(55)
                 else:
                     result_string = result_string.ljust(60)
@@ -397,7 +423,7 @@ class Runner(object):
                     LOG.error("Error in parsing template:")
                     break
                 test_cases = list(
-                    test_class.get_test_cases(file_path, req_str))
+                    test_class.get_test_cases(file_path, req_str, meta_vars))
                 total_tests = len(test_cases)
                 if total_tests > 0:
                     log_string = "[{test_id}]  :  {name}".format(

syntribos/tests/auth/auth.py

@@ -24,7 +24,7 @@ CONF = cfg.CONF
 class AuthTestCase(base.BaseTestCase):
     """Test for possible token misuse in keystone."""
     test_name = "AUTH"
-    test_type = "headers"
+    parameter_location = "headers"
 
     @classmethod
     def setUpClass(cls):
@@ -69,7 +69,7 @@ class AuthTestCase(base.BaseTestCase):
             )
 
     @classmethod
-    def get_test_cases(cls, filename, file_content):
+    def get_test_cases(cls, filename, file_content, meta_vars):
         """Generates the test cases
 
         For this particular test, only a single test

syntribos/tests/base.py

@@ -117,7 +117,7 @@ class BaseTestCase(unittest.TestCase):
         pass
 
     @classmethod
-    def get_test_cases(cls, filename, file_content):
+    def get_test_cases(cls, filename, file_content, meta_vars):
         """Returns tests for given TestCase class (overwritten by children)."""
         yield cls
 
@@ -136,6 +136,7 @@ class BaseTestCase(unittest.TestCase):
         cls.init_req = request_obj
         cls.init_resp = None
         cls.init_signals = None
+        cls.template_path = filename
 
     @classmethod
     def send_init_request(cls, filename, file_content, meta_vars):
@@ -189,7 +190,16 @@ class BaseTestCase(unittest.TestCase):
             if "EXCEPTION_RAISED" in cls.test_signals:
                 sig = cls.test_signals.find(
                     tags="EXCEPTION_RAISED")[0]
-                raise sig.data["exception"]
+                exc_name = type(sig.data["exception"]).__name__
+                if ("CONNECTION_FAIL" in sig.tags):
+                    six.raise_from(FatalHTTPError(
+                        "The remote target has forcibly closed the connection "
+                        "with Syntribos and resulted in exception '{}'. This "
+                        "could potentially mean that a fatal error was "
+                        "encountered within the target application or server"
+                        " itself.".format(exc_name)), sig.data["exception"])
+                else:
+                    raise sig.data["exception"]
 
     @classmethod
     def tearDown(cls):
@@ -249,7 +259,8 @@ class BaseTestCase(unittest.TestCase):
 
         issue.request = self.test_req
         issue.response = self.test_resp
-
+        issue.template_path = self.template_path
+        issue.parameter_location = self.parameter_location
         issue.test_type = self.test_name
         url_components = urlparse(self.init_resp.url)
         issue.target = url_components.netloc
@@ -261,3 +272,7 @@ class BaseTestCase(unittest.TestCase):
         self.failures.append(issue)
 
         return issue
+
+
+class FatalHTTPError(Exception):
+    pass

syntribos/tests/debug/dry_run.py

@@ -19,7 +19,7 @@ class DryRunTestCase(base.BaseTestCase):
     """Debug dry run test to run no logic and return no results."""
 
     test_name = "DEBUG_DRY_RUN"
-    test_type = "debug"
+    parameter_location = "debug"
 
     def test_case(self):
         pass

syntribos/tests/fuzz/base_fuzz.py

@@ -49,8 +49,8 @@ class BaseFuzzTestCase(base.BaseTestCase):
                 path = cls.data_key
             else:
                 path = os.path.join(payloads, file_name or cls.data_key)
-            with open(path, "rb") as fp:
+            with open(path, "r") as fp:
-                return str(fp.read()).splitlines()
+                return fp.read().splitlines()
         except (IOError, AttributeError, TypeError) as e:
             LOG.error("Exception raised: {}".format(e))
             print("\nPayload file for test '{}' not readable, "
@@ -124,7 +124,7 @@ class BaseFuzzTestCase(base.BaseTestCase):
         self.run_default_checks()
 
     @classmethod
-    def get_test_cases(cls, filename, file_content):
+    def get_test_cases(cls, filename, file_content, meta_vars):
         """Generates new TestCases for each fuzz string
 
         For each string returned by cls._get_strings(), yield a TestCase class
@@ -143,7 +143,8 @@ class BaseFuzzTestCase(base.BaseTestCase):
                 filename=filename, test_name=cls.test_name)
 
         fr = syntribos.tests.fuzz.datagen.fuzz_request(
-            cls.init_req, cls._get_strings(), cls.test_type, prefix_name)
+            cls.init_req, cls._get_strings(), cls.parameter_location,
+            prefix_name)
         for fuzz_name, request, fuzz_string, param_path in fr:
             yield cls.extend_class(fuzz_name, fuzz_string, param_path,
                                    {"request": request})
@@ -195,6 +196,8 @@ class BaseFuzzTestCase(base.BaseTestCase):
 
         issue.request = self.test_req
         issue.response = self.test_resp
+        issue.template_path = self.template_path
+
         issue.test_type = self.test_name
         url_components = urlparse(self.prepared_init_req.url)
         issue.target = url_components.netloc
@@ -209,7 +212,7 @@ class BaseFuzzTestCase(base.BaseTestCase):
 
         issue.impacted_parameter = ImpactedParameter(
             method=issue.request.method,
-            location=self.test_type,
+            location=self.parameter_location,
             name=self.param_path,
             value=self.fuzz_string)
 

syntribos/tests/fuzz/buffer_overflow.py

@@ -22,7 +22,7 @@ class BufferOverflowBody(base_fuzz.BaseFuzzTestCase):
     """Test for buffer overflow vulnerabilities in HTTP body."""
 
     test_name = "BUFFER_OVERFLOW_BODY"
-    test_type = "data"
+    parameter_location = "data"
     failure_keys = [
         '*** stack smashing detected ***:',
         'Backtrace:',
@@ -70,19 +70,19 @@ class BufferOverflowParams(BufferOverflowBody):
     """Test for buffer overflow vulnerabilities in HTTP params."""
 
     test_name = "BUFFER_OVERFLOW_PARAMS"
-    test_type = "params"
+    parameter_location = "params"
 
 
 class BufferOverflowHeaders(BufferOverflowBody):
     """Test for buffer overflow vulnerabilities in HTTP header."""
 
     test_name = "BUFFER_OVERFLOW_HEADERS"
-    test_type = "headers"
+    parameter_location = "headers"
 
 
 class BufferOverflowURL(BufferOverflowBody):
     """Test for buffer overflow vulnerabilities in HTTP URL."""
 
     test_name = "BUFFER_OVERFLOW_URL"
-    test_type = "url"
+    parameter_location = "url"
     url_var = "FUZZ"

syntribos/tests/fuzz/command_injection.py

@@ -23,7 +23,7 @@ class CommandInjectionBody(base_fuzz.BaseFuzzTestCase):
     """Test for command injection vulnerabilities in HTTP body."""
 
     test_name = "COMMAND_INJECTION_BODY"
-    test_type = "data"
+    parameter_location = "data"
     data_key = "command_injection.txt"
     failure_keys = [
         'uid=',
@@ -63,19 +63,19 @@ class CommandInjectionParams(CommandInjectionBody):
     """Test for command injection vulnerabilities in HTTP params."""
 
     test_name = "COMMAND_INJECTION_PARAMS"
-    test_type = "params"
+    parameter_location = "params"
 
 
 class CommandInjectionHeaders(CommandInjectionBody):
     """Test for command injection vulnerabilities in HTTP header."""
 
     test_name = "COMMAND_INJECTION_HEADERS"
-    test_type = "headers"
+    parameter_location = "headers"
 
 
 class CommandInjectionURL(CommandInjectionBody):
     """Test for command injection vulnerabilities in HTTP URL."""
 
     test_name = "COMMAND_INJECTION_URL"
-    test_type = "url"
+    parameter_location = "url"
     url_var = "FUZZ"

syntribos/tests/fuzz/integer_overflow.py

@@ -21,7 +21,7 @@ class IntOverflowBody(base_fuzz.BaseFuzzTestCase):
     """Test for integer overflow vulnerabilities in HTTP body."""
 
     test_name = "INTEGER_OVERFLOW_BODY"
-    test_type = "data"
+    parameter_location = "data"
     data_key = "integer-overflow.txt"
 
     def test_case(self):
@@ -41,19 +41,19 @@ class IntOverflowParams(IntOverflowBody):
     """Test for integer overflow vulnerabilities in HTTP params."""
 
     test_name = "INTEGER_OVERFLOW_PARAMS"
-    test_type = "params"
+    parameter_location = "params"
 
 
 class IntOverflowHeaders(IntOverflowBody):
     """Test for integer overflow vulnerabilities in HTTP header."""
 
     test_name = "INTEGER_OVERFLOW_HEADERS"
-    test_type = "headers"
+    parameter_location = "headers"
 
 
 class IntOverflowURL(IntOverflowBody):
     """Test for integer overflow vulnerabilities in HTTP URL."""
 
     test_name = "INTEGER_OVERFLOW_URL"
-    test_type = "url"
+    parameter_location = "url"
     url_var = "FUZZ"

syntribos/tests/fuzz/json_depth_overflow.py

@@ -22,7 +22,7 @@ class JSONDepthOverflowBody(base_fuzz.BaseFuzzTestCase):
     """Test for json depth overflow in HTTP body."""
 
     test_name = "JSON_DEPTH_OVERFLOW_BODY"
-    test_type = "data"
+    parameter_location = "data"
     failure_keys = [
         "maximum recursion depth exceeded",
         "RuntimeError",

syntribos/tests/fuzz/ldap.py

@@ -18,7 +18,7 @@ class LDAPInjectionBody(base_fuzz.BaseFuzzTestCase):
     """Test for LDAP injection vulnerabilities in HTTP body."""
 
     test_name = "LDAP_INJECTION_BODY"
-    test_type = "data"
+    parameter_location = "data"
     data_key = "ldap.txt"
 
 
@@ -26,19 +26,19 @@ class LDAPInjectionParams(LDAPInjectionBody):
     """Test for LDAP injection vulnerabilities in HTTP params."""
 
     test_name = "LDAP_INJECTION_PARAMS"
-    test_type = "params"
+    parameter_location = "params"
 
 
 class LDAPInjectionHeaders(LDAPInjectionBody):
     """Test for LDAP injection vulnerabilities in HTTP header."""
 
     test_name = "LDAP_INJECTION_HEADERS"
-    test_type = "headers"
+    parameter_location = "headers"
 
 
 class LDAPInjectionURL(LDAPInjectionBody):
     """Test for LDAP injection vulnerabilities in HTTP URL."""
 
     test_name = "LDAP_INJECTION_URL"
-    test_type = "url"
+    parameter_location = "url"
     url_var = "FUZZ"

syntribos/tests/fuzz/redos.py

@@ -20,7 +20,7 @@ class ReDosBody(base_fuzz.BaseFuzzTestCase):
     """Test for Regex DoS vulnerabilities in HTTP body."""
 
     test_name = "REDOS_BODY"
-    test_type = "data"
+    parameter_location = "data"
     data_key = "redos.txt"
 
     def test_case(self):
@@ -41,19 +41,19 @@ class ReDosParams(ReDosBody):
     """Test for Regex DoS vulnerabilities in HTTP params."""
 
     test_name = "REDOS_PARAMS"
-    test_type = "params"
+    parameter_location = "params"
 
 
 class ReDosHeaders(ReDosBody):
     """Test for Regex DoS vulnerabilities in HTTP header."""
 
     test_name = "REDOS_HEADERS"
-    test_type = "headers"
+    parameter_location = "headers"
 
 
 class ReDosURL(ReDosBody):
     """Test for Regex DoS vulnerabilities in HTTP URL."""
 
     test_name = "REDOS_URL"
-    test_type = "url"
+    parameter_location = "url"
     url_var = "FUZZ"

syntribos/tests/fuzz/sql.py

@@ -22,7 +22,7 @@ class SQLInjectionBody(base_fuzz.BaseFuzzTestCase):
     """Test for SQL injection vulnerabilities in HTTP body."""
 
     test_name = "SQL_INJECTION_BODY"
-    test_type = "data"
+    parameter_location = "data"
     data_key = "sql-injection.txt"
     failure_keys = [
         "SQL syntax", "mysql", "MySqlException (0x", "valid MySQL result",
@@ -65,19 +65,19 @@ class SQLInjectionParams(SQLInjectionBody):
     """Test for SQL injection vulnerabilities in HTTP params."""
 
     test_name = "SQL_INJECTION_PARAMS"
-    test_type = "params"
+    parameter_location = "params"
 
 
 class SQLInjectionHeaders(SQLInjectionBody):
     """Test for SQL injection vulnerabilities in HTTP header."""
 
     test_name = "SQL_INJECTION_HEADERS"
-    test_type = "headers"
+    parameter_location = "headers"
 
 
 class SQLInjectionURL(SQLInjectionBody):
     """Test for SQL injection vulnerabilities in HTTP URL."""
 
     test_name = "SQL_INJECTION_URL"
-    test_type = "url"
+    parameter_location = "url"
     url_var = "FUZZ"

syntribos/tests/fuzz/string_validation.py

@@ -19,7 +19,7 @@ class StringValidationBody(base_fuzz.BaseFuzzTestCase):
     """Test for string validation vulnerabilities in HTTP body."""
 
     test_name = "STRING_VALIDATION_BODY"
-    test_type = "data"
+    parameter_location = "data"
     data_key = "string_validation.txt"
 
 
@@ -27,19 +27,19 @@ class StringValidationParams(StringValidationBody):
     """Test for string validation vulnerabilities in HTTP params."""
 
     test_name = "STRING_VALIDATION_PARAMS"
-    test_type = "params"
+    parameter_location = "params"
 
 
 class StringValidationHeaders(StringValidationBody):
     """Test for string validation vulnerabilities in HTTP header."""
 
     test_name = "STRING_VALIDATION_HEADERS"
-    test_type = "headers"
+    parameter_location = "headers"
 
 
 class StringValidationURL(StringValidationBody):
     """Test for string validation vulnerabilities in HTTP URL."""
 
     test_name = "STRING_VALIDATION_URL"
-    test_type = "url"
+    parameter_location = "url"
     url_var = "FUZZ"

syntribos/tests/fuzz/user_defined.py

@@ -41,7 +41,7 @@ class UserDefinedVulnBody(base_fuzz.BaseFuzzTestCase):
     """Test for user defined vulnerabilities in HTTP body."""
 
     test_name = "USER_DEFINED_VULN_BODY"
-    test_type = "data"
+    parameter_location = "data"
     user_defined_config()
     data_key = CONF.user_defined.payload
     failure_keys = CONF.user_defined.failure_keys
@@ -74,7 +74,7 @@ class UserDefinedVulnBody(base_fuzz.BaseFuzzTestCase):
                                " provided strings.")))
 
     @classmethod
-    def get_test_cases(cls, filename, file_content):
+    def get_test_cases(cls, filename, file_content, meta_vars):
         """Generates test cases if a payload file is provided."""
         conf_var = CONF.user_defined.payload
         if conf_var is None or not os.path.isfile(conf_var):
@@ -85,7 +85,8 @@ class UserDefinedVulnBody(base_fuzz.BaseFuzzTestCase):
             test_name=cls.test_name,
             fuzz_file=cls.data_key)
         fr = syntribos.tests.fuzz.datagen.fuzz_request(
-            cls.init_req, cls._get_strings(), cls.test_type, prefix_name)
+            cls.init_req, cls._get_strings(), cls.parameter_location,
+            prefix_name)
         for fuzz_name, request, fuzz_string, param_path in fr:
             yield cls.extend_class(fuzz_name, fuzz_string, param_path,
                                    {"request": request})
@@ -95,19 +96,19 @@ class UserDefinedVulnParams(UserDefinedVulnBody):
     """Test for user defined vulnerabilities in HTTP params."""
 
     test_name = "USER_DEFINED_VULN_PARAMS"
-    test_type = "params"
+    parameter_location = "params"
 
 
 class UserDefinedVulnHeaders(UserDefinedVulnBody):
     """Test for user defined vulnerabilities in HTTP header."""
 
     test_name = "USER_DEFINED_VULN_HEADERS"
-    test_type = "headers"
+    parameter_location = "headers"
 
 
 class UserDefinedVulnURL(UserDefinedVulnBody):
     """Test for user defined vulnerabilities in HTTP URL."""
 
     test_name = "USER_DEFINED_VULN_URL"
-    test_type = "url"
+    parameter_location = "url"
     url_var = "FUZZ"

syntribos/tests/fuzz/xml_external.py

@@ -27,7 +27,7 @@ class XMLExternalEntityBody(base_fuzz.BaseFuzzTestCase):
     """Test for XML-external-entity injection vulnerabilities in HTTP body."""
 
     test_name = "XML_EXTERNAL_ENTITY_BODY"
-    test_type = "data"
+    parameter_location = "data"
     dtds_data_key = "xml-external.txt"
     failure_keys = [
         'root:',
@@ -41,7 +41,7 @@ class XMLExternalEntityBody(base_fuzz.BaseFuzzTestCase):
         'partition']
 
     @classmethod
-    def get_test_cases(cls, filename, file_content):
+    def get_test_cases(cls, filename, file_content, meta_vars):
         """Makes sure API call supports XML
 
         Overrides parent fuzz test generation, if API method does not support
@@ -49,7 +49,7 @@ class XMLExternalEntityBody(base_fuzz.BaseFuzzTestCase):
         """
         # Send request for different content-types
         request_obj = parser.create_request(
-            file_content, CONF.syntribos.endpoint)
+            file_content, CONF.syntribos.endpoint, meta_vars)
 
         prepared_copy = request_obj.get_prepared_copy()
         prepared_copy.headers['content-type'] = "application/json"
@@ -75,7 +75,7 @@ class XMLExternalEntityBody(base_fuzz.BaseFuzzTestCase):
                 filename=filename, test_name=cls.test_name,
                 fuzz_file=cls.dtds_data_key, d_index=d_num)
             fr = syntribos.tests.fuzz.datagen.fuzz_request(
-                request_obj, ["&xxe;"], cls.test_type, prefix_name)
+                request_obj, ["&xxe;"], cls.parameter_location, prefix_name)
             for fuzz_name, request, fuzz_string, param_path in fr:
                 request.data = "{0}\n{1}".format(dtd, request.data)
                 yield cls.extend_class(fuzz_name, fuzz_string, param_path,

syntribos/tests/fuzz/xss.py

@@ -20,7 +20,7 @@ class XSSBody(base_fuzz.BaseFuzzTestCase):
     """Test for cross-site-scripting vulnerabilities in HTTP body."""
 
     test_name = "XSS_BODY"
-    test_type = "data"
+    parameter_location = "data"
     data_key = "xss.txt"
 
     def test_case(self):

syntribos/tests/headers/cors.py

@@ -29,19 +29,19 @@ class CorsHeader(base.BaseTestCase):
     """Test for CORS wild character vulnerabilities in HTTP header."""
 
     test_name = "CORS_WILDCARD_HEADERS"
-    test_type = "headers"
+    parameter_location = "headers"
     client = client()
     failures = []
 
     @classmethod
-    def get_test_cases(cls, filename, file_content):
+    def get_test_cases(cls, filename, file_content, meta_vars):
-
         request_obj = parser.create_request(
-            file_content, CONF.syntribos.endpoint
+            file_content, CONF.syntribos.endpoint, meta_vars
         )
         prepared_copy = request_obj.get_prepared_copy()
         cls.test_resp, cls.test_signals = cls.client.send_request(
             prepared_copy)
+        cls.test_req = request_obj.get_prepared_copy()
         yield cls
 
     def test_case(self):

syntribos/tests/headers/xst.py

@@ -40,15 +40,15 @@ class XstHeader(base.BaseTestCase):
     """
 
     test_name = "XST_HEADERS"
-    test_type = "headers"
+    parameter_location = "headers"
     client = client()
     failures = []
 
     @classmethod
-    def get_test_cases(cls, filename, file_content):
+    def get_test_cases(cls, filename, file_content, meta_vars):
         xst_header = {"TRACE_THIS": "XST_Vuln"}
         request_obj = parser.create_request(
-            file_content, CONF.syntribos.endpoint, meta_vars=None)
+            file_content, CONF.syntribos.endpoint, meta_vars)
         prepared_copy = request_obj.get_prepared_copy()
         prepared_copy.method = "TRACE"
         prepared_copy.headers.update(xst_header)

syntribos/tests/transport_layer/ssl.py

@@ -22,7 +22,7 @@ class SSLTestCase(base.BaseTestCase):
     """Test if response body contains non-https links."""
 
     test_name = "SSL_ENDPOINT_BODY"
-    test_type = "body"
+    parameter_location = "data"
 
     def test_case(self):
         self.init_signals.register(https_check(self))

syntribos/utils/cli.py

@@ -11,8 +11,8 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-from __future__ import division
+
-from __future__ import unicode_literals
+
 from math import ceil
 import sys
 
@@ -39,7 +39,7 @@ def colorize(string, color="nocolor"):
     colors = dict(list(zip(color_names, list(range(31, 35)))))
     colors["nocolor"] = 0  # No Color
 
-    if CONF.no_colorize:
+    if not CONF.colorize:
         return string
     return "\033[0;{color}m{string}\033[0;m".format(string=string,
                                                     color=colors.setdefault(
@@ -95,10 +95,11 @@ class ProgressBar(object):
 
         :returns: formatted progress bar string
         """
-        bar_width = int(ceil(self.present_level / self.total_len * self.width))
+        bar_width = int(
+            ceil(self.present_level / float(self.total_len) * self.width))
         empty_char = self.empty_char * (self.width - bar_width)
         fill_char = self.fill_char * bar_width
-        percentage = int(self.present_level / self.total_len * 100)
+        percentage = int(self.present_level / float(self.total_len) * 100)
         return "{message}\t\t|{fill_char}{empty_char}|  {percentage} %".format(
             message=self.message, fill_char=fill_char,
             empty_char=empty_char, percentage=percentage)

syntribos/utils/config_fixture.py

@@ -64,7 +64,7 @@ class ConfFixture(config_fixture.Config):
         """config values for CLI options(default group)."""
         # TODO(unrahul): Add mock file path for outfile
         self.conf.set_default("test_types", [""])
-        self.conf.set_default("no_colorize", True)
+        self.conf.set_default("colorize", False)
         self.conf.set_default("output_format", "json")
         self.conf.set_default("min_severity", "LOW")
         self.conf.set_default("min_confidence", "LOW")

syntribos/utils/env.py

@@ -71,10 +71,12 @@ def get_venv_root():
 def get_syntribos_root():
     """This determines the proper path to use as syntribos' root directory."""
     path = ""
-    custom_root = CONF.syntribos.custom_root
+    try:
-
+        custom_root = CONF.syntribos.custom_root
-    if custom_root:
+        if custom_root:
-        return expand_path(custom_root)
+            return expand_path(custom_root)
+    except Exception:
+        pass
 
     home_root = get_user_home_root()
 
@@ -156,10 +158,10 @@ def create_conf_file(created_folders=None, remote_path=None):
             "# syntribos barebones configuration file\n"
             "# You should update this with your desired options!\n"
             "[syntribos]\n"
-            "endpoint=http://127.0.0.1:80\n"
+            "endpoint=http://127.0.0.1:8080\n"
             "payloads={payloads}\n"
             "templates={templates}\n"
-            "{custom_root}\n"
+            "{custom_root}\n\n"
             "[logging]\n"
             "log_dir={logs}\n"
         ).format(

syntribos/utils/file_utils.py

@@ -54,12 +54,16 @@ class ContentType(ExistingPathType):
     def _fetch_from_dir(self, string):
         for path, _, files in os.walk(string):
             for file_ in files:
-                file_path = os.path.join(path, file_)
+                try:
-                if path is not self._root:
+                    file_path = os.path.join(path, file_)
-                    subdir = os.path.relpath(path, self._root)
+                    if path is not self._root:
-                    yield self._fetch_from_file(file_path, subdir)
+                        subdir = os.path.relpath(path, self._root)
-                else:
+                        yield self._fetch_from_file(file_path, subdir)
-                    yield self._fetch_from_file(file_path)
+
+                    else:
+                        yield self._fetch_from_file(file_path)
+                except Exception:
+                    print("Skipped %s" % string)
 
     def _fetch_from_file(self, string, subdir=None):
         # Get the filename here

test-requirements.txt

@@ -1,19 +1,17 @@
 # The order of packages is significant, because pip processes them in the order
 # of appearance. Changing the order has an impact on the overall integration
 # process, which may cause wedges in the gate later.
+pylint<=2.1.0 # GPLv2
 unittest2>=1.1.0 # BSD
 coverage!=4.4,>=4.0 # Apache-2.0
 fixtures>=3.0.0 # Apache-2.0/BSD
-hacking>=1.1.0
+flake8 # MIT
-flake8<2.7.0,>=2.6.0 # MIT
 mock>=2.0.0 # BSD
 python-subunit>=1.0.0 # Apache-2.0/BSD
 testrepository>=0.0.18 # Apache-2.0/BSD
 testscenarios>=0.4 # Apache-2.0/BSD
 testtools>=2.2.0 # MIT
 requests-mock>=1.2.0 # Apache-2.0
-
 sphinx!=1.6.6,!=1.6.7,>=1.6.2 # BSD
 oslosphinx>=4.7.0 # Apache-2.0
 beautifulsoup4>=4.6.0 # MIT
-pylint>=1.5.0 # GPLv2

tests/unit/test_ascii_colors.py

@@ -20,7 +20,7 @@ from syntribos.utils.cli import CONF
 class TestColorize(testtools.TestCase):
 
     def test_colorize(self):
-        CONF.no_colorize = False
+        CONF.colorize = True
         string = "color this string"
         colors = {"red": 31,
                   "green": 32,
@@ -34,6 +34,6 @@ class TestColorize(testtools.TestCase):
                 colorize(string, color))
 
     def test_no_colorize(self):
-        CONF.no_colorize = True
+        CONF.colorize = False
         string = "No color"
         self.assertEqual(string, colorize(string))

tests/unit/test_content_validity.py

@@ -41,7 +41,7 @@ class TestValidContent(testtools.TestCase):
     """Tests valid_content check for both valid and invalid json/xml."""
 
     def test_valid_json(self, m):
-        text = u'{"text": "Sample json"}'
+        text = '{"text": "Sample json"}'
         headers = {"Content-type": "application/json"}
         m.register_uri("GET", "http://example.com", text=text, headers=headers)
         resp = requests.get("http://example.com")
@@ -50,7 +50,7 @@ class TestValidContent(testtools.TestCase):
         self.assertEqual("VALID_JSON", signal.slug)
 
     def test_invalid_json(self, m):
-        text = u'{"text""" "Sample json"}'
+        text = '{"text""" "Sample json"}'
         headers = {"Content-type": "application/json"}
         m.register_uri("GET", "http://example.com", text=text, headers=headers)
         resp = requests.get("http://example.com")
@@ -60,7 +60,7 @@ class TestValidContent(testtools.TestCase):
         self.assertIn("APPLICATION_FAIL", signal.tags)
 
     def test_valid_xml(self, m):
-        text = u"""<note>\n
+        text = """<note>\n
                      <to>Tove</to>\n
                      <from>Jani</from>\n
                      <heading>Reminder</heading>\n
@@ -78,7 +78,7 @@ class TestValidContent(testtools.TestCase):
         self.assertEqual("VALID_XML", signal.slug)
 
     def test_invalid_xml(self, m):
-        text = u"""<xml version=='1.0' encoding==UTF-8'?>
+        text = """<xml version=='1.0' encoding==UTF-8'?>
                      <!DOCTYPE note SYSTEM 'Note.dtd'>
                      <note>
                      <to>Tove</to>

tests/unit/test_http_checks.py

@@ -46,7 +46,7 @@ class HTTPCheckUnittest(testtools.TestCase):
     def _assert_has_tags(self, tags, signals):
         signal = self._get_one_signal(signals, tags=tags)
         self.assertEqual(len(tags), len(signal.tags))
-        list(map(lambda t: self.assertIn(t, signal.tags), tags))
+        list([self.assertIn(t, signal.tags) for t in tags])
 
 
 class HTTPFailureUnittest(HTTPCheckUnittest):
@@ -66,7 +66,7 @@ class HTTPFailureUnittest(HTTPCheckUnittest):
 
     def test_connect_timeout(self):
         signal = http_checks.check_fail(rex.ConnectTimeout())
-        self._assert_has_tags(self.timeout_tags, signal)
+        self._assert_has_tags(self.conn_fail_tags, signal)
         self._assert_has_slug("HTTP_FAIL_CONNECT_TIMEOUT", signal)
 
     def test_invalid_url(self):

tests/unit/test_http_parser.py

@@ -109,13 +109,13 @@ class HTTPParserUnittest(testtools.TestCase):
         string = 'GET /v1/CALL_EXTERNAL|'
         string += 'syntribos.extensions.random_data.client:get_uuid:[]|'
         parsed_string = parser.call_external_functions(string)
-        self.assertRegex(parsed_string, "GET /v1/[a-f0-9]+$")
+        self.assertRegex(parsed_string, r"GET /v1/[a-f0-9]+$")
 
     def test_call_external_uuid_uuid4(self):
         """Tests calling 'uuid.uuid4()' in URL string."""
         string = 'GET /v1/CALL_EXTERNAL|uuid:uuid4:[]|'
         parsed_string = parser.call_external_functions(string)
-        self.assertRegex(parsed_string, "GET /v1/[a-f0-9\-]+$")
+        self.assertRegex(parsed_string, r"GET /v1/[a-f0-9\-]+$")
 
     def test_call_external_invalid_module(self):
         """Tests calling invalid module in URL string."""

tests/unit/test_length.py

@@ -39,7 +39,7 @@ class FakeTestObject(object):
 class TestLength(testtools.TestCase):
     @requests_mock.Mocker()
     def test_percentage_difference(self, m):
-        content = u"""'Traceback (most recent call last):\n',
+        content = """'Traceback (most recent call last):\n',
                 File "<doctest...>", line 10, in <module>\n
                 lumberjack()\n',
                 File "<doctest...>", line 4, in lumberjack\n

tests/unit/test_progress_bar.py

@@ -35,7 +35,7 @@ class TestProgressBar(testtools.TestCase):
     def test_format_bar(self):
         pb = ProgressBar(total_len=5, width=5, fill_char="#", message="Test")
         pb.increment()  # increments progress bar by 1
-        self.assertEqual(u"Test\t\t|#----|  20 %", pb.format_bar())
+        self.assertEqual("Test\t\t|#----|  20 %", pb.format_bar())
 
     def test_print_bar(self):
         pb = ProgressBar(total_len=5, width=5, fill_char="#", message="Test")

tox.ini

@@ -35,7 +35,7 @@ commands = {posargs}
 [flake8]
 # E123, E125 skipped as they are invalid PEP-8.
 show-source = True
-ignore = E123,E125,H303,F403,H104,H302
+ignore = E123,E125,H303,F403,H104,H302,W504,H306
 exclude=.venv,.git,.tox,dist,doc,*lib/python*,*egg,build
 
 [testenv:pylint]