openstack/syntribos
Code Issues Proposed changes

Docstrings for more Syntribos components

Browse Source 
Adding a few more docstrings, populating code-docs page, modifying
Sphinx documentation generation.

Implements: blueprint docstring-add-to-framework
Change-Id: I9506c9fdeab19d9b4bf52ee3a51c27e38476e82e
This commit is contained in:
Charles Neill 2016-04-12 18:21:41 -05:00
parent a61b461594
commit 679cfd5d2f
11 changed files with 181 additions and 73 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

63
doc/source/code-docs.rst
View File

@ -7,20 +7,20 @@ Configuration
This section describes the configuration specified in your configuration file
(second argument to the runner).
.. autoclass:: syntribos.config.MainConfig
.. automodule:: syntribos.config
:members:
:undoc-members:
:show-inheritance:
..
Arguments
---------
.. autoclass:: syntribos.arguments.SyntribosCLI
.. automodule:: syntribos.arguments
:members:
..
Runner
------
.. autoclass:: syntribos.runners.Runner
:undoc-members:
:show-inheritance:
.. automodule:: syntribos.runner
:members:
:undoc-members:
:show-inheritance:
Tests
-----
@ -34,19 +34,26 @@ either directly, or through a subclass like
All tests are aggregated in the `syntribos.tests.base.test_table` variable
.. autoclass:: syntribos.tests.base.TestType
.. automodule:: syntribos.tests.base
:members:
.. autoclass:: syntribos.tests.base.BaseTestCase
:members:
.. autofunction:: syntribos.tests.base.replace_invalid_characters
:undoc-members:
:show-inheritance:
..
.. autoclass:: syntribos.tests.fuzz.base_fuzz.BaseFuzzTestCase
.. automodule:: syntribos.tests.fuzz.base_fuzz
:members:
:undoc-members:
:show-inheritance:
.. automodule:: syntribos.tests.fuzz.config
:members:
:undoc-members:
:show-inheritance:
.. automodule:: syntribos.tests.fuzz.datagen
:members:
:undoc-members:
:show-inheritance:
Issues
------
@ -54,8 +61,10 @@ Issues
This section describes the representation of issues that are uncovered by
Syntribos.
.. autoclass:: syntribos.issue.Issue
.. automodule:: syntribos.issue
:members:
:undoc-members:
:show-inheritance:
Results
-------
@ -63,8 +72,10 @@ Results
This section describes the representation of results (collections of issues)
from a given Syntribos run.
.. autoclass:: syntribos.result.IssueTestResult
.. automodule:: syntribos.result
:members:
:undoc-members:
:show-inheritance:
HTTP Requests
-------------
@ -72,13 +83,17 @@ HTTP Requests
This section describes the components related to generating, fuzzing, and making
HTTP requests.
.. autoclass:: syntribos.clients.http.parser.RequestCreator
.. automodule:: syntribos.clients.http.client
:members:
:private-members:
:undoc-members:
:show-inheritance:
.. autoclass:: syntribos.clients.http.models.RequestObject
.. automodule:: syntribos.clients.http.models
:members:
:undoc-members:
:show-inheritance:
.. autoclass:: syntribos.clients.http.models.RequestHelperMixin
.. automodule:: syntribos.clients.http.parser
:members:
:private-members:
:undoc-members:
:show-inheritance:

12
doc/source/conf.py
View File

@ -16,7 +16,6 @@ import os
import sys
sys.path.insert(0, os.path.abspath('../../'))
sys.path.insert(0, os.path.abspath('../../syntribos'))
# -- General configuration ----------------------------------------------------
@ -76,9 +75,10 @@ latex_documents = [
# Example configuration for intersphinx: refer to the Python standard library.
# intersphinx_mapping = {'http://docs.python.org/': None}
intersphinx_mapping = {'cafe': ('http://opencafe.readthedocs.org/en/latest', None)}
autodoc_mock_imports = [
'cafe',
'cafe.engine.http.client',
'cafe.drivers.unittest.arguments'
]
# autodoc_mock_imports = [
# 'cafe',
# 'cafe.engine.http.client',
# 'cafe.drivers.unittest.arguments'
# ]

28
doc/source/configuration.rst
View File

@ -1,14 +1,17 @@
Configuration
=============
Copy the data files from Syntribos data directory to .opencafe/data directory created during "cafe-config init". This directory contains the fuzz string files. Copy the example configuration file to .opencafe/configs directory created during "cafe-config init".
Copy the data files from Syntribos data directory to ``.opencafe/data``
directory created during ``cafe-config init``. This directory contains the fuzz
string files. Next, copy the example configuration file to the
``.opencafe/configs`` directory.
::
$ cp data/* .opencafe/data/
$ cp examples/configs/keystone.config .opencafe/configs/.
$ cp examples/configs/keystone.config .opencafe/configs/
Modify the configuration files to update your keystone URL, API endpoint
Modify the configuration files to update your Keystone URL, API endpoint
and user credentials.
::
@ -18,18 +21,29 @@ and user credentials.
Example configuration file:
::
[syntribos]
endpoint=<yourapiendpoint>
# End point URLs and versions of the services to be tested.
endpoint=http://localhost:5000
# Optional, api version if required.
# Used for cross auth tests (-t AUTH_WITH_SOMEONE_ELSE_TOKEN)
# version=v2
[user]
username=<yourusername>
password=<yourpassword>
user_id=<youruserid>
# Optional, if username is provided
# user_id=<youruserid>
[alt_user]
# Used for cross auth tests (-t AUTH_WITH_SOMEONE_ELSE_TOKEN)
username=<alt_username>
password=<alt_password>
user_id=<alt_userid>
[auth]
endpoint=<yourkeystoneurl>
# Config for authorization enpoint, so that the service can
# obtain a valid token, enter your keystone auth endpoint.
endpoint=http://localhost:5000
You can create a directory to store the request templates for the resources
being tested. The templates under the `examples` directory can give you a quick

1
doc/source/index.rst
View File

@ -34,6 +34,7 @@ For Developers
.. toctree::
:maxdepth: 1
structure
contributing
code-docs
unittests

20
doc/source/structure.rst Normal file
View File

@ -0,0 +1,20 @@
Project Structure
=================
- ``data/`` (textfiles containing data for use by Syntribos tests)
- ``doc/source/`` (Sphinx documentation files)
- ``examples/`` (example Syntribos request templates, config files)
- ``configs/`` (examples of Syntribos configs; currently only Keystone)
- ``templates/`` (examples of request templates; currently only Keystone/Solum)
- ``scripts/`` (?)
- ``syntribos/`` (core Syntribos code)
- ``clients/`` (clients for making calls, e.g. HTTP)
- ``http/`` (clients for making HTTP requests)
- ``extensions/`` (extensions that can be called in request templates)
- ``identity/`` (extension for interacting with Keystone/identity)
- ``random_data/`` (extension for generating random test data)
- ``formatters/`` (output formatters, e.g. JSON, XML/XUnit)
- ``tests/`` (location of tests that Syntribos can run against a target)
- ``auth/`` (tests related to authentication/authorization)
- ``fuzz/`` (tests that "fuzz" API requests)
- ``tests/unit/`` (unittests for testing Syntribos itself)

14
doc/source/test.anatomy.rst
View File

@ -1,4 +1,4 @@
Basic syntribos test anatomy
Basic Syntribos Test Anatomy
============================
**Test Types**
@ -33,7 +33,13 @@ For all tests against HTTP headers only, use:
Syntribos template files can be supplemented with variable data, or data
retrieved from external sources. This is handled using 'extensions.'
Extensions are found in ``syntribos/syntribos/extensions/`` .
Extensions are found in ``syntribos/extensions/`` .
Calls to extensions are made in this form:
::
CALL_EXTERNAL|{extension dot path}:{function}:{arguments}
One example packaged with Syntribos enables the tester to obtain an auth
token from keystone/identity. The code is located in
@ -54,7 +60,7 @@ usernames when fuzzing a create user call.
::
"username": "CALL_EXTERNAL|syntribos.extensions.random_data.client:get_uuid:[]|",
"username": "CALL_EXTERNAL|syntribos.extensions.random_data.client:get_uuid:[]|"
The extension function can return one value or be used as a generator if
you want it to change for each test.
@ -68,6 +74,6 @@ follows:
::
"ACTION_FIELD:id": "1a16f348-c8d5-42ec-a474-b1cdf78cf40f",
"ACTION_FIELD:id": "1a16f348-c8d5-42ec-a474-b1cdf78cf40f"
The ID provided will remain static for every test.

2
syntribos/config.py
View File

@ -27,5 +27,5 @@ class MainConfig(data_interfaces.ConfigSectionInterface):
@property
def version(self):
"""???"""
"""Used for base_auth test."""
return self.get("version")

8
syntribos/issue.py
View File

@ -30,15 +30,15 @@ class Issue(object):
:ivar response: The response object returned after sending the request
:ivar target: A hostname/IP/etc. to be tested
:ivar path: A specific REST API method, i.e. a URL path associated with a
Target
Target.
:ivar test_type: The type of vulnerability that is being tested for. This
is not necessarily the same as the Defect Type, which may be something
like 500 error or DoS.
:ivar impacted_parameter: For fuzz tests only, a
:class:`syntribos.tests.fuzz.base_fuzz.ImpactedParameter` that holds
data about what part of the request was affected by the fuzz test
data about what part of the request was affected by the fuzz test.
"""
def __init__(self, test, severity, text, confidence,
request=None, response=None, impacted_parameter=None):
self.defect_type = test
@ -86,7 +86,6 @@ class Issue(object):
"""Convert the request object to a dict of values for outputting.
:param req: The request object
:type req: (TODO)
:rtype: `dict`
:returns: dictionary of HTTP request data
"""
@ -102,7 +101,6 @@ class Issue(object):
"""Convert the response object to a dict of values for outputting.
:param res: The result object
:type res: (TODO)
:rtype: `dict`
:returns: dictionary of HTTP response data
"""

26
syntribos/tests/base.py
View File

@ -79,10 +79,7 @@ class BaseTestCase(cafe.drivers.unittest.fixtures.BaseTestFixture):
@classmethod
def get_test_cases(cls, filename, file_content):
"""Not sure what the point of this is.
TODO: FIGURE THIS OUT
"""
"""Returns tests for given TestCase class (overwritten by children)."""
yield cls
@classmethod
@ -93,6 +90,8 @@ class BaseTestCase(cafe.drivers.unittest.fixtures.BaseTestFixture):
read in by the test runner as the master list of tests to be run.
:param str new_name: Name of new class to be created
:param str fuzz_string: Fuzz string to insert
:param str param_path: String tracing location of the ImpactedParameter
:param dict kwargs: Keyword arguments to pass to the new class
:rtype: class
:returns: A TestCase class extending :class:`BaseTestCase`
@ -108,11 +107,23 @@ class BaseTestCase(cafe.drivers.unittest.fixtures.BaseTestFixture):
return new_cls
def run_test(self):
"""This kicks off the test(s) for a given TestCase class
After running the tests, an `AssertionError` is raised if any tests
were added to self.failures.
:raises: :exc:`AssertionError`
"""
self.test_case()
if self.failures:
raise AssertionError
def test_case(self):
"""This method is overwritten by individual TestCase classes
It represents the actual test that is called in :func:`run_test`,
and handles populating `self.failures`
"""
pass
def register_issue(self, issue):
@ -121,9 +132,10 @@ class BaseTestCase(cafe.drivers.unittest.fixtures.BaseTestFixture):
Registers a :class:`syntribos.issue.Issue` object as a failure and
associates the test's metadata to it.
:param Issue issue: issue object to update
:param issue: issue object to update
:type issue: :class:`syntribos.issue.Issue`
:returns: new issue object with metadata associated
:rtype: Issue
:rtype: :class:`syntribos.issue.Issue`
"""
# Still associating request and response objects with issue in event of
@ -142,7 +154,7 @@ class BaseTestCase(cafe.drivers.unittest.fixtures.BaseTestFixture):
return issue
def test_issues(self):
"""Run assertions for each test registered in test_case."""
"""(DEPRECATED) Run assertions for each test in test_case."""
for issue in self.issues:
try:
issue.run_tests()

2
syntribos/tests/fuzz/base_fuzz.py
View File

@ -21,7 +21,7 @@ from syntribos.tests import base
import syntribos.tests.fuzz.config
import syntribos.tests.fuzz.datagen
data_dir = os.environ.get("CAFE_DATA_DIR_PATH")
data_dir = os.environ.get("CAFE_DATA_DIR_PATH", "")
class BaseFuzzTestCase(base.BaseTestCase):

78
syntribos/tests/fuzz/datagen.py
View File

@ -20,11 +20,13 @@ from syntribos.clients.http import parser
class FuzzMixin(object):
"""FuzzMixin Class
FuzzBehavior provides the fuzz_data function which yields a test name
and all iterations of a given piece of data (currently supports dict,
ElementTree.Element, and basestring formats) with each string provided.
"""Mixin for fuzz tests
This class provides the :meth:`~.FuzzMixin._fuzz_data` function which
yields a test name and all iterations of a given piece of data (currently
supports `dict`, :class:`xml.etree.ElementTree.Element`, and `basestring`
formats) with each string provided.
"""
@classmethod
def _fuzz_data(cls, strings, data, skip_var, name_prefix):
@ -33,6 +35,11 @@ class FuzzMixin(object):
For each attribute in the model object, call the _build_combinations
method corresponding to the type of the data parameter, which replaces
the value with the fuzz string.
:param strings:
:param data:
:param skip_var:
:param name_prefix:
"""
param_path = ""
for str_num, stri in enumerate(strings, 1):
@ -50,14 +57,18 @@ class FuzzMixin(object):
yield (name, model, stri, param_path)
@classmethod
def _build_str_combinations(cls, string, data):
"""Places fuzz string in fuzz location for string data."""
def _build_str_combinations(cls, fuzz_string, data):
"""Places `fuzz_string` in fuzz location for string data.
:param str fuzz_string: Value to place in fuzz location
:param str data: Lines from the request template
"""
for match in re.finditer(r"{([\w]*):?([^}]*)}", data):
# Match either "{identifier:value}" or "{value}"
start, stop = match.span()
model = "{0}{1}{2}".format(
cls.remove_braces(data[:start]),
string, cls.remove_braces(data[stop:]))
fuzz_string, cls.remove_braces(data[stop:]))
if match.group(1):
# The string is of the format "{identifier:value}", so we just
# want the identifier as the param_path
@ -67,7 +78,12 @@ class FuzzMixin(object):
@classmethod
def _build_combinations(cls, stri, dic, skip_var):
"""Places fuzz string in fuzz location for object data."""
"""Places fuzz string in fuzz location for object data.
:param stri:
:param dic:
:param skip_var:
"""
for key, val in dic.iteritems():
if skip_var in key:
continue
@ -94,10 +110,15 @@ class FuzzMixin(object):
@staticmethod
def _merge_dictionaries(x, y):
"""merge the dictionaries
"""Merge `dicts` together
Uses the copy function to create a merged dictionary without squashing
the passed in objects
Create a copy of `x`, and update that with elements of `y`, to prevent
squashing of passed in dicts.
:param dict x: Dictionary 1
:param dict y: Dictionary 2
:returns: Merged dictionary
:rtype: `dict`
"""
z = x.copy()
@ -123,20 +144,28 @@ class FuzzMixin(object):
"{0}/{1}".format(ele.tag, param_path))
@staticmethod
def _update_element(ele, stri):
"""update element
def _update_element(ele, text):
"""Copies an XML element, updates its text attribute with `text`
Returns a copy of the element with the element text replaced by stri
:param ele: XML element to be copied, modified
:type ele: :class:`xml.ElementTree.Element`
:param str text: Text to populate `ele`'s text attribute with
:returns: XML element with "text" attribute set to `text`
:rtype: :class:`xml.ElementTree.Element`
"""
ret = ele.copy()
ret.text = stri
ret.text = text
return ret
@staticmethod
def _update_attribs(ele, attribs):
"""update attributes
"""Copies an XML element, populates attributes from `attribs`
Returns a copy of the element with the attributes replaced by attribs
:param ele: XML element to be copied, modified
:type ele: :class:`xml.ElementTree.Element`
:param dict attribs: Source of new attribute values for `ele`
:returns: XML element with all attributes overwritten by `attribs`
:rtype: :class:`xml.ElementTree.Element`
"""
ret = ele.copy()
ret.attrib = attribs
@ -144,9 +173,14 @@ class FuzzMixin(object):
@staticmethod
def _update_inner_element(ele, list_):
"""Update inner element
"""Copies an XML element, populates sub-elements from `list_`
Returns a copy of the element with the subelements given via list_
:param ele: XML element to be copied, modified
:type ele: :class:`xml.ElementTree.Element`
:param list list_: List of subelements to append to `ele`
:returns: XML element with new subelements from `list_`
:rtype: :class:`xml.ElementTree.Element`
"""
ret = ele.copy()
for i, v in enumerate(list_):
@ -155,6 +189,7 @@ class FuzzMixin(object):
@staticmethod
def remove_braces(string):
"""Remove braces from strings (in request templates)."""
return string.replace("}", "").replace("{", "")
@staticmethod
@ -175,6 +210,13 @@ class FuzzRequest(RequestObject, FuzzMixin, RequestHelperMixin):
Gets the name and the fuzzed request model from _fuzz_data, and
creates a request object from the parameters of the model.
:param strings:
:param fuzz_type:
:param name_prefix:
:returns: Generator of tuples:
(name, request, fuzzstring, ImpactedParameter name)
:rtype: `tuple`
"""
for name, data, stri, param_path in self._fuzz_data(
strings, getattr(self, fuzz_type), self.action_field,