Added hacking file

As part of adding the hacking file, this patch adds some files taken
from nova project that will help with the coding standards validation.

Tox now uses run_pep8 for the pep8 test

Change-Id: I0784390a0b13c9694e358563d43e5442592a785e

HACKING.rst

@@ -0,0 +1,220 @@
+Marconi Style Commandments
+==========================
+
+- Step 1: Read http://www.python.org/dev/peps/pep-0008/
+- Step 2: Read http://www.python.org/dev/peps/pep-0008/ again
+- Step 3: Read on
+
+
+General
+-------
+- Put two newlines between top-level code (funcs, classes, etc)
+- Put one newline between methods in classes and anywhere else
+- Do not write "except:", use "except Exception:" at the very least
+- Include your name with TODOs as in "#TODO(termie)"
+- Do not name anything the same name as a built-in or reserved word
+
+
+Imports
+-------
+- Do not make relative imports
+- Order your imports by the full module path
+- Import of classes is just allowed inside __init__ files.
+- Organize your imports according to the following template
+
+Example::
+
+  {{stdlib imports in human alphabetical order}}
+  \n
+  {{third-party lib imports in human alphabetical order}}
+  \n
+  {{marconi imports in human alphabetical order}}
+  \n
+  \n
+  {{begin your code}}
+
+
+Human Alphabetical Order Examples
+---------------------------------
+Example::
+
+  import logging
+  import time
+  import unittest
+
+  import eventlet
+
+  import marconi.common
+  from marconi import test
+  import marconi.transport
+
+
+More Import Examples
+--------------------
+
+**INCORRECT** ::
+
+    import marconi.transport.wsgi as wsgi
+
+**CORRECT** ::
+
+    from marconi.transport import wsgi
+
+Docstrings
+----------
+
+Docstrings are required for all functions and methods.
+
+Docstrings should ONLY use triple-double-quotes (``"""``)
+
+Single-line docstrings should NEVER have extraneous whitespace
+between enclosing triple-double-quotes.
+
+**INCORRECT** ::
+
+  """ There is some whitespace between the enclosing quotes :( """
+
+**CORRECT** ::
+
+  """There is no whitespace between the enclosing quotes :)"""
+
+Docstrings should document default values for named arguments
+if they're not None
+
+Docstrings that span more than one line should look like this:
+
+Example::
+
+  """
+  Start the docstring on the line following the opening triple-double-quote
+
+  If you are going to describe parameters and return values, use Sphinx, the
+  appropriate syntax is as follows.
+
+  :param foo: the foo parameter
+  :param bar: (Default True) the bar parameter
+  :param foo_long_bar: the foo parameter description is very
+    long so we have to split it in multiple lines in order to
+    keey things ordered
+  :returns: return_type -- description of the return value
+  :returns: description of the return value
+  :raises: AttributeError, KeyError
+  """
+
+**DO NOT** leave an extra newline before the closing triple-double-quote.
+
+
+Dictionaries/Lists
+------------------
+If a dictionary (dict) or list object is longer than 80 characters, its items
+should be split with newlines. Embedded iterables should have their items
+indented. Additionally, the last item in the dictionary should have a trailing
+comma. This increases readability and simplifies future diffs.
+
+Example::
+
+  my_dictionary = {
+      "image": {
+          "name": "Just a Snapshot",
+          "size": 2749573,
+          "properties": {
+               "user_id": 12,
+               "arch": "x86_64",
+          },
+          "things": [
+              "thing_one",
+              "thing_two",
+          ],
+          "status": "ACTIVE",
+      },
+  }
+
+
+Calling Methods
+---------------
+Calls to methods 80 characters or longer should format each argument with
+newlines. This is not a requirement, but a guideline::
+
+    unnecessarily_long_function_name('string one',
+                                     'string two',
+                                     kwarg1=constants.ACTIVE,
+                                     kwarg2=['a', 'b', 'c'])
+
+
+Rather than constructing parameters inline, it is better to break things up::
+
+    list_of_strings = [
+        'what_a_long_string',
+        'not as long',
+    ]
+
+    dict_of_numbers = {
+        'one': 1,
+        'two': 2,
+        'twenty four': 24,
+    }
+
+    object_one.call_a_method('string three',
+                             'string four',
+                             kwarg1=list_of_strings,
+                             kwarg2=dict_of_numbers)
+
+
+Internationalization (i18n) Strings
+-----------------------------------
+In order to support multiple languages, we have a mechanism to support
+automatic translations of exception and log strings.
+
+Example::
+
+    msg = _("An error occurred")
+    raise HTTPBadRequest(explanation=msg)
+
+If you have a variable to place within the string, first internationalize the
+template string then do the replacement.
+
+Example::
+
+    msg = _("Missing parameter: %s") % ("flavor",)
+    LOG.error(msg)
+
+If you have multiple variables to place in the string, use keyword parameters.
+This helps our translators reorder parameters when needed.
+
+Example::
+
+    msg = _("The server with id %(s_id)s has no key %(m_key)s")
+    LOG.error(msg % {"s_id": "1234", "m_key": "imageId"})
+
+
+Creating Unit Tests
+-------------------
+For every any change, unit tests should be created that both test and
+(implicitly) document the usage of said feature. If submitting a patch for a
+bug that had no unit test, a new passing unit test should be added. If a
+submitted bug fix does have a unit test, be sure to add a new one that fails
+without the patch and passes with the patch.
+
+NOTE: 100% coverage is required
+
+openstack-common
+----------------
+
+A number of modules from openstack-common are imported into the project.
+
+These modules are "incubating" in openstack-common and are kept in sync
+with the help of openstack-common's update.py script. See:
+
+  http://wiki.openstack.org/CommonLibrary#Incubation
+
+The copy of the code should never be directly modified here. Please
+always update openstack-common first and then run the script to copy
+the changes across.
+
+
+Logging
+-------
+Use __name__ as the name of your logger and name your module-level logger
+objects 'LOG'::
+
+    LOG = logging.getLogger(__name__)

marconi/__init__.py

@@ -13,7 +13,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from marconi.kernel import Kernel
+from marconi.kernel import Kernel  # NOQA
 import marconi.version
 
 __version__ = marconi.version.version_info.deferred_version_string()

marconi/kernel.py

@@ -13,10 +13,10 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from ConfigParser import SafeConfigParser
+import ConfigParser
 
-import marconi.transport.wsgi as wsgi
+from marconi.storage import reference as storage
-import marconi.storage.reference as reference
+from marconi.transport.wsgi import driver as wsgi
 
 
 class Kernel(object):
@@ -28,12 +28,12 @@ class Kernel(object):
     """
 
     def __init__(self, config_file):
-        # TODO(kgriffs) Error handling
+        #TODO(kgriffs): Error handling
-        cfg = SafeConfigParser()
+        cfg = ConfigParser.SafeConfigParser()
         cfg.read(config_file)
 
-        # TODO(kgriffs) Determine driver types from cfg
+        #TODO(kgriffs): Determine driver types from cfg
-        self.storage = reference.Driver(cfg)
+        self.storage = storage.Driver(cfg)
         self.transport = wsgi.Driver(cfg, self.storage.queue_controller,
                                      self.storage.message_controller,
                                      self.storage.claim_controller)

marconi/storage/__init__.py

@@ -1,3 +1,3 @@
 """Marconi Storage Drivers"""
 
-from .driver_base import DriverBase  # NOQA
+from marconi.storage.driver_base import DriverBase  # NOQA

marconi/storage/driver_base.py

@@ -13,20 +13,20 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from abc import ABCMeta, abstractproperty
+import abc
 
 
 class DriverBase:
-    __metaclass__ = ABCMeta
+    __metaclass__ = abc.ABCMeta
 
-    @abstractproperty
+    @abc.abstractproperty
     def queue_controller(self):
         pass
 
-    @abstractproperty
+    @abc.abstractproperty
     def message_controller(self):
         pass
 
-    @abstractproperty
+    @abc.abstractproperty
     def claim_controller(self):
         pass

marconi/storage/reference/__init__.py

@@ -4,4 +4,4 @@ In-memory reference Storage Driver for Marconi.
 Useful for automated testing and for prototyping storage driver concepts.
 """
 
-from .driver import Driver  # NOQA
+from marconi.storage.reference.driver import Driver  # NOQA

marconi/tests/test_simple.py

@@ -12,16 +12,16 @@
 # License for the specific language governing permissions and limitations
 # under the License.
 
-import marconi
+from marconi import kernel
-from marconi.tests import util
+from marconi.tests.util import suite
 
 
-class TestSimple(util.TestSuite):
+class TestSimple(suite.TestSuite):
 
     def test_simple(self):
-        """Doesn't really test much"""
+        """Doesn't really test much."""
         conf_file = self.conf_path('wsgi_reference.conf')
-        kernel = marconi.Kernel(conf_file)
+        k = kernel.Kernel(conf_file)
-        transport = kernel.transport
+        transport = k.transport
         wsgi_app = transport.app
         self.assertTrue(True)

marconi/tests/util/__init__.py

@@ -1,3 +1 @@
 """Test utilities"""
-
-from .test_suite import TestSuite  # NOQA

marconi/tests/util/suite.py

@@ -13,8 +13,8 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+import os
 import testtools
-import os.path
 
 
 class TestSuite(testtools.TestCase):
@@ -27,7 +27,7 @@ class TestSuite(testtools.TestCase):
     """
 
     def setUp(self):
-        """Initializer, unittest-style"""
+        """Initializer, unittest-style."""
 
         super(TestSuite, self).setUp()
 

marconi/transport/__init__.py

@@ -1,3 +1,3 @@
 """Marconi Transport Drivers"""
 
-from .driver_base import DriverBase  # NOQA
+from marconi.transport.driver_base import DriverBase  # NOQA

marconi/transport/driver_base.py

@@ -13,13 +13,14 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from abc import ABCMeta, abstractmethod
+import abc
 
 
 class DriverBase:
-    __metaclass__ = ABCMeta
+    __metaclass__ = abc.ABCMeta
 
-    @abstractmethod
+    @abc.abstractmethod
     def listen():
-        #TODO(kgriffs): If this is all there is to DriverBase, do we need it?
+        #TODO(kgriffs): If this is all there is to DriverBase, do we
+        # even need it?
         pass

marconi/transport/wsgi/__init__.py

@@ -1,3 +1,3 @@
 """WSGI Transport Driver"""
 
-from .driver import Driver  # NOQA
+from marconi.transport.wsgi.driver import Driver  # NOQA

marconi/transport/wsgi/driver.py

@@ -13,7 +13,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import marconi.transport as transport
+from marconi import transport
 
 
 class Driver(transport.DriverBase):
@@ -38,5 +38,5 @@ class Driver(transport.DriverBase):
         pass
 
     def app(self, env, start_response, exc_info=None):
-        """This will be replace by falcon.API()"""
+        """This will be replace by falcon.API()."""
         pass

setup.py

@@ -14,13 +14,14 @@
 # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
 # License for the specific language governing permissions and limitations
 # under the License.
-from setuptools import setup, find_packages
+import setuptools
+
 from marconi.openstack.common import setup as common_setup
 
 requires = common_setup.parse_requirements()
 dependency_links = common_setup.parse_dependency_links()
 
-setup(
+setuptools.setup(
     name='marconi',
     version=common_setup.get_post_version('marconi'),
     description='Message Bus for OpenStack',
@@ -28,7 +29,7 @@ setup(
     author='Kurt Griffiths',
     author_email='kurt.griffiths@rackspace.com',
     url='https://launchpad.net/marconi',
-    packages=find_packages(exclude=['bin']),
+    packages=setuptools.find_packages(exclude=['bin']),
     include_package_data=True,
     test_suite='nose.collector',
     install_requires=requires,

tools/flakes.py

@@ -0,0 +1,15 @@
+"""
+ wrapper for pyflakes to ignore gettext based warning:
+     "undefined name '_'"
+
+ Synced in from openstack-common
+"""
+import sys
+
+import pyflakes.checker
+from pyflakes.scripts import pyflakes
+
+if __name__ == "__main__":
+    orig_builtins = set(pyflakes.checker._MAGIC_GLOBALS)
+    pyflakes.checker._MAGIC_GLOBALS = orig_builtins | set(['_'])
+    sys.exit(pyflakes.main())

tools/hacking.py

@@ -0,0 +1,669 @@
+#!/usr/bin/env python
+# vim: tabstop=4 shiftwidth=4 softtabstop=4
+
+# Copyright (c) 2012, Cloudscaling
+# All Rights Reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+"""nova HACKING file compliance testing
+
+Built on top of pep8.py
+"""
+
+import imp
+import inspect
+import logging
+import os
+import re
+import subprocess
+import sys
+import tokenize
+import traceback
+
+import pep8
+
+# Don't need this for testing
+logging.disable('LOG')
+
+#N1xx comments
+#N2xx except
+#N3xx imports
+#N4xx docstrings
+#N5xx dictionaries/lists
+#N6xx calling methods
+#N7xx localization
+#N8xx git commit messages
+#N9xx other
+
+IMPORT_EXCEPTIONS = ['sqlalchemy', 'migrate', 'nova.db.sqlalchemy.session',
+                     'nova.db.sqlalchemy.migration.versioning_api']
+# Paste is missing a __init__ in top level directory
+START_DOCSTRING_TRIPLE = ['u"""', 'r"""', '"""', "u'''", "r'''", "'''"]
+END_DOCSTRING_TRIPLE = ['"""', "'''"]
+VERBOSE_MISSING_IMPORT = os.getenv('HACKING_VERBOSE_MISSING_IMPORT', 'False')
+
+_missingImport = set([])
+
+
+# Monkey patch broken excluded filter in pep8
+# See https://github.com/jcrocholl/pep8/pull/111
+def excluded(self, filename):
+    """Check if options.exclude contains a pattern that matches filename."""
+    basename = os.path.basename(filename)
+    return any((pep8.filename_match(filename, self.options.exclude,
+                               default=False),
+                pep8.filename_match(basename, self.options.exclude,
+                               default=False)))
+
+
+def input_dir(self, dirname):
+    """Check all files in this directory and all subdirectories."""
+    dirname = dirname.rstrip('/')
+    if self.excluded(dirname):
+        return 0
+    counters = self.options.report.counters
+    verbose = self.options.verbose
+    filepatterns = self.options.filename
+    runner = self.runner
+    for root, dirs, files in os.walk(dirname):
+        if verbose:
+            print('directory ' + root)
+        counters['directories'] += 1
+        for subdir in sorted(dirs):
+            if self.excluded(os.path.join(root, subdir)):
+                dirs.remove(subdir)
+        for filename in sorted(files):
+            # contain a pattern that matches?
+            if ((pep8.filename_match(filename, filepatterns) and
+                 not self.excluded(filename))):
+                runner(os.path.join(root, filename))
+
+
+def is_import_exception(mod):
+    return (mod in IMPORT_EXCEPTIONS or
+            any(mod.startswith(m + '.') for m in IMPORT_EXCEPTIONS))
+
+
+def import_normalize(line):
+    # convert "from x import y" to "import x.y"
+    # handle "from x import y as z" to "import x.y as z"
+    split_line = line.split()
+    if ("import" in line and line.startswith("from ") and "," not in line and
+           split_line[2] == "import" and split_line[3] != "*" and
+           split_line[1] != "__future__" and
+           (len(split_line) == 4 or
+           (len(split_line) == 6 and split_line[4] == "as"))):
+        return "import %s.%s" % (split_line[1], split_line[3])
+    else:
+        return line
+
+
+def nova_todo_format(physical_line, tokens):
+    """Check for 'TODO()'.
+
+    nova HACKING guide recommendation for TODO:
+    Include your name with TODOs as in "#TODO(termie)"
+
+    Okay: #TODO(sdague)
+    N101: #TODO fail
+    N101: #TODO (jogo) fail
+    """
+    # TODO(sdague): TODO check shouldn't fail inside of space
+    pos = physical_line.find('TODO')
+    pos1 = physical_line.find('TODO(')
+    pos2 = physical_line.find('#')  # make sure it's a comment
+    if (pos != pos1 and pos2 >= 0 and pos2 < pos and len(tokens) == 0):
+        return pos, "N101: Use TODO(NAME)"
+
+
+def nova_except_format(logical_line):
+    r"""Check for 'except:'.
+
+    nova HACKING guide recommends not using except:
+    Do not write "except:", use "except Exception:" at the very least
+
+    Okay: except Exception:
+    N201: except:
+    """
+    if logical_line.startswith("except:"):
+        yield 6, "N201: no 'except:' at least use 'except Exception:'"
+
+
+def nova_except_format_assert(logical_line):
+    r"""Check for 'assertRaises(Exception'.
+
+    nova HACKING guide recommends not using assertRaises(Exception...):
+    Do not use overly broad Exception type
+
+    Okay: self.assertRaises(NovaException)
+    N202: self.assertRaises(Exception)
+    """
+    if logical_line.startswith("self.assertRaises(Exception"):
+        yield 1, "N202: assertRaises Exception too broad"
+
+
+modules_cache = dict((mod, True) for mod in tuple(sys.modules.keys())
+                     + sys.builtin_module_names)
+
+RE_RELATIVE_IMPORT = re.compile('^from\s*[.]')
+
+
+def nova_import_rules(logical_line):
+    r"""Check for imports.
+
+    nova HACKING guide recommends one import per line:
+    Do not import more than one module per line
+
+    Examples:
+    Okay: from nova.compute import api
+    N301: from nova.compute import api, utils
+
+
+    Imports should usually be on separate lines.
+
+    nova HACKING guide recommends importing only modules:
+    Do not import objects, only modules
+
+    Examples:
+    Okay: from os import path
+    Okay: from os import path as p
+    Okay: from os import (path as p)
+    Okay: import os.path
+    Okay: from nova.compute import rpcapi
+    N302: from os.path import dirname as dirname2
+    N302: from os.path import (dirname as dirname2)
+    N303: from os.path import *
+    N304: from .compute import rpcapi
+    """
+    #NOTE(afazekas): An old style relative import example will not be able to
+    # pass the doctest, since the relativity depends on the file's locality
+
+    def is_module_for_sure(mod, search_path=sys.path):
+        mod = mod.replace('(', '')  # Ignore parentheses
+        try:
+            mod_name = mod
+            while '.' in mod_name:
+                pack_name, _sep, mod_name = mod.partition('.')
+                f, p, d = imp.find_module(pack_name, search_path)
+                search_path = [p]
+            imp.find_module(mod_name, search_path)
+        except ImportError:
+            try:
+                # NOTE(vish): handle namespace modules
+                module = __import__(mod)
+            except ImportError, exc:
+                # NOTE(vish): the import error might be due
+                #             to a missing dependency
+                missing = str(exc).split()[-1]
+                if (missing != mod.split('.')[-1] or
+                        "cannot import" in str(exc)):
+                    _missingImport.add(missing)
+                    return True
+                return False
+            except Exception, exc:
+                # NOTE(jogo) don't stack trace if unexpected import error,
+                # log and continue.
+                traceback.print_exc()
+                return False
+        return True
+
+    def is_module(mod):
+        """Checks for non module imports."""
+        if mod in modules_cache:
+            return modules_cache[mod]
+        res = is_module_for_sure(mod)
+        modules_cache[mod] = res
+        return res
+
+    current_path = os.path.dirname(pep8.current_file)
+    current_mod = os.path.basename(pep8.current_file)
+    if current_mod[-3:] == ".py":
+        current_mod = current_mod[:-3]
+
+    split_line = logical_line.split()
+    split_line_len = len(split_line)
+    if (split_line[0] in ('import', 'from') and split_line_len > 1 and
+        not is_import_exception(split_line[1])):
+        pos = logical_line.find(',')
+        if pos != -1:
+            if split_line[0] == 'from':
+                yield pos, "N301: one import per line"
+            return  # ',' is not supported by the N302 checker yet
+        pos = logical_line.find('*')
+        if pos != -1:
+            yield pos, "N303: No wildcard (*) import."
+            return
+
+        if split_line_len in (2, 4, 6) and split_line[1] != "__future__":
+            if 'from' == split_line[0] and split_line_len > 3:
+                mod = '.'.join((split_line[1], split_line[3]))
+                if is_import_exception(mod):
+                        return
+                if RE_RELATIVE_IMPORT.search(logical_line):
+                    yield logical_line.find('.'), ("N304: No "
+                        "relative imports. '%s' is a relative import"
+                        % logical_line)
+                    return
+
+                if (not is_module(mod) and
+                    not pep8.current_file.endswith("__init__.py")):
+                    yield 0, ("N302: import only modules."
+                              "'%s' does not import a module" % logical_line)
+                return
+
+        #NOTE(afazekas): import searches first in the package
+        # The import keyword just imports modules
+        # The guestfs module now imports guestfs
+        mod = split_line[1]
+        if (current_mod != mod and
+            not is_module(mod) and
+            is_module_for_sure(mod, [current_path])):
+                yield 0, ("N304: No relative imports."
+                          " '%s' is a relative import"
+                          % logical_line)
+
+
+#TODO(jogo): import template: N305
+
+
+def nova_import_alphabetical(logical_line, blank_lines, previous_logical,
+                             indent_level, previous_indent_level):
+    r"""Check for imports in alphabetical order.
+
+    nova HACKING guide recommendation for imports:
+    imports in human alphabetical order
+
+    Okay: import os\nimport sys\n\nimport nova\nfrom nova import test
+    N306: import sys\nimport os
+    """
+    # handle import x
+    # use .lower since capitalization shouldn't dictate order
+    split_line = import_normalize(logical_line.strip()).lower().split()
+    split_previous = import_normalize(previous_logical.strip()).lower().split()
+
+    if blank_lines < 1 and indent_level == previous_indent_level:
+        length = [2, 4]
+        if (len(split_line) in length and len(split_previous) in length and
+            split_line[0] == "import" and split_previous[0] == "import"):
+            if split_line[1] < split_previous[1]:
+                yield (0, "N306: imports not in alphabetical order (%s, %s)"
+                       % (split_previous[1], split_line[1]))
+
+
+def nova_import_no_db_in_virt(logical_line, filename):
+    """Check for db calls from nova/virt
+
+    As of grizzly-2 all the database calls have been removed from
+    nova/virt, and we want to keep it that way.
+
+    N307
+    """
+    if "nova/virt" in filename and not filename.endswith("fake.py"):
+        if logical_line.startswith("from nova import db"):
+            yield (0, "N307: nova.db import not allowed in nova/virt/*")
+
+
+def is_docstring(physical_line, previous_logical):
+    """Return True if found docstring
+    'A docstring is a string literal that occurs as the first statement in a
+    module, function, class,'
+    http://www.python.org/dev/peps/pep-0257/#what-is-a-docstring
+    """
+    line = physical_line.lstrip()
+    start = max([line.find(i) for i in START_DOCSTRING_TRIPLE])
+    end = max([line[-4:-1] == i for i in END_DOCSTRING_TRIPLE])
+    if (previous_logical.startswith("def ") or
+            previous_logical.startswith("class ")):
+        if start is 0:
+            return True
+        else:
+            # Handle multi line comments
+            return end and start in (-1, len(line) - 4)
+
+
+def nova_docstring_start_space(physical_line, previous_logical):
+    r"""Check for docstring not start with space.
+
+    nova HACKING guide recommendation for docstring:
+    Docstring should not start with space
+
+    Okay: def foo():\n    '''This is good.'''
+    Okay: def foo():\n    a = ''' This is not a docstring.'''
+    Okay: def foo():\n    pass\n    ''' This is not.'''
+    N401: def foo():\n    ''' This is not.'''
+    """
+    # short circuit so that we don't fail on our own fail test
+    # when running under external pep8
+    if physical_line.find("N401: def foo()") != -1:
+        return
+
+    # it's important that we determine this is actually a docstring,
+    # and not a doc block used somewhere after the first line of a
+    # function def
+    if is_docstring(physical_line, previous_logical):
+        pos = max([physical_line.find(i) for i in START_DOCSTRING_TRIPLE])
+        if physical_line[pos + 3] == ' ':
+            return (pos, "N401: docstring should not start with"
+                    " a space")
+
+
+def nova_docstring_one_line(physical_line, previous_logical):
+    r"""Check one line docstring end.
+
+    nova HACKING guide recommendation for one line docstring:
+    A one line docstring looks like this and ends in punctuation.
+
+    Okay: def foo():\n    '''This is good.'''
+    Okay: def foo():\n    '''This is good too!'''
+    Okay: def foo():\n    '''How about this?'''
+    Okay: def foo():\n    a = '''This is not a docstring'''
+    Okay: def foo():\n    pass\n    '''This is not a docstring'''
+    Okay: class Foo:\n    pass\n    '''This is not a docstring'''
+    N402: def foo():\n    '''This is not'''
+    N402: def foo():\n    '''Bad punctuation,'''
+    N402: class Foo:\n    '''Bad punctuation,'''
+    """
+    #TODO(jogo) make this apply to multi line docstrings as well
+    line = physical_line.lstrip()
+    if is_docstring(physical_line, previous_logical):
+        pos = max([line.find(i) for i in START_DOCSTRING_TRIPLE])  # start
+        end = max([line[-4:-1] == i for i in END_DOCSTRING_TRIPLE])  # end
+
+        if pos != -1 and end and len(line) > pos + 4:
+            if line[-5] not in ['.', '?', '!']:
+                return pos, "N402: one line docstring needs punctuation."
+
+
+def nova_docstring_multiline_end(physical_line, previous_logical, tokens):
+    r"""Check multi line docstring end.
+
+    nova HACKING guide recommendation for docstring:
+    Docstring should end on a new line
+
+    Okay: '''foobar\nfoo\nbar\n'''
+    Okay: def foo():\n    '''foobar\nfoo\nbar\n'''
+    Okay: class Foo:\n    '''foobar\nfoo\nbar\n'''
+    Okay: def foo():\n    a = '''not\na\ndocstring'''
+    Okay: def foo():\n    pass\n'''foobar\nfoo\nbar\n   d'''
+    N403: def foo():\n    '''foobar\nfoo\nbar\ndocstring'''
+    N403: class Foo:\n    '''foobar\nfoo\nbar\ndocstring'''\n\n
+    """
+    # if find OP tokens, not a docstring
+    ops = [t for t, _, _, _, _ in tokens if t == tokenize.OP]
+    if (is_docstring(physical_line, previous_logical) and len(tokens) > 0 and
+            len(ops) == 0):
+        pos = max(physical_line.find(i) for i in END_DOCSTRING_TRIPLE)
+        if physical_line.strip() not in START_DOCSTRING_TRIPLE:
+            return (pos, "N403: multi line docstring end on new line")
+
+
+def nova_docstring_multiline_start(physical_line, previous_logical, tokens):
+    r"""Check multi line docstring start with summary.
+
+    nova HACKING guide recommendation for docstring:
+    Docstring should start with A multi line docstring has a one-line summary
+
+    Okay: '''foobar\nfoo\nbar\n'''
+    Okay: def foo():\n    a = '''\nnot\na docstring\n'''
+    N404: def foo():\n'''\nfoo\nbar\n'''\n\n
+    """
+    if is_docstring(physical_line, previous_logical):
+        pos = max([physical_line.find(i) for i in START_DOCSTRING_TRIPLE])
+        # start of docstring when len(tokens)==0
+        if len(tokens) == 0 and pos != -1 and len(physical_line) == pos + 4:
+            if physical_line.strip() in START_DOCSTRING_TRIPLE:
+                return (pos, "N404: multi line docstring "
+                        "should start with a summary")
+
+
+def nova_no_cr(physical_line):
+    r"""Check that we only use newlines not carriage returns.
+
+    Okay: import os\nimport sys
+    # pep8 doesn't yet replace \r in strings, will work on an
+    # upstream fix
+    N901 import os\r\nimport sys
+    """
+    pos = physical_line.find('\r')
+    if pos != -1 and pos == (len(physical_line) - 2):
+        return (pos, "N901: Windows style line endings not allowed in code")
+
+
+FORMAT_RE = re.compile("%(?:"
+                            "%|"           # Ignore plain percents
+                            "(\(\w+\))?"   # mapping key
+                            "([#0 +-]?"    # flag
+                             "(?:\d+|\*)?"  # width
+                             "(?:\.\d+)?"   # precision
+                             "[hlL]?"       # length mod
+                             "\w))")        # type
+
+
+class LocalizationError(Exception):
+    pass
+
+
+def check_i18n():
+    """Generator that checks token stream for localization errors.
+
+    Expects tokens to be ``send``ed one by one.
+    Raises LocalizationError if some error is found.
+    """
+    while True:
+        try:
+            token_type, text, _, _, line = yield
+        except GeneratorExit:
+            return
+
+        if (token_type == tokenize.NAME and text == "_" and
+            not line.startswith('def _(msg):')):
+
+            while True:
+                token_type, text, start, _, _ = yield
+                if token_type != tokenize.NL:
+                    break
+            if token_type != tokenize.OP or text != "(":
+                continue  # not a localization call
+
+            format_string = ''
+            while True:
+                token_type, text, start, _, _ = yield
+                if token_type == tokenize.STRING:
+                    format_string += eval(text)
+                elif token_type == tokenize.NL:
+                    pass
+                else:
+                    break
+
+            if not format_string:
+                raise LocalizationError(start,
+                    "N701: Empty localization string")
+            if token_type != tokenize.OP:
+                raise LocalizationError(start,
+                    "N701: Invalid localization call")
+            if text != ")":
+                if text == "%":
+                    raise LocalizationError(start,
+                        "N702: Formatting operation should be outside"
+                        " of localization method call")
+                elif text == "+":
+                    raise LocalizationError(start,
+                        "N702: Use bare string concatenation instead"
+                        " of +")
+                else:
+                    raise LocalizationError(start,
+                        "N702: Argument to _ must be just a string")
+
+            format_specs = FORMAT_RE.findall(format_string)
+            positional_specs = [(key, spec) for key, spec in format_specs
+                                            if not key and spec]
+            # not spec means %%, key means %(smth)s
+            if len(positional_specs) > 1:
+                raise LocalizationError(start,
+                    "N703: Multiple positional placeholders")
+
+
+def nova_localization_strings(logical_line, tokens):
+    r"""Check localization in line.
+
+    Okay: _("This is fine")
+    Okay: _("This is also fine %s")
+    N701: _('')
+    N702: _("Bob" + " foo")
+    N702: _("Bob %s" % foo)
+    # N703 check is not quite right, disabled by removing colon
+    N703 _("%s %s" % (foo, bar))
+    """
+    # TODO(sdague) actually get these tests working
+    gen = check_i18n()
+    next(gen)
+    try:
+        map(gen.send, tokens)
+        gen.close()
+    except LocalizationError as e:
+        yield e.args
+
+#TODO(jogo) Dict and list objects
+
+
+def nova_is_not(logical_line):
+    r"""Check localization in line.
+
+    Okay: if x is not y
+    N901: if not X is Y
+    N901: if not X.B is Y
+    """
+    split_line = logical_line.split()
+    if (len(split_line) == 5 and split_line[0] == 'if' and
+            split_line[1] == 'not' and split_line[3] == 'is'):
+                yield (logical_line.find('not'), "N901: Use the 'is not' "
+                        "operator for when testing for unequal identities")
+
+
+def nova_not_in(logical_line):
+    r"""Check localization in line.
+
+    Okay: if x not in y
+    Okay: if not (X in Y or X is Z)
+    Okay: if not (X in Y)
+    N902: if not X in Y
+    N902: if not X.B in Y
+    """
+    split_line = logical_line.split()
+    if (len(split_line) == 5 and split_line[0] == 'if' and
+            split_line[1] == 'not' and split_line[3] == 'in' and not
+            split_line[2].startswith('(')):
+                yield (logical_line.find('not'), "N902: Use the 'not in' "
+                        "operator for collection membership evaluation")
+
+current_file = ""
+
+
+def readlines(filename):
+    """Record the current file being tested."""
+    pep8.current_file = filename
+    return open(filename).readlines()
+
+
+def add_nova():
+    """Monkey patch in nova guidelines.
+
+    Look for functions that start with nova_  and have arguments
+    and add them to pep8 module
+    Assumes you know how to write pep8.py checks
+    """
+    for name, function in globals().items():
+        if not inspect.isfunction(function):
+            continue
+        args = inspect.getargspec(function)[0]
+        if args and name.startswith("nova"):
+            exec("pep8.%s = %s" % (name, name))
+
+
+def once_git_check_commit_title():
+    """Check git commit messages.
+
+    nova HACKING recommends not referencing a bug or blueprint in first line,
+    it should provide an accurate description of the change
+    N801
+    N802 Title limited to 72 chars
+    """
+    #Get title of most recent commit
+
+    subp = subprocess.Popen(['git', 'log', '--no-merges', '--pretty=%s', '-1'],
+            stdout=subprocess.PIPE)
+    title = subp.communicate()[0]
+    if subp.returncode:
+        raise Exception("git log failed with code %s" % subp.returncode)
+
+    #From https://github.com/openstack/openstack-ci-puppet
+    #       /blob/master/modules/gerrit/manifests/init.pp#L74
+    #Changeid|bug|blueprint
+    git_keywords = (r'(I[0-9a-f]{8,40})|'
+                    '([Bb]ug|[Ll][Pp])[\s\#:]*(\d+)|'
+                    '([Bb]lue[Pp]rint|[Bb][Pp])[\s\#:]*([A-Za-z0-9\\-]+)')
+    GIT_REGEX = re.compile(git_keywords)
+
+    error = False
+    #NOTE(jogo) if match regex but over 3 words, acceptable title
+    if GIT_REGEX.search(title) is not None and len(title.split()) <= 3:
+        print ("N801: git commit title ('%s') should provide an accurate "
+               "description of the change, not just a reference to a bug "
+               "or blueprint" % title.strip())
+        error = True
+    # HACKING.rst recommends commit titles 50 chars or less, but enforces
+    # a 72 character limit
+    if len(title.decode('utf-8')) > 72:
+        print ("N802: git commit title ('%s') should be under 50 chars"
+                % title.strip())
+        error = True
+    return error
+
+imports_on_separate_lines_N301_compliant = r"""
+    Imports should usually be on separate lines.
+
+    Okay: import os\nimport sys
+    E401: import sys, os
+
+    N301: from subprocess import Popen, PIPE
+    Okay: from myclas import MyClass
+    Okay: from foo.bar.yourclass import YourClass
+    Okay: import myclass
+    Okay: import foo.bar.yourclass
+    """
+
+if __name__ == "__main__":
+    #include nova path
+    sys.path.append(os.getcwd())
+    #Run once tests (not per line)
+    once_error = once_git_check_commit_title()
+    #NOVA error codes start with an N
+    pep8.SELFTEST_REGEX = re.compile(r'(Okay|[EWN]\d{3}):\s(.*)')
+    pep8.ERRORCODE_REGEX = re.compile(r'[EWN]\d{3}')
+    add_nova()
+    pep8.current_file = current_file
+    pep8.readlines = readlines
+    pep8.StyleGuide.excluded = excluded
+    pep8.StyleGuide.input_dir = input_dir
+    # we need to kill this doctring otherwise the self tests fail
+    pep8.imports_on_separate_lines.__doc__ = \
+        imports_on_separate_lines_N301_compliant
+
+    try:
+        pep8._main()
+        sys.exit(once_error)
+    finally:
+        if len(_missingImport) > 0:
+            print >> sys.stderr, ("%i imports missing in this test environment"
+                    % len(_missingImport))

tools/install_venv.py

@@ -0,0 +1,74 @@
+# vim: tabstop=4 shiftwidth=4 softtabstop=4
+
+# Copyright 2010 United States Government as represented by the
+# Administrator of the National Aeronautics and Space Administration.
+# All Rights Reserved.
+#
+# Copyright 2010 OpenStack Foundation
+# Copyright 2013 IBM Corp.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+import os
+import sys
+
+import install_venv_common as install_venv
+
+
+def print_help(venv, root):
+    help = """
+    Marconi development environment setup is complete.
+
+    Marconi development uses virtualenv to track and manage Python dependencies
+    while in development and testing.
+
+    To activate the Marconi virtualenv for the extent of your current shell
+    session you can run:
+
+    $ source %s/bin/activate
+
+    Or, if you prefer, you can run commands in the virtualenv on a case by case
+    basis by running:
+
+    $ %s/tools/with_venv.sh <your command>
+
+    Also, make test will automatically use the virtualenv.
+    """
+    print help % (venv, root)
+
+
+def main(argv):
+    root = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))
+
+    if os.environ.get('tools_path'):
+        root = os.environ['tools_path']
+    venv = os.path.join(root, '.venv')
+    if os.environ.get('venv'):
+        venv = os.environ['venv']
+
+    pip_requires = os.path.join(root, 'tools', 'pip-requires')
+    test_requires = os.path.join(root, 'tools', 'test-requires')
+    py_version = "python%s.%s" % (sys.version_info[0], sys.version_info[1])
+    project = 'Marconi'
+    install = install_venv.InstallVenv(root, venv, pip_requires, test_requires,
+                             py_version, project)
+    options = install.parse_args(argv)
+    install.check_python_version()
+    install.check_dependencies()
+    install.create_virtualenv(no_site_packages=options.no_site_packages)
+    install.install_dependencies()
+    install.post_process()
+    print_help(venv, root)
+
+if __name__ == '__main__':
+    main(sys.argv)

tools/install_venv_common.py

@@ -0,0 +1,219 @@
+# vim: tabstop=4 shiftwidth=4 softtabstop=4
+
+# Copyright 2013 OpenStack Foundation
+# Copyright 2013 IBM Corp.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+"""Provides methods needed by installation script for OpenStack development
+virtual environments.
+
+Synced in from openstack-common
+"""
+
+import argparse
+import os
+import subprocess
+import sys
+
+
+class InstallVenv(object):
+
+    def __init__(self, root, venv, pip_requires, test_requires, py_version,
+                 project):
+        self.root = root
+        self.venv = venv
+        self.pip_requires = pip_requires
+        self.test_requires = test_requires
+        self.py_version = py_version
+        self.project = project
+
+    def die(self, message, *args):
+        print >> sys.stderr, message % args
+        sys.exit(1)
+
+    def check_python_version(self):
+        if sys.version_info < (2, 6):
+            self.die("Need Python Version >= 2.6")
+
+    def run_command_with_code(self, cmd, redirect_output=True,
+                              check_exit_code=True):
+        """Runs a command in an out-of-process shell.
+
+        Returns the output of that command. Working directory is self.root.
+        """
+        if redirect_output:
+            stdout = subprocess.PIPE
+        else:
+            stdout = None
+
+        proc = subprocess.Popen(cmd, cwd=self.root, stdout=stdout)
+        output = proc.communicate()[0]
+        if check_exit_code and proc.returncode != 0:
+            self.die('Command "%s" failed.\n%s', ' '.join(cmd), output)
+        return (output, proc.returncode)
+
+    def run_command(self, cmd, redirect_output=True, check_exit_code=True):
+        return self.run_command_with_code(cmd, redirect_output,
+                                          check_exit_code)[0]
+
+    def get_distro(self):
+        if (os.path.exists('/etc/fedora-release') or
+                os.path.exists('/etc/redhat-release')):
+            return Fedora(self.root, self.venv, self.pip_requires,
+                          self.test_requires, self.py_version, self.project)
+        else:
+            return Distro(self.root, self.venv, self.pip_requires,
+                          self.test_requires, self.py_version, self.project)
+
+    def check_dependencies(self):
+        self.get_distro().install_virtualenv()
+
+    def create_virtualenv(self, no_site_packages=True):
+        """Creates the virtual environment and installs PIP.
+
+        Creates the virtual environment and installs PIP only into the
+        virtual environment.
+        """
+        if not os.path.isdir(self.venv):
+            print 'Creating venv...',
+            if no_site_packages:
+                self.run_command(['virtualenv', '-q', '--no-site-packages',
+                                 self.venv])
+            else:
+                self.run_command(['virtualenv', '-q', self.venv])
+            print 'done.'
+            print 'Installing pip in venv...',
+            if not self.run_command(['tools/with_venv.sh', 'easy_install',
+                                    'pip>1.0']).strip():
+                self.die("Failed to install pip.")
+            print 'done.'
+        else:
+            print "venv already exists..."
+            pass
+
+    def pip_install(self, *args):
+        self.run_command(['tools/with_venv.sh',
+                         'pip', 'install', '--upgrade'] + list(args),
+                         redirect_output=False)
+
+    def install_dependencies(self):
+        print 'Installing dependencies with pip (this can take a while)...'
+
+        # First things first, make sure our venv has the latest pip and
+        # distribute.
+        # NOTE: we keep pip at version 1.1 since the most recent version causes
+        # the .venv creation to fail. See:
+        # https://bugs.launchpad.net/marconi/+bug/1047120
+        self.pip_install('pip==1.1')
+        self.pip_install('distribute')
+
+        # Install greenlet by hand - just listing it in the requires file does
+        # not
+        # get it installed in the right order
+        self.pip_install('greenlet')
+
+        self.pip_install('-r', self.pip_requires)
+        self.pip_install('-r', self.test_requires)
+
+    def post_process(self):
+        self.get_distro().post_process()
+
+    def parse_args(self, argv):
+        """Parses command-line arguments."""
+        parser = argparse.ArgumentParser()
+        parser.add_argument('-n', '--no-site-packages',
+                            action='store_true',
+                            help="Do not inherit packages from global Python "
+                                 "install")
+        return parser.parse_args(argv[1:])
+
+
+class Distro(InstallVenv):
+
+    def check_cmd(self, cmd):
+        return bool(self.run_command(['which', cmd],
+                    check_exit_code=False).strip())
+
+    def install_virtualenv(self):
+        if self.check_cmd('virtualenv'):
+            return
+
+        if self.check_cmd('easy_install'):
+            print 'Installing virtualenv via easy_install...',
+            if self.run_command(['easy_install', 'virtualenv']):
+                print 'Succeeded'
+                return
+            else:
+                print 'Failed'
+
+        self.die('ERROR: virtualenv not found.\n\n%s development'
+                 ' requires virtualenv, please install it using your'
+                 ' favorite package management tool' % self.project)
+
+    def post_process(self):
+        """Any distribution-specific post-processing gets done here.
+
+        In particular, this is useful for applying patches to code inside
+        the venv.
+        """
+        pass
+
+
+class Fedora(Distro):
+    """This covers all Fedora-based distributions.
+
+    Includes: Fedora, RHEL, CentOS, Scientific Linux
+    """
+
+    def check_pkg(self, pkg):
+        return self.run_command_with_code(['rpm', '-q', pkg],
+                                          check_exit_code=False)[1] == 0
+
+    def yum_install(self, pkg, **kwargs):
+        print "Attempting to install '%s' via yum" % pkg
+        self.run_command(['sudo', 'yum', 'install', '-y', pkg], **kwargs)
+
+    def apply_patch(self, originalfile, patchfile):
+        self.run_command(['patch', originalfile, patchfile])
+
+    def install_virtualenv(self):
+        if self.check_cmd('virtualenv'):
+            return
+
+        if not self.check_pkg('python-virtualenv'):
+            self.yum_install('python-virtualenv', check_exit_code=False)
+
+        super(Fedora, self).install_virtualenv()
+
+    def post_process(self):
+        """Workaround for a bug in eventlet.
+
+        This currently affects RHEL6.1, but the fix can safely be
+        applied to all RHEL and Fedora distributions.
+
+        This can be removed when the fix is applied upstream.
+
+        Marconi: https://bugs.launchpad.net/marconi/+bug/884915
+        Upstream: https://bitbucket.org/which_linden/eventlet/issue/89
+        """
+
+        # Install "patch" program if it's not there
+        if not self.check_pkg('patch'):
+            self.yum_install('patch')
+
+        # Apply the eventlet patch
+        self.apply_patch(os.path.join(self.venv, 'lib', self.py_version,
+                                      'site-packages',
+                                      'eventlet/green/subprocess.py'),
+                         'contrib/redhat-eventlet.patch')

tools/run_pep8.sh

@@ -0,0 +1,16 @@
+#!/bin/bash
+
+set -e
+# This is used by run_tests.sh and tox.ini
+python tools/hacking.py --doctest
+
+# Until all these issues get fixed, ignore.
+PEP8='python tools/hacking.py --ignore=E12,E711,E721,E712,N303,N403,N404'
+
+EXCLUDE='--exclude=.venv,.git,.tox,dist,doc,*openstack/common*,*lib/python*'
+EXCLUDE+=',*egg,build'
+${PEP8} ${EXCLUDE} .
+
+${PEP8} --filename=marconi* bin
+
+! pyflakes marconi/ | grep "imported but unused\|redefinition of function" | grep -v "__init__.py"

tools/test-requires

@@ -1,3 +1,5 @@
+distribute>=0.6.24
+
 coverage
 discover
 fixtures

tox.ini

@@ -14,12 +14,14 @@ deps = -r{toxinidir}/tools/pip-requires
 commands = nosetests {posargs}
 
 [tox:jenkins]
+sitepackages = True
 downloadcache = ~/cache/pip
 
 [testenv:pep8]
-deps = pep8==1.3.3
+# deps = pep8==1.3.3
-commands =
+# commands =
-  pep8 --ignore=E125,E126,E711,E712 --repeat --show-source --exclude=.venv,.tox,dist,doc,openstack .
+#  pep8 --ignore=E125,E126,E711,E712 --repeat --show-source --exclude=.venv,.tox,dist,doc,openstack .
+commands = bash tools/run_pep8.sh
 
 [testenv:cover]
 setenv = NOSE_WITH_COVERAGE=1