Move some common test and framework goo into neutron-lib

Also fix an issue with ignoring some of the unit tests, moving them all
under the right subdir, and do not overload the tests.base name.

This commit used to contain the move of the rpc code, but that is now
in a child commit.

Partially-Implements: blueprint neutron-lib
Change-Id: Iae9fe970cd2a76305a74804d93a912e476060f1e
This commit is contained in:
Doug Wiegley 2016-01-14 14:41:03 -06:00
parent 50546e95ec
commit 285236706a
20 changed files with 523 additions and 56 deletions

View File

@ -0,0 +1,78 @@
=======================
Neutron-Lib Conventions
=======================
Summary
-------
* Standard modules - current cycle + 2 deprecation policy
* Legacy modules - current cycle + 1 deprecation policy
* Private modules - no deprecation warnings of any kind
Interface Contract
------------------
The neutron-lib repo exists to provide code with a long-term stable interface
for subprojects. If we do need to deprecate something, a debtcollector
warning will be added, the neutron-lib core team will make every effort to
update any neutron stadium project for you, and you will get at least the
current release plus two cycles before it disappears.
In keeping with how hard it is to remove things, the change velocity of this
library will be slower than neutron. There are two notable cases where code
should go into standard neutron instead of this lib (or your repo):
* It is something common, but changes a lot. An example is something like
the neutron Port object. Everyone uses it, but it changes frequently.
You don't want to wait for a library release to tweak some neutron feature,
and we are not going to force releases quickly because you tried to put
it here. Those items will need to be addressed in some other manner
(in the case of the Port object, it'll be via an auto-magic container
object that mimics it.)
* It is something common, but you need it now. Put it in the repo that needs
it, get your stuff working. Then consider making it available in the lib,
and eventually remove it from your repo when the lib is publishing it.
An example would be a new neutron constant. The process would be, put it
in neutron along with your change, submit it to the lib, when that constant
is eventually available, remove it from neutron and use the lib version.
Private Interfaces
------------------
Private interfaces are *PRIVATE*. They will disappear, be renamed, and
absolutely no regard will be given to anyone that is using them. They are
for internal library use only.
DO NOT USE THEM. THEY WILL CHANGE.
Private interfaces in this library will always have a leading underscore,
on the module or function name.
Legacy Modules
--------------
This library has a special namespace called neutron_lib.legacy.
Anything in this directory will likely get a new interface in the top-level
library sometime in the near future, and then a debtcollector deprecation
notice. Expect to get current cycle plus one release of maintenance at that
point, and then they will be removed.
Why this intermediary step? Because neutron has some serious dependency
issues with its subprojects that need breaking, we do not want to rush
some of the refactors to our interfaces that need to happen, we have
limited resources, but we still need to make addressing those dependency
issues a high priority.
The legacy module is for those existing modules in neutron that are in
wide use by subprojects, but which are not super interfaces. The legacy
submodule is for routines that will still be maintained with a long-term
backwards compatibility interface contract, but which are not considered
"library worthy" by the neutron core team.
This can easily be abused as a kitchen sink to just move stuff and make
fast progress. Please do not do this, and do not expect this kind of thing
to be favorably reviewed. Good candidates for this area are things that
we want to refactor, but are lower priority, AND they have been around for
a long time with no changes (i.e. an existing history of stability).

View File

@ -40,6 +40,7 @@ Programming HowTos and Tutorials
:maxdepth: 3
readme
conventions
installation
usage
review-guidelines

View File

@ -225,7 +225,7 @@ def validate_ip_address(data, valid_values=None):
# e.g. 011 octal is 9 decimal. Since there is no standard saying
# whether IP address with leading '0's should be interpreted as octal
# or decimal, hence we reject leading '0's to avoid ambiguity.
if ip.version == 4 and str(ip) != data:
elif ip.version == 4 and str(ip) != data:
msg = _("'%(data)s' is not an accepted IP address, "
"'%(ip)s' is recommended") % {"data": data, "ip": ip}
except Exception:

63
neutron_lib/config.py Normal file
View File

@ -0,0 +1,63 @@
# Copyright 2011 VMware, Inc.
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import socket
import sys
from oslo_config import cfg
from oslo_db import options as db_options
from oslo_log import log as logging
from neutron_lib._i18n import _, _LI
from neutron_lib import version
LOG = logging.getLogger(__name__)
core_opts = [
cfg.StrOpt('host', default=socket.gethostname(),
sample_default='example.domain',
help=_("Hostname to be used by the Neutron server, agents and "
"services running on this machine. All the agents and "
"services running on this machine must use the same "
"host value.")),
]
# Register the configuration options
cfg.CONF.register_opts(core_opts)
def set_db_defaults():
# Update the default QueuePool parameters. These can be tweaked by the
# conf variables - max_pool_size, max_overflow and pool_timeout
db_options.set_defaults(
cfg.CONF,
connection='sqlite://',
sqlite_db='', max_pool_size=10,
max_overflow=20, pool_timeout=10)
logging.register_options(cfg.CONF)
def setup_logging():
"""Sets up the logging options for a log with supplied name."""
product_name = "neutron"
logging.setup(cfg.CONF, product_name)
LOG.info(_LI("Logging enabled!"))
LOG.info(_LI("%(prog)s version %(version)s"),
{'prog': sys.argv[0],
'version': version.version_info.release_string()})
LOG.debug("command line: %s", " ".join(sys.argv))

View File

@ -136,6 +136,9 @@ IPV6_PD_POOL_ID = 'prefix_delegation'
# Device names start with "tap"
TAP_DEVICE_PREFIX = 'tap'
# Linux interface max length
DEVICE_NAME_MAX_LEN = 15
# Time format
ISO8601_TIME_FORMAT = '%Y-%m-%dT%H:%M:%S.%f'

View File

@ -0,0 +1,7 @@
===========================
neutron-lib legacy routines
===========================
This is a special namespace with lower deprecation policy standards,
and specific rules for its use. Please refer to the 'Legacy Modules'
section of this library's 'conventions' document for more information.

View File

195
neutron_lib/tests/_base.py Normal file
View File

@ -0,0 +1,195 @@
# Copyright 2010-2011 OpenStack Foundation
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import logging as std_logging
import os
import os.path
import random
import fixtures
import mock
from oslo_config import cfg
from oslo_utils import strutils
import six
import testtools
from neutron_lib._i18n import _
from neutron_lib import config
from neutron_lib import constants
from neutron_lib.tests import _post_mortem_debug as post_mortem_debug
from neutron_lib.tests import _tools as tools
CONF = cfg.CONF
LOG_FORMAT = "%(asctime)s %(levelname)8s [%(name)s] %(message)s"
ROOTDIR = os.path.dirname(__file__)
ETCDIR = os.path.join(ROOTDIR, 'etc')
def etcdir(*p):
return os.path.join(ETCDIR, *p)
def fake_use_fatal_exceptions(*args):
return True
def fake_consume_in_threads(self):
return []
def get_rand_name(max_length=None, prefix='test'):
"""Return a random string.
The string will start with 'prefix' and will be exactly 'max_length'.
If 'max_length' is None, then exactly 8 random characters, each
hexadecimal, will be added. In case len(prefix) <= len(max_length),
ValueError will be raised to indicate the problem.
"""
if max_length:
length = max_length - len(prefix)
if length <= 0:
raise ValueError("'max_length' must be bigger than 'len(prefix)'.")
suffix = ''.join(str(random.randint(0, 9)) for i in range(length))
else:
suffix = hex(random.randint(0x10000000, 0x7fffffff))[2:]
return prefix + suffix
def get_rand_device_name(prefix='test'):
return get_rand_name(
max_length=constants.DEVICE_NAME_MAX_LEN, prefix=prefix)
def bool_from_env(key, strict=False, default=False):
value = os.environ.get(key)
return strutils.bool_from_string(value, strict=strict, default=default)
def get_test_timeout(default=0):
return int(os.environ.get('OS_TEST_TIMEOUT', default))
def sanitize_log_path(path):
# Sanitize the string so that its log path is shell friendly
return path.replace(' ', '-').replace('(', '_').replace(')', '_')
class AttributeDict(dict):
"""Provide attribute access (dict.key) to dictionary values."""
def __getattr__(self, name):
"""Allow attribute access for all keys in the dict."""
if name in self:
return self[name]
raise AttributeError(_("Unknown attribute '%s'.") % name)
class BaseTestCase(testtools.TestCase):
def setUp(self):
super(BaseTestCase, self).setUp()
config.set_db_defaults()
# Configure this first to ensure pm debugging support for setUp()
debugger = os.environ.get('OS_POST_MORTEM_DEBUGGER')
if debugger:
self.addOnException(post_mortem_debug.get_exception_handler(
debugger))
# Make sure we see all relevant deprecation warnings when running tests
self.useFixture(tools.WarningsFixture())
if bool_from_env('OS_DEBUG'):
_level = std_logging.DEBUG
else:
_level = std_logging.INFO
capture_logs = bool_from_env('OS_LOG_CAPTURE')
if not capture_logs:
std_logging.basicConfig(format=LOG_FORMAT, level=_level)
self.log_fixture = self.useFixture(
fixtures.FakeLogger(
format=LOG_FORMAT,
level=_level,
nuke_handlers=capture_logs,
))
test_timeout = get_test_timeout()
if test_timeout == -1:
test_timeout = 0
if test_timeout > 0:
self.useFixture(fixtures.Timeout(test_timeout, gentle=True))
# If someone does use tempfile directly, ensure that it's cleaned up
self.useFixture(fixtures.NestedTempfile())
self.useFixture(fixtures.TempHomeDir())
self.addCleanup(mock.patch.stopall)
if bool_from_env('OS_STDOUT_CAPTURE'):
stdout = self.useFixture(fixtures.StringStream('stdout')).stream
self.useFixture(fixtures.MonkeyPatch('sys.stdout', stdout))
if bool_from_env('OS_STDERR_CAPTURE'):
stderr = self.useFixture(fixtures.StringStream('stderr')).stream
self.useFixture(fixtures.MonkeyPatch('sys.stderr', stderr))
self.addOnException(self.check_for_systemexit)
self.orig_pid = os.getpid()
def check_for_systemexit(self, exc_info):
if isinstance(exc_info[1], SystemExit):
if os.getpid() != self.orig_pid:
# Subprocess - let it just exit
raise
# This makes sys.exit(0) still a failure
self.force_failure = True
def assertOrderedEqual(self, expected, actual):
expect_val = self.sort_dict_lists(expected)
actual_val = self.sort_dict_lists(actual)
self.assertEqual(expect_val, actual_val)
def sort_dict_lists(self, dic):
for key, value in six.iteritems(dic):
if isinstance(value, list):
dic[key] = sorted(value)
elif isinstance(value, dict):
dic[key] = self.sort_dict_lists(value)
return dic
def assertDictSupersetOf(self, expected_subset, actual_superset):
"""Checks that actual dict contains the expected dict.
After checking that the arguments are of the right type, this checks
that each item in expected_subset is in, and matches, what is in
actual_superset. Separate tests are done, so that detailed info can
be reported upon failure.
"""
if not isinstance(expected_subset, dict):
self.fail("expected_subset (%s) is not an instance of dict" %
type(expected_subset))
if not isinstance(actual_superset, dict):
self.fail("actual_superset (%s) is not an instance of dict" %
type(actual_superset))
for k, v in expected_subset.items():
self.assertIn(k, actual_superset)
self.assertEqual(v, actual_superset[k],
"Key %(key)s expected: %(exp)r, actual %(act)r" %
{'key': k, 'exp': v, 'act': actual_superset[k]})

View File

@ -0,0 +1,122 @@
# Copyright 2013 Red Hat, Inc.
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import functools
import traceback
def get_exception_handler(debugger_name):
debugger = _get_debugger(debugger_name)
return functools.partial(_exception_handler, debugger)
def _get_debugger(debugger_name):
try:
debugger = __import__(debugger_name)
except ImportError:
raise ValueError("can't import %s module as a post mortem debugger" %
debugger_name)
if 'post_mortem' in dir(debugger):
return debugger
else:
raise ValueError("%s is not a supported post mortem debugger" %
debugger_name)
def _exception_handler(debugger, exc_info):
"""Exception handler enabling post-mortem debugging.
A class extending testtools.TestCase can add this handler in setUp():
self.addOnException(post_mortem_debug.exception_handler)
When an exception occurs, the user will be dropped into a debugger
session in the execution environment of the failure.
Frames associated with the testing framework are excluded so that
the post-mortem session for an assertion failure will start at the
assertion call (e.g. self.assertTrue) rather than the framework code
that raises the failure exception (e.g. the assertTrue method).
"""
tb = exc_info[2]
ignored_traceback = get_ignored_traceback(tb)
if ignored_traceback:
tb = FilteredTraceback(tb, ignored_traceback)
traceback.print_exception(exc_info[0], exc_info[1], tb)
debugger.post_mortem(tb)
def get_ignored_traceback(tb):
"""Retrieve the first traceback of an ignored trailing chain.
Given an initial traceback, find the first traceback of a trailing
chain of tracebacks that should be ignored. The criteria for
whether a traceback should be ignored is whether its frame's
globals include the __unittest marker variable. This criteria is
culled from:
unittest.TestResult._is_relevant_tb_level
For example:
tb.tb_next => tb0.tb_next => tb1.tb_next
- If no tracebacks were to be ignored, None would be returned.
- If only tb1 was to be ignored, tb1 would be returned.
- If tb0 and tb1 were to be ignored, tb0 would be returned.
- If either of only tb or only tb0 was to be ignored, None would
be returned because neither tb or tb0 would be part of a
trailing chain of ignored tracebacks.
"""
# Turn the traceback chain into a list
tb_list = []
while tb:
tb_list.append(tb)
tb = tb.tb_next
# Find all members of an ignored trailing chain
ignored_tracebacks = []
for tb in reversed(tb_list):
if '__unittest' in tb.tb_frame.f_globals:
ignored_tracebacks.append(tb)
else:
break
# Return the first member of the ignored trailing chain
if ignored_tracebacks:
return ignored_tracebacks[-1]
class FilteredTraceback(object):
"""Wraps a traceback to filter unwanted frames."""
def __init__(self, tb, filtered_traceback):
"""Constructor.
:param tb: The start of the traceback chain to filter.
:param filtered_traceback: The first traceback of a trailing
chain that is to be filtered.
"""
self._tb = tb
self.tb_lasti = self._tb.tb_lasti
self.tb_lineno = self._tb.tb_lineno
self.tb_frame = self._tb.tb_frame
self._filtered_traceback = filtered_traceback
@property
def tb_next(self):
tb_next = self._tb.tb_next
if tb_next and tb_next != self._filtered_traceback:
return FilteredTraceback(tb_next, self._filtered_traceback)

View File

@ -13,9 +13,9 @@
# License for the specific language governing permissions and limitations
# under the License.
# Note: _safe_sort_key came from neutron/common/utils.py. For neutron-lib
# it is only used for testing, so is placed here.
import collections
import fixtures
import warnings
def _safe_sort_key(value):
@ -36,3 +36,17 @@ class UnorderedList(list):
def __neq__(self, other):
return not self == other
class WarningsFixture(fixtures.Fixture):
"""Filters out warnings during test runs."""
warning_types = (
DeprecationWarning, PendingDeprecationWarning, ImportWarning
)
def _setUp(self):
self.addCleanup(warnings.resetwarnings)
for wtype in self.warning_types:
warnings.filterwarnings(
"always", category=wtype, module='^neutron_lib\\.')

View File

@ -1,23 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2010-2011 OpenStack Foundation
# Copyright (c) 2013 Hewlett-Packard Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from oslotest import base
class TestCase(base.BaseTestCase):
"""Test case base class for all unit tests."""

View File

@ -17,11 +17,11 @@ import testtools
from neutron_lib.api import converters
from neutron_lib import exceptions as n_exc
from neutron_lib.tests import base
from neutron_lib.tests import tools
from neutron_lib.tests import _base as base
from neutron_lib.tests import _tools as tools
class TestConvertToBoolean(base.TestCase):
class TestConvertToBoolean(base.BaseTestCase):
def test_convert_to_boolean_bool(self):
self.assertIs(converters.convert_to_boolean(True), True)
@ -50,7 +50,7 @@ class TestConvertToBoolean(base.TestCase):
self.assertIs(converters.convert_to_boolean_if_not_none(1), True)
class TestConvertToInt(base.TestCase):
class TestConvertToInt(base.BaseTestCase):
def test_convert_to_int_int(self):
self.assertEqual(-1, converters.convert_to_int(-1))
@ -88,7 +88,7 @@ class TestConvertToInt(base.TestCase):
value, converters.convert_none_to_empty_list(value))
class TestConvertToFloat(base.TestCase):
class TestConvertToFloat(base.BaseTestCase):
# NOTE: the routine being tested here is a plugin-specific extension
# module. As the plugin split proceed towards its second phase this
# test should either be remove, or the validation routine moved into
@ -117,7 +117,7 @@ class TestConvertToFloat(base.TestCase):
self.assertIsNone(converters.convert_to_positive_float_or_none(None))
class TestConvertKvp(base.TestCase):
class TestConvertKvp(base.BaseTestCase):
def test_convert_kvp_list_to_dict_succeeds_for_missing_values(self):
result = converters.convert_kvp_list_to_dict(['True'])
@ -150,7 +150,7 @@ class TestConvertKvp(base.TestCase):
self.assertEqual(['a', 'a=a'], result)
class TestConvertToList(base.TestCase):
class TestConvertToList(base.BaseTestCase):
def test_convert_to_empty_list(self):
for item in (None, [], (), {}):

View File

@ -21,14 +21,14 @@ from neutron_lib.api import converters
from neutron_lib.api import validators
from neutron_lib import constants
from neutron_lib import exceptions as n_exc
from neutron_lib.tests import base
from neutron_lib.tests import _base as base
def dummy_validator(data, valid_values=None):
pass
class TestAttributeValidation(base.TestCase):
class TestAttributeValidation(base.BaseTestCase):
def _construct_dict_and_constraints(self):
"""Constructs a test dictionary and a definition of constraints.

View File

@ -22,13 +22,13 @@ Tests for `neutron_lib.callback.exceptions` module.
import functools
import neutron_lib._callbacks.exceptions as ex
from neutron_lib.tests import test_exceptions as base
from neutron_lib.tests.unit import test_exceptions
class TestCallbackExceptions(base.TestExceptions):
class TestCallbackExceptions(test_exceptions.TestExceptions):
def _check_exception(self, exc_class, expected_msg, **kwargs):
raise_exc_class = functools.partial(base._raise, exc_class)
raise_exc_class = functools.partial(test_exceptions._raise, exc_class)
e = self.assertRaises(exc_class, raise_exc_class, **kwargs)
self.assertEqual(expected_msg, str(e))

View File

@ -23,14 +23,14 @@ import functools
from neutron_lib._i18n import _
import neutron_lib.exceptions as ne
from neutron_lib.tests import base
from neutron_lib.tests import _base as base
def _raise(exc_class, **kwargs):
raise exc_class(**kwargs)
class TestExceptions(base.TestCase):
class TestExceptions(base.BaseTestCase):
def _check_nexc(self, exc_class, expected_msg, **kwargs):
raise_exc_class = functools.partial(_raise, exc_class)

View File

@ -19,10 +19,10 @@ test_neutron_lib
Tests for `neutron_lib` module.
"""
from neutron_lib.tests import base
from neutron_lib.tests import _base as base
class TestNeutron_lib(base.TestCase):
class TestNeutron_lib(base.BaseTestCase):
def test_something(self):
pass

View File

@ -2,9 +2,14 @@
# of appearance. Changing the order has an impact on the overall integration
# process, which may cause wedges in the gate later.
pbr>=1.6
Babel>=1.3
pbr>=1.6 # Apache-2.0
Babel>=1.3 # BSD
oslo.i18n>=1.5.0 # Apache-2.0
oslo.log>=1.12.0 # Apache-2.0
oslo.utils>=2.8.0 # Apache-2.0
debtcollector>=1.2.0 # Apache-2.0
oslo.config>=3.4.0 # Apache-2.0
oslo.db>=4.1.0 # Apache-2.0
oslo.i18n>=2.1.0 # Apache-2.0
oslo.log>=1.14.0 # Apache-2.0
oslo.messaging>=4.0.0 # Apache-2.0
oslo.service>=1.0.0 # Apache-2.0
oslo.utils>=3.4.0 # Apache-2.0

View File

@ -4,13 +4,13 @@
hacking<0.11,>=0.10.2
coverage>=3.6
discover
python-subunit>=0.0.18
sphinx!=1.2.0,!=1.3b1,<1.3,>=1.1.2
coverage>=3.6 # Apache-2.0
discover # BSD
python-subunit>=0.0.18 # Apache-2.0/BSD
sphinx!=1.2.0,!=1.3b1,<1.3,>=1.1.2 # BSD
oslosphinx!=3.4.0,>=2.5.0 # Apache-2.0
oslotest>=1.10.0 # Apache-2.0
os-testr>=0.4.1
testrepository>=0.0.18
testscenarios>=0.4
testtools>=1.4.0
os-testr>=0.4.1 # Apache-2.0
testrepository>=0.0.18 # Apache-2.0/BSD
testscenarios>=0.4 # Apache-2.0/BSD
testtools>=1.4.0 # MIT

View File

@ -34,5 +34,7 @@ commands = oslo_debug_helper {posargs}
[flake8]
show-source = True
builtins = _
exclude=.venv,.git,.tox,dist,doc,*lib/python*,*egg,build
[hacking]
import_exceptions = neutron_lib._i18n