This adds a pip install command to tox.ini that is only used when the
tox env is passed with the 'constraints' factor appended onto it.
As such this will not effect developer workflows or current unit tests.
The initial use of this will be in a non-voting job, to verify that the
constrained checks with tox are stable. DevStack is already running
constrained jobs, as such problems are no expected.
To run a tox with pip using constraints on a developer system a
developer should run the desired tox environment with -constraints.
For example: $(tox -epy27-constraints)
Pip will pull the current version of the upper-constraints.txt file down
from the git.openstack.org, however this method can be overriden to use
a local file setting the environment variable "UPPER_CONSTRAINTS_FILE"
to the local path or a different URL, it is passed directly to pip.
This is currently not enabled in the default tox run, however it is
possible to enable it as a default by adding it to 'envlist' in tox.ini
Implements Blueprint: Requirements-Management
doc/source/api and doc/build are build artifacts from building the docs
using the tox -e docs target. These are ignored in .gitignore since we
don't care about them from build to build - but leaving them around can
actually make the docs build fail if modules no longer exist but are
still sitting in these directories.
So this change just cleans those up each time you run tox -e docs.
If someone makes a change that breaks "tox -e genconfig"
it is not caught today by any of the gate/check jobs.
We should fail at least the docs job by running
oslo-config-generator to make sure it works.
`cmp` method was removed in Python 3, so it would be nice to use __lt__,
__le__, __eq__, __ne__, __gt__ and __ge__ instead, which are Py2 and Py3
Now that we require at least pbr 1.3 and that has
commit 61a3a1ecdf99c680e509742fdc6c441b499af130 in it,
let's remove the workaround.
Currently the tox py34 target uses testtools.run to run
a subset of our test harness. We need to be able to use
pretty_tox.sh just like py27 as we make progress with py34
The first step is to make sure we can discover all the
python -m subunit.run discover -t . ./nova/tests/ --list
So, we need to fix a bunch of things for the discovery
to work including updating to a new version of websockify.
In the xen code, we should keep the original import and
add except for py34 as xen uses an older python that does
not work with six.moves.
need to specify a key parameter which is a function that returns
a sorting key. since the original intention with pool_cmp was to
sort by number of items in each pool, we just use key parameter
with a lambda
needed list() so we could delete stuff inside the loop in py3
needed the import of test as some tests were failing in py3,
added noqa as test is not explicitly referenced within the
need BytesIO for py3, StringIO will fail in many tests
message is not available in py3, so we need to adjust
use test.nested already added in a previous commit
if we have a key which is a list, treat it just like a set
use reduce from six and explicitly typcast version_int as
The 'tox -epep8' command takes a long time as it checks
every single file in the Nova git repository (~1,400 at
time of writing).
This makes tox use a simple wrapper around flake8 which
can be told to restrict the check to only files changed
in the current command. This can be invoked in a simple
manner with 'tox -epep8 -- -HEAD'. Since most commits
only touch a handful of files, this will usually be
far faster than checking all 1,400 source files.
To check an entire branch for bisectability it can be
git rebase -i master -x 'tox -epep8 -- -HEAD'
* Fix sort(): dictionaries are no more comparable on Python 3, use a key
function building a sorted list of the dictionary items
* Replace __builtin__ with six.moves.builtins
* get_hash_str() now accepts Unicode: Unicode is encoded to UTF-8
before hashing the string. Mention also the hash method (MD5) in the
* LastBytesTestCase: use binary files instead of text files. Use also a
context manager on the TemporaryFile to ensure that the temporary file
is closed (even on error).
* Functions: use the __code__ attribute instead of func_code, and
__closure__ instead of func_closure. These attributes are also
available on Python 2.6.
* Replace unichr() with six.unichr()
* SafeTruncateTestCase(): write directly the chinese Unicode character
instead of using safe_decode() (which is not reliable, it depends on
the locale encoding!)
* sanitize_hostname(): On Python 3, decode the hostname from Latin1 to
get back Unicode. Add also a comment explaining the purpose of the
conversion to Latin1.
* last_bytes(): replace the hardcoded constant 22 with errno.EINVAL and
add a comment explaining how the function works.
* tox.ini: add nova.tests.unit.test_utils to Python 3.4
This includes 2 classes of fixes (and 1 skip) for mock.
The first is the change in allowed values for assert_has_calls -
The second is a yet unsolved bug around the use of autospec
The skip is because something has changed with mock.open that is
causing the vhduils test to fail. We don't know why, but it's one test
This also includes a brute force fix for tox -e docs, because pip is
no longer respecting the sphinx pin in test requirements. This has to
be landed with the other changes because they won't work without it.
Fix bytes vs Unicode issues:
* ssh_encrypt_text() now encodes text to UTF-8 if text type is Unicode.
* Encode Unicode to ASCII before encoding it to base64
* On Python 3, decode binascii.hexlify() and base64.encodestring()
output from ASCII to get Unicode (str type in Python 3) is required.
* convert_from_sshrsa_to_pkcs8(): reuse binascii.hexlify() instead of
a loop using struck.unpack() and "%02X" format.
* Replace str.encode('hex') with binascii.hexlify(str)
* Replace string.strip(text) with text.strip()
* Replace StringIO.StringIO with six.StringIO
* Add test on the output type of ssh_encrypt_text() and decrypt_text()
* Call utils.execute() with binary=True to not decode stdout/stderr
(to get them as raw bytes).
* Replace reduce() with six.moves.reduce()
* convert_version_to_str(): replace a/b with a//b to get integer
* tox.ini: add the following tests to Python 3.4
The desired OS_TEST_DBAPI_ADMIN_CONNECTION string is now reflected
in the oslo.db 1.12.0 release's default behavior, and so does not
need to be set any longer to achieve the same opportunistic backend
discovery for tests.
* FakeNovaException_Remote: on Python 3, define a __str__() method
instead of __unicode__() method. str(exc) now calls __str__() now
Pyton 3, __unicode__() is no more used.
* tox.ini: add nova.tests.unit.test_exception to Python 3.
This change adds a basic bandit config for Nova. It can be invoked
by running the tox environment for bandit;
tox -e bandit
This is intended as a starting point for using bandit with Nova
and it should be revisited to improve the testing as more is learned
about the specific needs of the Nova code base.
Tox is configured to only show results for high and medium severity
All tests in nova/tests/unit/objects/test*.py now run with
python3.4 tox target.
* Fix imports in limits.py and urlmap.py
* Add a list() as keys()/values() return iterator in
availability_zones.py, flavors.py and instance_numa_topology.py
* contextlib.nested is not present in python3, so whip up an
alternative using ExitStack(). Directly using ExitStack() wont
work for us.
* Add a few assertJsonEqual in the test cases
* Ensure fingerprinting generates the same exact value in both
python27 and python34
* Replace dict(obj.iteritems()) with dict(obj) where obj is a dictionary
or an oslo.db object
* Replace obj.iteritems() with obj.items() where obj is a dictionary:
obj.items() works on Python 2 and Python 3.
* Replace filter() and map() with list-comprehension using if when a list is
expected. On Python 3, filter() and map() return an iterator.
* Replace obj.keys() with list(obj.keys()) when a list is expected.
Replace obj.keys() with list(obj.keys()). On Python 3,
dict.keys() now returns an iterator.
* Replace (int, long) with six.integer_types
* Get the name of a function using the __name__ attribute, instead of
the func_name attribute. The func_name attribute was removed
in Python 3.
* InstanceTypeTestCase: use dict.update() to combine dictionaries
instead of using dict.items()+reduce()+dict()
* matchers.py: replace StringIO.StringIO with six.StringIO.
* tox.ini: add nova.tests.unit.db.test_db_api to Python 3.4
Oslo team is actively getting rid of the old style generator.py
in openstack/common/config. So we should switch to the newer
oslo-config-generator which is the way forward. For simplicity's
sake, i used one single list_opts in opts.py. However we can
split this up and add entrypoints in setup.cfg if we decide to
do that instead.
On Python 3, if LC_ALL is not passed then the interpreter will assume it
is C (which causes the interpreter to use ASCII). This, in turn, causes
the unicode in the README.rst for rfc3986 to trigger a UnicodeDecodeError
jlvillal (John Villalovos) pointed out
I340c88ea7d098ca5943d60adc73f63a0af79a405 in IRC as the fix to this bug.
With this change, "tox -e py34" now pass.
* Replace unichr() with six.unichr()
* Replace StringIO.StringIO() with six.Bytes() in
* On Python 3, replace UserDict.IterableUserDict with
* Replace __builtin__.open with six.moves.builtins.open
* Replace ConfigParser import with six.moves.configparser
* On Python 3, get the original "queue" module instead of the "Queue"
module in nova/virt/libvirt/host.py
* Replace urllib2 with six.moves.urllib
* tox.ini: "tox -e py34" now only runs nova.tests.unit.test_versions
Add test-requirements-py3.txt to use PyMySQL, instead of MySQL-python,
on Python 3.
Add a py34 test environment to tox.ini which uses
test-requirements-py3.txt and sets OS_TEST_DBAPI_ADMIN_CONNECTION to use
the PyMySQL connector instead of the MySQL-python connector.
This change is based on Ironic tox.ini, so I added Victor Sergeyev as
the coauthor of the patch.
Co-Authored-By: Victor Sergeyev <firstname.lastname@example.org>
The gate-nova-pip-missing-reqs job on the experimental queue is failing
and pointing out several missing runtime dependencies from
Also, we need to ignore the nova/test.py file during the scan since
pip-missing-reqs doesn't include test-requirements.txt.
The max-complexity is now set at 35. We are unable to lower this due to
the following offenders:
./nova/virt/xenapi/vmops.py:431:1: C901 'VMOps._spawn' is too complex (34)
./nova/network/manager.py:1226:1: C901 'NetworkManager._do_create_networks' is too complex (34)
./nova/network/neutronv2/api.py:370:1: C901 'API.allocate_for_instance' is too complex (34)
If your native locale is not en_US and you run tests, some will fail due
to the message translation that happens since the assertion messages are
in en_US (not translated, just hard-coded in the tests).
I recreated the bug by setting LANGUAGE=en_GB in my tox.ini and tested
Note that setting LANG=en_US had no effect.
In I7033847bab82b6d3cd8f2bdb66c553fec93d7acb, we disabled
N333 to allow changes in flight to still mix up oslo_ and
oslo. based imports. We should revert the change and enforce
every review use just the oslo_ version from now on.
There's lot of code in flight that will break if we switch
on the asserts to error out pep8/flake8 right now. So
we should temporarily switch it off. Doing this as a separate
review so it's easy to revery and stands out on its own.
- Add missing commas
- Replace single quote with double quote
- events field contains a list of events (event-create-req.json)
- Remove empty files
- Adds a JSON validation test to the docs pipeline
Co-Authored-By: Nicolas Simonds <email@example.com>
The _get_guest_config() method of libvirt's driver.py is currently the
worst offender in nova. This patch refactors the method to reduce the
complexity of the method from 40 (originally; it has since been dropped
to 35 by other patches) down to 24. With this change, the highest
complexity method is 37, so we can lower the limit in tox.ini to 38.
This is a refactor designed to reduce the cyclomatic complexity of the
current worst offender in nova: the servers.create() method of the nova
API. The complexity of the method was 46; it is now 22.
No logic changes were made.
This builds on the effort started with Change 129125 by Joe Gordon.
* Remove references in tox.ini to removed rules
* Fix minor changes (H501)
* Clarify that H306 (imports in alphabetical order shouldn't be skipped)
* Ignore H238: check for old style class declarations
* Temporary skip W292, fix in next patch.
The pylint tests have been broken for several months and
the Jenkins job has also been disabled. There has been
no sign of attempts to address this, so just delete what
remains of the pylint support so developers don't mistakenly
think it is supposed to work with patches they are submitting.
Start extracting all the setup code in note.test into fixtures which
will let us decompose test setup (and have test base classes that
don't setup everything).
This creates new OutputStreamCapture fixture and StandardLogging
fixture and unit tests for both of them plus tests for the exiting
H305/306 have to be turned off because hacking import rules are broken
and continue to believe that fixtures is a nova module not the
This moves the integrated tests into the functional tree as an initial
seed for the functional tests. It was agreed to as an early step at
the design summit. Work on cleaning these up will happen after the
PYTHON_HASH_SEED=0 is needed because these tests apparently remain
Also remove the use of test.ReplaceModule, which is an untested
mocking system in the nova tree that doesn't seem to reliably
work. Use stubs instead for the time being, as this will ensure
repeatability of the code.
Part of bp:functional-tests-for-nova
Tox functional job currently will fail because there are no tests,
which is makes it a little hard to atomically turn on voting with the
test move. To trick the system we can make tox functional always pass,
then change the tox definition to the real thing during the sideways
Part of bp:functional-tests-for-nova