This adds a pip install command to tox.ini that is only used when the
tox env is passed with the 'constraints' factor appended onto it.
As such this will not effect developer workflows or current unit tests.
The initial use of this will be in a non-voting job, to verify that the
constrained checks with tox are stable. DevStack is already running
constrained jobs, as such problems are no expected.
To run a tox with pip using constraints on a developer system a
developer should run the desired tox environment with -constraints.
For example: $(tox -epy27-constraints)
Pip will pull the current version of the upper-constraints.txt file down
from the git.openstack.org, however this method can be overriden to use
a local file setting the environment variable "UPPER_CONSTRAINTS_FILE"
to the local path or a different URL, it is passed directly to pip.
This is currently not enabled in the default tox run, however it is
possible to enable it as a default by adding it to 'envlist' in tox.ini
Depends-On: I17ac389f78af241917b6da7f049085f2b13d30f2
Change-Id: I8ea1710a4e3a287405978b467e0b2dfbb025b795
Implements Blueprint: Requirements-Management
doc/source/api and doc/build are build artifacts from building the docs
using the tox -e docs target. These are ignored in .gitignore since we
don't care about them from build to build - but leaving them around can
actually make the docs build fail if modules no longer exist but are
still sitting in these directories.
So this change just cleans those up each time you run tox -e docs.
Closes-Bug: #1488637
Change-Id: I646ae0b77a472c425d23815316e804b09bbe867d
If someone makes a change that breaks "tox -e genconfig"
it is not caught today by any of the gate/check jobs.
We should fail at least the docs job by running
oslo-config-generator to make sure it works.
Change-Id: I63fede4a7b5f358f2212383e2870cc64c4f9f1e4
`cmp` method was removed in Python 3, so it would be nice to use __lt__,
__le__, __eq__, __ne__, __gt__ and __ge__ instead, which are Py2 and Py3
compatible.
Change-Id: I1c89da0831b77b73f55d8681fd7d946535cc89b5
Now that we require at least pbr 1.3 and that has
commit 61a3a1ecdf99c680e509742fdc6c441b499af130 in it,
let's remove the workaround.
Change-Id: I4074b05a823d32f5db0051062527b0db4d379b62
Related-Bug: #1473401
Currently the tox py34 target uses testtools.run to run
a subset of our test harness. We need to be able to use
pretty_tox.sh just like py27 as we make progress with py34
support.
The first step is to make sure we can discover all the
tests using:
python -m subunit.run discover -t . ./nova/tests/ --list
So, we need to fix a bunch of things for the discovery
to work including updating to a new version of websockify.
In the xen code, we should keep the original import and
add except for py34 as xen uses an older python that does
not work with six.moves.
Depends-On: Ib4ef2e79b28b7180e564b3d6dc2188456c66c08a
Change-Id: I88b6746da6136a7386a173f6cacd42f0b470deee
nova/pci/stats.py:
need to specify a key parameter which is a function that returns
a sorting key. since the original intention with pool_cmp was to
sort by number of items in each pool, we just use key parameter
with a lambda
nova/scheduler/filters/trusted_filter.py
nova/scheduler/host_manager.py
needed list() so we could delete stuff inside the loop in py3
nova/tests/unit/scheduler/test_filter_scheduler.py
needed the import of test as some tests were failing in py3,
added noqa as test is not explicitly referenced within the
module
nova/tests/unit/scheduler/test_scheduler_options.py
need BytesIO for py3, StringIO will fail in many tests
nova/tests/unit/scheduler/test_scheduler_utils.py
message is not available in py3, so we need to adjust
use test.nested already added in a previous commit
nova/test.py
if we have a key which is a list, treat it just like a set
nova/utils.py
use reduce from six and explicitly typcast version_int as
an integer
Change-Id: Ic862873f38767f12b990a030b5812ae50a44c4d0
The 'tox -epep8' command takes a long time as it checks
every single file in the Nova git repository (~1,400 at
time of writing).
This makes tox use a simple wrapper around flake8 which
can be told to restrict the check to only files changed
in the current command. This can be invoked in a simple
manner with 'tox -epep8 -- -HEAD'. Since most commits
only touch a handful of files, this will usually be
far faster than checking all 1,400 source files.
To check an entire branch for bisectability it can be
automated via
git rebase -i master -x 'tox -epep8 -- -HEAD'
Change-Id: I157d1ccb883ca02402eee51fd7d6a50f86079389
* Fix sort(): dictionaries are no more comparable on Python 3, use a key
function building a sorted list of the dictionary items
* Replace __builtin__ with six.moves.builtins
* get_hash_str() now accepts Unicode: Unicode is encoded to UTF-8
before hashing the string. Mention also the hash method (MD5) in the
docstring.
* LastBytesTestCase: use binary files instead of text files. Use also a
context manager on the TemporaryFile to ensure that the temporary file
is closed (even on error).
* Functions: use the __code__ attribute instead of func_code, and
__closure__ instead of func_closure. These attributes are also
available on Python 2.6.
* Replace unichr() with six.unichr()
* SafeTruncateTestCase(): write directly the chinese Unicode character
instead of using safe_decode() (which is not reliable, it depends on
the locale encoding!)
* sanitize_hostname(): On Python 3, decode the hostname from Latin1 to
get back Unicode. Add also a comment explaining the purpose of the
conversion to Latin1.
* last_bytes(): replace the hardcoded constant 22 with errno.EINVAL and
add a comment explaining how the function works.
* tox.ini: add nova.tests.unit.test_utils to Python 3.4
Blueprint nova-python3
Change-Id: I96d9b0581ceaeccf9c35e0c9bada369e9d19fd15
This includes 2 classes of fixes (and 1 skip) for mock.
The first is the change in allowed values for assert_has_calls -
https://github.com/testing-cabal/mock/issues/263
The second is a yet unsolved bug around the use of autospec
https://github.com/testing-cabal/mock/issues/264
The skip is because something has changed with mock.open that is
causing the vhduils test to fail. We don't know why, but it's one test
to skip.
This also includes a brute force fix for tox -e docs, because pip is
no longer respecting the sphinx pin in test requirements. This has to
be landed with the other changes because they won't work without it.
Change-Id: Id835d080a1ada52cbd3f24dad9bab5eeb2f29a54
Partial-Bug: 1473401
Fix bytes vs Unicode issues:
* ssh_encrypt_text() now encodes text to UTF-8 if text type is Unicode.
* Encode Unicode to ASCII before encoding it to base64
* On Python 3, decode binascii.hexlify() and base64.encodestring()
output from ASCII to get Unicode (str type in Python 3) is required.
* convert_from_sshrsa_to_pkcs8(): reuse binascii.hexlify() instead of
a loop using struck.unpack() and "%02X" format.
* Replace str.encode('hex') with binascii.hexlify(str)
* Replace string.strip(text) with text.strip()
* Replace StringIO.StringIO with six.StringIO
* Add test on the output type of ssh_encrypt_text() and decrypt_text()
* Call utils.execute() with binary=True to not decode stdout/stderr
(to get them as raw bytes).
* Replace reduce() with six.moves.reduce()
* convert_version_to_str(): replace a/b with a//b to get integer
division
* tox.ini: add the following tests to Python 3.4
- nova.tests.unit.compute.test_keypairs
- nova.tests.unit.test_crypto
Blueprint nova-python3
Change-Id: I83d927166c0864020b205ac7495473795da7830d
The desired OS_TEST_DBAPI_ADMIN_CONNECTION string is now reflected
in the oslo.db 1.12.0 release's default behavior, and so does not
need to be set any longer to achieve the same opportunistic backend
discovery for tests.
Change-Id: I1af372f6f18038e8fc63076b4e399d0ae86f5a23
* FakeNovaException_Remote: on Python 3, define a __str__() method
instead of __unicode__() method. str(exc) now calls __str__() now
Pyton 3, __unicode__() is no more used.
* tox.ini: add nova.tests.unit.test_exception to Python 3.
Blueprint nova-python3
Change-Id: Ia3e3f253aa57b75a47f3ba044d2c539f9ce48aea
This change adds a basic bandit config for Nova. It can be invoked
by running the tox environment for bandit;
tox -e bandit
This is intended as a starting point for using bandit with Nova
and it should be revisited to improve the testing as more is learned
about the specific needs of the Nova code base.
Tox is configured to only show results for high and medium severity
results.
https://wiki.openstack.org/wiki/Security/Projects/Bandit
Change-Id: I3026b81317f0a6322acfc94784899a7453af586f
All tests in nova/tests/unit/objects/test*.py now run with
python3.4 tox target.
* Fix imports in limits.py and urlmap.py
* Add a list() as keys()/values() return iterator in
availability_zones.py, flavors.py and instance_numa_topology.py
* contextlib.nested is not present in python3, so whip up an
alternative using ExitStack(). Directly using ExitStack() wont
work for us.
* Add a few assertJsonEqual in the test cases
* Ensure fingerprinting generates the same exact value in both
python27 and python34
Blueprint nova-python3
Change-Id: I848c48475189c4b4ad8151e14509020ae7d110a4
* Replace dict(obj.iteritems()) with dict(obj) where obj is a dictionary
or an oslo.db object
* Replace obj.iteritems() with obj.items() where obj is a dictionary:
obj.items() works on Python 2 and Python 3.
* Replace filter() and map() with list-comprehension using if when a list is
expected. On Python 3, filter() and map() return an iterator.
* Replace obj.keys() with list(obj.keys()) when a list is expected.
Replace obj.keys()[0] with list(obj.keys())[0]. On Python 3,
dict.keys() now returns an iterator.
* Replace (int, long) with six.integer_types
* Get the name of a function using the __name__ attribute, instead of
the func_name attribute. The func_name attribute was removed
in Python 3.
* InstanceTypeTestCase: use dict.update() to combine dictionaries
instead of using dict.items()+reduce()+dict()
* matchers.py: replace StringIO.StringIO with six.StringIO.
* tox.ini: add nova.tests.unit.db.test_db_api to Python 3.4
Blueprint nova-python3
Change-Id: Iae4f6d971818d5b38e838a83753a0707e954bb04
Oslo team is actively getting rid of the old style generator.py
in openstack/common/config. So we should switch to the newer
oslo-config-generator which is the way forward. For simplicity's
sake, i used one single list_opts in opts.py. However we can
split this up and add entrypoints in setup.cfg if we decide to
do that instead.
blueprint oslo-config-generator
Change-Id: Ib7c72af7be47df08983708ac6da0ce103518b78c
When a development environment is under a proxy, tox is failed even if
environment variables of the proxy are set.
This patch fix this problem.
Change-Id: If0bc42891a0714cc9b37b9c211388db20555ea7e
On Python 3, if LC_ALL is not passed then the interpreter will assume it
is C (which causes the interpreter to use ASCII). This, in turn, causes
the unicode in the README.rst for rfc3986 to trigger a UnicodeDecodeError
exception.
jlvillal (John Villalovos) pointed out
I340c88ea7d098ca5943d60adc73f63a0af79a405 in IRC as the fix to this bug.
Closes-bug: 1460206
Change-Id: If05e3fe76450c06431413f63e46e154cf44fc1cf
Commit 7fc0dc0568 removed the
PYTHONHASHSEED environment variable usage so the comment no longer
applies.
Change-Id: I82591c6de9f81d1f72ba99d4fe9561ff60aefac6
With this change, "tox -e py34" now pass.
Changes:
* Replace unichr() with six.unichr()
* Replace StringIO.StringIO() with six.Bytes() in
nova.crypto.generate_key_pair()
* On Python 3, replace UserDict.IterableUserDict with
collections.UserDict
* Replace __builtin__.open with six.moves.builtins.open
* Replace ConfigParser import with six.moves.configparser
* On Python 3, get the original "queue" module instead of the "Queue"
module in nova/virt/libvirt/host.py
* Replace urllib2 with six.moves.urllib
* tox.ini: "tox -e py34" now only runs nova.tests.unit.test_versions
Blueprint nova-python3
Change-Id: I056769c7c5b32276894f7aade8c0a27af81c42ae
Add test-requirements-py3.txt to use PyMySQL, instead of MySQL-python,
on Python 3.
Add a py34 test environment to tox.ini which uses
test-requirements-py3.txt and sets OS_TEST_DBAPI_ADMIN_CONNECTION to use
the PyMySQL connector instead of the MySQL-python connector.
This change is based on Ironic tox.ini, so I added Victor Sergeyev as
the coauthor of the patch.
Co-Authored-By: Victor Sergeyev <vsergeyev@mirantis.com>
Blueprint nova-python3
Change-Id: I5a27c7d627105b614da3cab3a87296a8e186b555
The gate-nova-pip-missing-reqs job on the experimental queue is failing
and pointing out several missing runtime dependencies from
requirements.txt.
Also, we need to ignore the nova/test.py file during the scan since
pip-missing-reqs doesn't include test-requirements.txt.
requests
--------
nova/scheduler/filters/trusted_filter.py:50 dist=requests
nova/api/ec2/__init__.py:30 dist=requests
prettytable
-----------
nova/openstack/common/cliutils.py:29 dist=prettytable
python-keystoneclient
---------------------
nova/context.py:22 dist=python-keystoneclient
nova/network/neutronv2/api.py:21 dist=python-keystoneclient
nova/compute/manager.py:42 dist=python-keystoneclient
nova/volume/cinder.py:27 dist=python-keystoneclient
nova/context.py:23 dist=python-keystoneclient
nova/network/neutronv2/api.py:23 dist=python-keystoneclient
nova/network/neutronv2/api.py:24 dist=python-keystoneclient
nova/volume/cinder.py:28 dist=python-keystoneclient
nova/keymgr/barbican.py:25 dist=python-keystoneclient
nova/network/neutronv2/api.py:22 dist=python-keystoneclient
netifaces
---------
nova/compute/utils.py:21 dist=netifaces
setuptools
----------
nova/openstack/common/versionutils.py:26 dist=setuptools
nova/virt/xenapi/image/bittorrent.py:18 dist=setuptools
Closes-Bug: #1453857
Change-Id: If880d76cf3e47c14b370d63313330996e09f5ab8
As all bugs, related with that random
PYTHONHASHSEED fixed, final patch removes that
from tox.ini
Change-Id: Ice3920d1290bef46805842734d5bc1e381baf5ec
Closes-Bug: #1348818
The default tox cmd should also run the 'functional' target, this got
missed in adding that sub target.
Change-Id: I9a10b0c9798d9f90db03991c45b9f2b285d1e4d0
The max-complexity is now set at 35. We are unable to lower this due to
the following offenders:
./nova/virt/xenapi/vmops.py:431:1: C901 'VMOps._spawn' is too complex (34)
./nova/network/manager.py:1226:1: C901 'NetworkManager._do_create_networks' is too complex (34)
./nova/network/neutronv2/api.py:370:1: C901 'API.allocate_for_instance' is too complex (34)
Change-Id: I6101dcb594b88d346edd72eea2ef2250c4e2891f
If your native locale is not en_US and you run tests, some will fail due
to the message translation that happens since the assertion messages are
in en_US (not translated, just hard-coded in the tests).
I recreated the bug by setting LANGUAGE=en_GB in my tox.ini and tested
against nova.tests.unit.api.openstack.test_wsgi.
Note that setting LANG=en_US had no effect.
Closes-Bug: #1427745
Change-Id: Ib1c71d0d4170ba1a7ed2fe3eaf0e2d8071ec01d6
In I7033847bab82b6d3cd8f2bdb66c553fec93d7acb, we disabled
N333 to allow changes in flight to still mix up oslo_ and
oslo. based imports. We should revert the change and enforce
every review use just the oslo_ version from now on.
Change-Id: I1db83048c569bcdd62538cce9df076fc355f7b26
There's lot of code in flight that will break if we switch
on the asserts to error out pep8/flake8 right now. So
we should temporarily switch it off. Doing this as a separate
review so it's easy to revery and stands out on its own.
Change-Id: I7033847bab82b6d3cd8f2bdb66c553fec93d7acb
Commit a8a87900cf removed
H238 from the flake8 ignore list but missed removing the
comment about why we were ignoring it in the first place.
Change-Id: I1fd89aafeea50bead18c8bf4ffdc7577ded2bde3
- Add missing commas
- Replace single quote with double quote
- events field contains a list of events (event-create-req.json)
- Remove empty files
- Adds a JSON validation test to the docs pipeline
Co-Authored-By: Nicolas Simonds <nic@metacloud.com>
Change-Id: I9a4a9b6b8e2da73b5f25145be3b2faa44ebdb340
Fix H238 failures and enable check for H238
H238 old style class declaration, use new style
(inherit from `object`)
Change-Id: I71dd353ab260a513c5df5af0ee07754e23878570
The _get_guest_config() method of libvirt's driver.py is currently the
worst offender in nova. This patch refactors the method to reduce the
complexity of the method from 40 (originally; it has since been dropped
to 35 by other patches) down to 24. With this change, the highest
complexity method is 37, so we can lower the limit in tox.ini to 38.
Change-Id: I05de301bc3dfd84ed15ce3cbb8a7c50f2d1385ed
This is a refactor designed to reduce the cyclomatic complexity of the
current worst offender in nova: the servers.create() method of the nova
API. The complexity of the method was 46; it is now 22.
No logic changes were made.
This builds on the effort started with Change 129125 by Joe Gordon.
Closes-Bug: 1403586
Change-Id: Ib102091773f786b8c8dc5b79c6d776418ed1d57d
W292 no newline at end of file
W292 started failing with hacking 0.10.0, fix all cases and gate on the
rule.
Change-Id: I6196e8059e71730cc567d4dd211264cdd2272c40
Release notes:
http://git.openstack.org/cgit/openstack-dev/hacking/tag/?id=0.10.0
* Remove references in tox.ini to removed rules
* Fix minor changes (H501)
* Clarify that H306 (imports in alphabetical order shouldn't be skipped)
* Ignore H238: check for old style class declarations
* Temporary skip W292, fix in next patch.
Change-Id: I6d990a564df6a312bd09b2a152315bbdba732082
The pylint tests have been broken for several months and
the Jenkins job has also been disabled. There has been
no sign of attempts to address this, so just delete what
remains of the pylint support so developers don't mistakenly
think it is supposed to work with patches they are submitting.
Change-Id: Ifacc99e45acb0e9a72be6c31113e0cf073354f02
Start extracting all the setup code in note.test into fixtures which
will let us decompose test setup (and have test base classes that
don't setup everything).
This creates new OutputStreamCapture fixture and StandardLogging
fixture and unit tests for both of them plus tests for the exiting
Conf fixture.
H305/306 have to be turned off because hacking import rules are broken
and continue to believe that fixtures is a nova module not the
absolute one.
Change-Id: I97fdacdf5a6ad2957b0efa1e21ae084d4eb04ab9
This moves the integrated tests into the functional tree as an initial
seed for the functional tests. It was agreed to as an early step at
the design summit. Work on cleaning these up will happen after the
move.
PYTHON_HASH_SEED=0 is needed because these tests apparently remain
order dependent.
Also remove the use of test.ReplaceModule, which is an untested
mocking system in the nova tree that doesn't seem to reliably
work. Use stubs instead for the time being, as this will ensure
repeatability of the code.
Part of bp:functional-tests-for-nova
Change-Id: I742c487dd7bca9de3e811c545de477367fff112c
Tox functional job currently will fail because there are no tests,
which is makes it a little hard to atomically turn on voting with the
test move. To trick the system we can make tox functional always pass,
then change the tox definition to the real thing during the sideways
move.
Part of bp:functional-tests-for-nova
Change-Id: Ic276522851fba6d1d95dd1f3bdde1d48935568a3
Sachi King