354 Commits

Author SHA1 Message Date
Eric Fried
849c89d0e5 Remove pep8 whitespace ignores
We had carried over the full list of pep8 whitespace ignores from nova.
Trying to address them all in the entire nova repository would have been
too big a task; but it's tenable here in placement.

Do it now rather than letting these whitespace issues compound.

This change removes the E* whitespace ignores and fixes the pep8 issues
they were masking.

Change-Id: Icbabdb9b56fd5a3e9fd14ab537abf3d0d6456fee
2019-03-05 18:18:30 -06:00
Eric Fried
0b8475db68 Use tox 3.1.1 fixes
By 3.1.1, tox had fixed:
- [1] which mishandled the basepython directive.
- ignoring PYTHONDONTWRITEBYTECODE [2]

Require this version and remove the workarounds.

See ML thread [3] for some details

[1] https://github.com/tox-dev/tox/issues/477
[2] 336f4f6bd8
[3] http://lists.openstack.org/pipermail/openstack-dev/2018-July/thread.html#132075

Change-Id: I87cbf4b26bdd40a8b7a699966e1ace838a2491de
2019-02-14 09:59:07 -06:00
Eric Fried
9e5436efab tox: Don't write byte code (maybe)
In tox versions after 3.0.0rc1 [1], setting the environment variable
PYTHONDONTWRITEBYTECODE will cause tox not to write .pyc files, which
means you don't have to delete them, which makes things faster.

In older tox versions, the env var is ignored.

If we bump the minimum tox version to something later than 3.0.0rc1, we
can remove the commands that find and remove .pyc files.

[1] 336f4f6bd8

Change-Id: Id9a97af032a2dbefdc50057270368ad087e6bb5f
2019-02-14 09:54:44 -06:00
Chris Dent
4469a813cb Correct lower-constraints.txt and the related tox job
While exploring removing unused packages from lower-constraints.txt it
became clear that the lower-constraints job was not working as expected:
Because our tox config has usedevelop=True 'setup.py develop' is called
to install the placement package after the install command is called.
This means that the lower-constraints are clobbered.

I had mistakenly assumed that turning off 'usedevelop', which causes
'setup.py install' would not make any difference, because it usually
installs dependencies too. It turns out however, that when using pbr
and within a git working dir, it does not. That took some time to
figure out. Oh well.

This change makes it so that we create the tox environment using
usedevelop=False and with our own install_command, to avoid upper
constraints conflicting with lower constraints.

This flagged up a few changes, the main one being that we did not have
a new enough version of keystonemiddleware in order to require use of
www_authenticate_uri. requirements.txt is updated for this as well.

And PasteDeploy needed to be updated to work with Python 3's notion
of namespace packages.

psycopg2 need a newer version to work with Postgresql 10.

oslotest needs to be raised to 3.4.0 because the tests in
cmd.test_manage use features to control what is capture by the Output
fixture from oslotest. Note that the lower-constraints job found
this problem and also demonstrates why we must run the lower-constraints
job without upper-constraints being involved. upper-constraints will
"win" and we don't want that. The point of the job is find packages
where lower-constraints are wrong, so it must "win".

The end result here is a lower-constraints.txt file that starts from
the lower-constraints.txt defined by nova, and then is adapted to
update the versions of packages that were not up to date, remove
those packages which are no longer present, and add some that are
now required.

Change-Id: Id66a28f7ace6fc2adf0e1201d9de5f901234d870
2018-11-27 17:41:02 +00:00
Chris Dent
3b569321cc Clean up and clarify tox.ini
While looking into making py36 available when running tox without '-e'
and making py37 more available for both unit testing and functional
testing, it became clear that our tox.ini was a bit messy. This change
cleans it up to:

* Removes all py* jobs and lets tox define those automatically. Their
  commands come from the base [testenv].

  Non-unit tests need to define their commands specifically.

* move pep-related tox below both unit and functional

* remove some out of date comments

* set the default when using tox without -e to unit and functional
  in 2.7 and 3.6, and pep8

* removes the 'stestr slowest' from the unit and functional test env's.
  It's not used consistently and the output is far from consistent so
  it doesn't seem to have any particular value (for this project)

This is one of several ways to do this cleanup. This way assumes some
knowledge of the fact that tox defines a lot default jobs with default
python versions (via factors[1]).

Note that the releasenotes job does not currently work and did
not work before these changes, a conf.py is missing.

[1] https://tox.readthedocs.io/en/latest/config.html#factors-and-factor-conditional-settings

Change-Id: I9a877439e13eb753c7f98dc02e41db891ce33690
2018-11-06 16:18:14 +00:00
Zuul
ab62e9f4da Merge "Reduce max-complexity to 15" 2018-10-11 15:19:30 +00:00
Zuul
8c85050960 Merge "max-complexity=>16: refactor GET /a_c qs parsing" 2018-10-02 11:22:02 +00:00
Zuul
baaa0d023d Merge "Put stestr group_regex in .stestr.conf" 2018-09-25 21:52:49 +00:00
Zuul
28a5beaec3 Merge "Use both unit and functional for coverage testing" 2018-09-25 21:47:57 +00:00
Takashi NATSUME
e04ff2395b Fix missing specifying doctrees directory
In tox.ini file, the doctrees directory is specified
as an argument of sphinx-build command in api-ref,
releasenotes targets.
But it is missing in docs. So add it.

TrivialFix
Change-Id: I767b675b503cd627e80c7c1ff434d4a8ad53203a
2018-09-21 09:40:20 +09:00
Eric Fried
7ddf015238 Reduce max-complexity to 15
The dependencies knock down the four most complex methods in the
project, bringing our most complex method to 14 (and hence our
max-complexity limit to 15).

Ife303d018a26ddcf8ea41d5950dc774c5e898a56
placement/util.py: 'parse_qs_request_groups' (17)

I9f3aa66b4806cf2c34f5271a5f4426839cb872ce
placement/objects/resource_provider.py:
  'ResourceProviderList._get_all_by_filters_from_db' (16)
placement/objects/resource_provider.py:
  '_get_provider_ids_matching' (16)

I80f8b1a74b8afdabbb85afc6caef1abef8ac3751
placement/objects/resource_provider.py:
  'AllocationCandidates._get_by_requests' (16)

Change-Id: I11d3e3c0e1baf729f2acceafbe2f115212a08385
2018-09-18 14:47:23 +00:00
Eric Fried
95f87ca109 max-complexity=>16: refactor GET /a_c qs parsing
Before this patch, the project's max-complexity of 17 was perpetrated by
RequestGroup.dict_from_request, the request querystring parsing method
used for GET /allocation_candidates. This refactor-only patch splits
that method into smaller chunks, the most complex of which now comes in
at 8.

There is no functional change, hence no deltas to tests.

Change-Id: Ife303d018a26ddcf8ea41d5950dc774c5e898a56
2018-09-14 16:15:53 -06:00
Chris Dent
12a9e321bd Put stestr group_regex in .stestr.conf
This dries up tox.ini a bit while preserving the explanatory
comment. Having the grouping expression on test jobs that don't
require it has only minor impact: the regex is applied to the
list of tests and no groups are found.

Change-Id: I89330cf5b0ace25db6a7575cfa4f93d9575b08ac
2018-09-14 13:30:14 -06:00
Chris Dent
09905baab2 Use both unit and functional for coverage testing
Placement has always relied heavily on functional testing for both API
and database testing, so not including those tests in 'coverage' leads
to misleading results.

This change adjusts the cover job in tox to use a test path of
placement/tests, resulting in all the tests running.

In the process .coveragerc is updated to be placement oriented
and make sure we do not ignore branches.

This is in preparation for turning on a check job for coverage.
See this email for more on that:

http://lists.openstack.org/pipermail/openstack-dev/2018-September/134385.html

A followup patch will turn on a check job for it.

Change-Id: I6c122facc9c9f6e0b2a1d69c6aaf5faeee7cf253
2018-09-14 13:30:14 -06:00
Zuul
f2a242d457 Merge "s/placement-config-generator/config-generator/" 2018-09-14 16:34:23 +00:00
Eric Fried
f5a25d5fd6 s/placement-config-generator/config-generator/
Change [1] added support for tox -e genconfig via a configuration file
called placement-config-generator.conf. Since we're in the placement
project and only have one config generator, the placement- part is
redundant. In keeping with similar pieces (e.g. policy-generator.conf)
this patch renames the file and fixes references.

[1] https://review.openstack.org/#/c/601838/

Change-Id: I8ea9c00d26539feab141ada493bc825987046b3d
2018-09-14 16:10:58 +00:00
Zuul
3c30ed3cb4 Merge "Set upper bound on max-complexity in pep8" 2018-09-13 23:16:14 +00:00
Chris Dent
6cd3d800b9 Set upper bound on max-complexity in pep8
The current max-complexity in placement code is 17, so set the
upper bound in the pep8 tests to max of 18 so we get an error
when we're going to make it worse. The current high complexity
(but still under 18) functions are:

./placement/util.py:415:1: C901 'parse_qs_request_groups' is too complex (17)
./placement/objects/resource_provider.py:1341:1: C901 'ResourceProviderList._get_all_by_filters_from_db' is too complex (16)
./placement/objects/resource_provider.py:2848:1: C901 '_get_provider_ids_matching' is too complex (16)
./placement/objects/resource_provider.py:4076:1: C901 'AllocationCandidates._get_by_requests' is too complex (16)

Change-Id: I58dc5acb9bb13dcdb5623d13515820591ec5f974
2018-09-13 16:45:59 -06:00
Zuul
7e7dc3b930 Merge "Rename files to remove 'placement'" 2018-09-11 15:04:21 +00:00
Zuul
c53e4a92f5 Merge "Make docs build" 2018-09-07 20:34:48 +00:00
EdLeafe
5794b857b4 Rename files to remove 'placement'
When part of Nova, many files were named in such a way as to indicate
that they were separate from the nova files that did the same thing. Now
that that distinction is no longer needed, we can rename them to sound
more natural.

Additionally, references to these file names in other parts of the code
have been updated.

Change-Id: Id4439b4af827b64bbac213085ac74d1283c96db0
2018-09-07 20:10:06 +00:00
Eric Fried
9b584fcd1d Make docs build
This patch does pretty much the minimum possible to get `tox -e docs`
building and gating. Much, much more needs to be done to make the
content good.

Also corrects a link in the api-ref which, once these are publishing,
would have pointed to the wrong place.

Change-Id: I5cbc3a3cceeaeaa7be5593658b6a03fa25fb69d0
2018-09-06 11:45:11 -05:00
Chris Dent
15ca8bc758 Removing non-existent job from tox envlist
The pip-missing-reqs job does not exist but because of the way
we have tox configured, running 'tox' by itself will run what
amount to a null job, consuming time but testing nothing.

See also: I8355372da93b93cc1c7f7a1501af0fbd01eee615

Change-Id: I0969908ca7f5acbcde4a3d561366295bf2403c83
2018-09-04 20:49:54 +00:00
Chris Dent
eaeeea1cfe Inspect and correct tox.ini, .stestr.conf and setup.cfg
Begin the process of getting placement tests running, by getting the
test harness opertional.

Before we can expect the tests to be able to run the configuration for
the placement package and its test harness needs to be adjusted for
"nova" -> "placement". These changes are made by inspection and based
on previous iterations of doing these kinds of cleanups, it won't be
until later in this stack of change that actually passing tests will
show up. However, the logs for attempted tests runs should be getting
a little bit further than the parent.

setup.cfg changes quite a bit because there are many things in it that
are not relevant. placement (thus far) really only has one entry point.
As noted, by a TODO, a placement-manage and placement-status will be
desired in the near future.

Change-Id: Iaa385c6e8653a9646a21bb65203deaa05ec43217
2018-09-04 10:31:25 -05:00
Eric Fried
38dfc8f56f Remove blacklisted py3 xen tests
Xen UT failures seem to have cleared up, which may or may not have
anything to do with [1]. Remove the blacklist file and get these back
in the py3 jobs.

[1] https://review.openstack.org/#/c/591061/

Change-Id: Iacc1ae01f535caed64793d57b757ea07a8c46620
2018-08-14 16:34:19 +00:00
Chris Dent
640d49f283 Add explicit functional-py36 tox target
Without this, you can successfully run a 'functional-py36' environment,
but no actual tests are run. Python 3.6 is common in many environments
these days, and we want devs to test locally. This helps.

Change-Id: I2c04b7a8b6ee06638c6ce92dc51e200fa2b13539
2018-08-08 11:03:25 +01:00
Stephen Finucane
8424f21cfc tox: Ensure reused envdirs share the same deps
In commit df669caf, we modified tox.ini to reuse a shared directory for
environments with the same dependencies. The 'venv' directory was
chosen as this seemed like a suitably generic target. However, the
'venv' target actually uses a non-default values for 'deps' as doc
requirements are included to allow things like reno using this target.
This means switching between the 'venv' target and any of the other
other targets using this directory will result in the virtualenv being
rebuilt each time.

Resolve this by using an explicit directory for these shared
environments. A single setting of 'deps' for one of the targets is
removed to make this clear.

Change-Id: I239606f5f354a5536b20b1d31ee81d80b4572106
2018-08-02 11:31:41 +01:00
Zuul
5b474d1eb9 Merge "Convert 'placement_api_docs' into a Sphinx extension" 2018-08-01 13:45:35 +00:00
Zuul
7f2f183824 Merge "tox: Reuse envdirs" 2018-07-30 15:02:21 +00:00
Stephen Finucane
1e106d0cf4 tox: Silence psycopg2 warnings
There's nothing we can do to fix this so simply silence it until we no
longer support that version.

Change-Id: Ic7a199a1374e6f36c092e5ca6faa435aa5bf4cbd
2018-07-24 14:28:07 +01:00
Sean Mooney
a57b86d221 update tox venv env to install all requirements
- This change adds requirements.txt to the list
  of requirements files used when createing
  the tox venv environment.

Change-Id: Ib222b3cb68d2129c061400f51f50a19d186b9080
2018-07-17 19:44:05 +01:00
Stephen Finucane
ef7c2d24ed Convert 'placement_api_docs' into a Sphinx extension
This ensures it will get run in the gate.

Change-Id: I923c39d3115b595b8de94e85d2977eb7782d98e2
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
2018-07-02 11:21:47 +01:00
Stephen Finucane
c045217012 tox: Reuse envdirs
This massively reduces the amount of time needed to install dependencies
and saves a lot of disk space to boot #winning

Note that it's not possible to use 'envdir = {[testenv:py27]envdir}' as
that syntax doesn't work with implicit attributes.

Change-Id: I229e1961a66db08be2d4f7235d2c3724e875f13b
2018-06-28 16:36:33 +01:00
Stephen Finucane
2636cce7e1 tox: Document and dedupe mostly everything
Take advantage of tox's expressiveness to avoid duplication wherever
possible and document the many targets we have.

Change-Id: I33675b7eedcffcebf1115dd48757874cb73f3e2f
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
2018-06-28 15:24:17 +01:00
Stephen Finucane
3994476547 trivial: Remove 'tools/releasenotes_tox.sh'
This is not needed since reno 2.1.0 (commit f8fc8f97) [1].

[1] f8fc8f97ff

Change-Id: I66ba0e0835970b965acce00ba69d143bd4471037
2018-06-28 15:06:46 +01:00
Doug Hellmann
4a6cdbd3ab fix tox python3 overrides
We want to default to running all tox environments under python 3, so
set the basepython value in [testenv].

We do not want to specify a minor version number, because we do not
want to have to update the file every time we upgrade python.

We must explicitly set basepython in envs that are usually "automatic" -
otherwise due to https://github.com/tox-dev/tox/issues/425 these will
use `basepython` instead of their automatic equivalents.

Change-Id: I1026110f0f1736127cf0af8ec2ef791a0f999e3a
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2018-06-08 08:55:56 -05:00
Matt Riedemann
519e5a22d1 Implement granular policy rules for placement
This adds a granular policy checking framework for
placement based on nova.policy but with a lot of
the legacy cruft removed, like the is_admin and
context_is_admin rules.

A new PlacementPolicyFixture is added along with
a new configuration option, [placement]/policy_file,
which is needed because the default policy file
that gets used in config is from [oslo_policy]/policy_file
which is being used as the nova policy file. As
far as I can tell, oslo.policy doesn't allow for
multiple policy files with different names unless
I'm misunderstanding how the policy_dirs option works.

With these changes, we can have something like:

  /etc/nova/policy.json - for nova policy rules
  /etc/nova/placement-policy.yaml - for placement rules

The docs are also updated to include the placement
policy sample along with a tox builder for the sample.

This starts by adding granular rules for CRUD operations
on the /resource_providers and /resource_providers/{uuid}
routes which use the same descriptions from the placement
API reference. Subsequent patches will add new granular
rules for the other routes.

Part of blueprint granular-placement-policy

Change-Id: I17573f5210314341c332fdcb1ce462a989c21940
2018-05-17 11:12:16 -04:00
Zuul
5ff219c624 Merge "add lower-constraints job" 2018-05-15 01:12:25 +00:00
Doug Hellmann
84d58bf9d6 add lower-constraints job
Create a tox environment for running the unit tests against the lower
bounds of the dependencies.

Create a lower-constraints.txt to be used to enforce the lower bounds
in those tests.

Add openstack-tox-lower-constraints job to the zuul configuration.

See http://lists.openstack.org/pipermail/openstack-dev/2018-March/128352.html
for more details.

Change-Id: Ic28558ee6481d49a9b4e5dc2c4182504e330448f
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
Co-Authored-by: Eric Fried <efried@us.ibm.com>
Co-Authored-by: Jim Rollenhagen <jim@jimrollenhagen.com>
2018-05-14 16:47:07 +00:00
Stephen Finucane
d31e773383 Remove stale pip-missing-reqs tox test
pip_missing_reqs tool is no longer maintained and has broken with
release 10 of pip

Refer to:
 http://lists.openstack.org/pipermail/openstack-dev/2018-April/130027.html

Change-Id: I1b02384494ae9f440b72b98d9ae5f31d88dc8967
2018-05-01 17:40:03 +01:00
melanie witt
ea1c6f9d44 Default to py3 for the pep8 tox env because it's stricter
We recently fixed a couple of pep8 failures under py3 [0] which got
through because we're not gating pep8 on py3. This changes the python
version for the pep8 tox env to python3 because the py3 checks are
stricter. This updates the fast8 tox env as well.

[0] https://review.openstack.org/#/c/557633

Change-Id: I540377606a8224ba569b8c145773e7d057575bd8
2018-04-04 15:24:00 +00:00
Stephen Finucane
e3e3b97579 tox: Make everything work with Python 3
Nothing too fancy here except that we've to work around a really ugly
side-effect of blockdiag.

Change-Id: Ibd32d30aacae65702d0ccbdb8a02b1667ec4e8ee
2018-03-28 12:57:24 +01:00
Stephen Finucane
18e5240134 tox: Remove unnecessary configuration
These were either unnecessary or overrode default configuration.

Change-Id: Ie17998dc564f06e9133c8412c24bc041b1e9a603
2018-03-26 15:12:28 +01:00
Stephen Finucane
cd1e65831d tox: Fix indentation
Makes life easier for future, actually important changes.

TrivialFix

Change-Id: Idda28f153d5054efc885ef2bde0989841df29cd3
2018-03-26 15:05:08 +01:00
Stephen Finucane
7b01024591 Follow the new PTI for document build
The Project Testing Interface [1] asks that we list requirements in
'doc/requirements.txt' and build docs by calling 'sphinx-build' directly
instead of via the 'build_sphinx' setuptool/distutils wrapper. Start
doing this.

[1] https://governance.openstack.org/tc/reference/project-testing-interface.html

Change-Id: If9342c8ea757b1735f2488db751008984fb33baf
2018-03-16 14:39:36 +00:00
Zuul
1c903d7317 Merge "Remove single quotes from posargs on stestr run commands" 2018-02-17 15:49:01 +00:00
Matthew Treinish
32901b002a Remove single quotes from posargs on stestr run commands
This commit removes the single quotes from all the stestr run commands
in the tox.ini. These were originally added as a workaround from the
ostestr days to ensure people could run a subset of tests with a regex
like:

  tox -epy27 regex

With the switch to stestr this is no longer needed because stestr treats
extra arguments as a filter regex. Quoting the posargs has a negative
effect because '--' parameters are treated as a string on input instead
of a parameter.

Fixing this also uncovered another issue in the stestr commands for the
functional jobs. The CLI arg for stestr versions <=1.1.0 is
--group_regex not --group-regex. This commit fixes this to make those
jobs pass without the quotes.

Change-Id: I512a517fa6890b29de24060376b4d7e061dc3add
2018-02-16 22:07:22 -05:00
Zuul
c4aedef4f8 Merge "add whereto for testing redirect rules" 2018-01-25 06:48:57 +00:00
Chris Dent
656d2bcddf Enable py36 unit tests in tox
With the current super-explicit version of tox.ini, it is not possible
to run tox -epy<any version> and have tests discovered and run. This
is because the default testenv does not run tests. Therefore, the
easiest way to add another version is to simply add another testenv,
as done in this change.

Python3.6 is the default Python3 on recent distributions so it needs
to be present if people are expected to be able to run python3 unit
tests before pushing their changes.

Change-Id: I728e482af3142bb101fc6bf52ec7926e29a42594
2018-01-16 10:14:46 +00:00
Matthew Treinish
bc3d836f30 Finish stestr migration
Nova switched to using stestr for all it's in tree testing except for
the coverage job. This commit updates that job so it also uses stestr
and cleans up the last bits of testr setup left in tree. As part of
this change this moves all the one off per job configuration into the
tox job definitions instead of hiding it in either the .stestr.conf or
in a wrapper script.

Change-Id: I36e0e791d485b44641a1ed8770eca8e95aca19be
2017-11-24 16:51:12 -05:00