415 Commits

Author SHA1 Message Date
Chris Dent
bed7808b59 Don't use OVO with ResourceProvider and ResourceProviderList
Turn ResourceProvider and ResourceProviderList into
classical Python objects.

There are two changes here which deserve particular
attention because they are more changing than the other
OVO removals:

* There were two deepcopy calls in the allocation request
  handling. When using OVO, the __deepcopy__ handling
  built into it prevents deep recursion. I changed them
  to copy and things still work as expected. This will
  be because using the nested objects by reference is
  acceptable.

* The removal of OVO removes the detection of changed fields.
  This was being used when creating and saving resource
  providers (at the object level) to automagically detect
  and prevent writing fields we don't want to.

  This change removes that functionality. Instead if bad
  data has made it as far as the create or save calls, we
  simply don't write it. The HTTP layer continues to
  maintain the guards it already had in place to prevent
  badness. Tests which were testing the object layer
  are removed.

  The create_provider function in functional/db/test_base
  allowed (trying to) create a provider with a root
  but that functionality was only called from one place.
  That place and the functionality in create_provider
  is removed.

Other things to note:

* As with the other changes, where context is actually used
  by the object it is required in the constructor. This
  cascades some changes to tests.

* A test that checks to see if adding traits resets the
  changes on a rp has been removed, because we don't
  track that any more if we haven't got OVO.

* The last_modified handling in placement/util no longer
  need NotImplemented handling, that was an artifact of
  OVO.

oslo.versionedobjects is removed from requirements. This
doesn't have a huge impact on the size of the virtualenv:
114M -> 107M [1] but it does take us from 132 python
dependencies (via pip list) to 119, removing plenty of
stuff that was only being called in because stuff that
we don't use depended on it.

lower-constraints.txt is updated to reflect the removal
of dependencies that are no longer needed.

[1] Of note, 26M of that is babel. Do we need to translate
exceptions? See email disussion at
http://lists.openstack.org/pipermail/openstack-discuss/2019-February/thread.html#3002

Change-Id: Ie0a9351e0d7c762c9c96c45cbe50132a0fbd1486
2019-02-25 23:48:33 +00:00
Chris Dent
71a6aa21c6 Use one ConfigOpts in placement-status
Because of a conflict between how the oslo.upgradecheck library uses
oslo config and how we want to use it in placement, two different
ConfigOpts were needed to avoid an args already parsed error. The
new release of the oslo.upgradecheck library is change to allow
the two steps in main (registering the CLI opts and running the
command) to be called separately if desired. Doing so allows us
to use just one ConfigOpts.

Requirements and constraints are updated.

Change-Id: I792df18cb17da95659628bfe7f7a69897c6f37ab
2019-02-19 12:36:02 +00:00
Chris Dent
4acab4605f Update standard resource class counts in tests
The 0.2.0 release of os-resource-classes has happened, adding the
'PCPU' class. It's already updated in global upper-constraints,
so update it in requirements.txt and lower-constraints and change
the gabbi tests which count resource classes.

Change-Id: I4a189aca59485d65ad8a7c9bfbeca7ac995ed336
2019-01-23 13:26:07 +00:00
Chris Dent
a241fcee4c Use os-resource-classes in placement
os-resource-classes is a python library in which the standardized
resource classes are maintained. It is done as a library so that
multiple services (e.g., placement and nova) can use the same stuff.

It is used and managed here in the same way the os-traits library is
used: At system start up we compare the contents of the
resource_classes table with the classes in the library and add any
that are missing. CUSTOM resource classes are added with a high id
(and always were, even before this change). Because we need to
insert standard resource classes with an id of zero, so we need to
protect against mysql thinking 0 on a primary key id is "generate
the next one". We don't need a similar thing in os-traits because
we don't care about the ids there. And we don't need to guard
against postgresql or sqlite at this point because they do not have
the same behavior.

The resource_class_cache of id to string and string to id mappings
continues to be maintained, but now it looks solely in the database.
As part of confirming that code, it was discovered that the reader
context manager was being entered twice, this has been fixed.

Locking around every access to the resource class cache is fairly
expensive (changes the perfload job from <2s to >5s). Prior to this
change we would only go to cache if the resource classes in the
query were not standards. Now we always look at the cache so rather
than locking around reads and writes we only lock around writes.
This should be okay, because as long as we do a get (intead of the
previous two separate accesses) on the cache's dict that operation
is safe and if it misses (because something else destroyed the
cache) the fall through is to refresh the cache, which still has the
lock.

While updating the database fixture to ensure that the resource
classes are synched properly, it was discovered that addCleanup was
being called twice with the same args. That has been fixed.

In objects/resource_provider.py the ResourceClass field is changed to a
StringField. The field definition was in rc_field and we simply don't
need it anymore. This is satisfactory because we don't do any validation
on the field internal to the objects (but do elsewhere).

Based on initial feedback, 'os_resource_classes' is imported as 'orc'
throughout to avoid unwieldy length.

Change-Id: Ib7e8081519c3b310cd526284db28c623c8410fbe
2018-12-19 10:43:18 +00:00
Zuul
21e4001af6 Merge "Correct lower-constraints.txt and the related tox job" 2018-12-05 13:44:36 +00:00
Chris Dent
324e4f44da Allow placement to start without a config file
With oslo.config 6.7.0 configuration options may be found in environment
variables. Because of the small number of options that placement
requires to run it is straightforward to source all config from the
environment, useful in container-based environments.

However, prior to this change, placement-the-wsgi-app required that a
configuration file existed, placement.conf, either in /etc/placement or
in a custom directory.

This change makes it so that the placement.conf can be any of:

* in the custom directory
* in the default project locations (including /etc/placement)
* nowhere

The change maintains the behavior that if
[placement_database]/connection is not set the application will fail to
start.

Though this change is orthogonal to the oslo.config change, requirements
and lower-constraints are updated to ensure the desired behavior.

Closes-Bug: #1802925
Change-Id: Iefa8ad22dcb6a128293ea71ab77c377db56e8d70
2018-11-29 13:25:47 +00:00
Chris Dent
4469a813cb Correct lower-constraints.txt and the related tox job
While exploring removing unused packages from lower-constraints.txt it
became clear that the lower-constraints job was not working as expected:
Because our tox config has usedevelop=True 'setup.py develop' is called
to install the placement package after the install command is called.
This means that the lower-constraints are clobbered.

I had mistakenly assumed that turning off 'usedevelop', which causes
'setup.py install' would not make any difference, because it usually
installs dependencies too. It turns out however, that when using pbr
and within a git working dir, it does not. That took some time to
figure out. Oh well.

This change makes it so that we create the tox environment using
usedevelop=False and with our own install_command, to avoid upper
constraints conflicting with lower constraints.

This flagged up a few changes, the main one being that we did not have
a new enough version of keystonemiddleware in order to require use of
www_authenticate_uri. requirements.txt is updated for this as well.

And PasteDeploy needed to be updated to work with Python 3's notion
of namespace packages.

psycopg2 need a newer version to work with Postgresql 10.

oslotest needs to be raised to 3.4.0 because the tests in
cmd.test_manage use features to control what is capture by the Output
fixture from oslotest. Note that the lower-constraints job found
this problem and also demonstrates why we must run the lower-constraints
job without upper-constraints being involved. upper-constraints will
"win" and we don't want that. The point of the job is find packages
where lower-constraints are wrong, so it must "win".

The end result here is a lower-constraints.txt file that starts from
the lower-constraints.txt defined by nova, and then is adapted to
update the versions of packages that were not up to date, remove
those packages which are no longer present, and add some that are
now required.

Change-Id: Id66a28f7ace6fc2adf0e1201d9de5f901234d870
2018-11-27 17:41:02 +00:00
Matt Riedemann
3ff9d0a5a0 Add placement-status upgrade check command
This adds the basic framework for the
placement-status upgrade check command which
is a community-wide goal for the Stein release.

A simple placeholder check is added which should
be replaced later when we have a real upgrade
check.

Change-Id: I9291386fe7fcbfc035c104ea9fdbe5eb875c4776
Story: 2003657
Task: 27518
2018-11-20 13:53:07 -05:00
Chris Dent
d4d0e52100 Remove sqlalchemy-migrate from requirements.txt
With the migrations now gone, we can safely remove sqlalchemy-migrate
from requirements.txt and lower-constraints.txt.

Change-Id: Id8d03d9532bcf4c4f7ad1c1039fa6a42990f8d5d
2018-11-15 05:12:20 -05:00
Eric Fried
f94c92f8f0 Use uuidsentinel from oslo.utils
oslo.utils release 3.37.0 [1] introduced uuidsentinel [2]. This change
rips out placement's uuidsentinel and replaces it with the one from
oslo.utils.

[1] https://review.openstack.org/#/c/599754/
[2] https://review.openstack.org/#/c/594179/

Depends-On: https://review.openstack.org/600041
Change-Id: Iac8952e46e382d4380af380151cad483a31dfce3
2018-09-05 22:32:56 +00:00
Chris Dent
07bdda6148 Update requirements and test-requirements
With placement off by itself, the number of python packages required to
make things go shrinks.

This is the set of modules that are imported directly. There are plenty
of transitive dependencies not listed here.

test-requirements.txt also lists some of the optional things (database
drivers) that come up in devstack-based testing situations.

Change-Id: I3eb6aead544d1ef647ad8d94a18382d5fa795aa4
2018-09-05 21:30:42 +01:00
Stephen Finucane
4df41cb4b3 Revert "Don't use '_TransactionContextManager._async'"
This reverts commit bd7d991309ea2bea5d175cb1f2710519936fd0c2 and bumps
the minimum version of oslo.db to 4.40.0, as that is the first version
of the library to include the renamed attribute.

Change-Id: Ic9e7864be3af7ef362cad5648dfc7bdecd104465
Related-Bug: #1788833
2018-08-28 17:18:51 +01:00
Zuul
710ef9f4c8 Merge "Blacklist greenlet 0.4.14" 2018-07-24 16:06:41 +00:00
Matt Riedemann
a8176d4bb0 Blacklist greenlet 0.4.14
Due to an issue introduced with greenlet 0.4.14 and
gcc on ppc64le systems, we can't run powerkvm or
powervm third party CI, so this change blacklists the
0.4.14 version until the issue [1] is resolved.

The related upper-constraints change is at [2].

[1] https://github.com/python-greenlet/greenlet/issues/136
[2] https://review.openstack.org/584881/

Depends-On: https://review.openstack.org/584881
Change-Id: I3899a0597c75f19cf55d696790ac31a5103db18f
2018-07-24 03:01:54 +00:00
Kevin_Zheng
fd0dba4c9d Use ThreadPoolExecutor for max_concurrent_live_migrations
This changes the max_concurrent_live_migrations handling
to use a ThreadPoolExecutor so that we can control a bounded
pool of Futures in order to cancel queued live migrations
later in this series.

There is a slight functional difference in the unlimited
case since starting in python 3.5, ThreadPoolExecutor will
default to ncpu * 5 concurrently running threads. However,
max_concurrent_live_migrations defaults to 1 and assuming
compute hosts run with 32 physical CPUs on average, you'd
be looking at a maximum of 160 concurrently running live
migrations, which is probably way above what anyone would
consider sane.

Co-Authored-By: Matt Riedemann <mriedem.os@gmail.com>

Part of blueprint abort-live-migration-in-queued-status

Change-Id: Ia9ea1e164fb3b4a386405538eed58d94ad115172
2018-07-16 13:57:09 -04:00
Huang Rui
59f7f3b125 z/VM Driver: Initial change set of z/VM driver
This is the first change that implements basic virt.driver methods
to allow nova-compute process start successfully.

A set of subsequent changes will implement spawn, snapshot, destroy
and instance power actions.

Change-Id: Ica6117c2c64f7518b78b7fb02487622250638e88
blueprint: add-zvm-driver-rocky
2018-07-16 11:35:10 +08:00
Stephen Finucane
ae70025a3d Revert "docs: Disable smartquotes"
Bump the minimum version of oslo.config to 6.1.0, which adds proper
support for parsing Opt.help as rST [1]. This in turn allows us to
revert commit 75fc30090133c31316e6ae790568f1b622807d0c, which is a
temporary fix relying on deprecated features of Sphinx.

[1] https://review.openstack.org/#/c/553860/

Change-Id: I8f56bdce37cfc538348490052a24e463164c86a3
2018-07-11 14:02:32 +01:00
Sylvain Bauza
3df34654a0 Fix placement incompatible with webob 1.7
In Ie4d81fa178b3ed6b2a7b450b4978009486f07810 we started using a new WebOb API
for introspecting headers but since this new API isn't supported by older
versions than 1.8, we need to only accept 1.8.1 or 1.8.2 for Nova
(because 1.8.0 was having a bug fixed by 1.8.1 at least).

Change-Id: I345f372815aef5ac0fb6fc607812ce81587734bf
Closes-Bug: #1773225
2018-06-29 16:55:03 +02:00
Eric Fried
919699d5ed Bump keystoneauth1 minimum to 3.9.0
...so we can set raise_exc=False on Adapter creation instead of having
to do it on every request.

Change-Id: Icaf6e67de00108578a1a264570b60a9139a92f98
2018-06-22 09:13:19 -05:00
Eric Fried
59fbc8f2e2 Fix retrying lower bound in requirements.txt
Due to change [1], the retrying package in requirements.txt
must match the lower bound found in lower-constraints.txt.

[1] https://review.openstack.org/#/c/574367/

Change-Id: I05600e8c606099aea74aa032f92c4f44f947cb4c
2018-06-16 08:45:13 -04:00
Zuul
df9e8c8dd5 Merge "Use oslo.messaging per-call monitoring" 2018-06-14 21:55:37 +00:00
jianghua wang
32daaea287 XenAPI: define a new image handler to use vdi streaming
With the new image handler, it creates an image proxy which
will use the vdi streaming function from os-xenapi to
remotely export VHD from XenServer(image upload) or import
VHD to Xenerver(image download).
The existing GlanceStore uses custom functionality to directly
manipulate files on-disk, so it has the restriction that SR's
type must be file system based: e.g. ext or nfs. The new
image handler invokes APIs formally supported by XenServer
to export/import VDI remotely, it can support other SR
types also e.g. lvm, iscsi, etc.

Note:
vdi streaming would be supported by XenServer 6.5 or above.
The function of image handler depends on os-xenapi 0.3.3 or
above, so bump os-xenapi's version to 0.3.3 and also declare
depends on the patch which bump version in openstack/requirements.

Blueprint: xenapi-image-handler-option-improvement
Change-Id: I0ad8e34808401ace9b85e1b937a542f4c4e61690
Depends-On: Ib8bc0f837c55839dc85df1d1f0c76b320b9d97b8
2018-06-13 01:59:07 +00:00
Dan Smith
021419d665 Use oslo.messaging per-call monitoring
This change makes nova configure oslo.messaging's active call monitoring
feature if the operator increases the rpc_response_timeout configuration
option beyond the default of 60 seconds. If this happens, oslo.messaging will
heartbeat actively-running calls to indicate that they are still running,
avoiding a false timeout at the shorter interval, while still detecting
actual dead-service failures before the longer timeout value.

In addition, this adds a long_rpc_timeout configuration option that we
can use for known-to-run-long operations separately from the base
rpc_response_timeout value, and pre_live_migration() is changed to use
this, as it is known to suffer from early false timeouts.

Depends-On: Iecb7bef61b3b8145126ead1f74dbaadd7d97b407
Change-Id: Icb0bdc6d4ce4524341e70e737eafcb25f346d197
2018-06-11 14:44:10 -07:00
Zuul
07d2b69bd3 Merge "Implement granular policy rules for placement" 2018-06-01 21:06:42 +00:00
Rawan Herzallah
e933211aee Adding NVMEoF for libvirt driver
Adding NVMEoF libvirt driver for supporting NVMEoF initiator CLI.
Libvirt NVMe volume driver is added to handle required calls for
attaching and detaching volume from instaces through calling
os-brick's NVMe Connector.

Implements: blueprint nvme-over-fabirc-nova
Co-Authored-By: Ivan Kolodyazhny <e0ne@e0ne.info>
Change-Id: I67a72c4226e54c18b3a6e4a13b5055fa6e85af09
2018-05-29 11:26:01 +00:00
Matt Riedemann
519e5a22d1 Implement granular policy rules for placement
This adds a granular policy checking framework for
placement based on nova.policy but with a lot of
the legacy cruft removed, like the is_admin and
context_is_admin rules.

A new PlacementPolicyFixture is added along with
a new configuration option, [placement]/policy_file,
which is needed because the default policy file
that gets used in config is from [oslo_policy]/policy_file
which is being used as the nova policy file. As
far as I can tell, oslo.policy doesn't allow for
multiple policy files with different names unless
I'm misunderstanding how the policy_dirs option works.

With these changes, we can have something like:

  /etc/nova/policy.json - for nova policy rules
  /etc/nova/placement-policy.yaml - for placement rules

The docs are also updated to include the placement
policy sample along with a tox builder for the sample.

This starts by adding granular rules for CRUD operations
on the /resource_providers and /resource_providers/{uuid}
routes which use the same descriptions from the placement
API reference. Subsequent patches will add new granular
rules for the other routes.

Part of blueprint granular-placement-policy

Change-Id: I17573f5210314341c332fdcb1ce462a989c21940
2018-05-17 11:12:16 -04:00
Matt Riedemann
f95b74c9e5 Add retrying to requirements.txt
The nova.virt.powervm.media module imports this so we
should include it in our runtime requirements list.

Change-Id: I818b8957d791cbfbae1fc33cc296ebfb8c6a6651
2018-05-15 16:49:41 -04:00
Eric Fried
44c377488d Bump pypowervm minimum to 1.1.15
The referenced bug is fixed in pypowervm 1.1.15.  Without the fix,
PowerVM device detach during destroy (and therefore PowerVM CI) is
broken.

Change-Id: Icaabadfb17dd87207d99938224713a78dc925674
Depends-On: https://review.openstack.org/#/c/564275/
Closes-Bug: #1766692
2018-04-25 11:34:22 -05:00
jichen
1729ca6a1b uncap eventlet in nova
per commit 00eca5fdd352be7db52bbf89496762023db4a951
nova is failing here:
logs.openstack.org/87/523387/32/check/requirements-check/408e28c/job-output.txt.gz

Change-Id: Ie63f47fc403704af7ba16a29cc58cd914172f6bd
2018-04-11 21:35:49 +08:00
Zuul
37d3101d54 Merge "Support extending attached ScaleIO volumes" 2018-04-09 15:18:22 +00:00
Eric Young
d5b83194f8 Support extending attached ScaleIO volumes
Adding ability to extend ScaleIO volumes which are
attached to an instance.

Change-Id: Ib2debfe481d4c1921d14546f81b60d070e8f9f39
2018-04-05 09:01:16 -04:00
esberglu
47fe9af9de Move pypowervm requirement to 1.1.12
pypowervm needs to be 1.1.12 or later for PowerVM vSCSI cinder volume
support [1].

[1] https://review.openstack.org/#/c/526094/

Change-Id: I3bb4359c04ec8e1e9ece09670e108cd023d0f558
2018-04-02 13:39:21 -05:00
OpenStack Proposal Bot
e5c9eeeffd Updated from global requirements
Change-Id: I834ca07b3830ca9ea9f9064dbb2859d0231477bf
2018-03-26 08:15:10 +00:00
OpenStack Proposal Bot
6e3e501568 Updated from global requirements
Change-Id: If171c656296d779db44c9867cbd44bd45bfad09d
2018-03-23 22:44:04 +00:00
OpenStack Proposal Bot
7366b2c64e Updated from global requirements
Change-Id: I3d8b3d783c2ab37e1c907f7e6802ac00152d8439
2018-03-17 08:48:15 +00:00
OpenStack Proposal Bot
6688f557ec Updated from global requirements
Change-Id: Idaf1bdadf561fda7287bce34d9972fdd74fa75ae
2018-03-14 05:46:35 +00:00
Dan Smith
bc54f4de5e Add simple db purge command
This adds a simple purge command to nova-manage. It either deletes all
shadow archived data, or data older than a date if provided.

This also adds a post-test hook to run purge after archive to validate
that it at least works on data generated by a gate run.

Related to blueprint purge-db

Change-Id: I6f87cf03d49be6bfad2c5e6f0c8accf0fab4e6ee
2018-03-07 10:35:32 -08:00
OpenStack Proposal Bot
78bfbb895d Updated from global requirements
Change-Id: I788f833403d39d07c338d16b7002657eda22b801
2018-03-06 12:57:20 +00:00
OpenStack Proposal Bot
6a8ea9bd82 Updated from global requirements
Change-Id: I04c27a2a5b5c85861a84c4aed92d056e4d915399
2018-03-05 06:55:05 +00:00
OpenStack Proposal Bot
5207e30abd Updated from global requirements
Change-Id: Ia9254dd3d6c8d174c7d943430601c9121001c811
2018-03-04 10:16:00 +00:00
OpenStack Proposal Bot
b419ce6703 Updated from global requirements
Change-Id: I91e47a7c3c099f70303843a742474bae12133447
2018-03-02 15:15:34 +00:00
OpenStack Proposal Bot
bdd0dbc1e4 Updated from global requirements
Change-Id: If3af95c2c137c40a394cbdc445de91d1c0074d57
2018-02-17 09:52:04 +00:00
OpenStack Proposal Bot
4ab679580e Updated from global requirements
Change-Id: Iaf542e38017bf24cf80e7c0dff2c88c4b332de1b
2018-02-14 12:18:13 +00:00
OpenStack Proposal Bot
b8e0105140 Updated from global requirements
Change-Id: I2f8741902eefe1dafc16806bf8235a8eb038a1f2
2018-02-01 07:20:35 +00:00
Zuul
47942183b6 Merge "Updated from global requirements" 2018-01-31 06:04:43 +00:00
Zuul
53830a5c16 Merge "Add taskflow to requirements" 2018-01-31 05:19:53 +00:00
OpenStack Proposal Bot
6d27782eef Updated from global requirements
Change-Id: I371ee0cb4156ec4e0a20f55852b0756502d0dd90
2018-01-29 00:21:52 +00:00
OpenStack Proposal Bot
30c3412136 Updated from global requirements
Change-Id: I0b8fc724a05cfef267fa7a3a21ba4a83699f4e7e
2018-01-19 20:24:00 +00:00
Matthew Edmonds
194566ae2c Add taskflow to requirements
The powervm driver is dependent on taskflow, but that was not
indicated in requirements.txt. This corrects that omission.

Change-Id: Ic9a9376ae8e5054366f565175c797885b4e83c39
2018-01-16 13:57:09 +00:00
OpenStack Proposal Bot
050d82123c Updated from global requirements
Change-Id: Icada15d601dc5399eeb927bb7cba9bea828883da
2018-01-16 12:40:17 +00:00