Commit Graph

420 Commits

Author SHA1 Message Date
Zuul
3182721a85 Merge "Change sqlalchemy warnings filter to an error" 2019-02-14 21:46:37 +00:00
Zuul
38cefd9ece Merge "Switch to using os-resource-classes" 2019-02-14 19:08:52 +00:00
Matt Riedemann
0840bcccc0 Change sqlalchemy warnings filter to an error
The warning should be gone with change
I192e84ce757d12d33085a209dd58d8ea46fb90fb in
oslo.db 4.44.0 so this changes the warnings
filter from ignore to error and bumps the minimum
required version of oslo.db to include that change.

Change-Id: If7b1a9613b58476fab8409211512613a8863cdde
Related-Bug: #1813147
Related-Bug: #1814199
2019-02-14 08:47:53 -05:00
Matt Riedemann
c03cc26ee0 Fix deprecation warning for threadgroup.add_timer
This resolves the following deprecation warning:

b'/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/
lib/python3.5/site-packages/oslo_service/threadgroup.py:193:
DeprecationWarning: Calling add_timer() with arguments to the callback
function is deprecated. Use add_timer_args() instead.'

The add_timer_args method was added in 1.34.0:

  Ib2791342263e2b88c045bcc92adc8160f57a0ed6

So the required version of oslo.service is also updated.

Change-Id: Id54226dc926839686906d04ecf8d791c0881f82a
Partial-Bug: #1813147
2019-02-07 09:37:00 -05:00
Chris Dent
27617ee193 Switch to using os-resource-classes
With the extraction of placement we ended up with resource class names
being duplicated between nova and placement. To address that, the
os-resource-classes library [1] was created to provide a single
authority for standard resource classes and the format of custom
classes.

This patch changes nova to use it, removing the use of the rc_fields
module which used to have the information. A method left in it
(normalize_name) has been moved to utils.py, renamed as
normalize_rc_name, and callers and tests updated accordingly.

Because the placement code is being kept in nova for the time being,
that code's use of rc_fields is maintained, and the module too.
A note is added in the module explain that. Backporting the changes
from extracted-placement to placement-in-nova was considered but
because we no longer have placement tests in nova, that didn't seem
like the right thing to do.

requirements and lower-constraints have been updated.
os-resource-classes is already in global requirements.

For reference the related placement change is at [2].

[1] https://docs.openstack.org/os-resource-classes
[2] https://review.openstack.org/#/c/623556/

Change-Id: I8e579920c0eaca81b563a87429c930b21b3d4dc5
2019-02-07 11:11:09 +00:00
Jan Gutter
4d32b45c15 Add support for vrouter HW datapath offloads
Add plumbing for Contrail/Tungsten Fabric datapath offloads

* This change expands the VNIC type support for the vrouter VIF type by
  adding 'direct' and 'virtio-forwarder' plugging support.

* After this change, the vrouter VIF type will support the following modes:
  * 'normal': the 'classic' tap-style VNIC plugged into the instance,
  * 'direct': a PCI Virtual Function is passed through to the instance,
  * 'virtio-forwarder': a PCI Virtual Function is proxied to the
    instance via a vhost-user virtio forwarder.

* The os-vif conversion function was extended to support the two new
  plugging modes.

* Unit tests were added for the os-vif conversion functions.

* OpenContrail / Tungsten Fabric is planning to consume this
  functionality for the 5.1 release.

* os-vif 1.14.0 is required to pass the metadata

Change-Id: I327894839a892a976cf314d4292b22ce247b0afa
Depends-On: I401ee6370dad68e62bc2d089e786a840d91d0267
Signed-off-by: Jan Gutter <jan.gutter@netronome.com>
blueprint: vrouter-hw-offloads
2019-01-31 13:55:50 +00:00
Zuul
2e846ed0f3 Merge "Fix rfc3986.is_valid_uri deprecation warnings" 2018-12-27 18:15:04 +00:00
Takashi NATSUME
2d2dc78975 Fix rfc3986.is_valid_uri deprecation warnings
The rfc3986.is_valid_uri has been deprecated in 1.1.0.
It generates the following warnings.

DeprecationWarning: Please use rfc3986.validators.Validator instead.
This method will be eventually removed.

So this patch replaces rfc3986.is_valid_uri with
rfc3986.validators.Validator.

Even after applying this patch, the warnings are still output
because it is caused by oslo.config.
The fix for oslo.config will be done in another patch.

Change-Id: I70aebad6c6bd384dbd11ef732226356922bf1913
Closes-Bug: #1809755
2018-12-26 10:48:39 +09:00
Matt Riedemann
5316a8a0cd Fix jsonutils.to_primitive UserWarning
The CheatingSerializer fixture used in nova tests keeps
the RequestContext.db_connection set on the context object
which otherwise wouldn't normally happen. The context object
can show up in error notification payloads because of how
nova.exception_wrapper.wrap_exception works. That payload
is eventually serialized for notifications and since the
cheated RequestContext.db_connection is set and cannot be
serialized, it results in a UserWarning from the
jsonutils.to_primitive method (called via JsonPayloadSerializer).

This will eventually result in failures when that UserWarning
is made into an error.

To fix this, we can pass a fallback method to to_primitive()
which will serialize a RequestContext object the same way that
RequestContextSerializer serializes a context - by simply
converting it to dict form.

Since this only affects test runs, because of using the
CheatingSerializer fixture, it should have no impact on
runtime serializations.

Error logging is added to the FakeNotifier since it's hard
to know what is wrong in the payload unless it is logged.

Also, the WarningsFixture is updated to make sure we don't
introduce new UserWarnings for the serialization issue.

The jsonutils.to_primitive() fallback method was added to
oslo.serialization via commit cdb2f60d26e3b65b6370f87b2e9864045651c117
in 2.21.1 so we have to bump our minimum required version
of that library as well.

Change-Id: Id9f960a0c7c8897dbb9edf53b4723154341412d6
Closes-Bug: #1799249
2018-12-17 22:56:50 -05:00
Zuul
2e9356374e Merge "Migrate upgrade checks to oslo.upgradecheck" 2018-12-17 23:23:17 +00:00
Ben Nemec
c9436c3846 Migrate upgrade checks to oslo.upgradecheck
The common upgrade check code has been moved to oslo.upgradecheck.
This change switches the Nova upgrade checks to use the common code
and removes the tests for functionality that is now the responsibility
of the library.

Change-Id: I0dc2044286dbe78314c650a92c4654f7f50642d2
2018-12-14 21:45:34 +00:00
Balazs Gibizer
1f2a80c195 Replace ThreadPoolExecutor with GreenThreadPoolExecutor
Based on eventlet issue [1] ThreadPoolExecutor doesn't play nice with
eventlet in python 3.7. We saw deadlocks in the functional-py37
execution in live migration tests due to live migration using
ThreadPoolExecutor.

The [1] suggests to replaces ThreadPoolExecutor with
futurist.GreenThreadPoolExecutor to avoid deadlocks. So this patch does
the replacement and adjusts the unit tests accordingly.

As the ThreadPoolExecutor was the last used class from the futures
module we remove that from the requirements and add the futurist module
instead.

[1] https://github.com/eventlet/eventlet/issues/508

Change-Id: Ia56ab43be739e677760bbad5c40caad924425fa5
2018-12-13 10:41:42 -05:00
melanie witt
fded752b16 Use SleepFixture instead of mocking _ThreadingEvent.wait
Recently, the _ThreadingEvent class in oslo.service was removed [1] and
our unit test patching is preventing us from moving to a newer version
of oslo.service [2].

We have patching of the _ThreadingEvent.wait method to bypass the sleep
time in the looping call of RetryDecorator, which adds several seconds
to the run time of unit tests.

This changes things to use the new SleepFixture from oslo.service
instead.

Depends-On: https://review.openstack.org/616371

[1] I62e9f1a7cde8846be368fbec58b8e0825ce02079
[2] https://review.openstack.org/615676

Change-Id: I45dd7602068eb0ce1331cfefd5a0cf6418bc8e88
2018-11-08 16:50:39 -05:00
Balazs Gibizer
d4f8974f87 Add request_spec.RequestGroup versioned object
Later patches will introduce a field in RequestSpec using this type as
the field type to store the resource requests coming from outside of
Nova like the bandwidth request coming from the Neutron ports.

This patch refactors the usage of placement.lib.RequestGroup. Until now
this class was used both by placement and nova services and they used
it only as a util class. However after this series the nova services
would like to use such a class via RPC which requires an OVO. This
patch makes sure that the new OVO is used by nova and the old plain
object is used by placement. This way placement is not forced to use
an OVO where no OVO functionality is required.

The minimum required version of oslo.versionedobjects is updated to
1.33.3 to include the fix for bug 1771804.

Change-Id: I46c97d2641d9685ef59771314665a17a5236097d
blueprint: bandwidth-resource-provider
2018-11-05 17:47:06 -05:00
melanie witt
37d046c511 Bump os-brick version to 2.6.1
This will get us bug fixes for gate bug:

  https://bugs.launchpad.net/nova/+bug/1732199

and another bug that looks like it could affect the gate:

  https://bugs.launchpad.net/os-brick/+bug/1794829

Closes-Bug: #1732199

Change-Id: I709b4a1438aedcfc95d191462b4ec3f1f8c4355a
2018-10-16 20:29:51 +00:00
Eric Fried
8e1ca5bf34 Use uuidsentinel from oslo.utils
oslo.utils release 3.37.0 [1] introduced uuidsentinel [2]. This change
rips out nova's uuidsentinel and replaces it with the one from
oslo.utils.

[1] https://review.openstack.org/#/c/599754/
[2] https://review.openstack.org/#/c/594179/

Change-Id: I7f5f08691ca3f73073c66c29dddb996fb2c2b266
Depends-On: https://review.openstack.org/600041
2018-09-05 09:08:54 -05:00
Stephen Finucane
964832d37d Revert "Don't use '_TransactionContextManager._async'"
This reverts commit bd7d991309 and bumps
the minimum version of oslo.db to 4.40.0, as that is the first version
of the library to include the renamed attribute.

Change-Id: Ic9e7864be3af7ef362cad5648dfc7bdecd104465
Related-Bug: #1788833
2018-08-28 17:18:51 +01:00
Zuul
828f172c48 Merge "Blacklist greenlet 0.4.14" 2018-07-24 16:06:41 +00:00
Matt Riedemann
4aad363fe4 Blacklist greenlet 0.4.14
Due to an issue introduced with greenlet 0.4.14 and
gcc on ppc64le systems, we can't run powerkvm or
powervm third party CI, so this change blacklists the
0.4.14 version until the issue [1] is resolved.

The related upper-constraints change is at [2].

[1] https://github.com/python-greenlet/greenlet/issues/136
[2] https://review.openstack.org/584881/

Depends-On: https://review.openstack.org/584881
Change-Id: I3899a0597c75f19cf55d696790ac31a5103db18f
2018-07-24 03:01:54 +00:00
Kevin_Zheng
79dac41fee Use ThreadPoolExecutor for max_concurrent_live_migrations
This changes the max_concurrent_live_migrations handling
to use a ThreadPoolExecutor so that we can control a bounded
pool of Futures in order to cancel queued live migrations
later in this series.

There is a slight functional difference in the unlimited
case since starting in python 3.5, ThreadPoolExecutor will
default to ncpu * 5 concurrently running threads. However,
max_concurrent_live_migrations defaults to 1 and assuming
compute hosts run with 32 physical CPUs on average, you'd
be looking at a maximum of 160 concurrently running live
migrations, which is probably way above what anyone would
consider sane.

Co-Authored-By: Matt Riedemann <mriedem.os@gmail.com>

Part of blueprint abort-live-migration-in-queued-status

Change-Id: Ia9ea1e164fb3b4a386405538eed58d94ad115172
2018-07-16 13:57:09 -04:00
Huang Rui
e5acf4f961 z/VM Driver: Initial change set of z/VM driver
This is the first change that implements basic virt.driver methods
to allow nova-compute process start successfully.

A set of subsequent changes will implement spawn, snapshot, destroy
and instance power actions.

Change-Id: Ica6117c2c64f7518b78b7fb02487622250638e88
blueprint: add-zvm-driver-rocky
2018-07-16 11:35:10 +08:00
Stephen Finucane
d1820d14c6 Revert "docs: Disable smartquotes"
Bump the minimum version of oslo.config to 6.1.0, which adds proper
support for parsing Opt.help as rST [1]. This in turn allows us to
revert commit 75fc300901, which is a
temporary fix relying on deprecated features of Sphinx.

[1] https://review.openstack.org/#/c/553860/

Change-Id: I8f56bdce37cfc538348490052a24e463164c86a3
2018-07-11 14:02:32 +01:00
Sylvain Bauza
c4e0ad1433 Fix placement incompatible with webob 1.7
In Ie4d81fa178b3ed6b2a7b450b4978009486f07810 we started using a new WebOb API
for introspecting headers but since this new API isn't supported by older
versions than 1.8, we need to only accept 1.8.1 or 1.8.2 for Nova
(because 1.8.0 was having a bug fixed by 1.8.1 at least).

Change-Id: I345f372815aef5ac0fb6fc607812ce81587734bf
Closes-Bug: #1773225
2018-06-29 16:55:03 +02:00
Eric Fried
68caa292ce Bump keystoneauth1 minimum to 3.9.0
...so we can set raise_exc=False on Adapter creation instead of having
to do it on every request.

Change-Id: Icaf6e67de00108578a1a264570b60a9139a92f98
2018-06-22 09:13:19 -05:00
Eric Fried
f64e6eada3 Fix retrying lower bound in requirements.txt
Due to change [1], the retrying package in requirements.txt
must match the lower bound found in lower-constraints.txt.

[1] https://review.openstack.org/#/c/574367/

Change-Id: I05600e8c606099aea74aa032f92c4f44f947cb4c
2018-06-16 08:45:13 -04:00
Zuul
5b56434517 Merge "Use oslo.messaging per-call monitoring" 2018-06-14 21:55:37 +00:00
jianghua wang
f7593ded8f XenAPI: define a new image handler to use vdi streaming
With the new image handler, it creates an image proxy which
will use the vdi streaming function from os-xenapi to
remotely export VHD from XenServer(image upload) or import
VHD to Xenerver(image download).
The existing GlanceStore uses custom functionality to directly
manipulate files on-disk, so it has the restriction that SR's
type must be file system based: e.g. ext or nfs. The new
image handler invokes APIs formally supported by XenServer
to export/import VDI remotely, it can support other SR
types also e.g. lvm, iscsi, etc.

Note:
vdi streaming would be supported by XenServer 6.5 or above.
The function of image handler depends on os-xenapi 0.3.3 or
above, so bump os-xenapi's version to 0.3.3 and also declare
depends on the patch which bump version in openstack/requirements.

Blueprint: xenapi-image-handler-option-improvement
Change-Id: I0ad8e34808401ace9b85e1b937a542f4c4e61690
Depends-On: Ib8bc0f837c55839dc85df1d1f0c76b320b9d97b8
2018-06-13 01:59:07 +00:00
Dan Smith
fe26a52024 Use oslo.messaging per-call monitoring
This change makes nova configure oslo.messaging's active call monitoring
feature if the operator increases the rpc_response_timeout configuration
option beyond the default of 60 seconds. If this happens, oslo.messaging will
heartbeat actively-running calls to indicate that they are still running,
avoiding a false timeout at the shorter interval, while still detecting
actual dead-service failures before the longer timeout value.

In addition, this adds a long_rpc_timeout configuration option that we
can use for known-to-run-long operations separately from the base
rpc_response_timeout value, and pre_live_migration() is changed to use
this, as it is known to suffer from early false timeouts.

Depends-On: Iecb7bef61b3b8145126ead1f74dbaadd7d97b407
Change-Id: Icb0bdc6d4ce4524341e70e737eafcb25f346d197
2018-06-11 14:44:10 -07:00
Zuul
40a0b579c4 Merge "Implement granular policy rules for placement" 2018-06-01 21:06:42 +00:00
Rawan Herzallah
a833bcd05f Adding NVMEoF for libvirt driver
Adding NVMEoF libvirt driver for supporting NVMEoF initiator CLI.
Libvirt NVMe volume driver is added to handle required calls for
attaching and detaching volume from instaces through calling
os-brick's NVMe Connector.

Implements: blueprint nvme-over-fabirc-nova
Co-Authored-By: Ivan Kolodyazhny <e0ne@e0ne.info>
Change-Id: I67a72c4226e54c18b3a6e4a13b5055fa6e85af09
2018-05-29 11:26:01 +00:00
Matt Riedemann
0a461979df Implement granular policy rules for placement
This adds a granular policy checking framework for
placement based on nova.policy but with a lot of
the legacy cruft removed, like the is_admin and
context_is_admin rules.

A new PlacementPolicyFixture is added along with
a new configuration option, [placement]/policy_file,
which is needed because the default policy file
that gets used in config is from [oslo_policy]/policy_file
which is being used as the nova policy file. As
far as I can tell, oslo.policy doesn't allow for
multiple policy files with different names unless
I'm misunderstanding how the policy_dirs option works.

With these changes, we can have something like:

  /etc/nova/policy.json - for nova policy rules
  /etc/nova/placement-policy.yaml - for placement rules

The docs are also updated to include the placement
policy sample along with a tox builder for the sample.

This starts by adding granular rules for CRUD operations
on the /resource_providers and /resource_providers/{uuid}
routes which use the same descriptions from the placement
API reference. Subsequent patches will add new granular
rules for the other routes.

Part of blueprint granular-placement-policy

Change-Id: I17573f5210314341c332fdcb1ce462a989c21940
2018-05-17 11:12:16 -04:00
Matt Riedemann
9a80272307 Add retrying to requirements.txt
The nova.virt.powervm.media module imports this so we
should include it in our runtime requirements list.

Change-Id: I818b8957d791cbfbae1fc33cc296ebfb8c6a6651
2018-05-15 16:49:41 -04:00
Eric Fried
864acf7c8d Bump pypowervm minimum to 1.1.15
The referenced bug is fixed in pypowervm 1.1.15.  Without the fix,
PowerVM device detach during destroy (and therefore PowerVM CI) is
broken.

Change-Id: Icaabadfb17dd87207d99938224713a78dc925674
Depends-On: https://review.openstack.org/#/c/564275/
Closes-Bug: #1766692
2018-04-25 11:34:22 -05:00
jichen
62edffcff3 uncap eventlet in nova
per commit 00eca5fdd352be7db52bbf89496762023db4a951
nova is failing here:
logs.openstack.org/87/523387/32/check/requirements-check/408e28c/job-output.txt.gz

Change-Id: Ie63f47fc403704af7ba16a29cc58cd914172f6bd
2018-04-11 21:35:49 +08:00
Zuul
e2d5dc4e2c Merge "Support extending attached ScaleIO volumes" 2018-04-09 15:18:22 +00:00
Eric Young
ce90bec66f Support extending attached ScaleIO volumes
Adding ability to extend ScaleIO volumes which are
attached to an instance.

Change-Id: Ib2debfe481d4c1921d14546f81b60d070e8f9f39
2018-04-05 09:01:16 -04:00
esberglu
28b696f3d7 Move pypowervm requirement to 1.1.12
pypowervm needs to be 1.1.12 or later for PowerVM vSCSI cinder volume
support [1].

[1] https://review.openstack.org/#/c/526094/

Change-Id: I3bb4359c04ec8e1e9ece09670e108cd023d0f558
2018-04-02 13:39:21 -05:00
OpenStack Proposal Bot
2a1afa3ed4 Updated from global requirements
Change-Id: I834ca07b3830ca9ea9f9064dbb2859d0231477bf
2018-03-26 08:15:10 +00:00
OpenStack Proposal Bot
538ec87323 Updated from global requirements
Change-Id: If171c656296d779db44c9867cbd44bd45bfad09d
2018-03-23 22:44:04 +00:00
OpenStack Proposal Bot
38f7b68a64 Updated from global requirements
Change-Id: I3d8b3d783c2ab37e1c907f7e6802ac00152d8439
2018-03-17 08:48:15 +00:00
OpenStack Proposal Bot
f0a7990af7 Updated from global requirements
Change-Id: Idaf1bdadf561fda7287bce34d9972fdd74fa75ae
2018-03-14 05:46:35 +00:00
Dan Smith
ae241cc68f Add simple db purge command
This adds a simple purge command to nova-manage. It either deletes all
shadow archived data, or data older than a date if provided.

This also adds a post-test hook to run purge after archive to validate
that it at least works on data generated by a gate run.

Related to blueprint purge-db

Change-Id: I6f87cf03d49be6bfad2c5e6f0c8accf0fab4e6ee
2018-03-07 10:35:32 -08:00
OpenStack Proposal Bot
01333a7f18 Updated from global requirements
Change-Id: I788f833403d39d07c338d16b7002657eda22b801
2018-03-06 12:57:20 +00:00
OpenStack Proposal Bot
35414701c1 Updated from global requirements
Change-Id: I04c27a2a5b5c85861a84c4aed92d056e4d915399
2018-03-05 06:55:05 +00:00
OpenStack Proposal Bot
5d2add7453 Updated from global requirements
Change-Id: Ia9254dd3d6c8d174c7d943430601c9121001c811
2018-03-04 10:16:00 +00:00
OpenStack Proposal Bot
5799f5f6df Updated from global requirements
Change-Id: I91e47a7c3c099f70303843a742474bae12133447
2018-03-02 15:15:34 +00:00
OpenStack Proposal Bot
8fe87575e1 Updated from global requirements
Change-Id: If3af95c2c137c40a394cbdc445de91d1c0074d57
2018-02-17 09:52:04 +00:00
OpenStack Proposal Bot
68e17a8d10 Updated from global requirements
Change-Id: Iaf542e38017bf24cf80e7c0dff2c88c4b332de1b
2018-02-14 12:18:13 +00:00
OpenStack Proposal Bot
b4fbbbbe80 Updated from global requirements
Change-Id: I2f8741902eefe1dafc16806bf8235a8eb038a1f2
2018-02-01 07:20:35 +00:00
Zuul
eff52a6470 Merge "Updated from global requirements" 2018-01-31 06:04:43 +00:00