The current rootwrap.conf file is outdated and doesn't include some
parameters. This change updates the content to make it consistent with
the latest example file in oslo.rootwrap.
Change-Id: I989b7da5ff6afae0c1a476e01c3df07870be5f74
Local API and DB limits are limits on resources that are counted either
as an API request parameter (example: server metadata items) or as
records in the database (example: server key pairs).
Future patches will make use of this logic, and actually enforce the
limits. This patch just adds the infrastructure to allow for the
enforcement of the limits.
We are moving all existing quotas to be managed via Keystone's
unified limits.
To stop confusion between injected_file_path_length and
injected_file_path_bytes, the unified limit in Keystone will use the
latter name to match the name used the API.
These local limits are all about preventing excessive load on the API
and database and have little to do with resource usage. These limits
are represented by keystone registered limits only, accordingly.
Local limits include things that just limit items in an API request:
* metadata_items
* injected_files
* injected_file_content_bytes
* injected_file_path_bytes
Local limits also include things that are stored in the database:
* key_pairs
* server_groups
* server_group_members
Some resource names have been changed to prepend a prefix of "server_"
in order to disambiguate them from other potential unified limits in
keystone:
* metadata_items => server_metadata_items
* injected_files => server_injected_files
* injected_file_content_bytes => server_injected_file_content_bytes
* injected_file_path_bytes => server_injected_file_path_bytes
* key_pairs => server_key_pairs
Note that each of the above are counted via a different scope. This new
code ensures that key_pairs are counted per user, server_groups are
counted per project and server_group_members are counted per
server_group.
Note: Previously server_group_member were counted per user inside each
server_group, which has proved very confusing, as adding more users into
a project increases the maximum size of allowed for a server_group.
blueprint unified-limits-nova
Change-Id: I0b6f4d29aaee1d71541a95cbecfd0708aac325d2
This commit introduces the LightOS driver for nova. LightOS is a
software-defined disaggregated clustered storage solution running on
commodity servers with commodity SSDs. It it developed by Lightbits
Labs (https://www.lightbitslabs.com) and is actively developed and
maintained. LightOS is proprietary but the openstack drivers are
licensed under Apache v2.0.
The Cinder driver for LightOS currently supports the following
functionality:
Create volume
Delete volume
Attach volume
Detach volume
Create image from volume
create volume from image
Live migration
Volume replication
Thin provisioning
Multi-attach
Extend volume
Create snapshot
Delete snapshot
Create volume from snapshot
Create volume from volume (clone)
This driver has been developed and has been in use for a couple of
years by Lightbits and our clients. We have tested it extensively
internally with multiple openstack versions, including Queens, Rocky,
Stein, and Train. We have also tested it with master (19.1 xena) and we
are working to extend testing to cover additional openstack releases.
We are glad to join the openstack community and hope to get your
feedback and comments on this driver, and if it is acceptable, to see
it merged into the tree.
Note: the patch depends on os-brick 5.2.0. That version also increased
the lower constraints of several dependencies, thus needs nova to
increase those as well in requirements.txt, lower-constraints.txt and
setup.cfg.
Depends-On: I2e86fa84049053b7c75421d33ad1a1af459ef4e0
Signed-off-by: Yuval Brave yuval@lightbitslabs.com
Change-Id: Ic314b26695d9681d31a18adcec0794c2ff41fe71
The new version contains changes needed by the multi-architecture
support and off-path SmartNIC DPU support code.
Needed-By: I168d3ccc914f25a3d4255c9b319ee6b91a2f66e2
Needed-By: Ia070a29186c6123cf51e1b17373c2dc69676ae7c
Change-Id: Ic1179f3e5e2c1aeb069972f21edffe5b003eb525
...and start skipping the relevant tests in 'nova.tests.unit.virt' if
these dependencies are not present. Thanks to hacking rules N311 and
N312, which prevent virt drivers using code from other virt drivers,
simply skipping the unit tests for a virt driver module is enough to
ensure we never load those modules.
This means users that want to use the powervm driver can install the
required dependencies using e.g. 'pip install .[powervm]', and packagers
can choose to skip packaging a dependency safe in the knowledge that the
relevant tests will be skipped.
Change-Id: I3787f9afd78cd0a7c7feb4dfe1bcb21437b5a128
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
We no longer have a Xen driver. This is an unnecessary dependency.
Change-Id: Ic298fa9ac4a8935ce4e0dc17d8842d399d4eb808
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
The commit I168fffac8002f274a905cfd53ac4f6c9abe18803 added a wrapper
around fasteners.ReaderWriterLock to fix up an issue with eventlet. But
the wrapper was added to nova.utils module that is use not only by the
nova tests but also the nova production code. This made the fixture
library a dependency of the nova production code. While the current
ReaderWriterLock usage only limited to the nova test sub tree. The
I712f88fc1b6053fe6d1f13e708f3bd8874452a8f commit fix the issue of not
having fixtures in the nova requirements.txt. However I think a better
fix is to move the wrapper to the test subtree instead. This patch does
that and restores the state of the requirements.txt
Change-Id: I6903ce53b9b91325f7268cf2ebd02e4488579560
Related-Bug: #1958075
The commit 887c445a7a6a17b92a37b6ed1dcdcc7dd009f65d made the nova.utils
module dependent on the fixtures library but the change missed updating
requirements and the fixtures library is not installed automatically.
This change migrates the fixtures library from test-requirements.txt to
requirements.txt so that the library is installed without test codes.
Closes-Bug: #1958075
Change-Id: I712f88fc1b6053fe6d1f13e708f3bd8874452a8f
Some things that were missed in previous patches and are thrown together
here:
- Add alembic as an explicit dependency (we were getting it transitively
from oslo.db). We also bump the sqlalchemy dependency to a 1.4.x
release, which is the minimum supported by our chosen version of
alembic (more on this below)
- Remove tooling related to the old migrations
- Fix the tox whitelisting of the flaky MySQL tests
On the SQLAlchemy front, we opt for 1.4.13. Technically alembic should
support anything from 1.4.0, however, with SQLAlchemy >= 1.4.0, < 1.4.13
we see errors like the following in some tests:
sqlalchemy.exc.InvalidRequestError: Entity namespace for
"count(instance_mappings.id)" has no property "queued_for_delete"
There's nothing specific about this in the release notes for 1.4.13 [1]
but it definitely fixes things.
[1] https://docs.sqlalchemy.org/en/14/changelog/changelog_14.html#change-1.4.13
Change-Id: I4c8eb13f11aa7471c26a5ba326319aef245c9836
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
This makes it possible to partially revert
If48dd949ec4d69a09c87178f16d56a2517e21fd8 as one of the warning silenced
there now fixed in oslo.db.
Change-Id: I0051c0f8694699865bc16636953908d60ba456d2
Take advantage of the neutronclient bindings for the port binding APIs
added in neutronclient 7.1.0 to avoid having to vendor this stuff
ourselves.
Change-Id: Icc284203fb53658abe304f24a62705217f90b22b
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Since 3.7.0, oslo policy started the DeprecationWarning[1] if
deprecated_reason and deprecated_since param are not passed
in DeprecatedRule or they are passed in RuleDefault object.
[1] https://github.com/openstack/oslo.policy/blob/3.7.0/oslo_policy/policy.py#L1538
Change-Id: Idbbc203c6ae65aee29f9463a4911bae2bb541f41
Use the new oslo.confg type HostDomainOpt to support underscore in the
name. You can see the bugzilla[1] to have more information.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1868940
Related-Bug: #1892044
Change-Id: Ib6c8fc1e3d90b79b10066c429670fcb957bddd23
os-vif 1.15.0 added the ability to create an OVS port during plugging
by specifying the 'create_port' attribute in the 'port_profile' field.
By delegating port creation to os-vif, we can rely on it's 'isolate_vif'
config option [1] that will temporarily configure the VLAN to 4095
(0xfff), which is reserved for implementation use [2] and is used by
neutron to as a dead VLAN [3]. By doing this, we ensure VIFs are plugged
securely, preventing guests from accessing other tenants' networks
before the neutron OVS agent can wire up the port.
This change requires a little dance as part of the live migration flow.
Since we can't be certain the destination host has a version of os-vif
that supports this feature, we need to use a sentinel to indicate when
it does. Typically we would do so with a field in
'LibvirtLiveMigrateData', such as the 'src_supports_numa_live_migration'
and 'dst_supports_numa_live_migration' fields used to indicate support
for NUMA-aware live migration. However, doing this prevents us
backporting this important fix since o.vo changes are not backportable.
Instead, we (somewhat evilly) rely on the free-form nature of the
'VIFMigrateData.profile_json' string field, which stores JSON blobs and
is included in 'LibvirtLiveMigrateData' via the 'vifs' attribute, to
transport this sentinel. This is a hack but is necessary to work around
the lack of a free-form "capabilities" style dict that would allow us do
backportable fixes to live migration features.
Note that this change has the knock on effect of modifying the XML
generated for OVS ports: when hybrid plug is false will now be of type
'ethernet' rather than 'bridge' as before. This explains the larger than
expected test damage but should not affect users.
[1] https://opendev.org/openstack/os-vif/src/tag/2.4.0/vif_plug_ovs/ovs.py#L90-L93
[2] https://en.wikipedia.org/wiki/IEEE_802.1Q#Frame_format
[3] https://answers.launchpad.net/neutron/+question/231806
Change-Id: I11fb5d3ada7f27b39c183157ea73c8b72b4e672e
Depends-On: Id12486b3127ab4ac8ad9ef2b3641da1b79a25a50
Closes-Bug: #1734320
Closes-Bug: #1815989
The os-brick 4.3.1 release fixes a critical data loss bug that can be
triggered by reconfiguring [libvirt]/volume_use_multipath while volumes
are attached.
The following requirements and lower-requirements are also bumped as
introduced into os-brick by Ie7dcc859e1291bd1d50b1ae516be38d8732de836:
os-brick 4.3.1 depends on eventlet>=0.30.1
os-brick 4.3.1 depends on oslo.concurrency>=4.4.0
os-brick 4.3.1 depends on oslo.service>=2.5.0
os-brick 4.3.1 depends on oslo.serialization>=4.1.0
os-brick 4.3.1 depends on oslo.utils>=4.8.0
os-brick 4.3.1 depends on pbr>=5.5.1
os-brick 4.3.1 depends on requests>=2.25.1
os-brick 4.3.1 depends on tenacity>=6.3.1
Co-Authored-By: Lee Yarwood <lyarwood@redhat.com>
Related-Bug: #1921381
Change-Id: I0d5df271dec1c6035229b02ec45e03abad6f27e7
Ceph 16 (Pacific) will support attaching RBD images to Windows
hosts as well as Hyper-V VMs [1].
This patch updates the Hyper-V driver so that it may be able to
consume RBD volumes.
We're also updating the os-win requirement to 5.4.0. It should've
been bumped by [2], vms won't spawn without it. Note this introduces no
new or unmet direct or in-direct version dependencies and so is included
as part of this change.
[1] https://github.com/ceph/ceph/pull/33750
[2] Ibe6aff4edeb32208bc9865e9216a7432caddab2b
Implements: blueprint hyperv-rbd
Change-Id: I9ad90817648ca12f80a6b53f6ba728df15cbafab
This change simply bumps the version of os-brick required by Nova to
version 4.2.0. This is itself required by
I9ad90817648ca12f80a6b53f6ba728df15cbafab that introduces support for
rbd volumes within the HyperV driver.
After much back and fourth it was decided to bump the required version
of os-brick ahead of that change to allow the following requirements.txt
and lower-constraints.txt changes caused by direct and in-direct
dependency changes to be documented clearly:
os-brick 4.2.0 depends on oslo.log>=4.4.0
\_ oslo-log 4.4.0 depends on python-dateutil>=2.7.0
os-brick 4.2.0 depends on oslo.serialization>=4.0.1
os-brick 4.2.0 depends on pbr>=5.5.0
os-brick 4.2.0 depends on oslo.privsep>=2.4.0
\_ oslo.privsep 2.4.0 depends on msgpack>=0.6.0
os-brick 4.2.0 depends on oslo.service>=2.4.0
os-brick 4.2.0 depends on tenacity>=6.2.0
os-brick 4.2.0 depends on oslo.context>=3.1.1
os-brick 4.2.0 depends on oslo.concurrency>=4.3.0
os-brick 4.2.0 depends on oslo.i18n>=5.0.1
os-brick 4.2.0 depends on six>=1.15.0
os-brick 4.2.0 depends on os-win>=5.1.0
The above changes have been tested with pip 21.0.1 to ensure the new
resolver is happy and that nothing has been missed.
Change-Id: Ic83f3c7c955d0df89d75f700ee4fe2bd7f715794
Implements: blueprint hyperv-rbd
Since only Wallaby compute nodes will support the 'socket' PCI NUMA
affinity policy, this patch adds a ResourceRequest translator that adds
a required trait if the value of '(hw_|hw:)pci_numa_affinity_policy' is
'socket'.
The actual trait reporting by the libvirt driver will be added in a
future patch. Until then the 'socket' value remains a hidden no-op.
Implements: blueprint pci-socket-affinity
Change-Id: I908ff07e1107304ca5926cc04d2fdc8fe0da5ed9
This is now maintained as a Jazzband project [1].
[1] https://github.com/jazzband/prettytable
Change-Id: Ie422a243a97ffbe2f553934004369f01562b95b9
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Replace six.text_type with str.
This patch completes six removal.
Change-Id: I779bd1446dc1f070fa5100ccccda7881fa508d79
Implements: blueprint six-removal
Signed-off-by: Takashi Natsume <takanattie@gmail.com>
The dnspython 2.0.0 incompatibility[1] is worked around in eventlet 0.26.1
by restricting the dnspython version to < 2.0.0. So the issue affects
nova-novncproxy as well we need to bump our lower constraint as well.
[1] https://github.com/eventlet/eventlet/issues/619
Related-Bug: #1888237
Change-Id: Ie9cc718e654cb37a4f31ccb4b052b23878d476a2
md5 is not an approved algorithm in FIPS mode, and trying to
instantiate a hashlib.md5() will fail when the system is running in
FIPS mode.
md5 is allowed when in a non-security context. There is a plan to
add a keyword parameter (usedforsecurity) to hashlib.md5() to annotate
whether or not the instance is being used in a security context.
In the case where it is not, the instantiation of md5 will be allowed.
See https://bugs.python.org/issue9216 for more details.
Some downstream python versions already support this parameter. To
support these versions, a new encapsulation of md5() has been added to
oslo_utils. See https://review.opendev.org/#/c/750031/
This patch is to replace the instances of hashlib.md5() with this new
encapsulation, adding an annotation indicating whether the usage is
a security context or not.
The instances being replaced here appear to be used to provide
representations for paths. There is in fact already a sha256 version
of get_hash_str that is supposed to be used in security sensitive
usages.
With this change (and the related dependent changes), the unit and
functional tests pass when run on a FIPS enabled system.
Change-Id: If0ec11e7b7fcde4dacc57265c4dd77b0f536bfab
Depends-On: https://review.opendev.org/#/c/756432
Depends-On: https://review.opendev.org/#/c/756153
Depends-On: https://review.opendev.org/#/c/760160
Default value of 'CONF.oslo_policy.policy_file' config option
has been changed from 'policy.json' to 'policy.yaml'. If new default
file 'policy.yaml' does not exist but old default 'policy.json' exist
then fallback to use old default file.
An upgrade checks is added to check the policy_file format and
fail upgrade checks if it is JSON formatted.
Added a warning in policy doc about JSON formatted file is deprecated,
also removed all the reference to policy.json file in doc as well as
in tests.
Related Blueprint: policy-json-to-yaml
Closes-Bug: #1875418
Change-Id: Ic4d3b998bb9701cb1e3ef12d9bb6f4d91cc19c18
Several version specified in our lower-constraints files had conflicting
dependencies. This updates a few packages to avoid those conflicts. It
also removes the linters that are tracked in the global requirements
blocklist since we do not need to enforce lower-constraints for linters.
Change-Id: Iae93f0631d77794f093fd330e849d2fd805b238f
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
This series implements the referenced blueprint to allow for specifying
custom resource provider traits and inventories via yaml config files.
This first commit includes schema version 1.0 and functions to load and
validate against that schema.
Since this patch, package ddt>=1.2.1 is required to run tests.
Co-Author: Tony Su <tao.su@intel.com>
Author: Dustin Cowles <dustin.cowles@intel.com>
Blueprint: provider-config-file
Change-Id: I58099726f799c427e9174a0dcce9889344d51e7c
To support new AVX512 cpu feature, this patch adds the mapping
between cpu features reported by libvirt [1] and corresponding
os-traits [2]. The traits include 'AVX512VBMI', 'AVX512IFMA',
'AVX512VBMI2', 'AVX512BITALG', 'AVX512VAES', 'AVX512GFNI',
'AVX512VPCLMULQDQ' and 'AVX512VPOPCNTDQ'.
[1] https://github.com/libvirt/libvirt/blob/master/src/cpu_map/x86_features.xml
[2] https://review.opendev.org/#/c/728576/
Depends-On: I58cba02f204d2e19a1a838852c628694edd47d73
Change-Id: I212a84e5cae15bc133aafd538d55f85149ab360f
oslo has new flag to disable the warning for default check_str
change and keep logging warning if policy name is changed or it is
marked for removal.
Policy defaults refresh BP work changed the defaults for all the
policy so logging their warning is dangerousa and fill up the log file.
Once we switch to new defaults then we can enable them if needed.
Logging warning for name change and marked for removal make sense
when old rules are override by operator so the new flag does not control
those. For policy defaults refresh BP work, we do not have many policies
in this category so logging their warning is ok.
This commits reset the testing flag 'suppress_deprecation_warnings'
default value back to False. Test policy fixture sets that to True to
avoid any warnig during tests run.
Bumping the oslo policy version where this new flag is present
- https://review.opendev.org/#/c/719547/
Partial implement blueprint policy-defaults-refresh
Change-Id: Iaa6baf6877890babb6205bdb3ee2e1a2b28ebd9c
Add the validation framework necessary to verify extra specs along with
the definitions for every extra spec we currently recognize in-tree.
None of this is currently used since we don't have the API microversions
wired up, but that will come in a future patch.
Note that we must add the H238 hacking check to the ignore list here,
since this includes our first use of Python 3-type classes without the
explicit 'object' subclass. This can be removed when that check is
removed from hacking.
Part of blueprint flavor-extra-spec-validators
Change-Id: Ib64a1348cce1dca995746214616c4f33d9d664bd
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Building on the ``[workarounds]/disable_native_luksv1``
configurable introduced in Ia500eb614cf575ab846f64f4b69c9068274c8c1f
this change introduces another workaround configurable that when enabled
will connect RBD volumes to the compute host as block devices using
os-brick.
When used togther both options allow operators to workaround recently
discovered performance issues in the libgcrypt library used by QEMU when
natively decrypting LUKSv1 encrypted disks.
For now the extend_volume method raises a NotImplemented error in-line
with the underlying method in os-brick. Future work will be required to
both support this in os-brick and wire up the required calls in the
volume driver.
This workaround is temporary and will be removed during the W release
once all impacted distributions have been able to update their versions
of the libgcrypt library.
Finally os-brick 3.0.1 is now required as it provides the
Id507109df80391699074773f4787f74507c4b882 fix when attempting to
diconnect locally attached RBD volumes.
Closes-Bug: #1869184
Change-Id: Ied3732042738a6194b635c55e0304d71a6fb66e3
This change is a follow up to I0c3f14100a18107f7e416293f3d4fcc641ce5e55
and removes the direct call to nova.privsep.qemu with one to the images
API that now returns an oslo_utils.imageutils.QemuImgInfo object.
Version 4.1.0 of oslo.utils introducing support for the format-specific
data returned by qemu-img info for LUKSv1 based images.
Change-Id: I573396116e10cf87f80f1ded55f2cd8f498859e4
When the resource tracker has to lock a compute host for updates or
inspection, it uses a single semaphore. In most cases, this is fine, as
a compute process only is tracking one hypervisor. However, in Ironic, it's
possible for one compute process to track many hypervisors. In this
case, wait queues for instance claims can get "stuck" briefly behind
longer processing loops such as the update_resources periodic job. The
reason this is possible is because the oslo.lockutils synchronized
library does not use fair locks by default. When a lock is released, one
of the threads waiting for the lock is randomly allowed to take the lock
next. A fair lock ensures that the thread that next requested the lock
will be allowed to take it.
This should ensure that instance claim requests do not have a chance of
losing the lock contest, which should ensure that instance build
requests do not queue unnecessarily behind long-running tasks.
This includes bumping the oslo.concurrency dependency; fair locks were
added in 3.29.0 (I37577becff4978bf643c65fa9bc2d78d342ea35a).
Change-Id: Ia5e521e0f0c7a78b5ace5de9f343e84d872553f9
Related-Bug: #1864122
This includes a couple of fixes and removes the need for custom
workarounds, like the removal of the 'address_string' helper at [1].
[1] https://github.com/novnc/websockify/commit/be9823bf
Change-Id: I9b1d57fd513386e552afbbc8a59cf3a4c54b6908
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
This uses the COMPUTE_SAME_HOST_COLD_MIGRATE trait in the API during a
cold migration to filter out hosts that cannot support same-host cold
migration, which is all of them except for the hosts using the vCenter
driver.
For any nodes that do not report the trait, we won't know if they don't
because they don't support it or if they are not new enough to report
it, so the API has a service version check and will fallback to old
behavior using the config if the node is old. That compat code can be
removed in the next release.
As a result of this the FakeDriver capabilities are updated so the
FakeDriver no longer supports same-host cold migration and a new fake
driver is added to support that scenario for any tests that need it.
Change-Id: I7a4b951f3ab324c666ab924e6003d24cc8e539f5
Closes-Bug: #1748697
Related-Bug: #1811235