60836 Commits

Author SHA1 Message Date
Sean Mooney
b1cc39848e Add iso file format inspector
This change includes unit tests for the ISO
format inspector using mkisofs to generate
the iso files.

A test for stashing qcow content in the system_area
of an iso file is also included.

This change modifies format_inspector.detect_file_format
to evaluate all inspectors until they are complete and
raise an InvalidDiskInfo exception if multiple formats
match.

Related-Bug: #2059809
Change-Id: I7e12718fb3e1f77eb8d1cfcb9fa64e8ddeb9e712
2024-07-08 13:03:27 +01:00
Sean Mooney
b5a1d3b4b2 Reproduce iso regression with deep format inspection
This change adds a reproducer for the regression in iso
file support when
workarounds.disable_deep_image_inspection = False

Change-Id: I56d8b9980b4871941ba5de91e60a7df6a40106a8
2024-07-08 13:03:25 +01:00
Zuul
e4fca6f517 Merge "Server Rescue leads to Server ERROR state if original image is deleted" 2024-07-08 01:07:18 +00:00
Zuul
240296298c Merge "scheduler: fix _get_sharing_providers to support unlimited aggr" 2024-07-07 00:02:39 +00:00
Zuul
cff73f518b Merge "api: Don't do version check if nothing required" 2024-07-04 23:12:18 +00:00
Zuul
e4aaddc607 Merge "api: Add 'removed' decorator" 2024-07-04 23:12:11 +00:00
Zuul
5fbe975f6c Merge "api: Correct bug in flavors schema" 2024-07-04 23:06:08 +00:00
Zuul
49c4638fd5 Merge "api: Migrate to JSON Schema Draft 2020-12" 2024-07-04 21:36:50 +00:00
Zuul
9b0d691f0d Merge "port format inspector tests from glance" 2024-07-04 20:51:49 +00:00
Zuul
c39a425ba7 Merge "Fix vmdk_allowed_types checking" 2024-07-04 15:32:22 +00:00
Sean Mooney
838daa3cad port format inspector tests from glance
This commit is a direct port of the format inspector
unit tests from glance as of commit
0d8e79b713bc31a78f0f4eac14ee594ca8520999

the only changes to the test are as follows

"from glance.common import format_inspector" was updated to
"from nova.image import format_inspector"

"from glance.tests import utils as test_utils"
was replaced with "from nova import test"

"test_utils.BaseTestCase" was replaced with "test.NoDBTestCase"

"glance-unittest-formatinspector-" was replaced with
"nova-unittest-formatinspector-"

This makes the test funtional in nova.

TestFormatInspectors requries qemu-img to be installed on the
host which would be a new depency for executing unit tests.
to avoid that we skip TestFormatInspectors if qemu-img
is not installed.
TestFormatInspectorInfra and TestFormatInspectorsTargeted
do not have a qemu-img dependency so
no changes to the test assertions were required.

Change-Id: Ia34203f246f0bc574e11476287dfb33fda7954fe
2024-07-04 12:51:59 +01:00
Zuul
ad44cadfc2 Merge "Additional qemu safety checking on base images" 2024-07-04 03:06:22 +00:00
Zuul
6bf5f66833 Merge "Check images with format_inspector for safety" 2024-07-03 17:18:40 +00:00
Zuul
657e86585c Merge "Reject qcow files with data-file attributes" 2024-07-03 16:23:34 +00:00
Sahid Orentino Ferdjaoui
f254729a56 scheduler: fix _get_sharing_providers to support unlimited aggr
A Request-Too-Long error can be raised in case of too many aggregates
are attached to a host. The fix is to split the requests.

The request is executed by host via a periodic task. Result is set in
cache so no performances impact is expected.

The change introduce a new compute configuration option
`compute.sharing_providers_max_uuids_per_request = 200`.

Closes-bug: #2068893

Change-Id: I1355c0813cb99067ead60f9835938566289beb20
Signed-off-by: Sahid Orentino Ferdjaoui <sahid.ferdjaoui@industrialdiscipline.com>
2024-07-03 12:58:04 +00:00
Dan Smith
08be7b2a0d Fix vmdk_allowed_types checking
This restores the vmdk_allowed_types checking in create_image()
that was unintentionally lost by tightening the
qemu-type-matches-glance code in the fetch patch recently. Since we
are still detecting the format of base images without metadata, we
would have treated a vmdk file that claims to be raw as raw in fetch,
but then read it like a vmdk once it was used as a base image for
something else.

Change-Id: I07b332a7edb814f6a91661651d9d24bfd6651ae7
Related-Bug: #2059809
2024-07-02 13:08:40 -07:00
Dan Smith
b1b88bf001 Additional qemu safety checking on base images
There is an additional way we can be fooled into using a qcow2 file
with a data-file, which is uploading it as raw to glance and then
booting an instance from it. Because when we go to create the
ephemeral disk from a cached base image, we've lost the information
about the original source's format, we probe the image's file type
without a strict format specified. If a qcow2 file is listed in
glance as a raw, we won't notice it until it is too late.

This brings over another piece of code (proposed against) glance's
format inspector which provides a safe format detection routine. This
patch uses that to detect the format of and run a safety check on the
base image each time we go to use it to create an ephemeral disk
image from it.

This also detects QED files and always marks them as unsafe as we do
not support that format at all. Since we could be fooled into
downloading one and passing it to qemu-img if we don't recognize it,
we need to detect and reject it as unsafe.

Change-Id: I4881c8cbceb30c1ff2d2b859c554e0d02043f1f5
2024-07-02 13:08:40 -07:00
Dan Smith
9cdce71594 Check images with format_inspector for safety
It has been asserted that we should not be calling qemu-img info
on untrusted files. That means we need to know if they have a
backing_file, data_file or other unsafe configuration *before* we use
qemu-img to probe or convert them.

This grafts glance's format_inspector module into nova/images so we
can use it to check the file early for safety. The expectation is that
this will be moved to oslo.utils (or something) later and thus we will
just delete the file from nova and change our import when that happens.

NOTE: This includes whitespace changes from the glance version of
format_inspector.py because of autopep8 demands.

Change-Id: Iaefbe41b4c4bf0cf95d8f621653fdf65062aaa59
Closes-Bug: #2059809
2024-07-02 13:08:08 -07:00
Dan Smith
ec9c55cbbc Reject qcow files with data-file attributes
Change-Id: Ic3fa16f55acc38cf6c1a4ac1dce4487225e66d04
Closes-Bug: #2059809
2024-07-02 13:08:08 -07:00
Dan Smith
c0ff2386ed Fix disk_formats in ceph job tempest config
Tempest currently defaults to disk_formats[0] for images it creates,
which is 'ami'. However, it's actually using a qcow2 disk image by
default, which means we're lying to glance when we create those.

Change-Id: I737e9aa51c268a387f1eed24cf717618d057d747
2024-07-02 12:52:36 -07:00
Zuul
04ddddf018 Merge "tests: Ensure API schemas are valid" 2024-07-02 13:44:05 +00:00
Zuul
b84d939f59 Merge "api: Fix bugs in schemas" 2024-07-02 02:24:40 +00:00
Zuul
bb15d7aab8 Merge "tests: Ensure all APIs have a request body schema" 2024-07-01 14:19:58 +00:00
Stephen Finucane
64ac67fd19 api: Correct bug in flavors schema
The 'rxtx_factor' is a float, not an integer, and should allow values <
1 so long as they are > 0. Correct this.

Change-Id: I3567f13d7a1bb6f42755e3f5739fa9ecd827aa49
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2024-07-01 11:18:06 +01:00
Zuul
4c50398f3a Merge "pre-commit: Bump autopep8 version and update repo" 2024-06-29 02:31:27 +00:00
Zuul
f1dd70cff7 Merge "api: Add remaining missing request body schemas" 2024-06-28 16:21:16 +00:00
Zuul
f25065b470 Merge "retry write_sys call on device busy" 2024-06-27 19:49:52 +00:00
Zuul
fb2c9714d0 Merge "api: Add request body schemas for SG APIs" 2024-06-27 19:42:37 +00:00
Zuul
bc1febbc07 Merge "tweak emulation job to avoid OOM errors" 2024-06-27 19:07:50 +00:00
melanie witt
438c405394 pre-commit: Bump autopep8 version and update repo
Attempts to run autopep8 via pre-commit fail for Python 3.12 with
error: "ModuleNotFoundError: No module named 'lib2to3'".

The issue was fixed in v2.1.0:

https://github.com/hhatto/autopep8/issues/581

but v2.1.0 is not available at
https://github.com/pre-commit/mirrors-autopep8 as the mirror has been
deprecated.

The README instructs to use https://github.com/hhatto/autopep8 directly
instead.

Change-Id: Ifdac3dad368298720682ab256e9dbf0f2624884e
2024-06-27 08:11:33 +00:00
Stephen Finucane
ff040ada4f api: Don't do version check if nothing required
There's no reason to inspect the API version requested (and reject
things if unset) if we don't have version constraints on the API
resource. This lets us add schema decorators to unversioned API
resources (i.e. the "versions" API at '/').

Change-Id: Ib4d4585e3e4664eba8cd3daa47b4349eb2bfed74
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2024-06-24 11:38:55 +01:00
Stephen Finucane
b1824c5c97 api: Add 'removed' decorator
For APIs and resource actions that have been wholly removed and now
return a HTTP 4xx request regardless of microversion.

Change-Id: Ie3e90f4628dab4cc3e5983e0e8d8799dcf0465d8
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2024-06-24 11:38:27 +01:00
Stephen Finucane
7491417323 api: Migrate to JSON Schema Draft 2020-12
OpenAPI 3.1 is a superset of JSON Schema Draft 2020-12. As a result, we
wish to migrate our current schemas to this. There are a couple of
issues to address:

- 'exclusiveMinimum' is now an integer and allows you to define an
  minimum for an exclusive range, to complement 'minimum' which is used
  for an inclusive range. We can drop it and use 'minimum' with a larger
  number (since draft 6 [1]).

- 'integer' types can now accept a fractional part but only if it's zero
  (i.e. '1.0' is permissible, '1.1' is not) (since draft 6 [1]).

- 'items' has been replaced with 'prefixItems' for describing the format
  of an array item (since draft 2020-12 [2])

[1] https://json-schema.org/draft-06/json-schema-release-notes
[2] https://json-schema.org/draft/2020-12/release-notes

Change-Id: I1486701786960eef95c5c42674bff1b2d7d686e2
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2024-06-17 16:19:17 +00:00
Stephen Finucane
c382f036c3 tests: Ensure API schemas are valid
Validate the validation by validating our schemas against the JSON
Schema meta schema. This is an important first step in getting us
of JSON Schema Draft 4 and onto Draft 2019-09, which OpenAPI is a
superset of.

Change-Id: I3b5a05aa0aa058e92c6927c9e3bee3cdd4477f8f
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2024-06-17 16:19:10 +00:00
Stephen Finucane
7e78623ac5 api: Fix bugs in schemas
None of these should affect us at runtime but they are not technically
valid against JSON Schema Draft 4.

Change-Id: Iaae8998a34ff812b9f76a0cd1fa9be4555c89b98
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2024-06-17 16:19:03 +00:00
Stephen Finucane
917c4f0721 tests: Ensure all APIs have a request body schema
The idea here is to test that we are doing schema validation against
all routes that accept a request body (i.e. POST, PUT, PATCH). We can
use these schemas down the line to generate OpenAPI docs like all the
cool kids do.

Change-Id: Icdf20e7e5b38c9f5324eac39ec1d6327609bf6d4
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2024-06-17 16:18:57 +00:00
Stephen Finucane
f86cd38dfe api: Add remaining missing request body schemas
This provides us a path to generate OpenAPI schemas for the nova API.
Thankfully most of the API documentation is pretty darn simple. Note
that we need to fix a lot of unit tests so that they pass correct bodies
to the controller methods under test. Also note that the
'reset_server_state' module in 'nova.api.openstack.compute.schemas' is
renamed to 'admin_actions' to match the name of the module containing
the controller itself, as is the case for all other schemas.

Change-Id: I39098fb621b120589ed6a923ce64207ee26e91aa
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2024-06-17 16:18:51 +00:00
Stephen Finucane
847608e75a api: Add request body schemas for SG APIs
These are deprecated but there's value in having a proper - if loose -
schema in place for API documentation purposes. Also, doing things this
way allows us to remove a whole load of hand-rolled stuff.

Change-Id: I4106cfa2a09d135f12892ed6d1f42f4151dc72e4
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2024-06-17 17:18:25 +01:00
Zuul
7dc4b1ea62 Merge "add functional repoducer for bug 2065927" 2024-05-31 10:44:09 +00:00
Zuul
96268d4e7a Merge "libvirt: Ensure both swtpm and swtpm_setup exist for vTPM support" 2024-05-28 09:07:13 +00:00
Sean Mooney
44c1b48b31 retry write_sys call on device busy
This change adds a retry_if_busy decorator
to the read_sys and write_sys functions in the filesystem
module that will retry reads and writes up to 5 times with
an linear backoff.

This allows nova to tolerate short periods of time where
sysfs retruns device busy. If the reties are exausted
and offlineing a core fails a warning is log and the failure is
ignored. onling a core is always treated as a hard error if
retries are exausted.

Closes-Bug: #2065927
Change-Id: I2a6a9f243cb403167620405e167a8dd2bbf3fa79
2024-05-27 18:31:31 +01:00
Sean Mooney
3811c7f648 tweak emulation job to avoid OOM errors
This change increases the swap in the emulation job form
1G to 8G
This change updates the default cirrors image form 0.5.2
to 0.5.3 to avoid know kernel bugs
this change reduces the tb_cache_size: to 128
The tempest concurrency is reduced to 4 to avoid
no valid host error in the resize tests

Change-Id: Ic1dde3d54f5ca12408ef53218773a27d55760705
2024-05-27 14:19:02 +01:00
Zuul
ac4a67cbda Merge "docs: Follow up for persistent mdevs" 2024-05-27 11:01:32 +00:00
Zuul
3dfdc10f63 Merge "libvirt: Create persistent mdevs" 2024-05-27 11:01:26 +00:00
Zuul
bded279a00 Merge "docs: Add more information about unified limits" 2024-05-23 19:48:39 +00:00
melanie witt
c7e49dfa16 docs: Follow up for persistent mdevs
This address review feedback on change
I7e1d10e66a260efd0a3f2d6522aeb246c7582178 to add some clarifying text
to the docs and release note.

Related to blueprint persistent-mdevs

Change-Id: I472552c64cc2c2ce06896158664faac0199d90bd
2024-05-23 18:29:04 +00:00
Zuul
d7d2fb1edd Merge "scheduler: AggregateMultitenancyIsolation to support unlimited tenant" 2024-05-23 15:35:54 +00:00
Zuul
4e3a41f0a4 Merge "Stop using split UEC image (mostly)" 2024-05-22 19:03:32 +00:00
Zuul
3a53d715cd Merge "[doc] Improve description for nova-manage db purge" 2024-05-22 02:44:50 +00:00
Alexey Stupnikov
ac8729ac87 [doc] Improve description for nova-manage db purge
--before argument is currently described in ambiguous way: it
is not actually used to filter entries ARCHIVED before specified
date. Instead, it compares provided data with "deleted_at" value
for most rows and "updated_at" or "created_at" for remaining ones.

Since we already talk about time of deletion when describing
--before argument of "nova-manage db archive_deleted_rows" rows,
it make sense to not provide extra details here as well.

Change-Id: Ib5940e88a52dc8d32303e27237e567c3481fc3dc
2024-05-21 20:19:18 +02:00