Cinder is currently not able to upload a volume that is based on an
image back to glance. This bug is triggered if glance multistore is
enabled (devstack in this example).
When enabling multistore, the following properties will be stored in Cinder:
* os_glance_failed_import=''
* os_glance_importing_to_stores=''
Those properties will cause problems when Cinder tries to perform some
actions with Glance. Error msg:
```
cinderclient.exceptions.BadRequest: HTTP 403 Forbidden: Access was denied to this resource.: Attribute 'os_glance_failed_import' is reserved. (HTTP 400)
```
Nova had the same issue and solved it with:
50fdbc752a/releasenotes/notes/absolutely-non-inheritable-image-properties-85f7f304fdc20b61.yaml
and
dda179d3f9
Therefore, this patch is intended to apply a similar solution in Cinder.
Change-Id: I79d70543856c01a45e2d8c083ab8df6b9c047ebc
Closes-Bug: #1945500
This works in Python 3.7 or greater and is
cleaner looking.
See PEP-585 for more info.
https://peps.python.org/pep-0585/
Change-Id: I4c9da881cea1a3638da504c4b79ca8db13851b06
When a user tries to delete a volume in "awaiting-transfer" state ,
the error message received does not include the "awaiting-transfer"
state in invalid states of Volume for deletion.
This leads to the user believing the volume-delete request is valid
and is not able to debug the reason for failure of the volume deletion
for volume present in "awaiting-transfer" state.
Closes-Bug: #1971603
Change-Id: I78915c332169b26ffb2b97310efedec65bc25e4d
Call _is_encrypted() instead of volume_types.is_encrypted()
since we already have the volume type object.
Change-Id: Id82a5bc251a8cea4febdc429329cd136c805487d
Managing a volume to an encrypted volume type should not be allowed.
One reason is that there is no way for an operator to specify an
encryption key ID for the volume. Another is that we already don't
allow a volume of an encrypted type to be un-managed, so this change
will be symmetric.
Also update and correct the api-ref for this call.
Co-authored-by: Yadiel Xuan(轩艳东) <xuanyandong@inspur.com>
Co-authored-by: Brian Rosmaita <rosmaita.fossdev@gmail.com>
Change-Id: Ic2da41f3962c1108f974aca952bce3da6d6ac277
Closes-bug: #1944577
This patch adds volume re-image API to enable the ability to
re-image a specific volume.
Implements: blueprint add-volume-re-image-api
Co-Authored-by: Rajat Dhasmana <rajatdhasmana@gmail.com>
Change-Id: I031aae50ee82198648f46c503bba04c6e231bbe5
There is an initial policy check in the transfers accept API[1]
which validates correctly if the user is authorized to perform
the operation or not. However, we've a duplicate check in the volume
API layer which passes a target object (volume) while authorizing
which is wrong for this API. While authorizing, we enforce check on
the project id of the target object i.e. volume in this case which,
before the transfer operation is completed, contains the project id
of source project hence making the validation wrong.
In the case of transfers API, any project is able to accept the transfer
given they've the auth key required to secure the transfer accept
So this patch removes the duplicate policy check.
[1] https://opendev.org/openstack/cinder/src/branch/master/cinder/transfer/api.py#L225
Closes-Bug: #1950474
Change-Id: I3930bff90df835d9d8bbf7e6e91458db7e5654be
There are cases where requests to delete an attachment made by Nova can
race other third-party requests to delete the overall volume.
This has been observed when running cinder-csi, where it first requests
that Nova detaches a volume before itself requesting that the overall
volume is deleted once it becomes `available`.
This is a cinder race condition, and like most race conditions is not
simple to explain.
Some context on the issue:
- Cinder API uses the volume "status" field as a locking mechanism to
prevent concurrent request processing on the same volume.
- Most cinder operations are asynchronous, so the API returns before the
operation has been completed by the cinder-volume service, but the
attachment operations such as creating/updating/deleting an attachment
are synchronous, so the API only returns to the caller after the
cinder-volume service has completed the operation.
- Our current code **incorrectly** modifies the status of the volume
both on the cinder-volume and the cinder-api services on the
attachment delete operation.
The actual set of events that leads to the issue reported in this bug
are:
[Cinder-CSI]
- Requests Nova to detach volume (Request R1)
[Nova]
- R1: Asks cinder-api to delete the attachment and **waits**
[Cinder-API]
- R1: Checks the status of the volume
- R1: Sends terminate connection request (R1) to cinder-volume and
**waits**
[Cinder-Volume]
- R1: Ask the driver to terminate the connection
- R1: The driver asks the backend to unmap and unexport the volume
- R1: The last attachment is removed from the DB and the status of the
volume is changed in the DB to "available"
[Cinder-CSI]
- Checks that there are no attachments in the volume and asks Cinder to
delete it (Request R2)
[Cinder-API]
- R2: Check that the volume's status is valid. It doesn't have
attachments and is available, so it can be deleted.
- R2: Tell cinder-volume to delete the volume and return immediately.
[Cinder-Volume]
- R2: Volume is deleted and DB entry is deleted
- R1: Finish the termination of the connection
[Cinder-API]
- R1: Now that cinder-volume has finished the termination the code
continues
- R1: Try to modify the volume in the DB
- R1: DB layer raises VolumeNotFound since the volume has been deleted
from the DB
- R1: VolumeNotFound is converted to HTTP 404 status code which is
returned to Nova
[Nova]
- R1: Cinder responds with 404 on the attachment delete request
- R1: Nova leaves the volume as attached, since the attachment delete
failed
At this point the Cinder and Nova DBs are out of sync, because Nova
thinks that the attachment is connected and Cinder has detached the
volume and even deleted it.
Hardening is also being done on the Nova side [2] to accept that the
volume attachment may be gone.
This patch fixes the issue mentioned above, but there is a request on
Cinder-CSI [1] to use Nova as the source of truth regarding its
attachments that, when implemented, would also fix the issue.
[1]: https://github.com/kubernetes/cloud-provider-openstack/issues/1645
[2]: https://review.opendev.org/q/topic:%2522bug/1937084%2522+project:openstack/nova
Closes-Bug: #1937084
Change-Id: Iaf149dadad5791e81a3c0efd089d0ee66a1a5614
Our current `attachment_delete` methods in the volume API and the
manager are using DB methods directly, which makes the OVOs present in
those methods get out of sync with the latest data, which leads to
notifications having the wrong data when we send them on volume detach.
This patch replaces DB method calls with OVO calls and moves the
notification call to the end of the method, where we have the final
status on the volume.
It also adds the missing detach.start notification when deleting an
attachment in the reserved state.
Closes-Bug: #1916980
Closes-Bug: #1935011
Change-Id: Ie48cf55deacd08e7716201dac00ede8d57e6632f
Introduces API microversion 3.66, which allows
snapshot creation on in-use volumes without the
force flag being passed.
Co-authored-by: Eric Harney <eharney@redhat.com>
Co-authored-by: Brian Rosmaita <rosmaita.fossdev@gmail.com>
Implements: blueprint fix-snapshot-create-force
Change-Id: I6d45aeab065197a85ce62740fc95306bce9dfc45
This is a silly config option. We only have one database driver in-tree
and no plans to add more (SQLAlchemy is best in class). There's also no
way we'd be able to support out-of-tree drivers. Remove it entirely.
Change-Id: Ica3b2e8fcb079beca652e81d2230bcca82fb49d7
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Cinder creates temporary resources, volumes and snapshots, during some
of its operations, and these resources aren't counted towards quota
usage.
Cinder currently has a problem to track quota usage is when deleting
temporary resources.
Determining which volumes are temporary is a bit inconvenient because we
have to check the migration status as well as the admin metadata, so
they have been the source of several bugs, though they should be
properly tracked now.
For snapshots we don't have any way to track which ones are temporary,
which creates some issues:
- Quota sync mechanism will count them as normal snapshots.
- Manually deleting temporary snapshots after an operation fails will
mess the quota.
- If we are using snapshots instead of clones for backups of in-use
volumes the quota will be messed on completion.
This patch proposes the introduction of a new field for those database
resource tables where we create temporary resources: volumes and
snaphots.
The field will be called "use_quota" and will be set to False for
temporary resources to indicate that we don't want them to be counted
towards quota on deletion.
Instead of using "temporary" as the field name "use_quota" was used to
allow other cases that should not do quota in the future.
Moving from our current mechanism to the new one is a multi-release
process because we need to have backward compatibility code for rolling
upgrades.
This patch adds everything needed to complete the multi-release process
so that anybody can submit next release patches. To do so the patch
adds backward compatible code adding the feature in this release and
TODO comments with the exact changes that need to be done for the next
2 releases.
The removal of the compatibility code will be done in the next release,
and in the one after that we'll remove the temporary metadata rows that
may still exist in the database.
With this new field we'll be able to make our DB queries more efficient
for quota usage calculations, reduce the chances of introducing new
quota usage bugs in the future, and allow users to filter in/out
temporary volumes on listings.
Closes-Bug: #1923828
Closes-Bug: #1923829
Closes-Bug: #1923830
Implements: blueprint temp-resources
Change-Id: I98bd4d7a54906b613daaf14233d749da1e1531d5
File locks are never removed from the system, so they keep increasing in
the locks directory, which can become problematic.
In this patch we start trying to delete these lock files when we delete
a volume or a snapshot.
This affects the 2 type of file locks we currently have:
- Using oslo lockutils synchronized with external=True
- Using coordination.synchronized when deployed in Active-Passive and no
DLM
This will alleviate the ever increasing files in the locks directory.
Deployment tools should implement a service that runs when the host is
booting and cleans of the locks directory before the OpenStack services
are started.
Partial-Bug: #1432387
Change-Id: Ic73ee64257aeb024383c6cb79f2e8c04810aaf69
Both volumes and snapshots have a volume_type_id field in the DB, but
when we migrate a volume we leave the snapshots with the old type. This
can only happen for retypes that don't do migration, since we cannot do
a retype with migration if the volume has snapshots.
Leaving the snapshots with the old time makes a mess of the quota usage
when we do the retype as well as when we delete the snapshots.
This patch fixes the quota issue by making sure the snapshots are
retyped as well. This means that we will check quotas for the snapshots
when retyping a volume that have them and we will properly reserve and
set the quota on retype.
Closes-Bug: #1877164
Change-Id: I90e9f85d192e1f2fee4ec8615a5bc95851a90f8e
Attachment OVO automatically loads the Volume OVO into its `volume`
attribute, so we don't need to load it again in the volume's API
`attachment_delete` method.
Change-Id: I79f2e58de42fca69c08f3636d72b80bdf8457e9a
When we create a volume in the DB using the OVO interface we were
missing some fields in the volume present in memory after it has been
created.
This was caused by the create method not passing the expected attributes
to the _from_db_object.
Because of this missing information we have places in the code where we
forcefully a reload of the whole Volume OVO when it shouldn't be
necessary.
This patch fixes the create method of the Volume OVO and removes an
instance of the forceful reload of the volume, reducing our DB calls.
Change-Id: Ia59cbc5a4eb279e56f07ff9f44aa40b582aea829
In Change-ID Ic8a8ba2271d6ed672b694d3991dabd46bd9a69f4 we added:
vref.multiattach = self._is_multiattach(volume_type)
vref.save()
Then we remove the assignment but forgot to remove the save.
This patch removes that unnecessary save call.
Change-Id: I993444ba3b6e976d40ae7c5858b32999eb337c66
The old attachment API has a mix of OVO and DB method calls that can
result in admin metadata being removed.
When we automatically update the admin metadata using DB methods these
changes are not reflected in the volume OVO, so when we pass the OVO via
RPC the receptor will assume that admin metadata present in the OVO is
up to date, and will delete key-value pairs that exist in the DB and are
not in memory.
That is happening on the old `attach` method with the `readonly` key
that gets removed by the volume service after it was added (calling the
DB) on the API service.
Patch doesn't include a release note to avoid making unnecessary noise
in our release notes, because it is unlikely to affect existing users
since Nova will be using the new attachment API.
Change-Id: Id3c7783a80614e8a980d942343ecb9f47a5a805a
This patch fixes volume type retyping when multiple
availability zones are in place. This ensures that the
request_spec to the scheduler has the availability zone
in the right place for the AvailabilityZoneFilter to be
able to see it and use it for filtering.
Change-Id: I3f6cca6eb87ac4727b06b167b5aa12da07ba8fb5
Closes-Bug: #1883928
This patch includes 2 fix for the following issues:
1) ast.literal_eval() doesn't work with int and float values
See comment inline
2) Do not traverse and modify the same dict:
while traversing filters dict, we're modifying it inside the loop
which distorts the order of elements (adding modified elements in the
end). to fix this, i've used a temp dict that will be used to traverse
and modification will be done in the filters dict.
Closes-Bug: #1883490
Change-Id: I18b4b0b1b71904b766f7b89df49f5539e3c7662a
When a non user calls revert to snapshot using the snapshot name the
client returns a failure status:
$ cinder --os-volume-api-version 3.59 revert-to-snapshot snap1
ERROR: No snapshot with a name or ID of 'snap1' exists.
The revert to snapshot API requires an UUID, so the cinderclient lists
the snapshots passing "all_tenants=1" and the name of the snapshot to
find its first when it's not a UUID.
The problem that we were only removing the "all_tenants" filter for
admins, leaving it for normal users and passing it to the DB which
wouldn't understand that filter and return None.
This patch ensure we always remove the "all_tenants" filter (as we do in
other places) before calling the OVO method to list snapshots.
Change-Id: I5fd3d7840b36f6805143cd1b837258232e7bc58a
Fixes-Bug: #1889758
Allows support for setting a minimum and/or maximum vol size that
can be created in extra_specs for each volume_type. This allows
setting size restrictions on different "tiers" of storage.
If configured, the size restrictions will be checked at the API level
as part of volume creation or retype.
2 new volume type keys are supported for setting the minimum volume
size and maximum volume size for that type.
'provisioning:min_vol_size'
'provisioning:max_vol_size'
Implements: blueprint min-max-vol-size-by-vol-type
Change-Id: I222e778902a41e552e812896d7afd0516ee7fe68
When uploading a volume to an image, Cinder checks to see if there's
any image metadata already present on the volume, and if so, it adds
it to the image-create request so that Glance will persist these
additional properties on the new image. If during upload preparation,
Cinder wants to add specific metadata to the new image (for example,
a new encryption key id), we need to ensure that the new value for
this property is preferred to a value in the volume_glance_metadata
table.
Closes-bug: #1844725
Change-Id: Iba3c5fa4db87641a84eb22a0fc93294dd55a3132
When trying to upload volume to glance as image, currently all the
format transformation is done by software and the performance is not
good.
Leverage hardware accelerator to do the image compression, so as to
offload CPU, release CPU to do more common and complex thing.
Professional hardware accelerator will get better performance and
shorter conversion time than software solution. Currently hardware
accelerator is getting more popular and some are integrated in server
chipset by default.
This patch includes:
1. Uses the new image container_format 'compressed' introduced by Glance
in the Train release
2. Implemented a simple framework: if there's an accelerator detected in
system, then try to use it
3. Supported Intel QAT as one of the accelerator
4. Add command filter for command 'qzip' and 'gzip' in rootwrap
5. New configuration option 'allow_compression_on_image_upload' and
'compression_format' added
6. Releasenote added
Change-Id: I8460f58d2ad95a6654cf4d6a6bb367f3c537536b
Implements: blueprint leverage-compression-accelerator
Signed-off-by: Liang Fang <liang.a.fang@intel.com>
Much of our code renames this at import already --
just name it "volume_utils" for consistency, and
to make code that imports other modules named "utils"
less confusing.
Change-Id: I3cdf445ac9ab89b3b4c221ed2723835e09d48a53
This maintains consistency with our policy of not allowing the
combination of encryption and multiattach in commit
18327971ca
Change-Id: I1c22bc86d96bb1cab80c004d3450b1dac6c4a769
If multiple attachments are requested simultaneously,
a volume can be attached to an instance more than once.
This fixes this problem by detecting that a race has
occurred in _attachment_reserve, and backing out one of
the invalid attachment records.
This means that if the attachment API is called
repeatedly and quickly for the same volume, some requests
may fail, but this is much better than incorrectly
creating multiple attachments.
Closes-Bug: #1833736
Change-Id: Ic2463338b698c5cf805c0ae06d0229f54f64b3fc
The create function of GlanceImageService has called
the staticmethod _translate_to_glance. So when we call
create function, we don't need to call _translate_to_glance
again. Just remove it now.
Change-Id: I26cbbec1f5d330ffa72a075b7d8321f4d6da029c
Currently the new cinder attachment_create method allows volumes in
all status to be performed for single attachment. This may cause
problems if volume-attach operation is taken on volumes in creating
status.
In order to avoid these problems, we should only allow volumes
in 'in-use', 'reserved' status to check whether this attachment
creation is a override operation for a same instance.
Related-bug: #1694530
Change-Id: Ie59f609d38290147a67a936c61e0164fe278da99
Closes-Bug: #1813267
This patch fix bug: when revert volume to snapshot which status
is not available, the report msg error, '%s' should be behind
snapshot not volume.
Change-Id: Ia449f3619837f2050ddfbbbf006451d1aba5014a
Rafael Weingärtner