This patch addresses the scenario where an
incremental backup can be created having a
parent backup that was taken in a different
project. This scenario ultimately leads to
a silent error when creating/deleting the
incremental backup and quotas going out of
sync.
The proposed fix is to narrow the backup search
down to the same project. To achieve this, a
method's signature had to be updated to achieve
the desired optimized behavior of passing the
volume's project_id parameter.
Closes-bug: #1869746
Closes-bug: #1873518
Change-Id: Icb621ff5966133f59d9d43ca2dd9f8e1919b1149
(cherry picked from commit 8ebeafcbba)
The patch adds the following functionality:
* A default volume type (``__DEFAULT__``) will be
created during cinder DB migration.
* Admins can override the default type in cinder.conf
* Creating a new volume without any type (and
`default_volume_type` option is unset in cinder.conf)
will be assigned the ``__DEFAULT__`` type.
* The default volume type cannot be deleted
* All untyped volumes and snapshots will be migrated to the
``__DEFAULT__`` type.
Change-Id: I4da0c13b5b3f8174a30b8557f968d6b9e641b091
Implements: blueprint untyped-volumes-default-volume-type
Online data migrations are part of the rolling upgrades mechanism, and
they are run on the current installed version before we upgrade to the
next one to ensure that all DB records have been upgraded even if they
have not been accessed during the life of the current release service
life.
This means that online migrations from one release should be removed on
the next release, as they are no longer needed.
We still have the Queens online data migrations in our code, and we
should remove them in master, Stein, and Rocky.
This patch also adds checks to the status tool to check that these
online data migrations have completed.
Closes-bug: #1837703
Change-Id: I025789f92eb318b8e5a531bea6cf106d52f268ae
A lot of our config option reference documentation was out of date, with
deprecated and removed options and changed defaults. This changes the
"Additional options" configuration reference to use the config-table
directive so the published docs stay in sync with changes in the code.
Closes-bug: #1814910
Change-Id: Ib891435dc69eb795ea6d4d0fb6626259e95cf27f
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
This patch adds support for transfer pagination.
Co-Authored-By: Yikun Jiang <yikunkero@gmail.com>
Closes-bug: #1814195
Change-Id: Idb04f783b2287b2b45d626131648b0005a232fbe
This patch adds the option to split retrieved volumes and snapshots
by chunks during Volume Manager host initialization. Query results
will be obtained in batches from the database and not in one shot
to avoid extreme memory usage. Max number of volumes and snapshots
per batch is controlled by option init_host_max_objects_retrieval.
Option is disabled by default for backward compatibility.
To migrate any ConfKeyManager keys based on fixed_key to the currently
configured key manager Volume Manager will use special lightweight
object - VolumeMigration.
Change-Id: I53eccc77fdc2c35b27ed430af62dc18e7d1bde69
Closes-Bug: 1681374
This feature changes Cinder to transfer
snapshots with volumes at the same time by default.
If user doesn't want to transfer snapshots, they could use
a new optional argument '--no-snapshots' after microversion 3.55.
And we also introduce the new V3 api 'v3/volume_transfers'
to move this API out of contrib into Cinder V3 API.
The cinderclient patch: https://review.openstack.org/#/c/577611/
Change-Id: If848d131e5edcdb77d0b3c2ca45a99c4d5e14d1e
Implements: blueprint transfer-snps-with-vols
This adds the hook to the cinder-manage db online_data_migration
CLI for migrating attachment_specs entries to the volume_attachment
table.
Along the way, two changes have to be made in the VolumeAttachment
object:
1. Handle 'connector' in updates in VolumeAttachment.create(); this
makes the test work to create a volume attachment with the connector
already set, something we should support so you don't have to create()
and then save() the object.
2. In _from_db_object, we have to set the context on the attachment
object and reset the volume field changes *before* saving the
connector changes, otherwise save() fails because (a) it has an
empty context and (2) you can't change a volume through the attachment.
Change-Id: Ic0698e4fc0f6519c2676fe91e1a6c0fb47bbeb95
Related-Bug: #1737724
The attachment_specs table's key and value columns are strict
strings, which means things like a wwpns list value for a fibrechannel
connector can't get stored there and results in a DB error during
attach with the new volume attach flow in Nova.
The attachment_specs table is arguably not the best way to store
this data, which is just a dict like the connection_info.
A better way to store this is as a serialized json blob on the
volume_attachment record itself.
This patch adds the database migration to add the column and
an online data migration routine to migrate existing attachment_specs
entries when a volume attachment object is loaded from the database.
The volume manager attachment_update flow is changed to store
new connector attachments in the volume_attachment table directly.
An online data migration hook for the CLI will be added in a follow
up change.
Change-Id: Ica1f0e06adf0afcf740aad8cdc8d133ada1760c8
Closes-Bug: #1737724
This patch adds a service_uuid FK to the volumes table.
Up until now we've just done some host name parsing and
to match up service node with where the volume is being
serviced from.
With this, we now have a unique identifier that's user
visible to indicate what node the volume-service for a
particular volume is being serviced by.
We'll use this for things like share-target locks going
forward.
Change-Id: Ia5d1e988256246e3552e3a770146503ea7f7bf73
We upgraded the backup service attribute from
module name to class name. This results in
the failure of restoring earlier backups
with new code. This patch upgrades the older
backup records as well as adds compatibility
in code.
Change-Id: I0374ab9f2d37ecf8d8bce191c7535601266d432e
Closes-Bug: #1727912
os-host show API should only return the resouce count on the
specified host.
The logic in Cinder which counts the project's resouce on all host
is wrong.
Change-Id: I96cd285a82b44af8818514692818e739443dcc45
Closes-bug: #1699936
This patch adds support for display count info
in volume, backup and snapshot's list&detail APIs
since microversion 3.45, for instance:
1. /v3/{project_id}/volumes?with_count=True
2. /v3/{project_id}/volumes/detail?with_count=True
Depends-On: 1c8fe0ade4
Change-Id: I2e92b27c36357120fcf0ec5917c6484441c946a8
Implements: bp add-amount-info-in-list-api
In the 112 db migration I was being lazy and generating and
updating the newly added UUID column. I also didn't update
the Service object (left that for the follow on patch).
Turns out we're going to want/need the online_data_migration
pieces for some follow up work and we might as well do this at
least a little bit more efficiently/correctly now.
This patch modifies the db migration to NOT try and populate
the newly added UUID fields in the Service table, and it
updates the Service object properly including adding the gen UUID
components to service.create()
This also fixes up what we had in place for the online_data_migration
code in cmd/manage.py; note that we had a framework there/started but
it wasn't being used up to this point.
Change-Id: I6696a15cd2c8fbf851a59b8d6d60ae1981bb1b89
Closes-Bug: #1721837
The new attachment_update method in Cinder's API creates an attachment
object and populates it with the provided connector info. In addition,
we set the volumes status to in-use and update the attachment object
status to "attached".
This isn't really accurate though, because we don't know if the volume
is actually attached (connected) by the consumer or not. Also a big
side effect here is that currently all of our tests and automation
use volume-status to determine if a volume is fully connected/ready
for use and that everything went well. It's used as an ack in most
cases.
This change goes back to using multiple states to signify where a
an attachment is in it's life-cycle:
1. reserved
We've created an empty attachment record but haven't done anything
with it yet.
2. attaching
We provided a connector and set up the TGT so that everything is
ready for a consumer to connect/use it.
3. in-use
An ACK back from the consumer letting us know that they connected
it successfully and are doing their thing.
Some consumers don't need or care about this last step, and we're going
to provide a means to set it straight to attached/in-use, but for
this bug we don't need to introduce that particular *feature*.
Sadly, this requires a micro-version bump and a new API call to
toggle the state from the API, pushing us to 3.44.
closes-bug #1710295
Change-Id: I57631d3deddb2d7cd244584e82206ee17fe2dd78
Now only volume and snapshot has metadata property.
We should support it for backup as well.
This patch added/updated the related db and ovo model,
updated the related backup CRUD APIs.
Change-Id: I6c4c175ec3be9423cdc821ccb52578dcfe442cbe
Implements: blueprint metadata-for-backup
The docstring for conditional_update had ":returns" instead
of ":returns:" which was causing a Sphinx warning to be
generated and caused the documentation to be improperly
rendered.
This patch corrects the formatting.
Change-Id: I7388904593572cf61194ca1e7cabf5e7a461f3b0
If cluster is enabled, the group should be updated with cluster
as well.
Change-Id: I72dc6c61f10417f8382c7c488be27ed5199b75b9
Implements: blueprint cinder-volume-active-active-support
After transferring a volume without snapshots from one user project
to another user project, if the receiving user uses cascade deleting,
it will cause some exceptions in driver and volume will be error_deleting.
APIImpact
Change-Id: I9a870118c9c68799a4dab4bfa1f08b1e65a4796a
Closes-Bug: #1698310
This patch implements the spec of reverting volume to
latest snapshot.
Related tempest and client patches:
[1] https://review.openstack.org/#/c/463906/
[2] https://review.openstack.org/#/c/464903/
APIImpact
DocImpact
Partial-Implements: blueprint revert-volume-to-snapshot
Change-Id: Ib20d749c2118c350b5fa0361ed1811296d518a17
Probably the most common format for documenting arguments is reST field
lists [1]. This change updates some docstrings to comply with the field
lists syntax.
[1] http://sphinx-doc.org/domains.html#info-field-lists
Change-Id: I0fe2d2faa7a1abd6d5e84f05e7fdbb13661a94dc
Add filter, sorter and pagination support in group
snapshot with new microversion v3.29.
APIImpact
Closes-Bug: #1670540
Change-Id: I2ed1b87b022314b157fe432a97783ab50316367b
First DB schema migration in Pike ensures that all of the Ocata's data
migrations are completed (Iab993f93b4145a67ec2d6144acc61f2105586b3c). We
can then remove them from Pike's code.
The commit also removes checks that were being made in user messages
code. These are not needed as we're sure everything's migrated.
Change-Id: I882e953800c4f4473f175779b6a44ad55928db65
Currently get_capabilities accepts only host as a service identifier and
fails when it cannot find a service with such host property. In A/A
deployment, it should accept cluster_name as well, because
get_capabilities addresses backend and not individual host. This commit
fixes this up by making the call seek for available services using both
host and cluster_name properties.
Although it's an API change, it doesn't need a new microversion, because
it falls under "Did we silently fail to do what is asked?" category.
Change-Id: If7bca131c84fc997da8ad277eccd134e8ca4b316
Closes-Bug: 1660990
Current volumes summary code has unnecessary code duplication as well as
a variable name that doesn't follow the project guidelines.
The variable in question is `allTenants` that should have been named
`all_tenants`.
This patch removes the code duplication by merging the two DB and OVO
methods into one.
TrivialFix
Change-Id: Ia80568147462450a707bc316100e5289c4b02eed
Replaces the original patch:
https://review.openstack.org/#/c/387712/
Use the python-cinderclient WIP:
https://review.openstack.org/#/c/387716/
Here's what you can do currently:
`cinder attachment-create <volume-uuid>`
Currently only tested/implemented the reserve piece
Will create an attachment object, set volume to a
status of 'reserved'
`cinder attachment-list`
Simple list output of attachments
`cinder attachment-show <attachment-uuid>`
Detailed list of specified attachment
`cinder attachment-delete <attachment-uuid>`
Removes an attachment
Change-Id: Ie15233c99d91de67279b56d27a5508c5ea98d769
Freeze functionality in the replication feature doesn't work as
expected, since it is not being used on the scheduler to exclude
backends or used on the API or volume nodes so API-to-Vol operations
like delete and create snapshot will also work.
This patch fixes the freeze mechanism by excluding frozen backends in
the scheduler and checking the if the service is frozen on all other
modifying operations.
Since extend operation now goes through the scheduler it will be frozen
there.
Closes-Bug: #1616974
Change-Id: I4561500746c95b96136878ddfde8ca88e96b28c6
Image Volume Cache mechanism was not cluster aware and therefore cached
images would not be shared among different services in the same cluster.
This patch addresses this issue and makes sure that they share by
cluster if there is one.
This patch does not address any concurrency issues that may currently
exist in the caching mechanism's code.
Implements: blueprint cinder-volume-active-active-support
Change-Id: I9be2b3c6dc571ce2e0e4ccf7557123a7858c1990
This patch adds the 'VOLUME_' prefix to all message
event ids. This will prevent collisions and confusion
when other projects add user messages and create their
own event ids.
Also fix issue where request_id column is nullable in
sqlalchemy model but not properly set in the db
migration.
Implements blueprint summarymessage
Co-Authored-By: Sheel Rana <ranasheel2000@gmail.com>
Co-Authored-By: Michał Dulko <michal.dulko@gmail.com>
Change-Id: Ic23f898281870ad81c5a123302ddca50905952ea
Allow volume delete to take parameters "cascade",
or "force", or both.
A new policy field, "volume:force_delete" is added
with the default of "rule:admin_api".
Implements: blueprint volume-delete-parameters
APIImpact: New parameters to volume delete
Change-Id: Ic47cfcf1cc7d172d7f9d5b093233035f797982f5
Some of our DB methods have names that are not consistent with the
general naming we use to differentiate between methods that return a
single value -_get_- and those that return a list of values -_get_all_-.
This patch renames some of these methods to make them reflect whether
they are returning a single entity or a list of entities.
Change-Id: I3d32d65e573f71cf58c8956ef86debd4824d9f02
This patch provides script to migrate data from consistencygroups to
groups and from cgsnapshots to group_snapshots.
In the migration script, it creates a default_cgsnapshot_type
for migrating data and copies data from consistency groups to
groups and from cgsnapshots to group_snapshots. Migrated consistency
groups and cgsnapshots will be removed from the database.
It depends on the following patch that adds generic code for
online data migrations:
https://review.openstack.org/#/c/330391/
Run the following command to migrate CGs:
cinder-manage db online_data_migrations
--max_count <max>
--ignore_state
max_count is optional. Default is 50.
ignore_state is optional. Default is False.
UpgradeImpact
Partial-Implements: blueprint generic-volume-group
Related: blueprint online-schema-upgrades
Change-Id: I1cf31e4ba4acffe08e2c09cbfd5b50cf0ea7a6e0
One of the useful things that was missing from the
volume_attachments code was get_all methods.
This patch adds a get_all and a get_all_by_project, it
also goes ahead and adds some filtering capability to
the existing get_by_xxxx calls since we added the framework
for it in the get_all additions.
I also looked at refactoring our db methods for attach to just:
* attach_create
* attach_update
* attach_destroy
* attach_get
* attach_get_all
This would probably be good as an independent effort to
clean things up and bring these calls more in line with
others, but there's a lot of work to update the objects
and existing code, might be better to wait until after
implementing the new attach API.
Co-Authored-By: Michał Dulko <michal.dulko@intel.com>
Change-Id: I40614fe702f726c74ff05f93faaf6ee79253447f
To be able to support multiple hosts working with the same resources we
have added the workers table to keep track of which host is working with
each specific resource.
This patch makes c-vol service work with this new table by adding
entries on cleanable operations and removing them once these operations
have completed.
Service cleanup on initialization has also been changed to use this new
table so hosts will cleanup only resources from operations they left on
the air and leave any operations that are being processed by other
hosts.
Specs: https://review.openstack.org/236977
Implements: blueprint cinder-volume-active-active-support
Change-Id: I4e5440b8450558add372214fd1a0373ab4ad2434
Attachments_get_by_host|instance should be just that, associating
them with a specified volume-id doesn't really solve the problem.
If that relationship is needed a simple get_by_volume will work with
some inspection.
This patch removes the volume_id arg from those get methods. A follow
up patch will add get_all with filters for more complex relationships.
Change-Id: Ic5ffdced96fdf780cce2a1227c5f2a599860f0ca
Closes-Bug: #1632433
The patch enables the function to retype a volume
to a volume type with different encryptions.
The correspoding tempest is added in
https://review.openstack.org/#/c/343993/, and it
succeeds in master branch.
Change-Id: I66d1cfad7c37215cadeca9b7d07cb646fb35b50f
Implements: blueprint retype-encrypted-volume
This is the fifth patch that implements the generic-volume-group
bluerpint. It adds APIs for group snapshots and create group
from source.
This patch depends on the fourth patch which implements group
snapshots support in the volume manager:
https://review.openstack.org/#/c/361376/
Client side patch is here:
https://review.openstack.org/#/c/329770/
Current microversion is 3.14. The following CLI's are supported:
cinder --os-volume-api-version 3.14 group-create-from-src
--name my_group --group-snapshot <group snapshot uuid>
cinder --os-volume-api-version 3.14 group-create-from-src
--name my_group --source-group <source group uuid>
cinder --os-volume-api-version 3.14 group-snapshot-create
--name <name> <group uuid>
cinder --os-volume-api-version 3.14 group-snapshot-list
cinder --os-volume-api-version 3.14 group-snapshot-show
<group snapshot uuid>
cinder --os-volume-api-version 3.14 group-snapshot-delete
<group snapshot uuid>
APIImpact
DocImpact
Partial-Implements: blueprint generic-volume-group
Change-Id: I2e628968afcf058113e1f1aeb851570c7f0f3a08
This is the third patch that implements the generic-volume-group
bluerpint. It adds database and object changes in order to support
group snapshots and create group from source. The API changes will
be added in the next patch.
This patch depends on the second patch which adds create/delete/update
groups support which was already merged:
https://review.openstack.org/#/c/322459/
The next patch to add volume manager changes is here:
https://review.openstack.org/#/c/361376/
Partial-Implements: blueprint generic-volume-group
Change-Id: I2d11efe38af80d2eb025afbbab1ce8e6a269f83f
This is the second patch that implements the generic-volume-group
bluerpint. It adds the groups table and introduces create/delete/
update/list/show APIs for groups.
It depends on the first patch which adds group types and group specs:
https://review.openstack.org/#/c/320165/
Client side patch is here:
https://review.openstack.org/#/c/322627/
Current microversion is 3.13. The following CLI's are supported:
cinder --os-volume-api-version 3.13 group-create --name my_group
<group type uuid> <volume type uuid>
cinder --os-volume-api-version 3.13 group-list
cinder --os-volume-api-version 3.13 create --group-id <group uuid>
--volume-type <volume type uuid> <size>
cinder --os-volume-api-version 3.13 group-update <group uuid>
--name new_name description new_description
--add-volumes <uuid of volume to add>
--remove-volumes <uuid of volume to remove>
cinder --os-volume-api-version 3.13 group-show <group uuid>
cinder --os-volume-api-version 3.13 group-delete
--delete-volumes <group uuid>
APIImpact
DocImpact
Change-Id: I35157439071786872bc9976741c4ef75698f7cb7
Partial-Implements: blueprint generic-volume-group
Rodrigo Barbieri