This patch addresses the scenario where an
incremental backup can be created having a
parent backup that was taken in a different
project. This scenario ultimately leads to
a silent error when creating/deleting the
incremental backup and quotas going out of
sync.
The proposed fix is to narrow the backup search
down to the same project. To achieve this, a
method's signature had to be updated to achieve
the desired optimized behavior of passing the
volume's project_id parameter.
Closes-bug: #1869746
Closes-bug: #1873518
Change-Id: Icb621ff5966133f59d9d43ca2dd9f8e1919b1149
(cherry picked from commit 8ebeafcbba)
Incremental patch I516b7c82b05b26e81195f7f106d43a9e0804082d
introduced a regression, breaking the volume backup export
functionality.
Apparently, the newly introduced "parent" db field isn't
serialized properly.
This is the traceback from the launchpad bug:
oslo_versionedobjects.exception.ObjectActionError:
Object action obj_load_attr failed because: attribute
parent not lazy-loadable
Looking at the traceback, and the Backup OVO code it seems
like we have 2 problems:
- We don't support lazy loading the parent
- We try to serialize the parent
Co-Authored-By: Gorka Eguileor <geguileo@redhat.com>
Change-Id: I017602353e96cf9f0922074f94276002b17d1359
Closes-Bug: #1862635
Ceph RBD backend ignores the `--incremental` option when creating a
volume backup. The first backup of a given volume is always a full
backup, and each subsequent backup is always an incremental backup.
This behavior makes it impossible to remove old backups while
keeping at least one recent backup.
Since Cinder will not find the latest_backup id as parent_id if
'--incremental=False', so we can use the parent_id to ensure
whether do the full backup in rbd driver or not.
If the incremental flag '--incremental' is not specified, this
patch will always create a new full backup for rbd volume.
Change-Id: I516b7c82b05b26e81195f7f106d43a9e0804082d
Closes-Bug: #1810270
Closes-Bug: #1790713
Co-Authored-By: Sofia Enriquez <lsofia.enriquez@gmail.com>
Enhance the code that migrates the ConfKeyManager's fixed_key to
Barbican to also consider the Backup table. When the original key
migration feature was added, the encryption key ID was not stored in
the Backup table. But now the Backup table contains that field, so
the migration code needs to handle that table as well.
Whereas the cinder-volume service is responsible for migrating keys
in the Volume and Snapshot tables, the cinder-backup service handles
migrating keys in the Backup table. Each instance of the service
migrates its own entries by matching the "host" field in the
corresponding tables.
The Backup OVO now inherits from base.CinderComparableObject. This does
not affect the object's hash signature, and so the version number does
need to be incremented.
Closes-Bug: #1757235
Change-Id: Id4581eec80f82925c20c424847bff1baceda2349
As described in the launchpad bug [1], backup operations must take care
to ensure encryption key ID resources aren't lost, and that restored
volumes always have a unique encryption key ID.
[1] https://bugs.launchpad.net/cinder/+bug/1745180
This patch adds an 'encryption_key_id' column to the backups table. Now,
when a backup is created and the source volume's encryption key is
cloned, the cloned key ID is stored in the table. This makes it possible
to delete the cloned key ID when the backup is deleted. The code that
clones the volume's encryption key has been relocated from the common
backup driver layer to the backup manager. The backup manager now has
full responsibility for managing encryption key IDs.
When restoring a backup of an encrypted volume, the backup manager now
does this:
1) If the restored volume's encryption key ID has changed, delete the
key ID it had prior to the restore operation. This ensures no key IDs
are leaked.
2) If the 'encryption_key_id' field in the backup table is empty, glean
the backup's cloned key ID from the backup's "volume base metadata."
This helps populate the 'encryption_key_id' column for backup table
entries created prior to when the column existed.
3) Re-clone the backup's key ID to ensure the restored volume's key ID
is always unique.
Closes-Bug: #1745180
Change-Id: I6cadcbf839d146b2fd57d7019f73dce303f9e10b
Now only volume and snapshot has metadata property.
We should support it for backup as well.
This patch added/updated the related db and ovo model,
updated the related backup CRUD APIs.
Change-Id: I6c4c175ec3be9423cdc821ccb52578dcfe442cbe
Implements: blueprint metadata-for-backup
Probably the most common format for documenting arguments is reST field
lists [1]. This change updates some docstrings to comply with the field
lists syntax.
[1] http://sphinx-doc.org/domains.html#info-field-lists
Change-Id: I0fe2d2faa7a1abd6d5e84f05e7fdbb13661a94dc
Some of our DB methods have names that are not consistent with the
general naming we use to differentiate between methods that return a
single value -_get_- and those that return a list of values -_get_all_-.
This patch renames some of these methods to make them reflect whether
they are returning a single entity or a list of entities.
Change-Id: I3d32d65e573f71cf58c8956ef86debd4824d9f02
volume/snapshot/backup created/deleted time is offset-aware datetime,
while start_time and end_time are converted to offset-naive datatime
after passing in. Those need to be converted to same type before comparison.
Change-Id: I874f9c5d27b91dbff68644952b315badba0034c4
Closes-bug: #1646675
Co-author-by: Jeremy Liu <liujiong@gohighsec.com>
This commit introduces BackupDevice object to formalize data earlier
sent over RPC as an undefined dict. This is required to be able to make
non-backward compatible changes to data sent as this parameter while
maintaining compatibility with previous release - so to support rolling
upgrades.
Change-Id: Ie57d84e32ec1c5fcfac27a7bb6d4bbb189108a5b
Partial-Implements: blueprint cinder-objects
None of the Versioned Objects destroy methods we have update the
instance fields, so after doing a destroy on an instance the status,
deleted, and deleted_at fields will still be the same as they were
before, leaving the instance in an inconsistent state.
This is important because our workers decorator for the HA A-A cleanup
mechanism relies on the status of the resources being properly updated
by the operations.
Change-Id: I4f15356931007e2ca725e477327c1ef67c7db487
Closes-Bug: #1604347
In Cinder we don't have an indirection service to forward object methods
like Nova does (Conductor) so we shouldn't be defining all methods as
remotable since they are not.
If we ever decide to have a Conductor we can easily add them back where
appropriate, but for now they are only a hindrance:
- It's confusing, because most people don't understand what it actually
means in Cinder and they just add it to the methods because they see
it in other OVO methods.
- Every time we change a remote method's signature or add a new remote
method to an object the hash of that object changes and we need to
update our test_versions test, which is not necessary.
- This makes the linking of VOs with their list counterparts
problematic, because everyone will need to know that there is no need
to bump the version of a list object whenever they add or modify a
remotable method.
- We are executing unnecessary code with decorators, even if it's not
much, it's still wasted CPU time .
This patch removes all appearances of remotable and remotable_class
decorators thus making our OVOs consistent with our implementation and
removing the need of updating our tests on unrelated OVO changes.
Change-Id: I18a0f619b4b108c553d9567b5f21f978466f1670
To make objects that have other objects as fields compatible to an
earlier version, oslo versioned objects uses either a manifest passed to
obj_to_primitive or the object's obj_relationships mapping.
Which means that if we don't have any of those mechanisms in place our
rolling upgrades mechanism will fail whenever we try to backport a
Versioned Object that has set an ObjectField field because Oslo
Versioned Object will not know how to backport that related object.
This patch introduces the usage of manifests on backports when we are
doing rolling upgrades.
For the manifest, we use the data in our Objects History. Which means
that as long as we keep history in OBJ_VERSIONS right we will not have
to create and worry about keeping lists' child_versions field or our
versioned object's obj_relationships for fields with types
ListOfObjectsField and ObjectField.
We also don't have to worry about cascade version bumping, as in
changing the List OVO version whenever the OVO it contains gets bumped,
or bumping our OVO whenever one of the related OVO fields is bumped.
Closes-Bug: #1571566
Change-Id: Ibc1a1257830c925c10696c0b5aedd5f471c538d0
In some modules the global LOG is not used any more. And the import
of logging is not used. This patch removes the unused logging import
and LOG vars.
Co-Authored-By: ChangBo Guo(gcb) <eric.guo@easystack.cn>
Change-Id: Ia72f99688ce00aeecca9239f9ef123611259a2fa
The DB models define project_id and user_id as Column String fields of
255 characters because historically they weren't wasn't necessarily
UUIDs, but when we created the Versioned Object counterparts of the ORM<
models we forced them to be UUIDs, which will create problems for any
deployment where any of them is not using UUIDs.
We know cases where this is a problem for the project_id, moreover now
that Oslo Versioned Objects library is going to enforce the use of UUIDs
in UUIDFields.
This patch changes those fields to make them of StringField type as they
should have been all along.
This change has no implications on the serialization, so we don't need
to bump versions, we can simply change the hash in the test.
Change-Id: Id94ff7777260cc443f838aa3a8399582b2d00fc2
Currently there are some fields whose nullable in db
model and object don't match. This may lead object query
fails.
For example, for volume, if attach_status is nullable in
db schema, but is not nullable in object schema.
When writting data to db using db interface, the attach_status
can be null. But later when reading data from db to construct
volume object, it raises exception. As a result, all the
query related to the volume fails.
This patch is to make all the fields in corresponding db
model and object match.
Change-Id: I84fb16f0a67f9a1f7a09e91bb8ebe6298f3f49c4
Closes-Bug: #1549669
Related-Bug: #1549673
This patch is to add restore_volume_id in backup object.
When restoring a volume from a backup, it saves the
volume in backup object.
Currently volume service and backup service are in same host.
When backup service starts, it does cleanup tasks on both
backups and volumes on current host.
But with bp scalable-backup-service, backup service and
volume services can run on different hosts. When doing cleanup
tasks, we need to find out backing-up and restoring volumes
related to the backups on current host. Backing-up volumes can
be found with field backup.volume_id. Restoring volumes are found
by new field backup.restore_volume_id.
Change-Id: I757be7a5e47fc366c181400587b5a61fe3709a0b
Partial-Implements: bp scalable-backup-service
Co-Authored-By: Tom Barron <tpb@dyncloud.net>
* BaseObjectsTestCase._compare(): replace hasattr() with getattr()
and a white list of expected exceptions. On Python 2, hasattr()
ignores *all* exceptions and so may hide real bugs.
* Mocks: except a call to obj.__bool__() rather than
obj.__nonzero__() on Python 3. bool(obj) now calls obj.__bool__()
on Python 3.
* Replace dict.keys() with list(dict.keys()) to get a list on Python
3. On Python 3, dict.keys() now returns a view.
* JSON/base64:
* Replace jsonutils.dumps() with jsonutils.dump_as_bytes() to get
JSON as bytes
* Replace base64.encodestring() with oslo_serialization.base64.encode_as_text() to get
base64 as bytes
* Replace base64.decodestring() with
oslo_serialization.base64.decode_as_text() to accept Unicode and
return Unicode
* tests-py3.txt: add cinder.tests.unit.objects.test_* (11 files)
Partial-Implements: blueprint cinder-python3
Change-Id: I9ce7753eb947bf9a5c91e643f5c65156c8e3f7cc
This change adds a new enum and field, BackupStatus and
BackupStatusField, that will hold the constants for the 'status' field
of the Backup object. This enum and field are based off the base
oslo.versionedobjects enum and field. This also changes over the Backup
object to use the new field (and bumps the version so newer versions
know to enforce valid values). Finally, all uses of strings for
comparison and assignment to this field are changed over to use the
constants defined within the enum.
Change-Id: I3df71e361f2bd37417133a4c1189e37466ed0b56
Implements: bp cinder-object-fields
Today we can backup a volume, but not a snapshot.
This patch adds support to backup snapshots and
provide another layer of data protection for the
user.
DocImpact
implements blueprint backup-snapshots
Change-Id: Ib4ab9ca9dc72b30151154f3f96037f9ce3c9c540
Currently each Versioned Object needs to implement its own get_by_id,
with this patch they don't need anymore, since it will be included in
the base class CinderObject and it will work for all the objects.
This will help for other things like having a refresh method or
conditional updates in the objects.
Related-Bug: #1490944
Related-Bug: #1238093
Related-Bug: #1490946
Related-Bug: #1469659
Change-Id: I355dc8eaefed93003533ee083f74acd1315f057e
When importing backup metadata only required fields were being imported,
and all other fields were being ignored.
This led to some serious problems when importing backups back, since
you would lose not only the original Volume ID, but also relevant parent
id information needed to restore incremental backups.
This patch fixes this by importing everything back and creating backup
records with the right ID.
Closes-Bug: #1455043
Closes-Bug: #1476416
Depends-On: Id7ab6e174c1fe85772477f03059c4f457c5c8b17
Change-Id: Ia42ba73e9078d993c63f8e16308151ad11721ea9
Backups list/detail does not support pagination now.
Add pagination support like volume does by using marker,
limit, sort_key and sort_order.
Partial-Implements: blueprint extend-limit-implementations
DocImpact
APIImpact
Use marker, limit, etc. in list url like volume does.
(Pulled from gate, cinder can no longer pass unit tests)
Change-Id: I33dbdc34c61f78eab2a78a9cda08780068867a03
1. Add 'is_incremental=True' and 'has_dependent_backups=True/False' to
response body of querying.
2. Add parent_id to notification system.
Since we need to get volume has_dependent_backups value when querying
volume detail list, to reduce the performance impact, add index to
parent_id column in backup table.
APIImpact
When showing backup detail it will return additional info
"is_incremental": True/False and "has_dependent_backups": True/False
DocImpact
Change-Id: Id2fbf5616ba7bea847cf0443006800db89dd7c35
Implements: blueprint cinder-incremental-backup-improvements-for-l
The following patch is part of the cinder effort to
support rolling upgrade. This patch adds columns
to the services table to track the RPC and
oslo_versionedobjects versions of each service.
Follow up patches will be made to have each service:
register its RPC and oslo_versionedobjects versions on
startup, make the RPC and oslo_versionedobjects versions
compatible with an older release, and update the versions
once all services are updated to the latest release.
Change-Id: Ifa6c6ac230988c75dcc4e5fe220bfc5ee70ac338
Partial-Implements: blueprint rpc-object-compatibility
Encoding and decoding of backup metadata was being done in the backup
driver, but since this is not driver specific data we are talking about
it makes more sense to have this logic in the backup object.
This patch moves the logic to the object while still allowing to have
importing/exporting driver specific backup data, even when this is not
being used at the moment.
Related-Bug: #1455043
Related-Bug: #1476416
Change-Id: I690748e4542880d5e926f3cbf8cedfc86de5cb1f
This patch adds support for non-disruptive backup for
volumes in 'in-use' status as follows:
Adds a force flag in create backup API when backing up
an 'in-use' volume.
For the default implementation in volume/driver.py:
* Create a temporary volume from the original volume
* Backup the temporary volume
* Clean up the temporary volume
For the LVM driver:
* Create a temporary snapshot
* Obtain local_path for the temporary snapshot
* Backup the temporary snapshot
* Cleanup the temporary snapshot
Attach snapshot will be implemented in another patch.
Partial-implements blueprint non-disruptive-backup
Change-Id: I915c279b526e7268d68ab18ce01200ae22deabdd
Currently we cannot save date fields in Versioned Objects because they
are timezone aware and our database is not.
If we look at our objects definition, we are not giving any value to
tzinfo_aware argument, so it takes DateTime default value (True).
In our sqlalchemy model definitions we are defining our datetime fields
as Column(DateTime) which will take sqlalchemy default: class
sqlalchemy.types.DateTime(timezone=False)
In most cases we are accessing those fields directly through the DB
methods, so we don't see the problem, but if we were to modify any of
those fields using versioned object, for example scheduled_at, and then
save the object we'll get an exception:
File ".../sqlalchemy/sql/type_api.py", line 278, in compare_values
return x == y
TypeError: can't compare offset-naive and offset-aware datetimes
Because we are trying to save a timezone aware date to a timezone
unaware DB field.
Following Dan's advice we create a specific cinder_obj_get_changes,
instead of overwriting obj_get_changes, that returns all datefields as
timezone naive UTC datetimes ready for saving.
Change-Id: Ie7b0249e3f6850e7c70d23fc53cfaf27fe04afbb
Closes-Bug: #1469120
db.backup_destroy requires admin context to mark backup record as
deleted. Commit Icff37261b367463b71a1268be16f9c97f595bf0c removed admin
context passed to backup_destroy method.
This patch uses elevated context to destroy backup.
Change-Id: I75b9e1fff48569a8aa320f2f02914fc7b6665d79
Closes-Bug: #1467167
This patch adds VersionedObjects abstraction layer to volume backups.
The object derives from CinderObjectDictCompat, so it supports both
object (obj.prop) and dict (obj['prop']) syntax to access properties.
Complete move to object notation will be made in a follow up clean up
patch.
Co-Authored-By: Grzegorz Grasza <grzegorz.grasza@intel.com>
Change-Id: Icff37261b367463b71a1268be16f9c97f595bf0c
Partial-Implements: blueprint cinder-objects
Rodrigo Barbieri