Commit Graph

580 Commits (4ee754f359e8f7a4ef32df2ca87319c6bc381c47)

Author SHA1 Message Date
Zuul f63e726ad5 Merge "Allow global admins to operate sd resources from other projects" 4 years ago
huangtianhua 14fae9e7de Allow global admins to operate sc resources from other projects
Allow global admin users to operate the software configs from
other projects.

Change-Id: I9135da38c2f58f29b351bb1f5336e72a2f78d5fc
Partial-Bug: #1649759
4 years ago
huangtianhua 6fd6fefc47 Allow global admins to operate sd resources from other projects
Allow global admin users to operate the software deployments from
other projects.

Change-Id: I3eb2a7e24df68dec0ca638307ecf5a476e96f06a
Closes-Bug: #1649759
4 years ago
Zuul 378bf9dc64 Merge "Fix purging events from stacks converted to convergence" 5 years ago
Zuul 53e263d9d9 Merge "Fix an exception message" 5 years ago
Zane Bitter d12512ab17 Fix purging events from stacks converted to convergence
Stacks with convergence enabled are guaranteed not to have a backup stack,
but they may have previously had backup stacks (prior to being converted to
convergence) that remain only soft-deleted. In this case they may still
fail the foreign key constraints in the database when purging events, so
don't bail out of the workaround path when we encounter this situation.

This will allow events to be purged from these stacks without having to
wait for purge_deleted to purge the backup stacks first.

Change-Id: If541d421a0a2af17ab937ec0a27bee6e4e9c1f6b
Story: #2002643
Task: 22648
5 years ago
Zane Bitter a2c6f14c63 Fix an exception message
The message contained "<built-in function id>" instead of the row ID.

Change-Id: Ifd345a3098342c8ae5b7b64054f6a130dad2a03c
5 years ago
Zane Bitter 9526be6a32 Rename poorly named function
_query_all_by_stack() doesn't tell you what the query it returns will
query for. It's events.

Change-Id: If4b05a375f853873e90803be956c5e6b70470463
5 years ago
Zane Bitter 9709f71a86 Purge events that cross-reference with backup stack
In the legacy path, we have resource properties data rows that can be
referenced by events and resources both in the main stack and the backup
stack. In future we will stop doing this. However, to allow purging of
pre-existing events, fall back to excluding resource properties data that
is referenced by the other stack in the pair.

This is somewhat slower than the happy path, and only affects legacy
stacks, so only invoke it when we encounter an error.

Change-Id: Id0a4409d53d59ff9939b64ae14ac6c4360fb9299
Story: #2002643
Task: 22511
5 years ago
Zane Bitter 6169ed4660 Ignore errors in purging events
There is a known issue with purging the resource properties data for events
where the same resource properties data is referenced by a resource or
event in the backup stack.

It's more important that we are able to store a new event than that we are
able to purge old events, so catch any exceptions in purging and log an
error, rather than allowing the exception to abort the event creation.

Also use a transaction to ensure that we roll back the deletion of the
events when we encounter an error. This ensures that any resource
properties data *not* referenced by the backup stack is not orphaned
because of failures deleting other rows. This will allow us to properly
purge the database once the issue is fixed.

Change-Id: Iefa706f91382f7d1c47e09e8f67a9fad53a9390b
Story: #2002643
Task: 22334
5 years ago
Zuul 77f812daa4 Merge "Don't allow nested or stacks in FAILED state to be migrated" 5 years ago
Zuul f12426f777 Merge "Retry on deadlock in purge_deleted" 5 years ago
rabi abfec544d7 Don't allow nested or stacks in FAILED state to be migrated
Stack and its nested stacks should be in *_COMPLETE state for
it to be migrated. Also, we should not allow migration of a
nested stack.

This also changes stack_get_all_by_owner_id() to not select the
backup stacks. This seems to be used only in convergence code
path and migrate_convergence_1().

Change-Id: Icd54465d0c593557a12d853ddee4ee8ce6483499
Closes-Bug: #1767962
Story: #1767962
Task: #17363
5 years ago
ricolin 566dd9301b Add retry for resource_purge_deleted call
Retry when resource_purge_deleted failed with deadlock or other exceptions.

Change-Id: Iadab7b45e653fc6f465e257d3f450c4eaf509297
Story: #2001997
Task: #19629
5 years ago
Zane Bitter c06a0e469b Retry on deadlock in purge_deleted
Change-Id: I6dc7429b2564ab07dd66bf0cf0c8657ac1928508
Closes-Bug: #1721695
5 years ago
Zuul ab4c77bcc7 Merge "Remove db api and tables for CloudWatch" 5 years ago
Zuul df12b86b80 Merge "Retry on DB deadlock when updating resource" 5 years ago
rabi 9b6bcaf50d Remove db api and tables for CloudWatch
Change-Id: Ic728a994865a923b7cc56d9c6b5df0c0d38626e1
Closes-Bug: #1743707
5 years ago
Zuul 8f6c09df0f Merge "Speed up event listing with nested stacks" 6 years ago
Zuul 10179a88ec Merge "Eager-load properties for nested resource list" 6 years ago
Thomas Herve e1a26279ce Fix reset_stack_status
Fix reset_stack_status on legacy, by using a different session for each
stack reset, and handling lock duplicate errors.

Closes-Bug: #1735755
Change-Id: I6bcd7448052e86ec3e4eb4c49ef3139c20d4f919
6 years ago
Zane Bitter 23e06f66b9 Speed up event listing with nested stacks
When handling the command "openstack stack event list --nested-depth=n", we
obtain the list of nested stacks by querying the database for all resources
that share the root stack ID. However, since all we're getting is the stack
IDs, there's no need to query all of the fields and construct a versioned
object for each resource. Just get the set of stack IDs.

Change-Id: I12155433e2ac1af919aa4b5e780fb965cd5885d8
Related-Bug: #1588561
6 years ago
Zane Bitter dbc852b47a Eager-load properties for nested resource list
When doing "openstack stack resource list --nested-depth=n", we were
lazy-loading the resources' properties data. This is expensive, especially
when there are a large number of resources. Eager-load the data, as we
always use it to show the resources.

For consistency, always eager-load the resource 'data' (even in
resource_get), because the Resource versioned object accesses it

Change-Id: Idb871fddf77bf24828878c315e19e200c28841be
Related-Bug: #1665503
6 years ago
Zane Bitter e919b723bc Retry on DB deadlock when updating resource
Updating a row in the DB using UPDATE...WHERE... seems to be prone to
deadlocks. We already fixed added retries for updates of the Stack table
(bug 1578615); do the same with the Resource table.

Change-Id: I3928f1acc3c354711a27523215784a8cf8767581
Closes-Bug: #1732969
6 years ago
liyi 51aed6e41f Fix logic error in sqlarchemy api
Change-Id: Ic3074969b8a75f7b38aa3ba9a37b23dcf133aa8e
6 years ago
Zane Bitter c9792b96d2 Create replacement resource atomically
Use a single transaction to create the replacement resource and set it as
the replaced_by link in the old resource. Also, ensure that no other
traversal has taken a lock on the old resource before we modify it.

If we end up bailing out and not creating a replacement or sending an RPC
message to check it, make sure we retrigger any new traversal.

Change-Id: I23db4f06a4060f3d26a78f7b26700de426f355e3
Closes-Bug: #1727128
6 years ago
huangtianhua a532535b61 Correct the filter attribute of db model 'ResourcePropertiesData'
Make sure to delete the ResourcePropertiesData row
if the resource 'attr_data_id' has been set already.

Change-Id: I2df5dfc72d051b6619c769c1260ff24b016bc294
Closes-Bug: #1708378
6 years ago
Jenkins 0a9fcd9136 Merge "Eager load resource_properties_data in resource" 6 years ago
ricolin d0b7cd175e Adds 5 backport db migration place holder for Pike
Db migration version from 81 to 85 are made as place
holder of db backporting for stable/pike.

Change-Id: Iddd91173f4fa77f9ef8cf2a60f629d551f6dcdd5
6 years ago
Crag Wolfe 705625cee7 Eager load resource_properties_data in resource
Eager load resource_properties_data in resources in the typical
resource-loading scenarios where properties data will be
accessed. Thus, we can save an extra db query per resource when
loading all the resources in a stack, for instance. Fall back to lazy
loading properties data in other scenarios.

Also, the resource object doesn't need to store a copy of its
ResourcePropertiesData object in self.rsrc_prop_data, so don't.

Change-Id: Ib7684af3fe06f818628fd21f1216de5047872948
Closes-Bug: #1665503
6 years ago
Zane Bitter 118fc127b3 Fix a DB error when updating the resource attributes
This prevents the attributes from being stored sometimes, and is possibly
correlated with occasional random failures of the gate jobs.

Change-Id: I8f6567ec792bd015d59ffc04da9f2f37f5b0f13a
Related-Bug: #1660831
6 years ago
Crag Wolfe 0550659f5f Store resource attributes in the DB
Store resource attributes that may be cached in the DB, saving the
cost of re-resolving them later. This works for most resources,
specifically those that do not override the get_attribute() method.

Change-Id: I71f8aa431a60457326167b8c82adc03ca750eda6
Partial-Bug: #1660831
6 years ago
Jenkins df908f4417 Merge "Don't fetch stack before update in stack_update()" 6 years ago
Jenkins 3af92c84a7 Merge "Load "lighter" db resources when appropriate" 6 years ago
Crag Wolfe 450e3b2c09 Don't fetch stack before update in stack_update()
DB optimization where we no longer needlessly fetch a stack (and
its raw_template) before updating it.

Change-Id: I1ff5212f580e79838a8273065a94cb077e5aa2ff
Closes-Bug: #1672753
6 years ago
Crag Wolfe a6f4c6d2c6 Low-level db delete of events should be synchronous
Previously, synchronized_session=False was used in the call to prune
events. This was overly aggressive since it was possible (if rare)
that, during next db operation to delete resource_properties_data
rows, some of the referenced events could still have existed resulting
in a db referential integrity error.

Change-Id: I5c4cf6a162ff853f84d68e7b203ffa1aae684359
Closes-Bug: #1681772
6 years ago
Crag Wolfe 93fab308e8 Load "lighter" db resources when appropriate
Sometimes we know we will only access particular fields of a resource
object, rather than *all* of them. This commit allows the caller to
specify (optionally) the fields that should be populated when the
resource object is instantiated. This saves memory, trips to the db,
and in some cases avoids extra join queries (e.g. for or

Change-Id: I405888f46451d2657aa28f610f8ca555215ff5cf
Partial-Bug: #1680658
6 years ago
liyi 8f10215ffd Remove log translations
Log messages are no longer being translated. This removes all use of
the _LE, _LI, and _LW translation markers to simplify logging and to
avoid confusion with new contributions.


Change-Id: Ieec8028305099422e1b0f8fc84bc90c9ca6c694f
6 years ago
Jenkins 7a51c9c657 Merge "Squash database patches" 6 years ago
Jenkins 4dc101f10e Merge "Eager load event's resource prop. data when appropiate" 6 years ago
ricolin cb4640d67d Squash database patches
Remove patches up to the Mitaka release to speed up
initial deployment and tests, as we don't support
previous versions anymore.
Closes-bug: #1672630

Change-Id: I4f821176cb11ca3dc3af247238ba3c4fbc20a092
6 years ago
Jenkins cfbc624b75 Merge "Don't eager load raw_template in a few places" 6 years ago
Jenkins ab267e763a Merge "Remove unused event_get methods" 6 years ago
Crag Wolfe 283b2d3985 Eager load event's resource prop. data when appropiate
Eager load an event's resource properties data in the list_events
"show event" case, i.e. when an event uuid is specified. When showing
multiple events, the events' resource properties data should not be
eager loaded.

Change-Id: I12e371dcbdae398360e28200e68f4a75831f8f17
Closes-bug: #1665506
6 years ago
Crag Wolfe 019ae84d4f Remove unused event_get methods
Remove unused event get methods from the sqlalchemy db api and event

Change-Id: Ide5780d456755e180572194ad4d06277749c8c79
6 years ago
Crag Wolfe 02e7f70619 Don't eager load raw_template in a few places
Save the db and memory overhead associated with fetching a
raw_template when soft-deleting a stack or getting a root stack id.

Change-Id: I8cf575e4908764c4c5eb24185ec229254e30d58b
6 years ago
Crag Wolfe 9ebbd1e6f1 Data migration enabling storage of resource attributes
Working towards the goal of storing resource attributes in the db so
as to avoid re-resolving them when appropriate. Adds an 'attr_data'
object to the resource object, defined as a relationship on the
already existing resource_properties_data table.

Change-Id: I2104078d850da08b22547d7feab2bde00c543478
Partial-Bug: #1660831
6 years ago
Crag Wolfe 98505288b2 Add heat-manage subcommand to migrate legacy prop. data
Add a subcommand to heat-manage to migrate resource and events
properties data from the legacy db locations to the new. I.e., migrate
properties data out of the legacy columns in the resource and event
tables into the recently added resource_properties_data table. No
attempt at de-duplication between resources and events is made for the
migration: a new row is created in resource_properties_data for every
row that has legacy properties data in the resource or event tables.

Change-Id: I364d509c357539d1929eb2e40704e60049469ea2
6 years ago
Crag Wolfe 57c5aae88b De-duplicate properties_data between events, resources
Properties data (encrypted or not) is now stored in the
resource_properties_data table. We still support reading properties
data from the legacy locations (separate columns in the resource and
event tables) but all new properties data are written to

Instead of duplicating properties data for a given resource
and its events, we refer to the same resource_properties_data
object from both resources and events.

Notes about encrypting/decrypting properties data:

1. ResourcePropertiesData takes care of encrypting/decrypting, so
   consumers (events and resources) don't have to worry about it.
2. Previously (not anymore), heat.engine.resource would take care of
   the encrypting in _store() and _store_or_update().
3. Previously and currently, heat.object.resource decrypts the legacy
   properties_data field if needed.

Change-Id: I569e16d4a7d3f5ccc22d57fe881c4fd5994677ac
Closes-Bug: #1524013
6 years ago
Jenkins 2fa8cc09d3 Merge "Allow admins to get resource by physical resource id" 6 years ago