Stack and its nested stacks should be in *_COMPLETE state for
it to be migrated. Also, we should not allow migration of a
nested stack.
This also changes stack_get_all_by_owner_id() to not select the
backup stacks. This seems to be used only in convergence code
path and migrate_convergence_1().
Change-Id: Icd54465d0c593557a12d853ddee4ee8ce6483499
Closes-Bug: #1767962
Story: #1767962
Task: #17363
Retry when resource_purge_deleted failed with deadlock or other exceptions.
Change-Id: Iadab7b45e653fc6f465e257d3f450c4eaf509297
Story: #2001997
Task: #19629
Fix reset_stack_status on legacy, by using a different session for each
stack reset, and handling lock duplicate errors.
Closes-Bug: #1735755
Change-Id: I6bcd7448052e86ec3e4eb4c49ef3139c20d4f919
When handling the command "openstack stack event list --nested-depth=n", we
obtain the list of nested stacks by querying the database for all resources
that share the root stack ID. However, since all we're getting is the stack
IDs, there's no need to query all of the fields and construct a versioned
object for each resource. Just get the set of stack IDs.
Change-Id: I12155433e2ac1af919aa4b5e780fb965cd5885d8
Related-Bug: #1588561
When doing "openstack stack resource list --nested-depth=n", we were
lazy-loading the resources' properties data. This is expensive, especially
when there are a large number of resources. Eager-load the data, as we
always use it to show the resources.
For consistency, always eager-load the resource 'data' (even in
resource_get), because the Resource versioned object accesses it
unconditionally.
Change-Id: Idb871fddf77bf24828878c315e19e200c28841be
Related-Bug: #1665503
Updating a row in the DB using UPDATE...WHERE... seems to be prone to
deadlocks. We already fixed added retries for updates of the Stack table
(bug 1578615); do the same with the Resource table.
Change-Id: I3928f1acc3c354711a27523215784a8cf8767581
Closes-Bug: #1732969
Use a single transaction to create the replacement resource and set it as
the replaced_by link in the old resource. Also, ensure that no other
traversal has taken a lock on the old resource before we modify it.
If we end up bailing out and not creating a replacement or sending an RPC
message to check it, make sure we retrigger any new traversal.
Change-Id: I23db4f06a4060f3d26a78f7b26700de426f355e3
Closes-Bug: #1727128
Make sure to delete the ResourcePropertiesData row
if the resource 'attr_data_id' has been set already.
Change-Id: I2df5dfc72d051b6619c769c1260ff24b016bc294
Closes-Bug: #1708378
Eager load resource_properties_data in resources in the typical
resource-loading scenarios where properties data will be
accessed. Thus, we can save an extra db query per resource when
loading all the resources in a stack, for instance. Fall back to lazy
loading properties data in other scenarios.
Also, the resource object doesn't need to store a copy of its
ResourcePropertiesData object in self.rsrc_prop_data, so don't.
Change-Id: Ib7684af3fe06f818628fd21f1216de5047872948
Closes-Bug: #1665503
This prevents the attributes from being stored sometimes, and is possibly
correlated with occasional random failures of the gate jobs.
Change-Id: I8f6567ec792bd015d59ffc04da9f2f37f5b0f13a
Related-Bug: #1660831
Store resource attributes that may be cached in the DB, saving the
cost of re-resolving them later. This works for most resources,
specifically those that do not override the get_attribute() method.
Change-Id: I71f8aa431a60457326167b8c82adc03ca750eda6
Partial-Bug: #1660831
DB optimization where we no longer needlessly fetch a stack (and
its raw_template) before updating it.
Change-Id: I1ff5212f580e79838a8273065a94cb077e5aa2ff
Closes-Bug: #1672753
Previously, synchronized_session=False was used in the call to prune
events. This was overly aggressive since it was possible (if rare)
that, during next db operation to delete resource_properties_data
rows, some of the referenced events could still have existed resulting
in a db referential integrity error.
Change-Id: I5c4cf6a162ff853f84d68e7b203ffa1aae684359
Closes-Bug: #1681772
Sometimes we know we will only access particular fields of a resource
object, rather than *all* of them. This commit allows the caller to
specify (optionally) the fields that should be populated when the
resource object is instantiated. This saves memory, trips to the db,
and in some cases avoids extra join queries (e.g. for resource.data or
resource.rsrc_prop_data).
Change-Id: I405888f46451d2657aa28f610f8ca555215ff5cf
Partial-Bug: #1680658
Eager load an event's resource properties data in the list_events
"show event" case, i.e. when an event uuid is specified. When showing
multiple events, the events' resource properties data should not be
eager loaded.
Change-Id: I12e371dcbdae398360e28200e68f4a75831f8f17
Closes-bug: #1665506
Save the db and memory overhead associated with fetching a
raw_template when soft-deleting a stack or getting a root stack id.
Change-Id: I8cf575e4908764c4c5eb24185ec229254e30d58b
Add a subcommand to heat-manage to migrate resource and events
properties data from the legacy db locations to the new. I.e., migrate
properties data out of the legacy columns in the resource and event
tables into the recently added resource_properties_data table. No
attempt at de-duplication between resources and events is made for the
migration: a new row is created in resource_properties_data for every
row that has legacy properties data in the resource or event tables.
Change-Id: I364d509c357539d1929eb2e40704e60049469ea2
Properties data (encrypted or not) is now stored in the
resource_properties_data table. We still support reading properties
data from the legacy locations (separate columns in the resource and
event tables) but all new properties data are written to
resource_properties_data.
Instead of duplicating properties data for a given resource
and its events, we refer to the same resource_properties_data
object from both resources and events.
Notes about encrypting/decrypting properties data:
1. ResourcePropertiesData takes care of encrypting/decrypting, so
consumers (events and resources) don't have to worry about it.
2. Previously (not anymore), heat.engine.resource would take care of
the encrypting in _store() and _store_or_update().
3. Previously and currently, heat.object.resource decrypts the legacy
properties_data field if needed.
Change-Id: I569e16d4a7d3f5ccc22d57fe881c4fd5994677ac
Closes-Bug: #1524013
Initialize osprofiler in sqlalchemy/api.py failed with message:
'NoneType' object has no attribute 'get_engine'
Re-oder to init osprofiler after _facade will fix this issue.
Change-Id: I420f12f382c4712a692dfa440958b2e497dc22a1
Closes-Bug: #1657326
This change allows admin users to get resources from
other projects by physical resource id.
Partial-Bug: #1649759
Change-Id: I6e58ae8fd339bc50d41a5027cf610eb63dbd220c
Prepare for the future when both resources and events refer to
ResourcePropertiesData rather than store the data themselves.
Change-Id: I0c5aefa9d73e1281e6477d46fe181d8948aee71b
Previously, there was one (potentially quite large) transaction across
all batches of templates and resources when encrypting/decrypting
parameters and properties data via the rpc api.
Change-Id: Iba10b7c9e88a1ae3d94ab9ddf4a8e362e72c0d50
The db.api module provides a useless indirection to the only
implementation we ever had, sqlalchemy. Let's use that directly instead
of the wrapper.
Change-Id: I80353cfed801b95571523515fd3228eae45c96ae
It's possible that we could end up with multiple resources with the same
physical resource ID, but that would be undetectable since we return only
one from the database layer. This change allows us to detect the problem an
return an error where the result is rendered ambiguous.
Change-Id: I2c5ddbe6731c33a09ec7c4a7b91dcfe414da4385
Testing with devstack/mysql, given a stack with 350 resources,
stack_count_total_resources() now takes ~2.2ms vs ~7.6ms the old way.
Sqlalchemy was doing a bit more than we needed before, e.g.:
2016-12-03 16:41:58.455 2294 DEBUG sqlalchemy.orm.path_registry [req-b7cdfd64-4306-4145-adb2-dae13caaa9ab - - - - -] set 'memoized_setups' on path 'EntityRegistry((<Mapper at 0x44ddbd0; Resource>,))' to '{}' set /usr/lib64/python2.7/site-packages/sqlalchemy/orm/path_registry.py:63
Change-Id: I7def4093c7719a95e72d49925559bc926f168a01
We were a bit overzealous in our enforcement of
max_events_per_stack. Rather than count the number of events within a
stack every time an event is created, only do the count a small
percentage of the time (dependent on the inverse of
event_purge_batch_size). This also means that max_events_per_stack is
no longer a strict limit but a loose one.
Change-Id: I63b42a50cfc3ffa6b54413900b5e0a6c5c40943c
Avoid large sql "in" clauses by operating on smaller batches of stacks
at a time.
To avoid transaction overhead and contention on the resource table,
the first deletions occur outside of a transaction (are
autocommitted). This is OK because the purge is re-rentrant -- we
won't lose any stack_id's to delete if something goes wrong before the
conn.begin() block. That is, we will not orphan any rows if the purge
is run multiple times where an error occurs.
Change-Id: I9edf0558ed54820842193560e323df6501411d1d
Avoid a potentially large select in clause. synchronize_session may be
False here, as we don't have to worry about old stale pruned events in
the session (keeping in mind the prune gets triggered when new events
are created).
Change-Id: I27a00a1afb6518176c031551d6a74e8b12546ef2
Always loading the raw template in situations where we didn't need it -
e.g. in identify_stack, where we just want the name + id (given one of
them), or when getting the summary stack list - uses up DB bandwidth and
memory unnecessarily.
This partially reverts commit 3ab0ede98c.
* The eager_load option to get_stack() is reinstated, but with the default
flipped to True. In places where we explicitly do not want to load the
template, we pass False.
* stack_get_by_name() no longer eagerly loads the template. There were no
instances of this where we subsequently use the template.
* stack_get_all() acquires an eager_load option, with the default set to
False. Direct users of objects.stack.Stack.get_all() will not eagerly
load by default, but users of engine.stack.Stack.load_all() will get the
template eagerly loaded. This practically always corresponds to what you
want.
Change-Id: I1f156c25ea26322be5b606a61dd77d80cadd65fc
Related-Bug: #1626675
An exception could occur at any time, so we want to make sure we roll
back any uncompleted transaction when that happens.
Change-Id: Id0d5b65f44981bde24840b8a0d5fdc0b88f24d8a
Related-bug: #1546431
The heat cmd "heat-manage reset_stack_status {stack_id}" does
not work, because we don't release resource lock, so this change
to release lock to make sure it works.
Change-Id: Ic42ad40e279c4e40da95d9aee35125767e747124
Closes-Bug: #1622533
Implements mechanism to cancel existing workers (in_progress resources).
The stack-cancel-update request lands in one of the engines, and if
there are any workers in that engine which are working for the stack,
they are cancelled first and then other engines are requested to cancel
the workers.
Change-Id: I464c4fdb760247d436473af49448f7797dc0130d
This change takes a gradual approach to fixing bug #1479723 by
creating an enginefacade transaction_context, but mostly still using
the legacy facade for getting the default session.
Where a separate independent transaction is needed,
db_context.writer.independent.using is used instead of get_session.
The next change will decorate the db api functions with reader and
writer so queries in a function are performed in a single transaction.
Change-Id: If92788641d84b7314ed7ad1f8cb0e141fa1a8681
Partial-Bug: #1479723
Run `heat-manage migrate-convergence-1 <stack_id>` to migrate
legacy stack to convergence engine.
Heat engine is used for doing migration i.e. migration can't
be done offline.
Change-Id: Ie7c2498b37937438f16d154b154b3a6ecbf9ff74
Implements-bp: convergence-migrate-stack
We can use admin_context to have access to stacks
and software configs across projects. This removes
the tenant_safe flag from rpc and db api. This is
backward compatible with older rpc clients.
We still support use of global_tenant flag for listing
stacks and software configs. However, by default
an admin(user with admin role in admin_project)
would not need that.
Change-Id: I12303dbf30fb80290f95baba0c67cdf684f5f409
The vast majority of stack fetches are immediately followed by a
raw_template fetch, so this change always eagerly fetches the
raw_template for every stack fetch.
During stack versioned object creation the stack's raw_template object
is used to construct the versioned raw template object.
Change-Id: I1a6fb8fb7d069b50dd5d623c989acd5582818ae1
Related-Bug: #1479723
Related-Bug: #1523748
Add project-id argument to heat-manage purge_deleted command in order
to be able to hard delete DB entries for a specific project.
Change-Id: Ifffe5657a40ce97db9d059ff1516b8e1eb801132
Implements: bp heat-manage-purge-deleted-tenant
A stack with zero resources is not an exception worthy condition (for
example, a stack being deleted, a stack which really does have zero
resources).
Also its unusual that a collection query function would raise a
NotFound when the result count is zero.
This change fixes that for resource_get_all_by_stack and
resource_get_all_active_by_stack
Change-Id: I3420d5099830f5626095b17c01e9d886b477915c
Related-Bug: #1301320
Related-Bug: #1523748
- HeatBase.expire, not used anywhere
- HeatBase.save_and_update, moved to sqlalchemy.api
- SoftDelete.soft_delete moved to sqlalchemy.api
update_and_save creates a transaction, so it needs to be in
sqlalchemy.api so that a context manager can eventually manage the
transaction.
Change-Id: I84749f4fd0781ed9a2d62327b39ce6eee0f07b35
The HeatBase.delete method starts its own (sub) transaction with one
of three possible sessions. This change moves all delete calls to a
sqlalchemy.api function with the current context session. This will help with
bug #1479723 to always do deletes with a session provided by the
context manager.
Change-Id: I8dfd3bc6fdb44b0e3b06fab5d7dc8e06fa3d80a8
This change removes the refresh method from the parent db object, and
adds a refresh boolean argument to db_api.resource_get function (which is
the only known use of refresh). Constraining refresh calls to be
inside db_api methods helps with bug #1479723 where the refresh can
happen under a context manager session.
Change-Id: I6fc5c03e8572eee90f89455e0925d503514040b3
Related-Bug: #1479723
In convergence, wherein concurrent updates are possible, if a resource
is deleted (by previous traversal) after dependency graph is created
for new traversal, the resource remains in graph but wouldn't be
available in DB for processing.
It is prerequisite to have resources in DB before any action can be
taken on them.
Hence during convergence resource delete action, the resource entry
from DB is not deleted i.e soft deleted, so that the latest/new update
can find the entry.
All of these soft deleted resources will be deleted when the stack has
completed its operation.
Closes-Bug: #1528560
Change-Id: I0b36ce098022560d7fe01623ce7b66d1d5b38d55
rabi
ricolin
Zane Bitter
Thomas Herve
liyi
huangtianhua
Crag Wolfe
Tovin Seven
Anant Patil
Steve Baker
Oleksii Chuprykov
Ala Rezmerita
Rakesh H S