The db.api module provides a useless indirection to the only
implementation we ever had, sqlalchemy. Let's use that directly instead
of the wrapper.
Change-Id: I80353cfed801b95571523515fd3228eae45c96ae
When partially removing eager loading of raw_template in stacks
(260b79ed28), we didn't get into account
that accessing the sqlalchemy field would create an additional query
whereas it was previously eager loaded. This removes it by only filling
the field when it's already fetched.
Change-Id: Ifa17c74e3559adaef56593a205101d92e9a37da5
Closes-Bug: #1634127
Always loading the raw template in situations where we didn't need it -
e.g. in identify_stack, where we just want the name + id (given one of
them), or when getting the summary stack list - uses up DB bandwidth and
memory unnecessarily.
This partially reverts commit 3ab0ede98c.
* The eager_load option to get_stack() is reinstated, but with the default
flipped to True. In places where we explicitly do not want to load the
template, we pass False.
* stack_get_by_name() no longer eagerly loads the template. There were no
instances of this where we subsequently use the template.
* stack_get_all() acquires an eager_load option, with the default set to
False. Direct users of objects.stack.Stack.get_all() will not eagerly
load by default, but users of engine.stack.Stack.load_all() will get the
template eagerly loaded. This practically always corresponds to what you
want.
Change-Id: I1f156c25ea26322be5b606a61dd77d80cadd65fc
Related-Bug: #1626675
Run `heat-manage migrate-convergence-1 <stack_id>` to migrate
legacy stack to convergence engine.
Heat engine is used for doing migration i.e. migration can't
be done offline.
Change-Id: Ie7c2498b37937438f16d154b154b3a6ecbf9ff74
Implements-bp: convergence-migrate-stack
We can use admin_context to have access to stacks
and software configs across projects. This removes
the tenant_safe flag from rpc and db api. This is
backward compatible with older rpc clients.
We still support use of global_tenant flag for listing
stacks and software configs. However, by default
an admin(user with admin role in admin_project)
would not need that.
Change-Id: I12303dbf30fb80290f95baba0c67cdf684f5f409
The vast majority of stack fetches are immediately followed by a
raw_template fetch, so this change always eagerly fetches the
raw_template for every stack fetch.
During stack versioned object creation the stack's raw_template object
is used to construct the versioned raw template object.
Change-Id: I1a6fb8fb7d069b50dd5d623c989acd5582818ae1
Related-Bug: #1479723
Related-Bug: #1523748
In Python 3 __ne__ by default delegates to __eq__ and inverts the
result, but in Python 2 they urge you to define __ne__ when you
define __eq__ for it to work properly [1].There are no implied
relationships among the comparison operators. The truth of x==y
does not imply that x!=y is false. Accordingly, when defining
__eq__(), one should also define __ne__() so that the operators
will behave as expected.
[1]https://docs.python.org/2/reference/datamodel.html#object.__ne__
Change-Id: I3ae84db585355fef864a8beea4ad388ed09e6ff7
Having such a long positional args list which is then passed through
with *args is error prone and hard to maintain.
Change-Id: I525b00c113099f2f8e4648317cfcbd3cc05398c4
Tags for a stack need to be loaded in the following scenarios:
- To display during a stack show
- To store during a create or update
All other operations which require a loaded stack will likely not
reference the tags, so the extra SQL query which always loads them is
often not necessary.
This change removes the tags loading from stack_object.Stack and
stack.Stack.load and implements annotated properties functions for a
Stack.tags property so that they are loaded from the database on
command.
Change-Id: Iecbf4f101b7f264a42b57f720a8428b0aed706e2
Partial-Bug: #1578851
Currently calls to get a collection of stacks via the stack object
have the following data access behaviour:
- One query to fetch the stack records
- One query per stack to fetch the raw template
- One query per stack to fetch the tags
This causes excessive database round trips when there are many stacks.
In addition, the list_stacks call results in a collection of full
stack objects to be created which builds a fully parsed stack - most
of this information is then discarded by the RPC and middleware
formatters so this overhead is for no benefit.
This change is the first step in changing the data access patterns by
querying the Stack versioned object directly instead of via the full
Stack object. A future change will apply an eager fetch and caching
approach to avoid unnecessary queries.
This change does the following:
- Service list_stacks calls stack_object.Stack.get_all directly
- Service list_stacks formats with new function
api.format_stack_db_object with the exact fields required by the CFN
and REST API list stacks formatters
- Since the description field is the only one that requires full stack
parsing, it is now set to an empty string for stack listings via API
or calls to aws cloudformation list-stacks [1]
This last point may be controversial, my attempts to find uses of the
stack description in stack listings found none that do, and the following
API uses which *don't* show the description:
- heat stack-list
- openstack stack list
- horizon stack list
- rackspace control panel stack list
If we want to add the description back later we could always add a
description field back to the Stack table to store the denormalised
description.
[1] http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-describing-stacks.html
Partial-Bug: #1578851
Change-Id: I88b6f705e87ee7ff4acb7830dcddfc188ae6b3b2
Currently, StackResource loads the whole stack when checking for status
(in check_*_complete method), but only care about the state of the
stack. This is a fairly expensive operation, as it retrieves the
template and reparses everything. This simplifies it with a new API that
simply query the stack status from the database.
Closes-Bug: #1549213
Change-Id: I18df89a2b9959241ddbec2593a53c5e2aa6a4717
Retrieving the field from the db object triggers the query to
stack_tags, so we don't need to do it a second time to retrieve the list
of tags. Instead, use the data directly.
Change-Id: Ibe6ac45d54afca6588b1b71ffa7278977d2f4d8a
When refreshing a Stack object, we first retrive the object from the
database (which handle NotFound errors) and then call refresh in a
different transaction. By that time, the db object can be gone and
refresh would fail, and it also doesn't bring any benefit, so let's
remove that call.
Change-Id: I63af65d2a409ddf4179672ebb054852a8732a1fe
Closes-Bug: #1543922
As we don't have transactions and we're loading the templates
seperately, they can be gone by the time we load the stack. Let's ignore
NotFound errors in that case.
Change-Id: I048b4fb50512478a514f245b618784f1d7eb0c1d
Closes-Bug: #1537627
If we get passed a non-string stack name, e.g a map or list, we
fail with a DB error associated with looking up the existing stack.
So instead force all stack lookups to use string identifiers, and
make the name validation for new stacks robust to fail gracefully
when there is an invalid (non string) argument passed.
Change-Id: I052dc4a715773895d070e1e9f26183c6a1cf3d7f
Closes-Bug: #1533065
Fix raw_template purge query on MySQL, and handle stack tags before
removing stacks. This also removes a bunch of race conditions where we
deleted incorrect data.
Change-Id: I7b7a1d94acefbaeeed86f1833c979819361c8988
Closes-Bug: #1524387
To avoid certain concurrency related issues, the DB update API needs to
be given the traversal ID of the stack intended to be updated. By making
this change, we can void having following at all the places:
if current_traversal != stack.current_traversal:
return
The check for current traversal should be implicit, as a part of stack's
store and state_set methods, where self.current_traversal should be used
as expected traversal to be updated. All the state changes or updates in
DB to the stack object go through this implicit check (using
update...where).
When stack updates are triggered, the current traversal should be backed
up as previous traversal, a new traversal should be generated and the
stack should be stored in DB with expected traversal as the previous
traversal. This will ensure that no two updates can simultaneously
succeed on same stack with same traversal ID. This was one of our
primary goal.
Following example cases describe the issues we encounter:
1. When 2 updates, U1 and U2 try to update a stack concurrently:
1. Current traversal(CT) is X
2. U1 loads stack with CT=X
3. U2 loads stack with CT=X
4. U2 stores the stack and updates CT=Y
5. U1 stores the stack and updates the CT=Z
Both the updates have succeeded, and both would be running until
one of the workers does stack.current_traversal == current_traversal
and bail out.
Ideally, U1 should have failed: only one should be allowed in case
of concurrent update. When both U1 and U2 pass X as the expected
traversal ID of the stack, then this problem is solved.
2. A resource R is being provisioned for stack with current traversal
CT=X:
1. An new update U is issued, it loads the stack with CT=X.
2. Resource R fails and loads the stack with CT=X to mark it as FAILED.
3. Update U updates the stack with CT=Y and goes ahead with sync_point
etc., marks stack as UPDATE_IN_PROGRESS
4. Resource marks the stack as UPDATE_FAILED, which to user means that
update U has failed, but it actually is going on.
With this patch, when Resource R fails, it will supply CT=X as
expected traversal to be updated and will eventually fail because
update U with CT=Y has taken over.
Partial-Bug: #1512343
Change-Id: I6ca11bed1f353786bb05fec62c89708d98159050
We uselessly refresh the stack object when we save it, as we don't use
the result anywhere. Let's remove that useless query.
Change-Id: Idbc737a0d88aa98e69979f6edd88b8b148030388
When StackResource 'update_with_template' is called and there
is no existing nested_stack, we call create_stack with an empty
template(TemplateResource CREATE_FAILED->UPDATE flow). We check
db for CREATE_COMPLETE before calling update_stack.There is a
possibility that the create_stack green thread has not finished
and released the stack lock. By persisting the stack state for
COMPLETE/FAILED in the same db session as releasing the lock,
when green thread finishes it's run, we can avoid this race
condition.
Change-Id: I1442a7bcedb4fa340ee4f0e6ebc3705544c2a9dd
Closes-Bug: #1498495
The stack update API needs to know the expected traversal ID of the
stack (each update is associated with a unique traversal ID) it is
updating.
When two updates are triggered simultaneously - for example, in a case
where two resources at same level try to rollback the stack when they
fail at same time - then the updates need to give the traversal ID they
intend to update. If it happens that the resource A triggers the
rollback immediately after the rollback of A has started, but before A
has persisted in DB, then the system is in erroneous state.
Steps to describe the problem:
1. Resource A and B of a Stack S fail.
2. Resource A loads the stack.
3. Resource B loads the stack.
4. Resource A triggers rollback - it takes the previous template as
current template, updates the DB.
5. Resource B now again triggers the rollback. It is not aware of update
triggered by resource A.
If the updates triggered by A and B give the stack traversal they
expect to be updated, then update from resource B will fail, since the
expected traversal ID will not be found in DB as a result of resource A
already updating it.
Change-Id: I13b2f5afe8cadf9a2d9ab4139a14718a5481dec4
1. don't return DB models from the objects API
(only return Objects)
2. delete shouldn't return anything
3. update_and_save should return the refreshed object.
Note: there is still some inconsistency in what is returned by
update_by_id() some return an object and some return a bool.
Related-bug: #1432936
Change-Id: I1a0a38773d4fc4a62af5e0a98076396f39187b6c
The count is performed in 2 parts
- A function which builds a list of all nested stacks
by recursively calling stack_get_all_by_owner_id
- A query which counts resources which belong to the list
of stacks
Considering this will be the basis for fixing bug #1455589 then this
approach is appropriate for backporting to stable/kilo, but master would
ideally replace this soon with a new Stack attribute that stores the
calculated resource count.
Partial-Bug: #1455589
Change-Id: Ifa2e5609fd9a6853e20037e94a5e16696bb378ee
This change adds a (currently unused) database function
stack_get_root_id to find the root stack ID for any stack.
Scheduler hints are moved to using Stack.root_stack_id() in this change.
Remaining uses of Stack.root_stack() will switch to using
Stack.root_stack_id() later in the series.
Change-Id: I9914b0df5af119edea6346db0d7c62124fbb1313
Partial-Bug: #1455589
1. Add a configuration option to enable/disable template parameters
encryption.
2. Encrypt hidden parameters before storing them in the database and
decrypt on stack load.
Change-Id: Ie46c6a149f414f655600616da8deee463e55671c
Implements: blueprint encrypt-hidden-parameters
Co-Authored-By: Jason Dunsmore <jasondunsmore@gmail.com>
We are persisting for a number of reasons:
- so we don't have to pass this through ever rpc call
- the API exposes parent_resource (currently always None as
it is not persisted)
Closes-bug: #1438978
Change-Id: Id2db36c0234a085ec4f0ce2ab114ec483ea29d81
The current Stack object refresh doesn't actually do any refreshing,
which explains the functional test speed regression since the RPC
nested stack code relies on stack.refresh() to poll for state change.
With this change AutoScalingSignalTest.test_signal_with_policy_update
locally takes ~60s consistently. Without this change ~50% of test runs
have some form of execution delay, taking 80s -> 300s+.
Change-Id: Ie06851fd5b0d8b802ebf701e8bf621562d9b6d88
Closes-Bug: #1433555
Adds support for versioning object support for StackTag
and add it to the Stack object
Closes-bug: #1432477
Implements blueprint versioned-object
Change-Id: I7cc39c1e1ce913c35862a52592e55b9f484c7e68
Few object methods misses _from_db_object. To enable it to make
schema valid, we need to fix that.
Change-Id: I2e5ff22cd40be56ebc5f3a7095887be773ddb809
Closes-Bug: #1431353
Implementation of oslo.versionedobjects.
This commit consists basic mechanism and first objects.
This should be base of implementation versoning to other objects
Implements: blueprint versioned-objects
Co-Authored-By: ShaoHe Feng <shaohe.feng@intel.com>
Co-Authored-By: Grzegorz Grasza <grzegorz.grasza@intel.com>
Change-Id: I554162cf3681fe559c75f54c61c6f32c91f5c2f8
Thomas Herve