Commit Graph

41 Commits (cb80df4f4ab0e8de16d53c7fddff7c44dedd3f94)

Author SHA1 Message Date
Thomas Herve 84067dba88 Remove db.api wrapper
The db.api module provides a useless indirection to the only
implementation we ever had, sqlalchemy. Let's use that directly instead
of the wrapper.

Change-Id: I80353cfed801b95571523515fd3228eae45c96ae
2016-12-13 09:40:29 +01:00
Thomas Herve c59865870b Don't query raw_template in stacks uselessly
When partially removing eager loading of raw_template in stacks
(260b79ed28), we didn't get into account
that accessing the sqlalchemy field would create an additional query
whereas it was previously eager loaded. This removes it by only filling
the field when it's already fetched.

Change-Id: Ifa17c74e3559adaef56593a205101d92e9a37da5
Closes-Bug: #1634127
2016-10-18 21:37:49 +02:00
Zane Bitter 260b79ed28 Don't always eagerly load the raw_template for a stack
Always loading the raw template in situations where we didn't need it -
e.g. in identify_stack, where we just want the name + id (given one of
them), or when getting the summary stack list - uses up DB bandwidth and
memory unnecessarily.

This partially reverts commit 3ab0ede98c.

* The eager_load option to get_stack() is reinstated, but with the default
  flipped to True. In places where we explicitly do not want to load the
  template, we pass False.
* stack_get_by_name() no longer eagerly loads the template. There were no
  instances of this where we subsequently use the template.
* stack_get_all() acquires an eager_load option, with the default set to
  False. Direct users of objects.stack.Stack.get_all() will not eagerly
  load by default, but users of engine.stack.Stack.load_all() will get the
  template eagerly loaded. This practically always corresponds to what you

Change-Id: I1f156c25ea26322be5b606a61dd77d80cadd65fc
Related-Bug: #1626675
2016-10-04 08:25:01 -04:00
Oleksii Chuprykov 68944d2230 Migrate stacks from legacy to convergence engine
Run `heat-manage migrate-convergence-1 <stack_id>` to migrate
legacy stack to convergence engine.

Heat engine is used for doing migration i.e. migration can't
be done offline.

Change-Id: Ie7c2498b37937438f16d154b154b3a6ecbf9ff74
Implements-bp: convergence-migrate-stack
2016-08-22 10:40:29 +03:00
rabi 1f88c675c4 Remove tenant_safe flag from rpc and db api
We can use admin_context to have access to stacks
and software configs across projects. This removes
the tenant_safe flag from rpc and db api. This is
backward compatible with older rpc clients.

We still support use of global_tenant flag for listing
stacks and software configs. However, by default
an admin(user with admin role in admin_project)
would not need that.

Change-Id: I12303dbf30fb80290f95baba0c67cdf684f5f409
2016-08-16 12:38:33 +05:30
Steve Baker 3ab0ede98c Always eager load the raw_template for a stack
The vast majority of stack fetches are immediately followed by a
raw_template fetch, so this change always eagerly fetches the
raw_template for every stack fetch.

During stack versioned object creation the stack's raw_template object
is used to construct the versioned raw template object.

Change-Id: I1a6fb8fb7d069b50dd5d623c989acd5582818ae1
Related-Bug: #1479723
Related-Bug: #1523748
2016-08-15 22:17:30 +00:00
yuyafei 456fe5fca6 Add __ne__ built-in function
In Python 3 __ne__ by default delegates to __eq__ and inverts the
result, but in Python 2 they urge you to define __ne__ when you
define __eq__ for it to work properly [1].There are no implied
relationships among the comparison operators. The truth of x==y
does not imply that x!=y is false. Accordingly, when defining
__eq__(), one should also define __ne__() so that the operators
will behave as expected.

Change-Id: I3ae84db585355fef864a8beea4ad388ed09e6ff7
2016-07-15 10:30:52 +00:00
Steve Baker c2208f1fc6 Declare and use keyword args for Stack.get_all
Having such a long positional args list which is then passed through
with *args is error prone and hard to maintain.

Change-Id: I525b00c113099f2f8e4648317cfcbd3cc05398c4
2016-05-26 10:36:01 +12:00
Steve Baker 56b95a00d8 Lazy-load stack tags
Tags for a stack need to be loaded in the following scenarios:
- To display during a stack show
- To store during a create or update

All other operations which require a loaded stack will likely not
reference the tags, so the extra SQL query which always loads them is
often not necessary.

This change removes the tags loading from stack_object.Stack and
stack.Stack.load and implements annotated properties functions for a
Stack.tags property so that they are loaded from the database on

Change-Id: Iecbf4f101b7f264a42b57f720a8428b0aed706e2
Partial-Bug: #1578851
2016-05-25 11:59:13 +12:00
Steve Baker b3c228d707 Stack list does direct stack object query
Currently calls to get a collection of stacks via the stack object
have the following data access behaviour:
- One query to fetch the stack records
- One query per stack to fetch the raw template
- One query per stack to fetch the tags

This causes excessive database round trips when there are many stacks.
In addition, the list_stacks call results in a collection of full
stack objects to be created which builds a fully parsed stack - most
of this information is then discarded by the RPC and middleware
formatters so this overhead is for no benefit.

This change is the first step in changing the data access patterns by
querying the Stack versioned object directly instead of via the full
Stack object. A future change will apply an eager fetch and caching
approach to avoid unnecessary queries.

This change does the following:
- Service list_stacks calls stack_object.Stack.get_all directly
- Service list_stacks formats with new function
  api.format_stack_db_object with the exact fields required by the CFN
  and REST API list stacks formatters
- Since the description field is the only one that requires full stack
  parsing, it is now set to an empty string for stack listings via API
  or calls to aws cloudformation list-stacks [1]

This last point may be controversial, my attempts to find uses of the
stack description in stack listings found none that do, and the following
API uses which *don't* show the description:
- heat stack-list
- openstack stack list
- horizon stack list
- rackspace control panel stack list

If we want to add the description back later we could always add a
description field back to the Stack table to store the denormalised


Partial-Bug: #1578851
Change-Id: I88b6f705e87ee7ff4acb7830dcddfc188ae6b3b2
2016-05-25 11:58:40 +12:00
Thomas Herve b862908d4c Optimize nested stack status check
Currently, StackResource loads the whole stack when checking for status
(in check_*_complete method), but only care about the state of the
stack. This is a fairly expensive operation, as it retrieves the
template and reparses everything. This simplifies it with a new API that
simply query the stack status from the database.

Closes-Bug: #1549213
Change-Id: I18df89a2b9959241ddbec2593a53c5e2aa6a4717
2016-02-24 11:03:43 +01:00
Jenkins 0746e81102 Merge "Remove db refresh in Stack object get" 2016-02-16 00:19:35 +00:00
Thomas Herve 39b772844d Don't query stack tags twice
Retrieving the field from the db object triggers the query to
stack_tags, so we don't need to do it a second time to retrieve the list
of tags. Instead, use the data directly.

Change-Id: Ibe6ac45d54afca6588b1b71ffa7278977d2f4d8a
2016-02-15 05:53:17 +01:00
Thomas Herve 5c03782555 Remove db refresh in Stack object get
When refreshing a Stack object, we first retrive the object from the
database (which handle NotFound errors) and then call refresh in a
different transaction. By that time, the db object can be gone and
refresh would fail, and it also doesn't bring any benefit, so let's
remove that call.

Change-Id: I63af65d2a409ddf4179672ebb054852a8732a1fe
Closes-Bug: #1543922
2016-02-10 07:36:55 +01:00
Jenkins 4e28b9bfad Merge "Add OBJ_PROJECT_NAMESPACE for Heat project" 2016-02-08 21:10:05 +00:00
Thomas Herve 3f519dfe0f Handle NotFound in Stack get_all
As we don't have transactions and we're loading the templates
seperately, they can be gone by the time we load the stack. Let's ignore
NotFound errors in that case.

Change-Id: I048b4fb50512478a514f245b618784f1d7eb0c1d
Closes-Bug: #1537627
2016-01-25 08:53:37 +01:00
Grzegorz Grasza dfdbc99d8f Add OBJ_PROJECT_NAMESPACE for Heat project
This code will be used when objects are registered and serialized.

Change-Id: I9d9095bc40e61a104ebea5042f4eb75d16d27d69
Partial-Bug: 1491258
2016-01-22 21:53:51 +00:00
Jenkins 336f0f8f34 Merge "Fix database purge query" 2016-01-18 13:14:41 +00:00
Steven Hardy 1636b3b053 Handle invalid stack names which are non-string
If we get passed a non-string stack name, e.g a map or list, we
fail with a DB error associated with looking up the existing stack.

So instead force all stack lookups to use string identifiers, and
make the name validation for new stacks robust to fail gracefully
when there is an invalid (non string) argument passed.

Change-Id: I052dc4a715773895d070e1e9f26183c6a1cf3d7f
Closes-Bug: #1533065
2016-01-14 15:02:17 +00:00
Thomas Herve 6cece5bb7e Fix database purge query
Fix raw_template purge query on MySQL, and handle stack tags before
removing stacks. This also removes a bunch of race conditions where we
deleted incorrect data.

Change-Id: I7b7a1d94acefbaeeed86f1833c979819361c8988
Closes-Bug: #1524387
2015-12-15 11:58:01 +01:00
Anant Patil 634c24ecfe Convergence: Concurrency subtle issues
To avoid certain concurrency related issues, the DB update API needs to
be given the traversal ID of the stack intended to be updated. By making
this change, we can void having following at all the places:

    if current_traversal != stack.current_traversal:

The check for current traversal should be implicit, as a part of stack's
store and state_set methods, where self.current_traversal should be used
as expected traversal to be updated. All the state changes or updates in
DB to the stack object go through this implicit check (using

When stack updates are triggered, the current traversal should be backed
up as previous traversal, a new traversal should be generated and the
stack should be stored in DB with expected traversal as the previous
traversal. This will ensure that no two updates can simultaneously
succeed on same stack with same traversal ID. This was one of our
primary goal.

Following example cases describe the issues we encounter:

1. When 2 updates, U1 and U2 try to update a stack concurrently:

    1. Current traversal(CT) is X
    2. U1 loads stack with CT=X
    3. U2 loads stack with CT=X
    4. U2 stores the stack and updates CT=Y
    5. U1 stores the stack and updates the CT=Z

    Both the updates have succeeded, and both would be running until
    one of the workers does stack.current_traversal == current_traversal
    and bail out.

    Ideally, U1 should have failed: only one should be allowed in case
    of concurrent update. When both U1 and U2 pass X as the expected
    traversal ID of the stack, then this problem is solved.

2. A resource R is being provisioned for stack with current traversal

    1. An new update U is issued, it loads the stack with CT=X.
    2. Resource R fails and loads the stack with CT=X to mark it as FAILED.
    3. Update U updates the stack with CT=Y and goes ahead with sync_point
       etc., marks stack as UPDATE_IN_PROGRESS
    4. Resource marks the stack as UPDATE_FAILED, which to user means that
       update U has failed, but it actually is going on.

    With this patch, when Resource R fails, it will supply CT=X as
    expected traversal to be updated and will eventually fail because
    update U with CT=Y has taken over.

Partial-Bug: #1512343
Change-Id: I6ca11bed1f353786bb05fec62c89708d98159050
2015-11-26 09:45:49 +00:00
Jenkins b2031196c9 Merge "Remove refresh in Stack update_and_save" 2015-11-17 18:35:23 +00:00
Jenkins a089c6724a Merge "Fix code compatibility with Python 3" 2015-11-02 05:08:38 +00:00
Thomas Herve 1ddb770086 Remove refresh in Stack update_and_save
We uselessly refresh the stack object when we save it, as we don't use
the result anywhere. Let's remove that useless query.

Change-Id: Idbc737a0d88aa98e69979f6edd88b8b148030388
2015-10-29 11:39:28 +01:00
Peter Razumovsky 9176a36f10 Fix [H405] pep rule in heat/objects
Implements bp docstring-improvements

Change-Id: I4038f6695e4d4e9aae2cc6ac9b8830fccd9bfa53
2015-10-08 15:13:52 +03:00
Rabi Mishra e2294da861 Fix action (CREATE) in progress error for TemplateResource/RG
When StackResource 'update_with_template' is called and there
is no existing nested_stack, we call create_stack with an empty
template(TemplateResource CREATE_FAILED->UPDATE flow). We check
db for CREATE_COMPLETE before calling update_stack.There is a
possibility that the create_stack green thread has not finished
and released the stack lock. By persisting the stack state for
COMPLETE/FAILED in the same db session as releasing the lock,
when green thread finishes it's run, we can avoid this race

Change-Id: I1442a7bcedb4fa340ee4f0e6ebc3705544c2a9dd
Closes-Bug: #1498495
2015-10-01 22:29:00 +05:30
Sergey e75b42b56e Add missing value types for logging messages
Change-Id: I0a6c6c577311d16f43afe54202122f6821cb9708
Closes-Bug: #1494574
2015-09-11 04:58:58 -04:00
Grzegorz Grasza eb067db724 Fix code compatibility with Python 3
This code will be used when objects are registered.

Change-Id: I5399d8111fa144eb1e46623041bcc52ddee63c98
Partial-Bug: 1491258
2015-09-09 15:56:01 +02:00
Anant Patil fe119a89f6 Convergence: Stack concurrent update
The stack update API needs to know the expected traversal ID of the
stack (each update is associated with a unique traversal ID) it is

When two updates are triggered simultaneously - for example, in a case
where two resources at same level try to rollback the stack when they
fail at same time - then the updates need to give the traversal ID they
intend to update. If it happens that the resource A triggers the
rollback immediately after the rollback of A has started, but before A
has persisted in DB, then the system is in erroneous state.

Steps to describe the problem:
1. Resource A and B of a Stack S fail.
2. Resource A loads the stack.
3. Resource B loads the stack.
4. Resource A triggers rollback - it takes the previous template as
current template, updates the DB.
5. Resource B now again triggers the rollback. It is not aware of update
triggered by resource A.

If the updates triggered by A and B give the stack traversal they
expect to be updated, then update from resource B will fail, since the
expected traversal ID will not be found in DB as a result of resource A
already updating it.

Change-Id: I13b2f5afe8cadf9a2d9ab4139a14718a5481dec4
2015-09-04 12:00:37 +05:30
Pavlo Shchelokovskyy 51bbf37314 Fix some files permissions
these files in Heat repo had been eroneousely set as executables

Change-Id: Ibcb142e30461aefb321d60d311a8b72c90db52a4
2015-08-26 12:11:27 +03:00
Angus Salkeld bbe08b18eb Make the objects behave more consistently
1. don't return DB models from the objects API
   (only return Objects)
2. delete shouldn't return anything
3. update_and_save should return the refreshed object.

Note: there is still some inconsistency in what is returned by
update_by_id() some return an object and some return a bool.

Related-bug: #1432936
Change-Id: I1a0a38773d4fc4a62af5e0a98076396f39187b6c
2015-06-19 11:42:04 +10:00
Steve Baker be41d08a44 Count all nested stack resources with DB operations
The count is performed in 2 parts
- A function which builds a list of all nested stacks
  by recursively calling stack_get_all_by_owner_id
- A query which counts resources which belong to the list
  of stacks

Considering this will be the basis for fixing bug #1455589 then this
approach is appropriate for backporting to stable/kilo, but master would
ideally replace this soon with a new Stack attribute that stores the
calculated resource count.

Partial-Bug: #1455589

Change-Id: Ifa2e5609fd9a6853e20037e94a5e16696bb378ee
2015-06-02 12:04:06 +12:00
Steve Baker 71a1e26140 Find root stack ID with database operations
This change adds a (currently unused) database function
stack_get_root_id to find the root stack ID for any stack.

Scheduler hints are moved to using Stack.root_stack_id() in this change.
Remaining uses of Stack.root_stack() will switch to using
Stack.root_stack_id() later in the series.

Change-Id: I9914b0df5af119edea6346db0d7c62124fbb1313
Partial-Bug: #1455589
2015-05-19 10:26:32 -07:00
Vijendar Komalla 1b451c2be1 Encrypt Heat template hidden parameters
1. Add a configuration option to enable/disable template parameters
2. Encrypt hidden parameters before storing them in the database and
   decrypt on stack load.

Change-Id: Ie46c6a149f414f655600616da8deee463e55671c
Implements: blueprint encrypt-hidden-parameters
Co-Authored-By: Jason Dunsmore <>
2015-05-12 10:50:49 -05:00
Angus Salkeld edf86aeac2 Persist parent_resource_name and make sure it's available
We are persisting for a number of reasons:
- so we don't have to pass this through ever rpc call
- the API exposes parent_resource (currently always None as
  it is not persisted)

Closes-bug: #1438978
Change-Id: Id2db36c0234a085ec4f0ce2ab114ec483ea29d81
2015-04-10 14:15:38 +10:00
Jason Dunsmore 9817ed77ef Database API and engine changes for stack tags
Allow tagging of stacks with simple string tags.

blueprint stack-tags

Change-Id: I65e1e8e87515595edae332c2ff7e0e82ded409ce
2015-04-08 15:23:50 -05:00
Steve Baker 9bb663d81b Call db refresh in Stack object refresh
The current Stack object refresh doesn't actually do any refreshing,
which explains the functional test speed regression since the RPC
nested stack code relies on stack.refresh() to poll for state change.

With this change AutoScalingSignalTest.test_signal_with_policy_update
locally takes ~60s consistently. Without this change ~50% of test runs
have some form of execution delay, taking 80s -> 300s+.

Change-Id: Ie06851fd5b0d8b802ebf701e8bf621562d9b6d88
Closes-Bug: #1433555
2015-03-20 11:33:03 +13:00
Jenkins ed99ce3826 Merge "StackTag versioned-object" 2015-03-17 06:48:18 +00:00
Kanagaraj Manickam 47fa307c10 StackTag versioned-object
Adds support for versioning object support for StackTag
and add it to the Stack object

Closes-bug: #1432477

Implements blueprint versioned-object

Change-Id: I7cc39c1e1ce913c35862a52592e55b9f484c7e68
2015-03-16 14:54:58 +05:30
Michal Jastrzebski (inc0) 9b4511d0a0 Add missing versioned mechanism to Stack object
Few object methods misses _from_db_object. To enable it to make
schema valid, we need to fix that.

Change-Id: I2e5ff22cd40be56ebc5f3a7095887be773ddb809
Closes-Bug: #1431353
2015-03-13 13:16:02 +01:00
Michal Jastrzebski dea0897f45 First objects - stack and raw template
Implementation of oslo.versionedobjects.
This commit consists basic mechanism and first objects.
This should be base of implementation versoning to other objects

Implements: blueprint versioned-objects

Co-Authored-By: ShaoHe Feng <>
Co-Authored-By: Grzegorz Grasza <>

Change-Id: I554162cf3681fe559c75f54c61c6f32c91f5c2f8
2015-03-11 13:56:38 +01:00