Commit Graph

49 Commits (cb80df4f4ab0e8de16d53c7fddff7c44dedd3f94)

Author SHA1 Message Date
Zane Bitter c9792b96d2 Create replacement resource atomically
Use a single transaction to create the replacement resource and set it as
the replaced_by link in the old resource. Also, ensure that no other
traversal has taken a lock on the old resource before we modify it.

If we end up bailing out and not creating a replacement or sending an RPC
message to check it, make sure we retrigger any new traversal.

Change-Id: I23db4f06a4060f3d26a78f7b26700de426f355e3
Closes-Bug: #1727128
2017-11-03 09:14:46 +05:30
Jenkins 0a9fcd9136 Merge "Eager load resource_properties_data in resource" 2017-10-13 18:25:36 +00:00
rabi f849b4deb2 Set resource._properties_data=None when loading from db
In I462ce7161497306483286b78416f9037ac80d6fa we changed to use the
frozen_defintion properties for delete. However, When deleting a
resource from backup stack, where the resource is in INIT_COMPLETE,
setting the _stored_properties_data(_properties_data) to {} when
loading the resource from the db, results in error, when resources
access properties in handle_delete.

Change-Id: If76372c7ef9aee258efb1bfbc724d8637bc6a32c
Closes-Bug: #1709682
2017-08-22 18:06:05 +05:30
Crag Wolfe 705625cee7 Eager load resource_properties_data in resource
Eager load resource_properties_data in resources in the typical
resource-loading scenarios where properties data will be
accessed. Thus, we can save an extra db query per resource when
loading all the resources in a stack, for instance. Fall back to lazy
loading properties data in other scenarios.

Also, the resource object doesn't need to store a copy of its
ResourcePropertiesData object in self.rsrc_prop_data, so don't.

Change-Id: Ib7684af3fe06f818628fd21f1216de5047872948
Closes-Bug: #1665503
2017-07-31 21:13:49 -04:00
Crag Wolfe 0550659f5f Store resource attributes in the DB
Store resource attributes that may be cached in the DB, saving the
cost of re-resolving them later. This works for most resources,
specifically those that do not override the get_attribute() method.

Change-Id: I71f8aa431a60457326167b8c82adc03ca750eda6
Partial-Bug: #1660831
2017-06-21 20:18:46 -04:00
Jenkins 3af92c84a7 Merge "Load "lighter" db resources when appropriate" 2017-06-05 09:50:38 +00:00
Zane Bitter e37d9fab8f Corrected max secs for concurrent trans retries
This was most likely meant as a max 2s delay here, not a max 2ms

Also includes a related change: when retries for metadata updates are
attempted, make sure we do not have a stale value of the atomic_key
(otherwise we'll just inevitably hit the ConcurrentTransaction issue).

Co-Authored-By: Crag Wolfe <>
Partial-Bug: #1651768
Change-Id: Ie56e0e4ff93633db1f4752859d2b2a9506922911
2017-05-04 12:32:22 -04:00
Crag Wolfe a7376f7494 Consolidate resource locking with state changes
Change-Id: I261b2f0968e16d35b7d5d791a3edb4b265a4f1d1
Closes-Bug: #1662585
2017-04-25 08:21:42 -07:00
Crag Wolfe 93fab308e8 Load "lighter" db resources when appropriate
Sometimes we know we will only access particular fields of a resource
object, rather than *all* of them. This commit allows the caller to
specify (optionally) the fields that should be populated when the
resource object is instantiated. This saves memory, trips to the db,
and in some cases avoids extra join queries (e.g. for or

Change-Id: I405888f46451d2657aa28f610f8ca555215ff5cf
Partial-Bug: #1680658
2017-04-19 20:51:02 -04:00
liyi 8f10215ffd Remove log translations
Log messages are no longer being translated. This removes all use of
the _LE, _LI, and _LW translation markers to simplify logging and to
avoid confusion with new contributions.


Change-Id: Ieec8028305099422e1b0f8fc84bc90c9ca6c694f
2017-03-25 17:11:50 +08:00
Crag Wolfe 9ebbd1e6f1 Data migration enabling storage of resource attributes
Working towards the goal of storing resource attributes in the db so
as to avoid re-resolving them when appropriate. Adds an 'attr_data'
object to the resource object, defined as a relationship on the
already existing resource_properties_data table.

Change-Id: I2104078d850da08b22547d7feab2bde00c543478
Partial-Bug: #1660831
2017-02-16 16:08:31 -05:00
Crag Wolfe 791c245c14 Add resource_properties_data assoc. to resource, event objs
Add the resource_properties_data association to resource and event
objects. The resource and event engine objects do not use yet
it but will soon.

Change-Id: Idecaafffbc5e9bfcd2355e2a165836a5ed89b16f
2016-12-22 10:59:01 -08:00
Thomas Herve 84067dba88 Remove db.api wrapper
The db.api module provides a useless indirection to the only
implementation we ever had, sqlalchemy. Let's use that directly instead
of the wrapper.

Change-Id: I80353cfed801b95571523515fd3228eae45c96ae
2016-12-13 09:40:29 +01:00
Zane Bitter f310a1f6bc Handle ambiguous physical resource IDs
It's possible that we could end up with multiple resources with the same
physical resource ID, but that would be undetectable since we return only
one from the database layer. This change allows us to detect the problem an
return an error where the result is rendered ambiguous.

Change-Id: I2c5ddbe6731c33a09ec7c4a7b91dcfe414da4385
2016-12-05 14:58:20 -05:00
Crag Wolfe a673ee2d56 Refactor, add encrypt/decrypt data dict functions to crypt
Just a refactor, no change in functionality.

The functions added to crypt are used to encrypt / decrypt resource
properties data dicts. Note that they should not be used for
encrypting / decrypting other things such as params or user creds
(which are just strings).  An intermediate json conversion of each
value in a dict takes place before it is encrypted/decrypted.

Change-Id: Id6bcc90cbf430095719315ac7e9d3e8c9e745012
2016-11-28 20:31:49 -08:00
Boden R 756dda72d4 Replace retrying with tenacity
We are replacing all usages of the 'retrying' package with
'tenacity' as the author of retrying is not actively maintaining
the project. Tenacity is a fork of retrying, but has improved the
interface and extensibility (see [1] for more details). Our end
goal here is removing the retrying package from our requirements.

Tenacity provides the same functionality as retrying, but has the
following major differences to account for:
- Tenacity uses seconds rather than ms as retrying did.
- Tenacity has different kwargs for the decorator and
Retrying class itself.
- Tenacity has a different approach for retrying args by
using classes for its stop/wait/retry kwargs.
- By default tenacity raises a RetryError if a retried callable
times out; retrying raises the last exception from the callable.
Tenacity provides backwards compatibility here by offering
the 'reraise' kwarg.
- Tenacity defines 'time.sleep' as a default value for a kwarg.
That said consumers who need to mock patch time.sleep
need to account for this via mocking of time.sleep before
tenacity is imported.
- For retries that check a result, tenacity will raise if the retried
function raises, whereas retrying retried on all exceptions.

This patch updates all usages of retrying with tenacity.
Unit tests will be added/removed where applicable.


Closes-Bug: #1635388

Change-Id: Iec0822cc0d5589b04c1764db518478d286455031
2016-11-15 04:34:17 +00:00
Steve Baker bc3b84fb60 A context cache for Resource objects
A context cache which memoizes the resources fetched by calls to
Resource.get_all_by_root_stack(..., cache=True)
which are recalled by subsequent calls to Resource.get_all_by_stack.

Because get_all_by_stack returns a collection instead of a single
resource, there is no way of taking advantage of the SQLAlchemy
identity map [1].


Change-Id: Ia5aae0c86a586041020e9798566c9e0af48c180d
Partial-Bug: #1578854
2016-08-16 01:32:44 +00:00
Steve Baker b125b81298 Remove remaining methods from db objects
- HeatBase.expire, not used anywhere
- HeatBase.save_and_update, moved to sqlalchemy.api
- SoftDelete.soft_delete moved to sqlalchemy.api

update_and_save creates a transaction, so it needs to be in
sqlalchemy.api so that a context manager can eventually manage the

Change-Id: I84749f4fd0781ed9a2d62327b39ce6eee0f07b35
2016-07-13 11:58:27 +12:00
Steve Baker 845075031a Remove the db object delete method
The HeatBase.delete method starts its own (sub) transaction with one
of three possible sessions. This change moves all delete calls to a
sqlalchemy.api function with the current context session. This will help with
bug #1479723 to always do deletes with a session provided by the
context manager.

Change-Id: I8dfd3bc6fdb44b0e3b06fab5d7dc8e06fa3d80a8
2016-07-13 11:58:27 +12:00
Steve Baker b8a529ac97 Only call session.refresh from within db_api functions
This change removes the refresh method from the parent db object, and
adds a refresh boolean argument to db_api.resource_get function (which is
the only known use of refresh). Constraining refresh calls to be
inside db_api methods helps with bug #1479723 where the refresh can
happen under a context manager session.

Change-Id: I6fc5c03e8572eee90f89455e0925d503514040b3
Related-Bug: #1479723
2016-07-13 11:58:13 +12:00
Rakesh H S 906a0ed6fd Convergence: Fix concurrent update resource delete
In convergence, wherein concurrent updates are possible, if a resource
is deleted (by previous traversal) after dependency graph is created
for new traversal, the resource remains in graph but wouldn't be
available in DB for processing.
It is prerequisite to have resources in DB before any action can be
taken on them.

Hence during convergence resource delete action, the resource entry
from DB is not deleted i.e soft deleted, so that the latest/new update
can find the entry.
All of these soft deleted resources will be deleted when the stack has
completed its operation.

Closes-Bug: #1528560
Change-Id: I0b36ce098022560d7fe01623ce7b66d1d5b38d55
2016-07-05 12:27:37 +05:30
Steve Baker de99472c75 DB query to get all resources by the root stack
To prevent a resource query for every nested stack during a
resource-list, there needs to be a way to fetch every resource in a
single query.

Change-Id: Ib05b2166d6c7584a844e1ab4a5dd6e35437c96c4
Related-Bug: #1578854
2016-06-03 16:30:03 +12:00
Rakesh H S 9cf021ac7e Convergence: DB api to get all active resources of stack
This patch reverts change I6a212da19a774239f014163774e75fe11dfe272c
and adds new DB api resource_get_all_active_by_stack to be used by

The new DB api will be used while generating graph for convergence
stack and will fetch all resources of stack from DB excluding
the DELETE COMPLETE resources, if any.

Change-Id: I303ef2c9b5b6a0a49253425c00565c8981cc6825
Partial-Bug: #1528560
2016-05-31 15:13:24 +00:00
Kanagaraj Manickam 3d173834b1 Migrate nova_instance to physical_resource_id
migrate to physical_resource_id from nova_instance
at application layer, while the db schema still holds
nova_instance label. This is the first phase of
migration. Next patch will take care of db schema.

Change-Id: I6ebbe3d71d5fb9a7dd3c68ff13777982eb5bbbef
Partial-bug: #1346742
2016-05-17 13:33:43 +05:30
Steve Baker 50f5142eca Remove stack ObjectField from resource
This causes the stack record to be loaded every time a resource is
loaded, and it is not used for anything other than getting the stack
ID, which is already available via the stack_id field.

Change-Id: I45ce9d18984f4881151dba496482713a62c9eae9
Partial-Bug: #1578854
2016-05-11 10:27:07 +12:00
Sergey Kraynev 82b744042e Handling concurrent transactions in metadata_set method
There are follow changes in this patch:
 - Using exception ConcurrentTransactions for processing
   concurrent transactions during writing metadata.
 - wrapper @retry_on_conflict was used for metadata_set method to
   allow retrying in the event of a race. The same wrapper was added for
   _push_metadata_software_deployments method.
 - added new parameter for metadata_set method - merge_metadata.
   When RetryRequest exception is raised, oslo_db_api.wrap_db_retry
   re-call metadata_set method and in this case we need to refresh
   old metadata. It's mostly need for signals without data and id.
   For example:
     A and B signals come in the same moment and both get number 1,
   because metadata was empty. Then during write in db RetryRequest
   exception was raised for signal B. Metadata of this signal stores old
   number - 1. So we should re-calculate this value using new length
   of metadata and set number - 2.

Change-Id: I1ddbad7cde3036cfa9310c670609fcde607ffcac
Co-Authored-By: Zane Bitter <>
Partially-Bug: #1497274
2016-03-23 03:47:46 -04:00
Jenkins 8d07c7dc8d Merge "Don't useless fetch resources on update" 2016-02-24 15:06:18 +00:00
Jenkins 471a34b2ba Merge "Stack resource search" 2016-02-17 15:40:09 +00:00
Kanagaraj Manickam 6c4709696b Stack resource search
heat resource-list -f <filters>

imlements blueprint heat-stack-resource-search


Change-Id: Iaae88f3b32bc2ba7b41a4078ef3aa8ffc07079b7
2016-02-16 17:52:24 +05:30
Thomas Herve 28c07fdfdb Don't useless fetch resources on update
Currently when updating resource in the database, we fetch the row many
times uselessly. This adds and uses an API to do less queries.

Change-Id: Ic50f8646fba6a578634e4e869ab5155756b0a1aa
2016-02-15 13:06:26 +01:00
Grzegorz Grasza dfdbc99d8f Add OBJ_PROJECT_NAMESPACE for Heat project
This code will be used when objects are registered and serialized.

Change-Id: I9d9095bc40e61a104ebea5042f4eb75d16d27d69
Partial-Bug: 1491258
2016-01-22 21:53:51 +00:00
Jenkins a089c6724a Merge "Fix code compatibility with Python 3" 2015-11-02 05:08:38 +00:00
Peter Razumovsky 9176a36f10 Fix [H405] pep rule in heat/objects
Implements bp docstring-improvements

Change-Id: I4038f6695e4d4e9aae2cc6ac9b8830fccd9bfa53
2015-10-08 15:13:52 +03:00
Steve Baker 1b2cd7495d Add resource.root_stack_id column
This change adds a root_stack_id column to the resource
record to allow a subsequent change enforce
max_resources_per_stack with a single query instead of the
many it currently requires.

This change includes the following:
- Data migration to add the resource.root_stack_id column
  and populate all existing resources with their calculated
  root stack
- Make new resources aquire and set their root_stack_id on
  store or update.
- StackResource._validate_nested_resources use the stored
  root_stack_id resulting in a ~15% performance improvement
  for the creation time of a test stack containing 40 nested

Change-Id: I2b00285514235834131222012408d2b5b2b37d30
Partial-Bug: 1489548
2015-09-22 15:43:15 +12:00
Grzegorz Grasza eb067db724 Fix code compatibility with Python 3
This code will be used when objects are registered.

Change-Id: I5399d8111fa144eb1e46623041bcc52ddee63c98
Partial-Bug: 1491258
2015-09-09 15:56:01 +02:00
Pavlo Shchelokovskyy 51bbf37314 Fix some files permissions
these files in Heat repo had been eroneousely set as executables

Change-Id: Ibcb142e30461aefb321d60d311a8b72c90db52a4
2015-08-26 12:11:27 +03:00
Thomas Herve f3101bf9e4 Fixes Resource object refresh to use db object
Fix Resource object _refresh method to actually pass a database object
to _from_db_object, instead of wrapped one. This fixes properties

Change-Id: I3d8d54fa7441c95fc3de5354f80ce2e7e2ba7054
Closes-Bug: #1481644
2015-08-05 10:48:35 +02:00
Rakesh H S ca34802c76 Convergence: Fix concurrent update resource cleanup
If there are multiple resources with same name(in case of
update-replace), only one resource is returned by
resource_get_all_by_stack, due to which convergence cleanup
was breaking.

An additional argument is added to resource_get_all_by_stack to return
resources dictionary with resource id(unique) as key.

Change-Id: I6a212da19a774239f014163774e75fe11dfe272c
2015-08-03 15:50:22 +05:30
Thomas Herve f3bd4fa466 Add a new crypt method using cryptography
This updates the default crypt method to use the cryptography module
instead of the oslo crypto utils module. It also refactors decrypt to
remove some duplication.

This new patch fixes an issue with small keys.

Change-Id: I3ef166d15306693f0589903785102a359834c307
Closes-Bug: #1468025
2015-07-07 10:06:47 +02:00
Thomas Herve 9cba80c091 Revert "Add a new crypt method using cryptography"
This reverts commit d8f3ef910c.

Change-Id: I76dc18baa0d71b1819b765c6ee5840bd7e2ef59d
Related-Bug: #1471150
2015-07-03 09:00:08 +00:00
Thomas Herve d8f3ef910c Add a new crypt method using cryptography
This updates the default crypt method to use the cryptography module
instead of the oslo crypto utils module. It also refactors decrypt to
remove some duplication.

Change-Id: Ie24aebcb3080725c250a4f3ba726b23a9c995965
Closes-Bug: #1468025
2015-07-02 09:42:06 +02:00
Angus Salkeld 56012a3584 Convergence: concurrent workflow
implements blueprint convergence-concurrent-workflow
Depends-On: I7586bc16ce492144f617ee3adc31a2bc19a62173
Change-Id: If968e46ed54f92c1bbfe3ce3bef241802a993ce6
2015-06-26 13:18:36 +10:00
Angus Salkeld bbe08b18eb Make the objects behave more consistently
1. don't return DB models from the objects API
   (only return Objects)
2. delete shouldn't return anything
3. update_and_save should return the refreshed object.

Note: there is still some inconsistency in what is returned by
update_by_id() some return an object and some return a bool.

Related-bug: #1432936
Change-Id: I1a0a38773d4fc4a62af5e0a98076396f39187b6c
2015-06-19 11:42:04 +10:00
Jason Dunsmore 89d4206954 Encrypt properties data
Encrypt properties data before storing it in database and decrypt it
when the resource is being loaded from the database.

Change-Id: I646542b1d03296f62a83041dc2a0ca2719775289
Implements: blueprint encrypt-hidden-parameters
2015-06-01 09:52:46 -05:00
Sirushti Murugesan 11f835b8ef Replace dict.itervalues/dict.iteritems and use len over filter function
partial blueprint heat-python34-support

Change-Id: I2b23f068e485fd778e1d7b14d16cadb6d401163f
2015-04-23 19:45:36 +05:30
Kanagaraj Manickam 39368e2f6a Cleans default declaration nullable=False from heat objects
In Heat objects, the default value of nullable attribute is
set in many of the object's fields, which is not required
as oslo object Field defines nullable as False by default

Change-Id: I0164b64c043816f624aeba19561a4a5f8d36689d
Closes-bug: #1439957
2015-04-06 10:33:48 +05:30
Angus Salkeld 0b25435867 Correct confusing "cls" argument on non-static method

Change-Id: Ie4ac74fe4e35d8d22376f2f9fa0623b1c80d1148
2015-03-20 09:10:02 +10:00
Sergey Kraynev 2c21d660b3 Add extra columns for resource table
This patch adds following columns for resource table:
- `needed_by` (a list of Resource keys)
- `requires` (a list of Resource keys)
- `replaces` (a single Resource key, Null by default)
- `replaced_by` (a single Resource key, Null by default)
- `current_template_id` (a single RawTemplate key)

Co-Authored-by: Angus Salkeld <>
Co-Authored-By: Qiming Teng <>
Change-Id: I65e1032e84b40cb7ae3126fa6b63c914988cc970
Implements: blueprint convergence-resource-table
2015-03-17 06:09:26 -04:00
ShaoHe Feng 49b30027a5 Versioned objects - Resource
implementation for versioned objects. This contains Resource

Implements: blueprint versioned-objects
Co-Authored-By: Michal Jastrzebski (inc0) <>
Change-Id: Id4a5724d68e93b51aaf45d75e5e5e564b5d87789
2015-03-12 18:38:49 +01:00