Commit Graph

150 Commits (cb80df4f4ab0e8de16d53c7fddff7c44dedd3f94)

Author SHA1 Message Date
Zane Bitter c9792b96d2 Create replacement resource atomically
Use a single transaction to create the replacement resource and set it as
the replaced_by link in the old resource. Also, ensure that no other
traversal has taken a lock on the old resource before we modify it.

If we end up bailing out and not creating a replacement or sending an RPC
message to check it, make sure we retrigger any new traversal.

Change-Id: I23db4f06a4060f3d26a78f7b26700de426f355e3
Closes-Bug: #1727128
6 years ago
Jenkins 0a9fcd9136 Merge "Eager load resource_properties_data in resource" 6 years ago
rabi f849b4deb2 Set resource._properties_data=None when loading from db
In I462ce7161497306483286b78416f9037ac80d6fa we changed to use the
frozen_defintion properties for delete. However, When deleting a
resource from backup stack, where the resource is in INIT_COMPLETE,
setting the _stored_properties_data(_properties_data) to {} when
loading the resource from the db, results in error, when resources
access properties in handle_delete.

Change-Id: If76372c7ef9aee258efb1bfbc724d8637bc6a32c
Closes-Bug: #1709682
6 years ago
Crag Wolfe 705625cee7 Eager load resource_properties_data in resource
Eager load resource_properties_data in resources in the typical
resource-loading scenarios where properties data will be
accessed. Thus, we can save an extra db query per resource when
loading all the resources in a stack, for instance. Fall back to lazy
loading properties data in other scenarios.

Also, the resource object doesn't need to store a copy of its
ResourcePropertiesData object in self.rsrc_prop_data, so don't.

Change-Id: Ib7684af3fe06f818628fd21f1216de5047872948
Closes-Bug: #1665503
6 years ago
Crag Wolfe 0550659f5f Store resource attributes in the DB
Store resource attributes that may be cached in the DB, saving the
cost of re-resolving them later. This works for most resources,
specifically those that do not override the get_attribute() method.

Change-Id: I71f8aa431a60457326167b8c82adc03ca750eda6
Partial-Bug: #1660831
6 years ago
Jenkins 3af92c84a7 Merge "Load "lighter" db resources when appropriate" 6 years ago
Zane Bitter e37d9fab8f Corrected max secs for concurrent trans retries
This was most likely meant as a max 2s delay here, not a max 2ms

Also includes a related change: when retries for metadata updates are
attempted, make sure we do not have a stale value of the atomic_key
(otherwise we'll just inevitably hit the ConcurrentTransaction issue).

Co-Authored-By: Crag Wolfe <>
Partial-Bug: #1651768
Change-Id: Ie56e0e4ff93633db1f4752859d2b2a9506922911
6 years ago
Crag Wolfe a7376f7494 Consolidate resource locking with state changes
Change-Id: I261b2f0968e16d35b7d5d791a3edb4b265a4f1d1
Closes-Bug: #1662585
6 years ago
Crag Wolfe 93fab308e8 Load "lighter" db resources when appropriate
Sometimes we know we will only access particular fields of a resource
object, rather than *all* of them. This commit allows the caller to
specify (optionally) the fields that should be populated when the
resource object is instantiated. This saves memory, trips to the db,
and in some cases avoids extra join queries (e.g. for or

Change-Id: I405888f46451d2657aa28f610f8ca555215ff5cf
Partial-Bug: #1680658
6 years ago
liyi 8f10215ffd Remove log translations
Log messages are no longer being translated. This removes all use of
the _LE, _LI, and _LW translation markers to simplify logging and to
avoid confusion with new contributions.


Change-Id: Ieec8028305099422e1b0f8fc84bc90c9ca6c694f
6 years ago
Jenkins 4dc101f10e Merge "Eager load event's resource prop. data when appropiate" 6 years ago
Jenkins ab267e763a Merge "Remove unused event_get methods" 6 years ago
Jenkins a77ef39048 Merge "Data migration enabling storage of resource attributes" 6 years ago
Crag Wolfe 283b2d3985 Eager load event's resource prop. data when appropiate
Eager load an event's resource properties data in the list_events
"show event" case, i.e. when an event uuid is specified. When showing
multiple events, the events' resource properties data should not be
eager loaded.

Change-Id: I12e371dcbdae398360e28200e68f4a75831f8f17
Closes-bug: #1665506
6 years ago
Crag Wolfe 019ae84d4f Remove unused event_get methods
Remove unused event get methods from the sqlalchemy db api and event

Change-Id: Ide5780d456755e180572194ad4d06277749c8c79
6 years ago
Crag Wolfe 9ebbd1e6f1 Data migration enabling storage of resource attributes
Working towards the goal of storing resource attributes in the db so
as to avoid re-resolving them when appropriate. Adds an 'attr_data'
object to the resource object, defined as a relationship on the
already existing resource_properties_data table.

Change-Id: I2104078d850da08b22547d7feab2bde00c543478
Partial-Bug: #1660831
6 years ago
Crag Wolfe fdddc28f9d Lazy-load resource_properties_data for events
Typically when retrieving a large number of events, we don't need to
load resource_properties_data for each event.

Change-Id: I6564fea91d19c24409075152a614cc036dc1ee34
6 years ago
Crag Wolfe 791c245c14 Add resource_properties_data assoc. to resource, event objs
Add the resource_properties_data association to resource and event
objects. The resource and event engine objects do not use yet
it but will soon.

Change-Id: Idecaafffbc5e9bfcd2355e2a165836a5ed89b16f
7 years ago
Crag Wolfe 1ea6bb8bf6 Add the ResourcePropertiesData object
Prepare for the future when both resources and events refer to
ResourcePropertiesData rather than store the data themselves.

Change-Id: I0c5aefa9d73e1281e6477d46fe181d8948aee71b
7 years ago
Thomas Herve 84067dba88 Remove db.api wrapper
The db.api module provides a useless indirection to the only
implementation we ever had, sqlalchemy. Let's use that directly instead
of the wrapper.

Change-Id: I80353cfed801b95571523515fd3228eae45c96ae
7 years ago
Zane Bitter f310a1f6bc Handle ambiguous physical resource IDs
It's possible that we could end up with multiple resources with the same
physical resource ID, but that would be undetectable since we return only
one from the database layer. This change allows us to detect the problem an
return an error where the result is rendered ambiguous.

Change-Id: I2c5ddbe6731c33a09ec7c4a7b91dcfe414da4385
7 years ago
Crag Wolfe a673ee2d56 Refactor, add encrypt/decrypt data dict functions to crypt
Just a refactor, no change in functionality.

The functions added to crypt are used to encrypt / decrypt resource
properties data dicts. Note that they should not be used for
encrypting / decrypting other things such as params or user creds
(which are just strings).  An intermediate json conversion of each
value in a dict takes place before it is encrypted/decrypted.

Change-Id: Id6bcc90cbf430095719315ac7e9d3e8c9e745012
7 years ago
Jenkins 8625175277 Merge "Don't query raw_template in stacks uselessly" 7 years ago
Boden R 756dda72d4 Replace retrying with tenacity
We are replacing all usages of the 'retrying' package with
'tenacity' as the author of retrying is not actively maintaining
the project. Tenacity is a fork of retrying, but has improved the
interface and extensibility (see [1] for more details). Our end
goal here is removing the retrying package from our requirements.

Tenacity provides the same functionality as retrying, but has the
following major differences to account for:
- Tenacity uses seconds rather than ms as retrying did.
- Tenacity has different kwargs for the decorator and
Retrying class itself.
- Tenacity has a different approach for retrying args by
using classes for its stop/wait/retry kwargs.
- By default tenacity raises a RetryError if a retried callable
times out; retrying raises the last exception from the callable.
Tenacity provides backwards compatibility here by offering
the 'reraise' kwarg.
- Tenacity defines 'time.sleep' as a default value for a kwarg.
That said consumers who need to mock patch time.sleep
need to account for this via mocking of time.sleep before
tenacity is imported.
- For retries that check a result, tenacity will raise if the retried
function raises, whereas retrying retried on all exceptions.

This patch updates all usages of retrying with tenacity.
Unit tests will be added/removed where applicable.


Closes-Bug: #1635388

Change-Id: Iec0822cc0d5589b04c1764db518478d286455031
7 years ago
Thomas Herve c59865870b Don't query raw_template in stacks uselessly
When partially removing eager loading of raw_template in stacks
(260b79ed28), we didn't get into account
that accessing the sqlalchemy field would create an additional query
whereas it was previously eager loaded. This removes it by only filling
the field when it's already fetched.

Change-Id: Ifa17c74e3559adaef56593a205101d92e9a37da5
Closes-Bug: #1634127
7 years ago
Zane Bitter 260b79ed28 Don't always eagerly load the raw_template for a stack
Always loading the raw template in situations where we didn't need it -
e.g. in identify_stack, where we just want the name + id (given one of
them), or when getting the summary stack list - uses up DB bandwidth and
memory unnecessarily.

This partially reverts commit 3ab0ede98c.

* The eager_load option to get_stack() is reinstated, but with the default
  flipped to True. In places where we explicitly do not want to load the
  template, we pass False.
* stack_get_by_name() no longer eagerly loads the template. There were no
  instances of this where we subsequently use the template.
* stack_get_all() acquires an eager_load option, with the default set to
  False. Direct users of objects.stack.Stack.get_all() will not eagerly
  load by default, but users of engine.stack.Stack.load_all() will get the
  template eagerly loaded. This practically always corresponds to what you

Change-Id: I1f156c25ea26322be5b606a61dd77d80cadd65fc
Related-Bug: #1626675
7 years ago
Jenkins d86ea51bbd Merge "A context cache for Resource objects" 7 years ago
Oleksii Chuprykov 68944d2230 Migrate stacks from legacy to convergence engine
Run `heat-manage migrate-convergence-1 <stack_id>` to migrate
legacy stack to convergence engine.

Heat engine is used for doing migration i.e. migration can't
be done offline.

Change-Id: Ie7c2498b37937438f16d154b154b3a6ecbf9ff74
Implements-bp: convergence-migrate-stack
7 years ago
rabi 1f88c675c4 Remove tenant_safe flag from rpc and db api
We can use admin_context to have access to stacks
and software configs across projects. This removes
the tenant_safe flag from rpc and db api. This is
backward compatible with older rpc clients.

We still support use of global_tenant flag for listing
stacks and software configs. However, by default
an admin(user with admin role in admin_project)
would not need that.

Change-Id: I12303dbf30fb80290f95baba0c67cdf684f5f409
7 years ago
Steve Baker bc3b84fb60 A context cache for Resource objects
A context cache which memoizes the resources fetched by calls to
Resource.get_all_by_root_stack(..., cache=True)
which are recalled by subsequent calls to Resource.get_all_by_stack.

Because get_all_by_stack returns a collection instead of a single
resource, there is no way of taking advantage of the SQLAlchemy
identity map [1].


Change-Id: Ia5aae0c86a586041020e9798566c9e0af48c180d
Partial-Bug: #1578854
7 years ago
Steve Baker 3ab0ede98c Always eager load the raw_template for a stack
The vast majority of stack fetches are immediately followed by a
raw_template fetch, so this change always eagerly fetches the
raw_template for every stack fetch.

During stack versioned object creation the stack's raw_template object
is used to construct the versioned raw template object.

Change-Id: I1a6fb8fb7d069b50dd5d623c989acd5582818ae1
Related-Bug: #1479723
Related-Bug: #1523748
7 years ago
Jenkins f9b47c61c3 Merge "Remove remaining methods from db objects" 7 years ago
Jenkins 0bdd21bd11 Merge "Remove the db object delete method" 7 years ago
Jenkins a72954acea Merge "Only call session.refresh from within db_api functions" 7 years ago
Jenkins 490fc208a0 Merge "Add __ne__ built-in function" 7 years ago
yuyafei 456fe5fca6 Add __ne__ built-in function
In Python 3 __ne__ by default delegates to __eq__ and inverts the
result, but in Python 2 they urge you to define __ne__ when you
define __eq__ for it to work properly [1].There are no implied
relationships among the comparison operators. The truth of x==y
does not imply that x!=y is false. Accordingly, when defining
__eq__(), one should also define __ne__() so that the operators
will behave as expected.

Change-Id: I3ae84db585355fef864a8beea4ad388ed09e6ff7
7 years ago
Steve Baker b125b81298 Remove remaining methods from db objects
- HeatBase.expire, not used anywhere
- HeatBase.save_and_update, moved to sqlalchemy.api
- SoftDelete.soft_delete moved to sqlalchemy.api

update_and_save creates a transaction, so it needs to be in
sqlalchemy.api so that a context manager can eventually manage the

Change-Id: I84749f4fd0781ed9a2d62327b39ce6eee0f07b35
7 years ago
Steve Baker 845075031a Remove the db object delete method
The HeatBase.delete method starts its own (sub) transaction with one
of three possible sessions. This change moves all delete calls to a
sqlalchemy.api function with the current context session. This will help with
bug #1479723 to always do deletes with a session provided by the
context manager.

Change-Id: I8dfd3bc6fdb44b0e3b06fab5d7dc8e06fa3d80a8
7 years ago
Steve Baker b8a529ac97 Only call session.refresh from within db_api functions
This change removes the refresh method from the parent db object, and
adds a refresh boolean argument to db_api.resource_get function (which is
the only known use of refresh). Constraining refresh calls to be
inside db_api methods helps with bug #1479723 where the refresh can
happen under a context manager session.

Change-Id: I6fc5c03e8572eee90f89455e0925d503514040b3
Related-Bug: #1479723
7 years ago
Steve Baker 26b0dd2a3e Do not populate unused stack and watch rule objects
These fields are lazy loaded, so cause extra unnecessary database

The watch rules are needed when formatting every watch data metric for
the cloudwatch API, so an id->name map of all rules is built to avoid
repeated lazy loading of every rule.

Change-Id: I36dd43172ccd402f08a45b254ae47e208b7b8c08
Related-Bug: #1479723
7 years ago
Rakesh H S 906a0ed6fd Convergence: Fix concurrent update resource delete
In convergence, wherein concurrent updates are possible, if a resource
is deleted (by previous traversal) after dependency graph is created
for new traversal, the resource remains in graph but wouldn't be
available in DB for processing.
It is prerequisite to have resources in DB before any action can be
taken on them.

Hence during convergence resource delete action, the resource entry
from DB is not deleted i.e soft deleted, so that the latest/new update
can find the entry.
All of these soft deleted resources will be deleted when the stack has
completed its operation.

Closes-Bug: #1528560
Change-Id: I0b36ce098022560d7fe01623ce7b66d1d5b38d55
7 years ago
Steve Baker 9e007acc44 Add context to stack lock function arguments
Context is added as the first argument to stack_lock_* db functions
but is currently ignored for getting the session used for stack lock

This is required later for bug 1479723 but is added in its own change
here to ease reviewing load.

Change-Id: Ieb3e4e2ee67150777cbe1e961d0d1806cf1f7e46
Related-Bug: #1479723
7 years ago
Steve Baker 4ed43674c1 Be consistent about context first argument in db api
By ensuring every db api function has the context as its first
argument, this raises the possiblity of using oslo.db recommended
session handling:

Change-Id: I3649bdbf254e2834e79984cc381a1db1ea9e78b8
Related-Bug: #1479723
7 years ago
Steve Baker 7bfbace79f Use Event versioned object for event_list formatting
This change removes the event.Event.load method, and switches
api.format_event to using the Event versioned object instead.

Since the only purpose of loading the stack is to get its identifier,
this is now also coming from a Stack versioned object.

Related-Bug: #1588561
Change-Id: I7b1fc99894818b44cde08cf08b010b1d1fb94e9f
7 years ago
Steve Baker de99472c75 DB query to get all resources by the root stack
To prevent a resource query for every nested stack during a
resource-list, there needs to be a way to fetch every resource in a
single query.

Change-Id: Ib05b2166d6c7584a844e1ab4a5dd6e35437c96c4
Related-Bug: #1578854
7 years ago
Jenkins 9852d79ac7 Merge "Convergence: DB api to get all active resources of stack" 7 years ago
Jenkins 2c05a61328 Merge "Declare and use keyword args for Stack.get_all" 7 years ago
Rakesh H S 9cf021ac7e Convergence: DB api to get all active resources of stack
This patch reverts change I6a212da19a774239f014163774e75fe11dfe272c
and adds new DB api resource_get_all_active_by_stack to be used by

The new DB api will be used while generating graph for convergence
stack and will fetch all resources of stack from DB excluding
the DELETE COMPLETE resources, if any.

Change-Id: I303ef2c9b5b6a0a49253425c00565c8981cc6825
Partial-Bug: #1528560
7 years ago
Jenkins c7bdf30bac Merge "Lazy-load stack tags" 7 years ago
Jenkins 70238b3d61 Merge "Stack list does direct stack object query" 7 years ago