Commit Graph

556 Commits (cb80df4f4ab0e8de16d53c7fddff7c44dedd3f94)

Author SHA1 Message Date
liyi 51aed6e41f Fix logic error in sqlarchemy api
Change-Id: Ic3074969b8a75f7b38aa3ba9a37b23dcf133aa8e
6 years ago
Zane Bitter c9792b96d2 Create replacement resource atomically
Use a single transaction to create the replacement resource and set it as
the replaced_by link in the old resource. Also, ensure that no other
traversal has taken a lock on the old resource before we modify it.

If we end up bailing out and not creating a replacement or sending an RPC
message to check it, make sure we retrigger any new traversal.

Change-Id: I23db4f06a4060f3d26a78f7b26700de426f355e3
Closes-Bug: #1727128
6 years ago
huangtianhua a532535b61 Correct the filter attribute of db model 'ResourcePropertiesData'
Make sure to delete the ResourcePropertiesData row
if the resource 'attr_data_id' has been set already.

Change-Id: I2df5dfc72d051b6619c769c1260ff24b016bc294
Closes-Bug: #1708378
6 years ago
Jenkins 0a9fcd9136 Merge "Eager load resource_properties_data in resource" 6 years ago
ricolin d0b7cd175e Adds 5 backport db migration place holder for Pike
Db migration version from 81 to 85 are made as place
holder of db backporting for stable/pike.

Change-Id: Iddd91173f4fa77f9ef8cf2a60f629d551f6dcdd5
6 years ago
Crag Wolfe 705625cee7 Eager load resource_properties_data in resource
Eager load resource_properties_data in resources in the typical
resource-loading scenarios where properties data will be
accessed. Thus, we can save an extra db query per resource when
loading all the resources in a stack, for instance. Fall back to lazy
loading properties data in other scenarios.

Also, the resource object doesn't need to store a copy of its
ResourcePropertiesData object in self.rsrc_prop_data, so don't.

Change-Id: Ib7684af3fe06f818628fd21f1216de5047872948
Closes-Bug: #1665503
6 years ago
Zane Bitter 118fc127b3 Fix a DB error when updating the resource attributes
This prevents the attributes from being stored sometimes, and is possibly
correlated with occasional random failures of the gate jobs.

Change-Id: I8f6567ec792bd015d59ffc04da9f2f37f5b0f13a
Related-Bug: #1660831
6 years ago
Crag Wolfe 0550659f5f Store resource attributes in the DB
Store resource attributes that may be cached in the DB, saving the
cost of re-resolving them later. This works for most resources,
specifically those that do not override the get_attribute() method.

Change-Id: I71f8aa431a60457326167b8c82adc03ca750eda6
Partial-Bug: #1660831
6 years ago
Jenkins df908f4417 Merge "Don't fetch stack before update in stack_update()" 6 years ago
Jenkins 3af92c84a7 Merge "Load "lighter" db resources when appropriate" 6 years ago
Crag Wolfe 450e3b2c09 Don't fetch stack before update in stack_update()
DB optimization where we no longer needlessly fetch a stack (and
its raw_template) before updating it.

Change-Id: I1ff5212f580e79838a8273065a94cb077e5aa2ff
Closes-Bug: #1672753
6 years ago
Crag Wolfe a6f4c6d2c6 Low-level db delete of events should be synchronous
Previously, synchronized_session=False was used in the call to prune
events. This was overly aggressive since it was possible (if rare)
that, during next db operation to delete resource_properties_data
rows, some of the referenced events could still have existed resulting
in a db referential integrity error.

Change-Id: I5c4cf6a162ff853f84d68e7b203ffa1aae684359
Closes-Bug: #1681772
6 years ago
Crag Wolfe 93fab308e8 Load "lighter" db resources when appropriate
Sometimes we know we will only access particular fields of a resource
object, rather than *all* of them. This commit allows the caller to
specify (optionally) the fields that should be populated when the
resource object is instantiated. This saves memory, trips to the db,
and in some cases avoids extra join queries (e.g. for or

Change-Id: I405888f46451d2657aa28f610f8ca555215ff5cf
Partial-Bug: #1680658
6 years ago
liyi 8f10215ffd Remove log translations
Log messages are no longer being translated. This removes all use of
the _LE, _LI, and _LW translation markers to simplify logging and to
avoid confusion with new contributions.


Change-Id: Ieec8028305099422e1b0f8fc84bc90c9ca6c694f
6 years ago
Jenkins 7a51c9c657 Merge "Squash database patches" 6 years ago
Jenkins 4dc101f10e Merge "Eager load event's resource prop. data when appropiate" 6 years ago
ricolin cb4640d67d Squash database patches
Remove patches up to the Mitaka release to speed up
initial deployment and tests, as we don't support
previous versions anymore.
Closes-bug: #1672630

Change-Id: I4f821176cb11ca3dc3af247238ba3c4fbc20a092
6 years ago
Jenkins cfbc624b75 Merge "Don't eager load raw_template in a few places" 6 years ago
Jenkins ab267e763a Merge "Remove unused event_get methods" 6 years ago
Crag Wolfe 283b2d3985 Eager load event's resource prop. data when appropiate
Eager load an event's resource properties data in the list_events
"show event" case, i.e. when an event uuid is specified. When showing
multiple events, the events' resource properties data should not be
eager loaded.

Change-Id: I12e371dcbdae398360e28200e68f4a75831f8f17
Closes-bug: #1665506
6 years ago
Crag Wolfe 019ae84d4f Remove unused event_get methods
Remove unused event get methods from the sqlalchemy db api and event

Change-Id: Ide5780d456755e180572194ad4d06277749c8c79
6 years ago
Crag Wolfe 02e7f70619 Don't eager load raw_template in a few places
Save the db and memory overhead associated with fetching a
raw_template when soft-deleting a stack or getting a root stack id.

Change-Id: I8cf575e4908764c4c5eb24185ec229254e30d58b
6 years ago
Crag Wolfe 9ebbd1e6f1 Data migration enabling storage of resource attributes
Working towards the goal of storing resource attributes in the db so
as to avoid re-resolving them when appropriate. Adds an 'attr_data'
object to the resource object, defined as a relationship on the
already existing resource_properties_data table.

Change-Id: I2104078d850da08b22547d7feab2bde00c543478
Partial-Bug: #1660831
6 years ago
Crag Wolfe 98505288b2 Add heat-manage subcommand to migrate legacy prop. data
Add a subcommand to heat-manage to migrate resource and events
properties data from the legacy db locations to the new. I.e., migrate
properties data out of the legacy columns in the resource and event
tables into the recently added resource_properties_data table. No
attempt at de-duplication between resources and events is made for the
migration: a new row is created in resource_properties_data for every
row that has legacy properties data in the resource or event tables.

Change-Id: I364d509c357539d1929eb2e40704e60049469ea2
6 years ago
Crag Wolfe 57c5aae88b De-duplicate properties_data between events, resources
Properties data (encrypted or not) is now stored in the
resource_properties_data table. We still support reading properties
data from the legacy locations (separate columns in the resource and
event tables) but all new properties data are written to

Instead of duplicating properties data for a given resource
and its events, we refer to the same resource_properties_data
object from both resources and events.

Notes about encrypting/decrypting properties data:

1. ResourcePropertiesData takes care of encrypting/decrypting, so
   consumers (events and resources) don't have to worry about it.
2. Previously (not anymore), heat.engine.resource would take care of
   the encrypting in _store() and _store_or_update().
3. Previously and currently, heat.object.resource decrypts the legacy
   properties_data field if needed.

Change-Id: I569e16d4a7d3f5ccc22d57fe881c4fd5994677ac
Closes-Bug: #1524013
6 years ago
Jenkins 2fa8cc09d3 Merge "Allow admins to get resource by physical resource id" 6 years ago
Tovin Seven 936b3c3714 Change order of init _facade and osprofiler
Initialize osprofiler in sqlalchemy/ failed with message:

'NoneType' object has no attribute 'get_engine'

Re-oder to init osprofiler after _facade will fix this issue.

Change-Id: I420f12f382c4712a692dfa440958b2e497dc22a1
Closes-Bug: #1657326
6 years ago
huangtianhua c4c5dfdb69 Allow admins to get resource by physical resource id
This change allows admin users to get resources from
other projects by physical resource id.

Partial-Bug: #1649759
Change-Id: I6e58ae8fd339bc50d41a5027cf610eb63dbd220c
6 years ago
Crag Wolfe 1ea6bb8bf6 Add the ResourcePropertiesData object
Prepare for the future when both resources and events refer to
ResourcePropertiesData rather than store the data themselves.

Change-Id: I0c5aefa9d73e1281e6477d46fe181d8948aee71b
7 years ago
Crag Wolfe d80c23f409 Add resource_properties_data data migration
Prepare for the future when both resources and events refer to
properties data in this table, rather than storing duplicate copies.

Change-Id: Idb7ed0ddd76d2780993a677114f8e10ccce56862
7 years ago
Crag Wolfe f3e6f47661 One transaction per encrypt/decrypt batch
Previously, there was one (potentially quite large) transaction across
all batches of templates and resources when encrypting/decrypting
parameters and properties data via the rpc api.

Change-Id: Iba10b7c9e88a1ae3d94ab9ddf4a8e362e72c0d50
7 years ago
Jenkins eb932ace15 Merge "Trivial refactor of session var in reset_stack_status" 7 years ago
Crag Wolfe 664735470b Refactor batch encrypt/decrpyts of params, props
A little more DRY.

Change-Id: I062987faa19db493f15ed71c207b7f81ef86cfd8
7 years ago
Thomas Herve 84067dba88 Remove db.api wrapper
The db.api module provides a useless indirection to the only
implementation we ever had, sqlalchemy. Let's use that directly instead
of the wrapper.

Change-Id: I80353cfed801b95571523515fd3228eae45c96ae
7 years ago
Jenkins 1190b1bead Merge "Handle ambiguous physical resource IDs" 7 years ago
Zane Bitter f310a1f6bc Handle ambiguous physical resource IDs
It's possible that we could end up with multiple resources with the same
physical resource ID, but that would be undetectable since we return only
one from the database layer. This change allows us to detect the problem an
return an error where the result is rendered ambiguous.

Change-Id: I2c5ddbe6731c33a09ec7c4a7b91dcfe414da4385
7 years ago
Crag Wolfe b158dcf78e More efficient db resources-per-stack count
Testing with devstack/mysql, given a stack with 350 resources,
stack_count_total_resources() now takes ~2.2ms vs ~7.6ms the old way.

Sqlalchemy was doing a bit more than we needed before, e.g.:
2016-12-03 16:41:58.455 2294 DEBUG sqlalchemy.orm.path_registry [req-b7cdfd64-4306-4145-adb2-dae13caaa9ab - - - - -] set 'memoized_setups' on path 'EntityRegistry((<Mapper at 0x44ddbd0; Resource>,))' to '{}' set /usr/lib64/python2.7/site-packages/sqlalchemy/orm/

Change-Id: I7def4093c7719a95e72d49925559bc926f168a01
7 years ago
Crag Wolfe f3302c38a7 Do not count events for every event created
We were a bit overzealous in our enforcement of
max_events_per_stack. Rather than count the number of events within a
stack every time an event is created, only do the count a small
percentage of the time (dependent on the inverse of
event_purge_batch_size). This also means that max_events_per_stack is
no longer a strict limit but a loose one.

Change-Id: I63b42a50cfc3ffa6b54413900b5e0a6c5c40943c
7 years ago
Crag Wolfe 882a640f18 Refactor purge_deleted, operate on batches of stacks
Avoid large sql "in" clauses by operating on smaller batches of stacks
at a time.

To avoid transaction overhead and contention on the resource table,
the first deletions occur outside of a transaction (are
autocommitted). This is OK because the purge is re-rentrant -- we
won't lose any stack_id's to delete if something goes wrong before the
conn.begin() block. That is, we will not orphan any rows if the purge
is run multiple times where an error occurs.

Change-Id: I9edf0558ed54820842193560e323df6501411d1d
7 years ago
Jenkins dd2e69a794 Merge "More efficient pruning of events" 7 years ago
Jenkins 19133101e1 Merge "Use enginefacade instead of db_session" 7 years ago
Crag Wolfe f1bc509702 More efficient pruning of events
Avoid a potentially large select in clause. synchronize_session may be
False here, as we don't have to worry about old stale pruned events in
the session (keeping in mind the prune gets triggered when new events
are created).

Change-Id: I27a00a1afb6518176c031551d6a74e8b12546ef2
7 years ago
Jenkins fcbe932772 Merge "Squash database patches" 7 years ago
Jenkins db83e3b636 Merge "Replace retrying with tenacity" 7 years ago
Jenkins 25e1677c40 Merge "Add Apache 2.0 license to source file" 7 years ago
Boden R 756dda72d4 Replace retrying with tenacity
We are replacing all usages of the 'retrying' package with
'tenacity' as the author of retrying is not actively maintaining
the project. Tenacity is a fork of retrying, but has improved the
interface and extensibility (see [1] for more details). Our end
goal here is removing the retrying package from our requirements.

Tenacity provides the same functionality as retrying, but has the
following major differences to account for:
- Tenacity uses seconds rather than ms as retrying did.
- Tenacity has different kwargs for the decorator and
Retrying class itself.
- Tenacity has a different approach for retrying args by
using classes for its stop/wait/retry kwargs.
- By default tenacity raises a RetryError if a retried callable
times out; retrying raises the last exception from the callable.
Tenacity provides backwards compatibility here by offering
the 'reraise' kwarg.
- Tenacity defines 'time.sleep' as a default value for a kwarg.
That said consumers who need to mock patch time.sleep
need to account for this via mocking of time.sleep before
tenacity is imported.
- For retries that check a result, tenacity will raise if the retried
function raises, whereas retrying retried on all exceptions.

This patch updates all usages of retrying with tenacity.
Unit tests will be added/removed where applicable.


Closes-Bug: #1635388

Change-Id: Iec0822cc0d5589b04c1764db518478d286455031
7 years ago
Kanagaraj Manickam 7dd06fc2c5 Adds 5 backport db migration place holder for newton
Db migration version from 74 to 78 are made as place
holder of db backporting for stable/newton.


Change-Id: I353a7452d724c1cb1ae2892dc561b3c2afe8b7cf
7 years ago
Kanagaraj Manickam f9fe6abdf9 Squash database patches
Remove patches up to the Liberty release to speed up
initial deployment and tests, as we don't support
previous versions anymore.

Closes-bug: #1633975

Change-Id: Icdab0db3f3371cd4eb8a8fb11cbc2328c0a830e7
7 years ago
Zane Bitter 260b79ed28 Don't always eagerly load the raw_template for a stack
Always loading the raw template in situations where we didn't need it -
e.g. in identify_stack, where we just want the name + id (given one of
them), or when getting the summary stack list - uses up DB bandwidth and
memory unnecessarily.

This partially reverts commit 3ab0ede98c.

* The eager_load option to get_stack() is reinstated, but with the default
  flipped to True. In places where we explicitly do not want to load the
  template, we pass False.
* stack_get_by_name() no longer eagerly loads the template. There were no
  instances of this where we subsequently use the template.
* stack_get_all() acquires an eager_load option, with the default set to
  False. Direct users of objects.stack.Stack.get_all() will not eagerly
  load by default, but users of engine.stack.Stack.load_all() will get the
  template eagerly loaded. This practically always corresponds to what you

Change-Id: I1f156c25ea26322be5b606a61dd77d80cadd65fc
Related-Bug: #1626675
7 years ago
Cao Xuan Hoang 2e40bccd15 Add Apache 2.0 license to source file
As per OpenStack licensing guide lines [1]:
[H102 H103] Newly contributed Source Code should be licensed under
the Apache 2.0 license.
[H104] Files with no code shouldn't contain any license header nor
comments, and must be left completely empty.


Change-Id: I82387fec7ac94001a6c2379321ebf1f2e3f40c12
7 years ago