Use a single transaction to create the replacement resource and set it as
the replaced_by link in the old resource. Also, ensure that no other
traversal has taken a lock on the old resource before we modify it.
If we end up bailing out and not creating a replacement or sending an RPC
message to check it, make sure we retrigger any new traversal.
Change-Id: I23db4f06a4060f3d26a78f7b26700de426f355e3
Closes-Bug: #1727128
Make sure to delete the ResourcePropertiesData row
if the resource 'attr_data_id' has been set already.
Change-Id: I2df5dfc72d051b6619c769c1260ff24b016bc294
Closes-Bug: #1708378
Eager load resource_properties_data in resources in the typical
resource-loading scenarios where properties data will be
accessed. Thus, we can save an extra db query per resource when
loading all the resources in a stack, for instance. Fall back to lazy
loading properties data in other scenarios.
Also, the resource object doesn't need to store a copy of its
ResourcePropertiesData object in self.rsrc_prop_data, so don't.
Change-Id: Ib7684af3fe06f818628fd21f1216de5047872948
Closes-Bug: #1665503
This prevents the attributes from being stored sometimes, and is possibly
correlated with occasional random failures of the gate jobs.
Change-Id: I8f6567ec792bd015d59ffc04da9f2f37f5b0f13a
Related-Bug: #1660831
Store resource attributes that may be cached in the DB, saving the
cost of re-resolving them later. This works for most resources,
specifically those that do not override the get_attribute() method.
Change-Id: I71f8aa431a60457326167b8c82adc03ca750eda6
Partial-Bug: #1660831
DB optimization where we no longer needlessly fetch a stack (and
its raw_template) before updating it.
Change-Id: I1ff5212f580e79838a8273065a94cb077e5aa2ff
Closes-Bug: #1672753
Previously, synchronized_session=False was used in the call to prune
events. This was overly aggressive since it was possible (if rare)
that, during next db operation to delete resource_properties_data
rows, some of the referenced events could still have existed resulting
in a db referential integrity error.
Change-Id: I5c4cf6a162ff853f84d68e7b203ffa1aae684359
Closes-Bug: #1681772
Sometimes we know we will only access particular fields of a resource
object, rather than *all* of them. This commit allows the caller to
specify (optionally) the fields that should be populated when the
resource object is instantiated. This saves memory, trips to the db,
and in some cases avoids extra join queries (e.g. for resource.data or
resource.rsrc_prop_data).
Change-Id: I405888f46451d2657aa28f610f8ca555215ff5cf
Partial-Bug: #1680658
Remove patches up to the Mitaka release to speed up
initial deployment and tests, as we don't support
previous versions anymore.
Closes-bug: #1672630
Change-Id: I4f821176cb11ca3dc3af247238ba3c4fbc20a092
Eager load an event's resource properties data in the list_events
"show event" case, i.e. when an event uuid is specified. When showing
multiple events, the events' resource properties data should not be
eager loaded.
Change-Id: I12e371dcbdae398360e28200e68f4a75831f8f17
Closes-bug: #1665506
Save the db and memory overhead associated with fetching a
raw_template when soft-deleting a stack or getting a root stack id.
Change-Id: I8cf575e4908764c4c5eb24185ec229254e30d58b
Working towards the goal of storing resource attributes in the db so
as to avoid re-resolving them when appropriate. Adds an 'attr_data'
object to the resource object, defined as a relationship on the
already existing resource_properties_data table.
Change-Id: I2104078d850da08b22547d7feab2bde00c543478
Partial-Bug: #1660831
Add a subcommand to heat-manage to migrate resource and events
properties data from the legacy db locations to the new. I.e., migrate
properties data out of the legacy columns in the resource and event
tables into the recently added resource_properties_data table. No
attempt at de-duplication between resources and events is made for the
migration: a new row is created in resource_properties_data for every
row that has legacy properties data in the resource or event tables.
Change-Id: I364d509c357539d1929eb2e40704e60049469ea2
Properties data (encrypted or not) is now stored in the
resource_properties_data table. We still support reading properties
data from the legacy locations (separate columns in the resource and
event tables) but all new properties data are written to
resource_properties_data.
Instead of duplicating properties data for a given resource
and its events, we refer to the same resource_properties_data
object from both resources and events.
Notes about encrypting/decrypting properties data:
1. ResourcePropertiesData takes care of encrypting/decrypting, so
consumers (events and resources) don't have to worry about it.
2. Previously (not anymore), heat.engine.resource would take care of
the encrypting in _store() and _store_or_update().
3. Previously and currently, heat.object.resource decrypts the legacy
properties_data field if needed.
Change-Id: I569e16d4a7d3f5ccc22d57fe881c4fd5994677ac
Closes-Bug: #1524013
Initialize osprofiler in sqlalchemy/api.py failed with message:
'NoneType' object has no attribute 'get_engine'
Re-oder to init osprofiler after _facade will fix this issue.
Change-Id: I420f12f382c4712a692dfa440958b2e497dc22a1
Closes-Bug: #1657326
This change allows admin users to get resources from
other projects by physical resource id.
Partial-Bug: #1649759
Change-Id: I6e58ae8fd339bc50d41a5027cf610eb63dbd220c
Prepare for the future when both resources and events refer to
ResourcePropertiesData rather than store the data themselves.
Change-Id: I0c5aefa9d73e1281e6477d46fe181d8948aee71b
Prepare for the future when both resources and events refer to
properties data in this table, rather than storing duplicate copies.
Change-Id: Idb7ed0ddd76d2780993a677114f8e10ccce56862
Previously, there was one (potentially quite large) transaction across
all batches of templates and resources when encrypting/decrypting
parameters and properties data via the rpc api.
Change-Id: Iba10b7c9e88a1ae3d94ab9ddf4a8e362e72c0d50
The db.api module provides a useless indirection to the only
implementation we ever had, sqlalchemy. Let's use that directly instead
of the wrapper.
Change-Id: I80353cfed801b95571523515fd3228eae45c96ae
It's possible that we could end up with multiple resources with the same
physical resource ID, but that would be undetectable since we return only
one from the database layer. This change allows us to detect the problem an
return an error where the result is rendered ambiguous.
Change-Id: I2c5ddbe6731c33a09ec7c4a7b91dcfe414da4385
Testing with devstack/mysql, given a stack with 350 resources,
stack_count_total_resources() now takes ~2.2ms vs ~7.6ms the old way.
Sqlalchemy was doing a bit more than we needed before, e.g.:
2016-12-03 16:41:58.455 2294 DEBUG sqlalchemy.orm.path_registry [req-b7cdfd64-4306-4145-adb2-dae13caaa9ab - - - - -] set 'memoized_setups' on path 'EntityRegistry((<Mapper at 0x44ddbd0; Resource>,))' to '{}' set /usr/lib64/python2.7/site-packages/sqlalchemy/orm/path_registry.py:63
Change-Id: I7def4093c7719a95e72d49925559bc926f168a01
We were a bit overzealous in our enforcement of
max_events_per_stack. Rather than count the number of events within a
stack every time an event is created, only do the count a small
percentage of the time (dependent on the inverse of
event_purge_batch_size). This also means that max_events_per_stack is
no longer a strict limit but a loose one.
Change-Id: I63b42a50cfc3ffa6b54413900b5e0a6c5c40943c
Avoid large sql "in" clauses by operating on smaller batches of stacks
at a time.
To avoid transaction overhead and contention on the resource table,
the first deletions occur outside of a transaction (are
autocommitted). This is OK because the purge is re-rentrant -- we
won't lose any stack_id's to delete if something goes wrong before the
conn.begin() block. That is, we will not orphan any rows if the purge
is run multiple times where an error occurs.
Change-Id: I9edf0558ed54820842193560e323df6501411d1d
Avoid a potentially large select in clause. synchronize_session may be
False here, as we don't have to worry about old stale pruned events in
the session (keeping in mind the prune gets triggered when new events
are created).
Change-Id: I27a00a1afb6518176c031551d6a74e8b12546ef2
We are replacing all usages of the 'retrying' package with
'tenacity' as the author of retrying is not actively maintaining
the project. Tenacity is a fork of retrying, but has improved the
interface and extensibility (see [1] for more details). Our end
goal here is removing the retrying package from our requirements.
Tenacity provides the same functionality as retrying, but has the
following major differences to account for:
- Tenacity uses seconds rather than ms as retrying did.
- Tenacity has different kwargs for the decorator and
Retrying class itself.
- Tenacity has a different approach for retrying args by
using classes for its stop/wait/retry kwargs.
- By default tenacity raises a RetryError if a retried callable
times out; retrying raises the last exception from the callable.
Tenacity provides backwards compatibility here by offering
the 'reraise' kwarg.
- Tenacity defines 'time.sleep' as a default value for a kwarg.
That said consumers who need to mock patch time.sleep
need to account for this via mocking of time.sleep before
tenacity is imported.
- For retries that check a result, tenacity will raise if the retried
function raises, whereas retrying retried on all exceptions.
This patch updates all usages of retrying with tenacity.
Unit tests will be added/removed where applicable.
[1] https://github.com/jd/tenacity
Closes-Bug: #1635388
Change-Id: Iec0822cc0d5589b04c1764db518478d286455031
Db migration version from 74 to 78 are made as place
holder of db backporting for stable/newton.
Closes-bug:#1633977
Change-Id: I353a7452d724c1cb1ae2892dc561b3c2afe8b7cf
Remove patches up to the Liberty release to speed up
initial deployment and tests, as we don't support
previous versions anymore.
Closes-bug: #1633975
Change-Id: Icdab0db3f3371cd4eb8a8fb11cbc2328c0a830e7
Always loading the raw template in situations where we didn't need it -
e.g. in identify_stack, where we just want the name + id (given one of
them), or when getting the summary stack list - uses up DB bandwidth and
memory unnecessarily.
This partially reverts commit 3ab0ede98c.
* The eager_load option to get_stack() is reinstated, but with the default
flipped to True. In places where we explicitly do not want to load the
template, we pass False.
* stack_get_by_name() no longer eagerly loads the template. There were no
instances of this where we subsequently use the template.
* stack_get_all() acquires an eager_load option, with the default set to
False. Direct users of objects.stack.Stack.get_all() will not eagerly
load by default, but users of engine.stack.Stack.load_all() will get the
template eagerly loaded. This practically always corresponds to what you
want.
Change-Id: I1f156c25ea26322be5b606a61dd77d80cadd65fc
Related-Bug: #1626675
As per OpenStack licensing guide lines [1]:
[H102 H103] Newly contributed Source Code should be licensed under
the Apache 2.0 license.
[H104] Files with no code shouldn't contain any license header nor
comments, and must be left completely empty.
[1] http://docs.openstack.org/developer/hacking/#openstack-licensing
Change-Id: I82387fec7ac94001a6c2379321ebf1f2e3f40c12
liyi