Commit Graph

690 Commits (abfec544d7938ad8b576e1d0fa78f8a5dca1df00)

Author SHA1 Message Date
rabi abfec544d7 Don't allow nested or stacks in FAILED state to be migrated
Stack and its nested stacks should be in *_COMPLETE state for
it to be migrated. Also, we should not allow migration of a
nested stack.

This also changes stack_get_all_by_owner_id() to not select the
backup stacks. This seems to be used only in convergence code
path and migrate_convergence_1().

Change-Id: Icd54465d0c593557a12d853ddee4ee8ce6483499
Closes-Bug: #1767962
Story: #1767962
Task: #17363
2018-05-11 11:06:23 +05:30
Zuul 49a5e4d676 Merge "Use stack_id of None for service timer" 2018-05-01 22:27:42 +00:00
Thomas Herve 9f73a232d7 Return nested parameters for resource group.
This refactors the building of schema from parameter validation to use a
new method (which doesn't keep stacks in memory), and use that new
method for providing proper schema for resource group when the size is

Change-Id: Id3020e8f3fd94e2cef413d5eb9de9d1cd16ddeaa
Closes-Bug: #1751074
Closes-Bug: #1626025
2018-02-26 14:57:01 +01:00
rabi 8db1b3ea41 Remove stack watch service
This removes the rpc api and related code.

Change-Id: Ib89bcc3ff6a542f49467e2ad6c7e2a716a0dc2b4
Partial-Bug: #1743707
2018-01-28 09:35:05 +05:30
Zuul 8f6c09df0f Merge "Speed up event listing with nested stacks" 2017-12-06 17:40:25 +00:00
Zuul 10179a88ec Merge "Eager-load properties for nested resource list" 2017-12-06 11:25:51 +00:00
Zuul 3fb7b4eb2e Merge "[policy in code] part3 (resource types)" 2017-12-06 06:43:05 +00:00
Thomas Herve e1a26279ce Fix reset_stack_status
Fix reset_stack_status on legacy, by using a different session for each
stack reset, and handling lock duplicate errors.

Closes-Bug: #1735755
Change-Id: I6bcd7448052e86ec3e4eb4c49ef3139c20d4f919
2017-12-01 17:15:55 +01:00
ricolin 46f0e16d11 [policy in code] part3 (resource types)
Allow use policy in code to resource type's rule.
Also add test for override the in-code resource type rule in json
Partially-Implements: bp policy-in-code

Change-Id: Id6c21732e66de6c421427ded98de52f5da0a4db2
2017-12-01 01:34:55 +08:00
Zane Bitter 23e06f66b9 Speed up event listing with nested stacks
When handling the command "openstack stack event list --nested-depth=n", we
obtain the list of nested stacks by querying the database for all resources
that share the root stack ID. However, since all we're getting is the stack
IDs, there's no need to query all of the fields and construct a versioned
object for each resource. Just get the set of stack IDs.

Change-Id: I12155433e2ac1af919aa4b5e780fb965cd5885d8
Related-Bug: #1588561
2017-11-20 15:06:13 -05:00
Zane Bitter dbc852b47a Eager-load properties for nested resource list
When doing "openstack stack resource list --nested-depth=n", we were
lazy-loading the resources' properties data. This is expensive, especially
when there are a large number of resources. Eager-load the data, as we
always use it to show the resources.

For consistency, always eager-load the resource 'data' (even in
resource_get), because the Resource versioned object accesses it

Change-Id: Idb871fddf77bf24828878c315e19e200c28841be
Related-Bug: #1665503
2017-11-20 14:29:38 -05:00
Steven Hardy 313faad5f1 Return Environment from validate_template
This is to enable preview of the merged environment
without merging the environment on the client side.

Related-Bug: #1635409
Change-Id: I7ec3af729a65164230153021f438bf226cc5e858
2017-11-14 17:35:52 +00:00
rabi 01b5878129 Add REST api support for cancel without rollback
We already have REST api support for cancelling a
UPDATE_IN_PROGRESS stack with rollback. This adds a
new action 'cancel_without_rollback' to allow for
canceling a create/update in_progress stack without


Change-Id: I6c6ffa0502ab8745cfb2f9c5ef263f1e02dfc4ca
Closes-Bug: #1709041
2017-11-08 17:08:43 +05:30
rabi 3adca03db6 Set stack.thread_group_mgr for cancel_update
In change I84d2b34d65b3ce7d8d858de106dac531aff509b7, we changed to
call self._converge_create_or_update() in a sub-thread. However,
thread_group_mgr is not set for cancel_update (with rollback),
which in turn calls converge_stack.

This also enables test_cancel_update_server_with_port, as
bug #1607714 seems to be fixed now.

Change-Id: Ie674fd556418f6aa8e79654458cbe43648851db2
Closes-Bug: #1713952
2017-11-03 09:14:46 +05:30
Zane Bitter 8aca0e99ff Use stack_id of None for service timer
In order to keep the engine service alive, we add a timer that periodically
does nothing. Calls to add_timer() require a stack_id, and currently we
pass cfg.CONF.periodic_interval. This is highly misleading, because the
value you pass for the stack_id has no effect on the interval. The cause
was a copy-paste error in 07884448fe, when
the code changed from calling ThreadGroup.add_timer() to
ThreadGroupManager.add_timer(). Use None as the stack ID instead.

Change-Id: Ia24a0d3ae9a0295fc811eb5300656399f426408b
2017-10-23 14:32:39 -04:00
Zane Bitter 9b5f66351f Don't load resource data for list_outputs
This RPC call only generates a list of the outputs defined in the template,
not their values, so don't load the resource data needed to calculate the
output values (which can be very slow).

Also, explicitly pass resolve_value=False instead of relying on the default
argument (which is different for format_stack_output() and
format_stack_outputs()), to reduce future confusion.

Change-Id: I79aae94b6552d465db6707cd4a40cd53ff18455b
Closes-Bug: #1719340
2017-09-25 12:49:45 -04:00
Zane Bitter 49d833f9ac Speed up show-stack with outputs in legacy path
When we show a stack including the outputs, we calculate all of the
resource attributes that are referenced anywhere in the stack. In
convergence, these are either already cached (and therefore fast) or need
to be cached (and therefore the initial slowness will pay off in future).
This isn't the case in the legacy path though, since we are not doing
caching of attributes in the database in that path. So this is
unnecessarily calculating all of the referenced attribute values, which are
potentially very slow to get.

For legacy stacks, only calculate the attribute values needed to show the

Change-Id: I35800c7f87b58daf05cbabd05bcbcd75d0c0fadb
Partial-Bug: #1719333
2017-09-25 10:53:38 -04:00
Jenkins bef2455080 Merge "Add converge flag in stack update for observing on reality" 2017-08-10 21:56:40 +00:00
Jenkins 91bb3e8781 Merge "Release stack lock after export stack" 2017-08-07 08:26:26 +00:00
ricolin 552f94b928 Add converge flag in stack update for observing on reality
Add converge parameter for stack update API and RPC call,
that allow triggering observe on reality. This will be
triggered by API call with converge argument (with True
or False value) within. This flag also works for resources
within nested stack.
Implements bp get-reality-for-resources

Change-Id: I151b575b714dcc9a5971a1573c126152ecd7ea93
2017-08-07 05:39:29 +00:00
Jenkins 9b6b65b3d1 Merge "Only check service availability during validation" 2017-08-04 14:34:14 +00:00
Jenkins 2963d81613 Merge "Don't get resource twice in resource_signal()" 2017-08-03 06:07:06 +00:00
Zane Bitter 165fb8ace4 Only check service availability during validation
There's no need to check it every time we create a Resource object.

Change-Id: I92082d64e6060c2d8d29d866fdabeea2846253e9
Closes-Bug: #1706197
2017-08-01 11:47:14 -04:00
Jenkins 28cfcd3177 Merge "Update StackDefinition resource data when getting outputs" 2017-07-26 05:43:01 +00:00
Jenkins b4d88e05e9 Merge "Update StackDefinition resource data on metadata refresh" 2017-07-25 18:02:38 +00:00
Zane Bitter d45de801b2 Don't get resource twice in resource_signal()
For no reason at all, we were calling Stack.resource_get() twice in
handle_signal(). In the worst case, this function does a database query
that loads every resource.

Change-Id: I9e3f1e77146c71ae35a0896f915f6f9c9fac3fd3
Related-Bug: #1635610
2017-07-24 17:31:27 -04:00
Jenkins 0518e906ed Merge "Keep existing stack tags for patch update" 2017-07-24 13:52:51 +00:00
Zane Bitter cc3c60647b Update StackDefinition resource data when getting outputs
Before returning the stack outputs, update the StackDefinition with the
resource data needed to resolve them. This will enable us to switch to
resolving outputs via the StackDefinition in a subsequent patch.

Change-Id: Ide3920385d9c4841c18024e1eb1456eed5c8eef5
Partially-Implements: blueprint stack-definition
2017-07-21 10:30:34 -04:00
Zane Bitter 119f3e4aa5 Update StackDefinition resource data on metadata refresh
After refreshing the metadata of a resource its attributes may change, so
we need to update the StackDefinition with the new data.

Change-Id: Iec521317bffc0b381a0d965daef7939b1146ce50
Partially-Implements: blueprint stack-definition
2017-07-21 10:30:34 -04:00
Zane Bitter 76aa7c87cb Check for circular dependencies in template validation
We ought to iterate over the dependency list in the validate_template call,
to check for any circular dependencies. This check was removed by
d12cbe7959 in order to avoid trying to
calculate implicit dependencies added by resource plugins (as e.g. many
Neutron resources do) at a time when we don't have sufficient data to do a
good job of it anyway.

However, the previous patch d12cbe7959 split
apart the calculation of explicit and implicit dependencies, so we can
easily do this while still checking for circular references in the
explicit dependencies. This patch splits the dependency calculation into
two phases - the first using only data available from the template, and the
second querying the resources for implicit dependencies.

Use strict_validate = False as the criterion for when to avoid calculating
implicit dependencies, since this corresponds exactly to when we don't have
enough data to do so.

Change-Id: I21c63dcc8d1cad20dbc237b472670753779b0ff8
Closes-bug: #1691575
2017-07-21 10:27:52 -04:00
Jenkins 5e283bcf08 Merge "Always cache attributes in outputs in Resource.node_data()" 2017-07-19 08:28:06 +00:00
Zane Bitter cfff39ce68 Always cache attributes in outputs in Resource.node_data()
Ensure that attributes that are referenced in outputs get cached (and
therefore stored in the database) even when they are not referenced by
other resources.

Change-Id: I667ab04f91edddef5c5dbec0a89d465110c312b4
Closes-Bug: #1660831
2017-07-10 22:06:45 -04:00
Zane Bitter dcf2463f1f Fix ResourceActionNotSupported errors in log
In 37ef669705 (in Kilo) we made signalling
a resource asynchronous because things like e.g. scaling an autoscaling
group were too slow to do synchronously (bug 1394095).

However, for resources that don't have a handle_signal() method at all,
this means that if a signal other than one handled by Heat (e.g. to
clear a hook) is received then the user won't be notified of the error.
There will instead be an error message in the logs warning of an
unhandled exception in a thread.

(Previously, similar problems occurred if the user was attempting to
clear a hook and some error occurred. This has since been fixed as bug
1472515, in Liberty.)

If there's no handle_signal() method defined, then calling
Resource.signal() is quick (i.e. it's only going to clear a hook, or
raise an error). So handle this case synchronously. This ensures that
any error is reported to the user and not in the log.

Change-Id: I917b4c7d4ab2cde46148197a3a232d5dc70f63d7
Related-Bug: #1472515
2017-07-07 20:38:17 -04:00
rabi ed0beb6871 Keep existing stack tags for patch update
When updating a stack, we overwrite the existing tags.
So if no tags are specified during update, all existing
tags are removed. It would atleast be good to keep the
existing tags when doing a patch update (with `--existing`).

Change-Id: I910fe5237605405791eb6daa81c422bf7fd9eaa1
Closes-Bug: #1701098
2017-07-04 12:15:00 +05:30
Thomas Herve fda92763c9 Don't store attributes if resource not stored
If stack-show is called at an inopportune moment in convergence, we can
attempt to store the attributes of a resource that isn't actually in the
database.  This causes occasional gate failures due to an assert.

Closes-Bug: #1701569
Change-Id: I50de08ae6b40958781ef4223c82a5c195b0757bf
2017-06-30 22:15:24 +02:00
Jenkins ff8d741dd5 Merge "Fix snapshot delete in convergence" 2017-06-27 09:31:24 +00:00
Thomas Herve df1708b1a8 Fix snapshot delete in convergence
The patch at I1cb321a3878a0abce9b41832f76bf77c25bf7cb4 properly deleted
the snapshots from the database, but as delete in convergence sets the
template to an empty template, stack.resources is empty. This works
around the problem by deleting the snapshots beforehand.

Related-Bug: #1508299
Change-Id: Id1c2c1a293fdcda07c527f29fedc00b716b303bc
2017-06-26 09:34:15 +02:00
Crag Wolfe 0550659f5f Store resource attributes in the DB
Store resource attributes that may be cached in the DB, saving the
cost of re-resolving them later. This works for most resources,
specifically those that do not override the get_attribute() method.

Change-Id: I71f8aa431a60457326167b8c82adc03ca750eda6
Partial-Bug: #1660831
2017-06-21 20:18:46 -04:00
Jenkins 7698eb0501 Merge "Load all templates for generating parameter schema" 2017-06-19 04:57:30 +00:00
Jenkins 06b3ba11a2 Merge "Fix restore with convergence" 2017-06-05 03:51:11 +00:00
Thomas Herve 7221cd2eba Load all templates for generating parameter schema
To be able to handle parameter defaults properly, we need to load all
possible templates when merging the environments, to generate an
approximate schema.

Closes-Bug: #1669571
Change-Id: Idab781d3e6af92990c357a727aa60ec0aa8f5f05
2017-05-30 13:15:57 +02:00
Saravanan KR b15b7e568c Add ParameterGroups for the nested stack during stack.validate
ParameterGroups is added only for the parent stack and all
the nested stack are missing this information. Added
ParameterGroups for all the nested stack also, if present.

Change-Id: I032144733bde916f8de8644121b9fb1ef29baef2
2017-05-18 15:10:32 +05:30
Thomas Herve 42e3d02dbf Fix restore with convergence
Handle the restore operation as a normal convergence update instead of a
legacy one.

Change-Id: I6ee46cdf7a8fdf89c58c9812d08af21c97fb0f9e
Related-Bug: #1687006
2017-05-17 16:07:35 +02:00
ChangBo Guo(gcb) 381a84a4e8 Remove usage of parameter enforce_type
Oslo.config deprecated parameter enforce_type and change its default
value to True in Ifa552de0a994e40388cbc9f7dbaa55700ca276b0. Remove
the usage of it to avoid DeprecationWarning: "Using the 'enforce_type'
argument is deprecated in version '4.0' and will be removed in version
'5.0': The argument enforce_type has changed its default value to True
and then will be removed completely."

Change-Id: I91b0f0a52b5ce8654702510eed76d5dea8cc8fe4
Related--Bug: #1517839
2017-05-08 17:55:32 +08:00
Zane Bitter 41ae901c3b Get rid of resource_validate flag in Stack
Why pass a parameter to a method to influence what it does when you can set
some state on the object that the programmer always has to keep in their
head to figure out what any given call will do?

Change-Id: I020238de0a351dd2eaf51c6f970ae36ad0f931be
2017-05-01 17:26:05 -04:00
Zane Bitter ea2673fb9a Validate property values in nested stacks again
In ced6f78aa0 we stopped doing validations of
nested stacks at stack creation time, on the assumption that they had been
validated when the parent stack was created. This assumption was incorrect;
for children of the stack being created, the strict_validate global is
always set during validation, so property values of resources will not be
validated until it comes time to create the resource.

Instead, prevent only redundant non-strict validations of stacks at nested
depth 2 and greater. This means that every stack other than the root will
be validated exactly twice - once without validating property values when
the root is created, and again including property validation when the
nested stack itself is created.

Most of the performance benefits should remain; in the case of a large
ResourceGroup using index substitution, we will now have to validate a lot
of nearly-identical resource properties, however we still will not load
into memory and validate a nested stack for each one as we originally did.
Since that happens synchronously, it was likely the main contributor to RPC
timeouts when dealing with large scaling groups. (During the validation at
the creation of the root stack, only a single member of a ResourceGroup is
validated even when index substitution is used. For scaling groups with
identical members, only one member is validated since

This change reverts commit ced6f78aa0.

Change-Id: I97cf789cee75931edef58b78c88f02da204d2a08
Closes-Bug: #1675589
Related-Bug: #1645336
2017-03-31 15:57:56 -04:00
huangtianhua 736be1fe27 Release stack lock after export stack
Change-Id: I69e3c8f20530a538bd39f3bc13cd86c923ce8f8f
Closes-Bug: #1664159
2017-03-29 15:01:13 +00:00
Zane Bitter 331df3abe8 Use pydoc for formatting docstrings
pydoc is part of the standard library and is much more robust at formatting
docstings than any trivial hand-rolled munger could be. For example, it
correctly preserves indentation within the docstring. Use it when
generating descriptions from docstrings for the API.

Change-Id: Ib565a64d990a45c652a9475255c37805f86070b4
2017-03-28 15:36:53 +00:00
liyi 8f10215ffd Remove log translations
Log messages are no longer being translated. This removes all use of
the _LE, _LI, and _LW translation markers to simplify logging and to
avoid confusion with new contributions.


Change-Id: Ieec8028305099422e1b0f8fc84bc90c9ca6c694f
2017-03-25 17:11:50 +08:00
Crag Wolfe 3e30b16624 Omit rsrc prop data in rpc api for multiple events
The list_events rpc api no longer includes resource properties data
when returning multiple events. In the "show event" case where an event
uuid is included in the list_events request, the resource properties data
for that single event continues to be included in the output.

Note that there is no change in behaviour on the heat-api side (see
heat/api/openstack/v1/ Previously when listing multiple
events, it had just been ignoring the resource properties data it
fetched from the list_events rpc api, not including it in the output
it returned (e.g., to python-heatclient).

Change-Id: I7ac83d848cdd0e6c313870c0a4d59a5d9b2301f5
Partial-Bug: #1665506
2017-03-01 10:05:37 -05:00