Commit Graph

36 Commits (4a500125b350b46dee0d3c9f01c3cac7223d9c80)

Author SHA1 Message Date
Tanvir Talukder d6a90cc6ac Fix for resources stuck in progress after engine crash
When a stack is IN_PROGRESS and an UPDATE or RESTORE is called
after an engine crash, we set status of the stack and all of its

Change-Id: Ia3adbfeff16c69719f9e5365657ab46a0932ec9b
Closes-Bug: #1570576
6 years ago
Thomas Herve 84067dba88 Remove db.api wrapper
The db.api module provides a useless indirection to the only
implementation we ever had, sqlalchemy. Let's use that directly instead
of the wrapper.

Change-Id: I80353cfed801b95571523515fd3228eae45c96ae
7 years ago
Zane Bitter ced6f78aa0 Avoid re-validating resources in nested stacks
Before creating a nested stack we validate the resources in the parent
stack. Therefore it's wasteful to validate them again upon creating the
stack - prior to this patch the number or validation calls for each
resource would be equal to the depth of the stack in the tree (so twice for
a child of the root, and three times for a grandchild). With this patch,
each resource in a nested stack is validated only once, by the parent

ResourceGroup and its subclasses are currently the only StackResources that
do not validate all of their members; they validates a stack containing a
single representative resource. This means that in the case of e.g. index
substitution, the nested stack could be created without having validated
some resources. However, in practice this is unlikely to lead to validity
problems not causing a failure until later than they previously would have.

Change-Id: Iaa36ae5543b3ce30ae8df5a05b48fe987bc0ffdc
Closes-Bug: #1645336
7 years ago
Zane Bitter e2ba3390cd Don't acquire the lock when cancelling a stack
We used to try to acquire the stack lock in order to find out which engine
to cancel a running update on, in the misguided belief that it could never
succeed. Accordingly, we never released the lock.

Since it is entirely possible to encounter a race where the lock has
already been released, use the get_engine_id() method instead to look up
the ID of the engine holding the lock without attempting to acquire it.

Change-Id: I1d026f8c67dddcf840ccbc2f3f1537693dc266fb
Closes-Bug: #1624538
7 years ago
rabi 35a2dd9196 Refactor to use param_schemata with env merge
This refactors the service module to use template_schemata
for environment merging.

Change-Id: I86a28d0496e05f978fa1b734818404639541761b
Blueprint: environment-merging
7 years ago
rabi 59b2a559ec Move merge_environment to environment_util module
This moves the merge_environment utility function
from to

Change-Id: Ia005cf47d5e655e60359f8da397a712e749ce13c
Blueprint: environment-merging
7 years ago
Zane Bitter b12ec30368 Use a queue instead of an event to send messages
This has a couple of effects. One is that we can send more than one message
to an action (since the action need not terminate immediately upon
receiving a message). The other is that we only ever receive the message
once, instead of every time we check for it once it has been sent.

Change-Id: Ib3b33534c0e8b03dfbbb1acde6f7b2d3dcfa82b6
7 years ago
Steven Hardy b47c002556 Keep encrypted_param_names environment internal to heat
Currently this is reflected in the user_env_as_dict output, which means
not only is it stored in the DB (which we want), but also passed around
to nested stack (which we don't want because it's derived from the parent
stack hidden parameters), and also to the user via stack environment show
(the API returns it but heatclient currently hides it), which we also don't
want because it's not a valid key in user-provided environments.

Change-Id: If5821ccb4a8bbf98012a2541ddd3c8e91455e5cc
Closes-Bug: #1590507
7 years ago
Steve Baker 419c9ab994 del _session(), use context.session directly
There should always be a context supplied, lets use its session
directly so we can be sure that the session is the shared one from the

Any remaining functions which call get_session() directly will be
handled in their own changes as some require special behaviour.

Change-Id: I3e52d2ff6bb1014f201715a1b1ba32c122c2391d
Related-Bug: #1479723
7 years ago
Rabi Mishra ef4cf621fb Move some functions to service_utils
This moves some utility functions to service_utils
from stack_lock.

Change-Id: Iebc39a7f7b15dd5eea4be3cd3fdaea22607dd72e
7 years ago
Crag Wolfe fef94d0d73 De-duplicate raw_template.files
Save space in the db by allowing templates (especially nested
templates) to reference the same template files dictionary objects,
now stored in the table raw_template_files. Also, cache template files
per heat-engine process as a RAM/performance optimization.

Also, begin to allow for future rolling upgrades with

Change-Id: I1842ea6ba6773a135b1007b793a5f0884417747d
Closes-Bug: 1570983
7 years ago
Zane Bitter 9a650a5e2f RPC API: Add a template_id parameter to stack_create/update
This allows us to pre-store a template in the database before
creating/updating a stack over RPC, so that we don't need to send the
contents over RPC. This will help us safely de-duplicate files dicts in
the database, reduce the size of messages, and prevent unnecessary
copies of files (in particular) being loaded into memory.

Change-Id: Id1990d5cbac8b3954ce224af73425d4fd8db17d6
7 years ago
Angus Salkeld 3ed5948b1c Set enforce_type=True on config overrides in tests
This makes sure that type checking is done by oslo.config
on the test override values.

Change-Id: Ia8c1cb55fe98e9d06b9b9ff13e5c2d25aa67bff3
Closes-bug: #1517839
7 years ago
Thomas Herve 90fc4fe89e Remove unknown parameters in patched update
When updating a stack with the existing flag, we keep the parameters
from the old template to be used against the new version. Sometimes
parameters will get remove and won't make sense anymore, and keeping
them would break update with a 'Parameter was not defined' error. This
filters out such parameters so that the updates succeed.

Change-Id: I6f2aa77da28d271dd001a137bb574b5470292f15
Closes-Bug: #1558610
7 years ago
Rabi Mishra 9305478141 Resolve name/id using translation rules for Server
This changes OS::Nova::Server resource to use
translation rules to resolve name_id properties.

As most of the tests in and are impacted by this
change, this also changes all the tests to use mock
library. Though the change is very big, it's mostly
mox to mock conversion for the tests.

This reduces the LOC drastically. Some of these test
modules have become very big to manage. We can split
them into smaller ones in the future.

Change-Id: Iaa630ab3b013dfb65e9131b1f265e846c8274037
Partial-Bug: #1514680
Blueprint: mox-to-mock-conversion
7 years ago
PanFengyun 415175ca8a Modify the message of ERROR in stack_cancel_update()
Difficult to understand the ERROR.


Change-Id: Idb19dc95ddb0c74915d9dfd28c1c20e122a7d217
Closes-Bug: #1555425
7 years ago
Jenkins 571393a63b Merge "Immutable parameters" 7 years ago
Jason Dunsmore 98262c10d8 Immutable parameters
Add a new "immutable" boolean field to the parameters section in a HOT
template.  A value of True will result in the engine rejecting
stack-updates that include changes to that parameter.  When not
specified in the template, "immutable" defaults to False to ensure
backwards compatibility with old templates.

blueprint: immutable-parameters
Change-Id: If17df0b97e96ff2a1926451634e4744d6c3e26f1
7 years ago
Jay Dobies d454caf00a Hook into environment merging on server
If environment_files is specified, look in the files dict for each
specified environment file and merge into the stack's environment. This
is the same workflow that previously occurred client-side.

Change-Id: Ibe46fd35de7988920c101a83259c06c8f8a3ed0b
Implements: blueprint multi-environments
7 years ago
Steven Hardy 0a4249f5d1 Fix update preview to consider type comparison
Currently the update preview code only considers _needs_update, but
the actual update code does a type comparison which is used to decide
whether to attempt an in-place update vs a replacement.

Thus, we currently erroneously say a resource is either unchanged or
updated during update --dry-run when the type changes, but during
the actual update we replace the resource.  So instead use the same
type comparison used during update to provide a more accurate preview.

Change-Id: Idb2d9a55d47748007a438f84bb1f2259e553c067
Partial-Bug: #1521971
7 years ago
Thomas Herve 4f3246db45 Event transport
Implement a new mechanism to allow specifying a target in the
environment to send events to. It adds zaqar as the first

Depends-On: Ie04f9204f3ba0f75de32253f096f439c512cddee
Change-Id: Icfc3864e08693cb4b4f921641af380b39bcf0bc0
8 years ago
Steven Hardy b646c6b4ab Fix mocking in several tests causing excessive run-times
Several tests have missing mocking resulting in many minutes wasted
waiting for either RPC or client validation to time out.

Strangely this doesn't cause the tests to fail, only waste a lot of
time (tox -e py27 was taking over 10 mins for me, now nearer 5)

There are more broken tests to fix, but some have logical errors
so will be handled via subsequent patches.

Change-Id: I2021505930d6547a16531de6b2f9fbf33855fa3a
Partial-Bug: #1524047
8 years ago
Anant Patil 634c24ecfe Convergence: Concurrency subtle issues
To avoid certain concurrency related issues, the DB update API needs to
be given the traversal ID of the stack intended to be updated. By making
this change, we can void having following at all the places:

    if current_traversal != stack.current_traversal:

The check for current traversal should be implicit, as a part of stack's
store and state_set methods, where self.current_traversal should be used
as expected traversal to be updated. All the state changes or updates in
DB to the stack object go through this implicit check (using

When stack updates are triggered, the current traversal should be backed
up as previous traversal, a new traversal should be generated and the
stack should be stored in DB with expected traversal as the previous
traversal. This will ensure that no two updates can simultaneously
succeed on same stack with same traversal ID. This was one of our
primary goal.

Following example cases describe the issues we encounter:

1. When 2 updates, U1 and U2 try to update a stack concurrently:

    1. Current traversal(CT) is X
    2. U1 loads stack with CT=X
    3. U2 loads stack with CT=X
    4. U2 stores the stack and updates CT=Y
    5. U1 stores the stack and updates the CT=Z

    Both the updates have succeeded, and both would be running until
    one of the workers does stack.current_traversal == current_traversal
    and bail out.

    Ideally, U1 should have failed: only one should be allowed in case
    of concurrent update. When both U1 and U2 pass X as the expected
    traversal ID of the stack, then this problem is solved.

2. A resource R is being provisioned for stack with current traversal

    1. An new update U is issued, it loads the stack with CT=X.
    2. Resource R fails and loads the stack with CT=X to mark it as FAILED.
    3. Update U updates the stack with CT=Y and goes ahead with sync_point
       etc., marks stack as UPDATE_IN_PROGRESS
    4. Resource marks the stack as UPDATE_FAILED, which to user means that
       update U has failed, but it actually is going on.

    With this patch, when Resource R fails, it will supply CT=X as
    expected traversal to be updated and will eventually fail because
    update U with CT=Y has taken over.

Partial-Bug: #1512343
Change-Id: I6ca11bed1f353786bb05fec62c89708d98159050
8 years ago
ricolin 220e44a3ac Use EntityNotFound instead of StackNotFound.
replace and remove StackNotFound

Change-Id: I0bf4e35e4acf8bcfdd933c7fb862c63f7550f6c2
Partial-Bug: #1515603
8 years ago
Jason Dunsmore 72ac4def9f Assign updated stack ID during stack update preview
Change-Id: Ibb85263023bb6e7d2420ac219306b344e584461e
Closes-Bug: #1504319
8 years ago
Peter Razumovsky c4b0ced813 Fix [H405] pep rule in heat/tests
Implements bp docstring-improvements

Change-Id: I2fd82d698f42aca3b34822875cf4d706fedbf5f2
8 years ago
huangtianhua a2d7ef1a22 Correct expected and observed order of assertEqual
Correct order of expected and observed of assertEqual
in service/

Change-Id: I66809a76580456d2070d277464223d3d3ee0c489
8 years ago
Rabi Mishra e2294da861 Fix action (CREATE) in progress error for TemplateResource/RG
When StackResource 'update_with_template' is called and there
is no existing nested_stack, we call create_stack with an empty
template(TemplateResource CREATE_FAILED->UPDATE flow). We check
db for CREATE_COMPLETE before calling update_stack.There is a
possibility that the create_stack green thread has not finished
and released the stack lock. By persisting the stack state for
COMPLETE/FAILED in the same db session as releasing the lock,
when green thread finishes it's run, we can avoid this race

Change-Id: I1442a7bcedb4fa340ee4f0e6ebc3705544c2a9dd
Closes-Bug: #1498495
8 years ago
Jenkins ec337e1381 Merge "Revert "Additional unit tests for update-preview"" 8 years ago
Steve Baker bab3b3afaa Revert "Additional unit tests for update-preview"
Both of these tests appear to have races which cause gate
failures. This is affecting RC1 changes landing so lets
revert for now while it is investigated.

This reverts commit 92107789ad.
Closes-Bug: #1496886

Change-Id: I411be70ab88e05ad9dc79b3bf147b282bbda34b0
8 years ago
Jenkins 8616c992ca Merge "PATCH update reuse existing template" 8 years ago
Jenkins 86731f1dac Merge "Additional unit tests for update-preview" 8 years ago
Steven Hardy a5cb850841 PATCH update reuse existing template
This is a replacement for the approach outlined in, we reuse the previous
template, even if the stack is in a failed state, by making use
of the prev_raw_template now stored at the start of each update.

Closes-Bug: 1224828
Change-Id: I0294745dea0d25b91dbcc50a8bd55b3e1ea87d81
8 years ago
Jason Dunsmore 92107789ad Additional unit tests for update-preview
Now that OS::Heat::TestResource has a _needs_update() method,
update-preview correctly shows the resource as "replaced" whenever the
update_replace property is set to true.  Also, add an update_not_allowed
property to OS::Heat::TestResource and a corresponding unit test to test
properties with update_allowed set to False.

Change-Id: I75c4982f0c8eb29ec5ce4287b63ccbb8af25bfef
Partial-Bug: #1494436
8 years ago
Steven Hardy 53c4707bf6 Save previous template on non convergence updates
We already store the previous template for convergence based
updates, but for non-convergence updates we can reuse the same
column to enable recovery from a failed update when using
PATCH updates, e.g where the error is in the environment or
nested templates and thus the top-level template may not be
provided & it's valid to use the previously stored one.

Change-Id: I79248bedb01260a0c4b1edf183db55fc0bffbdf6
8 years ago
tengqm 7ec066db4b Split engine service test cases (9)
This patch separates out the stack-update unit tests into a dedicated
module. Some mox calls are replaced with mock calls where appropriate.

Change-Id: I01793f4a4970fef49b935c6be07903b723d6126f
8 years ago