When a stack is IN_PROGRESS and an UPDATE or RESTORE is called
after an engine crash, we set status of the stack and all of its
IN_PROGRESS resources to FAILED
Change-Id: Ia3adbfeff16c69719f9e5365657ab46a0932ec9b
Closes-Bug: #1570576
The db.api module provides a useless indirection to the only
implementation we ever had, sqlalchemy. Let's use that directly instead
of the wrapper.
Change-Id: I80353cfed801b95571523515fd3228eae45c96ae
Before creating a nested stack we validate the resources in the parent
stack. Therefore it's wasteful to validate them again upon creating the
stack - prior to this patch the number or validation calls for each
resource would be equal to the depth of the stack in the tree (so twice for
a child of the root, and three times for a grandchild). With this patch,
each resource in a nested stack is validated only once, by the parent
resource.
ResourceGroup and its subclasses are currently the only StackResources that
do not validate all of their members; they validates a stack containing a
single representative resource. This means that in the case of e.g. index
substitution, the nested stack could be created without having validated
some resources. However, in practice this is unlikely to lead to validity
problems not causing a failure until later than they previously would have.
Change-Id: Iaa36ae5543b3ce30ae8df5a05b48fe987bc0ffdc
Closes-Bug: #1645336
We used to try to acquire the stack lock in order to find out which engine
to cancel a running update on, in the misguided belief that it could never
succeed. Accordingly, we never released the lock.
Since it is entirely possible to encounter a race where the lock has
already been released, use the get_engine_id() method instead to look up
the ID of the engine holding the lock without attempting to acquire it.
Change-Id: I1d026f8c67dddcf840ccbc2f3f1537693dc266fb
Closes-Bug: #1624538
This refactors the service module to use template_schemata
for environment merging.
Change-Id: I86a28d0496e05f978fa1b734818404639541761b
Blueprint: environment-merging
This moves the merge_environment utility function
from service.py to environment_util.py.
Change-Id: Ia005cf47d5e655e60359f8da397a712e749ce13c
Blueprint: environment-merging
This has a couple of effects. One is that we can send more than one message
to an action (since the action need not terminate immediately upon
receiving a message). The other is that we only ever receive the message
once, instead of every time we check for it once it has been sent.
Change-Id: Ib3b33534c0e8b03dfbbb1acde6f7b2d3dcfa82b6
Currently this is reflected in the user_env_as_dict output, which means
not only is it stored in the DB (which we want), but also passed around
to nested stack (which we don't want because it's derived from the parent
stack hidden parameters), and also to the user via stack environment show
(the API returns it but heatclient currently hides it), which we also don't
want because it's not a valid key in user-provided environments.
Change-Id: If5821ccb4a8bbf98012a2541ddd3c8e91455e5cc
Closes-Bug: #1590507
There should always be a context supplied, lets use its session
directly so we can be sure that the session is the shared one from the
context.
Any remaining functions which call get_session() directly will be
handled in their own changes as some require special behaviour.
Change-Id: I3e52d2ff6bb1014f201715a1b1ba32c122c2391d
Related-Bug: #1479723
Save space in the db by allowing templates (especially nested
templates) to reference the same template files dictionary objects,
now stored in the table raw_template_files. Also, cache template files
per heat-engine process as a RAM/performance optimization.
Also, begin to allow for future rolling upgrades with
oslo.versionedobjects.
Change-Id: I1842ea6ba6773a135b1007b793a5f0884417747d
Closes-Bug: 1570983
This allows us to pre-store a template in the database before
creating/updating a stack over RPC, so that we don't need to send the
contents over RPC. This will help us safely de-duplicate files dicts in
the database, reduce the size of messages, and prevent unnecessary
copies of files (in particular) being loaded into memory.
Change-Id: Id1990d5cbac8b3954ce224af73425d4fd8db17d6
This makes sure that type checking is done by oslo.config
on the test override values.
Change-Id: Ia8c1cb55fe98e9d06b9b9ff13e5c2d25aa67bff3
Closes-bug: #1517839
When updating a stack with the existing flag, we keep the parameters
from the old template to be used against the new version. Sometimes
parameters will get remove and won't make sense anymore, and keeping
them would break update with a 'Parameter was not defined' error. This
filters out such parameters so that the updates succeed.
Change-Id: I6f2aa77da28d271dd001a137bb574b5470292f15
Closes-Bug: #1558610
This changes OS::Nova::Server resource to use
translation rules to resolve name_id properties.
As most of the tests in test_server.py and
test_rackspace_cloud_server.py are impacted by this
change, this also changes all the tests to use mock
library. Though the change is very big, it's mostly
mox to mock conversion for the tests.
This reduces the LOC drastically. Some of these test
modules have become very big to manage. We can split
them into smaller ones in the future.
Change-Id: Iaa630ab3b013dfb65e9131b1f265e846c8274037
Partial-Bug: #1514680
Blueprint: mox-to-mock-conversion
Add a new "immutable" boolean field to the parameters section in a HOT
template. A value of True will result in the engine rejecting
stack-updates that include changes to that parameter. When not
specified in the template, "immutable" defaults to False to ensure
backwards compatibility with old templates.
blueprint: immutable-parameters
Change-Id: If17df0b97e96ff2a1926451634e4744d6c3e26f1
If environment_files is specified, look in the files dict for each
specified environment file and merge into the stack's environment. This
is the same workflow that previously occurred client-side.
Change-Id: Ibe46fd35de7988920c101a83259c06c8f8a3ed0b
Implements: blueprint multi-environments
Currently the update preview code only considers _needs_update, but
the actual update code does a type comparison which is used to decide
whether to attempt an in-place update vs a replacement.
Thus, we currently erroneously say a resource is either unchanged or
updated during update --dry-run when the type changes, but during
the actual update we replace the resource. So instead use the same
type comparison used during update to provide a more accurate preview.
Change-Id: Idb2d9a55d47748007a438f84bb1f2259e553c067
Partial-Bug: #1521971
Implement a new mechanism to allow specifying a target in the
environment to send events to. It adds zaqar as the first
implementation.
Depends-On: Ie04f9204f3ba0f75de32253f096f439c512cddee
Change-Id: Icfc3864e08693cb4b4f921641af380b39bcf0bc0
Several tests have missing mocking resulting in many minutes wasted
waiting for either RPC or client validation to time out.
Strangely this doesn't cause the tests to fail, only waste a lot of
time (tox -e py27 was taking over 10 mins for me, now nearer 5)
There are more broken tests to fix, but some have logical errors
so will be handled via subsequent patches.
Change-Id: I2021505930d6547a16531de6b2f9fbf33855fa3a
Partial-Bug: #1524047
To avoid certain concurrency related issues, the DB update API needs to
be given the traversal ID of the stack intended to be updated. By making
this change, we can void having following at all the places:
if current_traversal != stack.current_traversal:
return
The check for current traversal should be implicit, as a part of stack's
store and state_set methods, where self.current_traversal should be used
as expected traversal to be updated. All the state changes or updates in
DB to the stack object go through this implicit check (using
update...where).
When stack updates are triggered, the current traversal should be backed
up as previous traversal, a new traversal should be generated and the
stack should be stored in DB with expected traversal as the previous
traversal. This will ensure that no two updates can simultaneously
succeed on same stack with same traversal ID. This was one of our
primary goal.
Following example cases describe the issues we encounter:
1. When 2 updates, U1 and U2 try to update a stack concurrently:
1. Current traversal(CT) is X
2. U1 loads stack with CT=X
3. U2 loads stack with CT=X
4. U2 stores the stack and updates CT=Y
5. U1 stores the stack and updates the CT=Z
Both the updates have succeeded, and both would be running until
one of the workers does stack.current_traversal == current_traversal
and bail out.
Ideally, U1 should have failed: only one should be allowed in case
of concurrent update. When both U1 and U2 pass X as the expected
traversal ID of the stack, then this problem is solved.
2. A resource R is being provisioned for stack with current traversal
CT=X:
1. An new update U is issued, it loads the stack with CT=X.
2. Resource R fails and loads the stack with CT=X to mark it as FAILED.
3. Update U updates the stack with CT=Y and goes ahead with sync_point
etc., marks stack as UPDATE_IN_PROGRESS
4. Resource marks the stack as UPDATE_FAILED, which to user means that
update U has failed, but it actually is going on.
With this patch, when Resource R fails, it will supply CT=X as
expected traversal to be updated and will eventually fail because
update U with CT=Y has taken over.
Partial-Bug: #1512343
Change-Id: I6ca11bed1f353786bb05fec62c89708d98159050
When StackResource 'update_with_template' is called and there
is no existing nested_stack, we call create_stack with an empty
template(TemplateResource CREATE_FAILED->UPDATE flow). We check
db for CREATE_COMPLETE before calling update_stack.There is a
possibility that the create_stack green thread has not finished
and released the stack lock. By persisting the stack state for
COMPLETE/FAILED in the same db session as releasing the lock,
when green thread finishes it's run, we can avoid this race
condition.
Change-Id: I1442a7bcedb4fa340ee4f0e6ebc3705544c2a9dd
Closes-Bug: #1498495
Both of these tests appear to have races which cause gate
failures. This is affecting RC1 changes landing so lets
revert for now while it is investigated.
This reverts commit 92107789ad.
Closes-Bug: #1496886
Change-Id: I411be70ab88e05ad9dc79b3bf147b282bbda34b0
This is a replacement for the approach outlined in
https://review.openstack.org/#/c/154620, we reuse the previous
template, even if the stack is in a failed state, by making use
of the prev_raw_template now stored at the start of each update.
Closes-Bug: 1224828
Change-Id: I0294745dea0d25b91dbcc50a8bd55b3e1ea87d81
Now that OS::Heat::TestResource has a _needs_update() method,
update-preview correctly shows the resource as "replaced" whenever the
update_replace property is set to true. Also, add an update_not_allowed
property to OS::Heat::TestResource and a corresponding unit test to test
properties with update_allowed set to False.
Change-Id: I75c4982f0c8eb29ec5ce4287b63ccbb8af25bfef
Partial-Bug: #1494436
We already store the previous template for convergence based
updates, but for non-convergence updates we can reuse the same
column to enable recovery from a failed update when using
PATCH updates, e.g where the error is in the environment or
nested templates and thus the top-level template may not be
provided & it's valid to use the previously stored one.
Change-Id: I79248bedb01260a0c4b1edf183db55fc0bffbdf6
This patch separates out the stack-update unit tests into a dedicated
module. Some mox calls are replaced with mock calls where appropriate.
Change-Id: I01793f4a4970fef49b935c6be07903b723d6126f
Tanvir Talukder