In convergence, if stack state_set returns None, it indicates failed
concurrent update.
Fix rollback logic to follow convergence code path while saving the
stack state in DB, so that convergence post stack action code is
executed.
Change-Id: I9c4910c9fa56aa0070c752fc419e5e5fa7f13e99
Closes-Bug: #1544949
This prevents stacks with forbidden resources start
to create/update or stuck in DELETE_IN_PROGRESS state,
if delete the stack with admin resources in nested stacks.
Also we need to allow get id of resource that in SUSPEND
state, because of we use stack preview in SUSPEND state.
Change-Id: I328891e62b4f4bcf620c52ef9d4d8ab60801a651
Closes-Bug: #1539145
If enable re-auth, we will use the stored context
instead of request context, then we can't create
ceilometer alarm resource. There are two problems
when create ceilometer client:
1. the stored context has no domain info, an error
raised from keystone: BadRequest: Expecting to find
domain in project. So this patch will retrive the
user/project domain ids from the auth_ref.
2. after fix the first problem, then another error
raised from keystone: Forbidden: You are not authorized
to perform the requested action. Due keystone doesn't
allow to create a token by a trust-scoped token when
get aodh endpoint. So this patch will pass 'aodh_endpoint'
to ceilometer client to avoid this.
Change-Id: I44ed5c10b6dec6f39714f4f74cf51a10ef6104a6
Closes-Bug: #1531406
We used to create the Stack object for the backup stack with the same name
as the actual stack, but store it in the database with an asterisk appended
to the name. This meant that a backup stack loaded from the database would
have an asterisk at the end of its name (e.g. in log messages), but one
newly created would not. Resolve this inconsistency by appending the
asterisk to the name at the time the Stack object is created, and don't
change it when we store it in the database.
Change-Id: I4eafddc7555fb148b160bdbe5a10ef649d4e27f2
There are many neutron and nova resource has in-built
dependencies by using add_dependencies() method. Its not
required to validate them while valding the user template,
as
1. user interest will be validaing their template
2. in-built dependencies are not known to user.
Co-Authored-By: Rabi Mishra <ramishra@redhat.com>
Change-Id: I4a3043fd17b69a346d447dfbd17488040cf9b387
Related-bug: #1554380
We allow using 'depends_on' instead of 'get_resource'
function to explictly indicate the dependency, and some
resources override the get_reference_id() method to return
the physical_resource_name, in this case the resource name
won't match the physical_resource_name, so we should allow
to find resource in stack with resource name as reference.
Change-Id: Icd37e0d7b341c996f62d78d42cc81da00ec0e7d3
Closes-Bug: #1554372
Uncaught exceptions in an overridden add_dependencies() method of a plugin
can prevent a stack being loaded from the database. To protect against
programming errors that can result in uncaught exceptions, separate the
calculation of implicit dependencies out from the calculation of explicit
dependencies, and ignore exceptions in the latter when dealing with an
existing stack.
Change-Id: I939dba57eeba5710bb77a1b30a872fca5d38ad71
Closes-Bug: #1554625
Move the cleanup code (that sets the status to FAILED and merges the new
environment into the old one) so that it runs on any exception, including
exit exceptions (due to e.g. the thread being cancelled or the engine
restarted). In the case of the environment this isn't as good as merging
them before the update starts (since it's possible for Heat to die without
time to run exception handlers), but at least it leaves things consistent
in the orderly case.
Change-Id: Ia4d2deca393ac9e250932c5cdb3691b6ee2b3ef4
Partial-bug: #1544348
Related-bug: #1492433
While selecting best existing resource from DB, we select the resource
only if it belongs to current or previous template.
But in case of multiple concurrent updates, we could still have the
candidate resource from old template. Hence do not ignore them.
Change-Id: I33243a12aca43242825a256dde2c2969ddb5ef73
Closes-Bug: #1495544
Check that length of stack name is 255 characters
or less so that it can fail with a BadRequest Error
instead of making it to the database to fail and
giving the user an Internal Server Error.
Change-Id: I0b6d55a9ee29aed4eaed63f9ad8958e9e2768b50
Closes-Bug: 1542955
It enhance the validate_template to ignore the given
set of errors while the template is validated.
implements blueprint heat-template-validate-improvements
Change-Id: Id6d85664ae576b6a5d18dbbe84ddf0122fbfaa40
This prevents stacks with forbidden resources start to
create/update or stuck in DELETE_IN_PROGRESS state, if
delete the stack with admin resources in nested stacks.
Also we need to allow get id of resource that in SUSPEND
state, because of we use stack preview in SUSPEND state.
Closes-Bug: #1539145
Change-Id: Ic0a6d6c334aa5fdb6d2c3ba549df86aeb1263e9e
If we get passed a non-string stack name, e.g a map or list, we
fail with a DB error associated with looking up the existing stack.
So instead force all stack lookups to use string identifiers, and
make the name validation for new stacks robust to fail gracefully
when there is an invalid (non string) argument passed.
Change-Id: I052dc4a715773895d070e1e9f26183c6a1cf3d7f
Closes-Bug: #1533065
Some catched exception with ``except XXX as e:`` raise with exception like
``raise e``, but in this way we lose the traceback. I propose we use
``raise`` to keep traceback.
Change-Id: I89fe43ef1af5374e4b2c4151ddc4b2d4f7729601
Currently we leave the context roles empty when loading the stored
context, even though there are roles associated with e.g the trust
scoped token used via loading the stored context. Loading the auth
ref and populating the roles from the token ensure any RBAC performed
on the context roles will work as expected.
Change-Id: I7d699bcf947940357a6eb6ae2d17027ec8d6bd04
Closes-Bug: #1529354
Implement a new mechanism to allow specifying a target in the
environment to send events to. It adds zaqar as the first
implementation.
Depends-On: Ie04f9204f3ba0f75de32253f096f439c512cddee
Change-Id: Icfc3864e08693cb4b4f921641af380b39bcf0bc0
UPDATE flow doesn't call 'state_set', to set the state.
However, this is called in 'set_stack_and_resource_to_failed',
to set the stack status to FAILED. This includes reset during
stack UPDATE.
With https://review.openstack.org/#/c/245810/5, we removed UPDATE
from persisting the state as part of releasing the lock. This has
resulted in the state not being persisted for UPDATE when
'reset_stack_status' is called.
Change-Id: I8a80e329604fdec6498acca0a71723100d412384
Closes-Bug: #1523717
Ensure that an unbound method is never logged as "<method name> from None".
Also report the actual action name when doing stack_action().
Change-Id: Iff37860525d182e6eee01a7cc52c6599f05bff8c
Related-Bug: #1339759
The change Ic948c2fe5baf23c9c4ced33060f672ca9c278a19 aim
to prevent the stacks hanging in IN_PROGRESS, but the code
is incorrect, this patch fix them.
Change-Id: I0e9655b5a931452d810c38b7ddf6a2ce742bb7ed
Closes-Bug: #1521881
Related-Bug: #1492433
Stack create is called internally to complete stack adopt action.
Hence stack adopt will not use stack lock in convergence, so persist
state when state_set is called.
Stack adopt logic is already taken care by convergence.
Change-Id: I2bcfdd7d8b7d9a0ce141ea29ce90253dce0402a8
To avoid certain concurrency related issues, the DB update API needs to
be given the traversal ID of the stack intended to be updated. By making
this change, we can void having following at all the places:
if current_traversal != stack.current_traversal:
return
The check for current traversal should be implicit, as a part of stack's
store and state_set methods, where self.current_traversal should be used
as expected traversal to be updated. All the state changes or updates in
DB to the stack object go through this implicit check (using
update...where).
When stack updates are triggered, the current traversal should be backed
up as previous traversal, a new traversal should be generated and the
stack should be stored in DB with expected traversal as the previous
traversal. This will ensure that no two updates can simultaneously
succeed on same stack with same traversal ID. This was one of our
primary goal.
Following example cases describe the issues we encounter:
1. When 2 updates, U1 and U2 try to update a stack concurrently:
1. Current traversal(CT) is X
2. U1 loads stack with CT=X
3. U2 loads stack with CT=X
4. U2 stores the stack and updates CT=Y
5. U1 stores the stack and updates the CT=Z
Both the updates have succeeded, and both would be running until
one of the workers does stack.current_traversal == current_traversal
and bail out.
Ideally, U1 should have failed: only one should be allowed in case
of concurrent update. When both U1 and U2 pass X as the expected
traversal ID of the stack, then this problem is solved.
2. A resource R is being provisioned for stack with current traversal
CT=X:
1. An new update U is issued, it loads the stack with CT=X.
2. Resource R fails and loads the stack with CT=X to mark it as FAILED.
3. Update U updates the stack with CT=Y and goes ahead with sync_point
etc., marks stack as UPDATE_IN_PROGRESS
4. Resource marks the stack as UPDATE_FAILED, which to user means that
update U has failed, but it actually is going on.
With this patch, when Resource R fails, it will supply CT=X as
expected traversal to be updated and will eventually fail because
update U with CT=Y has taken over.
Partial-Bug: #1512343
Change-Id: I6ca11bed1f353786bb05fec62c89708d98159050
When stack actions such as suspend/resume/snapshot/restore are
completed, two events are logged in DB.
The above stack actions still use stack lock even under convergence.
Hence when setting the stack action as complete/failed, state_set
should not persist state.
Sending notification, events and saving Complete/Failed state in DB
is done at the time of releasing lock for these stack actions.
Change-Id: Ib3e6c1de2f2f17502049e06ae484cb2e50867fab
Closes-Bug: #1516089
Start using self.user_creds_id attribute instead of passing it as
parameter in the fucntion _try_get_user_creds.
Change-Id: I5a0bd3a1d6734cccce4c15c9b34ef3aeb89a577d
Make sure that any unexpected exceptions are handled by moving the stack to
the FAILED state.
Change-Id: Ic948c2fe5baf23c9c4ced33060f672ca9c278a19
Closes-Bug: #1492433
Rakesh H S