Update git submodules
* Update heat from branch 'master' - Merge "Add heat-agents as required project" - Add heat-agents as required project We removed[1] these scripts from heat-templates. [1] https://review.openstack.org/#/c/584777/ Change-Id: I6f0989a5cf4cf6253ff64a8c3c1ea023dd8b6db7 - Merge "Remove duplicate tempest.conf setup for grenade" - Remove duplicate tempest.conf setup for grenade Change-Id: I25930a4af9b67e98c7705ad8801bfc6e5a7c0d72 - Merge "Add tags support for ProviderNet resource" - Merge "Bump minimal_image_ref" - Bump minimal_image_ref Devstack bumped[1] the cirros image version. Changes to source the old openrc file for grenade before configuring tempest. [1] https://review.openstack.org/#/c/624716/ Change-Id: I9481bbcb03609db6d2224998a4f83d3a8790e96c - Incorrect project name in installation instructions for CentOS Return the instructions to using the "service" project as it was before 5ecee29572556140e57ed667ef275d8429c994b6, and as it is in other manual installation instructions. Add a note that describes the circumstances in which you need to use "services" as the project name. Change-Id: I76de8960ac640b9e8037879af85b84ce39b4cdcb Co-Authored-By: Zane Bitter <zbitter@redhat.com> Task: 28520 Story: 2003872 - Merge "Delete db resources not in template" - Imported Translations from Zanata For more information about this automatic import see: https://docs.openstack.org/i18n/latest/reviewing-translation-import.html Change-Id: I27716c190459bd3fb89322567279b122be626db5 - Merge "Fall back to www_authenthicate_uri" - Merge "Change openstack-dev to openstack-discuss" - Change openstack-dev to openstack-discuss Mailinglists have been updated. Openstack-discuss replaces openstack-dev. Change-Id: Ia2033b46b8a5bde404c3765bb32e125646be3ffe - Pass role ids to trust if possible currently Heat fails to create a stack requiring deferred auth with trusts when user happens to have a role assigned for which a different but similar named role exist in another Keystone domain. This patch makes Heat to attempt extract role IDs from the auth_token_info first and pass those to Keystone trust if found. Change-Id: Ic9207e3f027f9c56c14d548d404ecc0c2427f326 Story: 2002613 Task: 22237 - Merge "Revert "Use OutputDefinition to generate attributes schema"" - Merge "Remove tripleo newton and ocata jobs" - Merge "Use fedora 29 image from nodepool mirror" - Add tags support for ProviderNet resource Edit OS::Neutron::ProviderNet resource to add tag attribute as allowed in Neutron API Change-Id: I792dadbd671edbf1fc5706df524a5335777be800 Story: 2004181 Task: 27669 - Merge "Add Release Notes in README" - Revert "Use OutputDefinition to generate attributes schema" This reverts commit 85eff5fc3d03dce7d0d17d965f27246bd3306c53. Change-Id: I325593df0cd7a48cbc85edf7838cddbc0911b4b6 Closes-Bug: #1805589 Depends-On: https://review.openstack.org/620457 Depends-On: https://review.openstack.org/619786 - Fall back to www_authenthicate_uri The [keystone_authtoken]/auth_uri setting was deprecated in favour of [keystone_authtoken]/www_authenticate_uri. This commit uses the new setting as the fallback if [clients_keystone]/auth_uri has not been set. Change-Id: I4b6e254b1344865e3775948a8bbf1e9aac196e4f Story: #2004434 Task: #28097 - Use fedora 29 image from nodepool mirror It seems fedora 27 cloud images are not available. Change-Id: I199afcbaa5050226a333779ac22aea3451f94b04 - Merge "Use OutputDefinition to generate attributes schema" - Merge "Ensure Functions can be created without stack definition" - Remove tripleo newton and ocata jobs Non-containerized multinode scenario jobs are no longer supported and should be removed from zuul config. Change-Id: I0c98aecd82478fbf7e2b2a45d5b2664290155ed9 - Merge "Return execution state_info in ExternalResource" - Merge "Try deleting stack if not soft deleted yet" - Return execution state_info in ExternalResource When a workflow execution fails, we don't return anything information to the user. Let's propage what mistral sends us in the resource failure. Change-Id: Ib412346f12450fa5e6a1291d635debe1a4add380 Story: #2004396 Task: #28025 - Merge "convert py35 jobs to py3" - Merge "Consider volume in 'reserved' status as detachment in progress" - Add heat-status to man page output Change-Id: I7afd5274b6f5e6a4ad0cb32fa2f5b1512e4f07f7 - Add Release Notes in README Add Release notes in README.rst Change-Id: I3507bf3b824e0580a501dd48db11fca6366c4fbe - Merge "Don't depend on string interning" - convert py35 jobs to py3 Update the jobs that were explicitly using python 3.5 and have them just use python 3 without specifying a minor version. Change-Id: I38734f2956a30c01ff51e2d74a9c248e4304d08b - Merge "Add heat-status upgrade check command framework" - Merge "change the dist name to 'openstack-heat'" - Imported Translations from Zanata For more information about this automatic import see: https://docs.openstack.org/i18n/latest/reviewing-translation-import.html Change-Id: Iad33b2d2266fa14cfd615667fcaa78b45406db73 - Merge "Clarify resource deprecation process" - change the dist name to 'openstack-heat' We are updating all Python projects to publish artifacts to PyPI. The name "heat" is already taken there by another project, and they have rejected our request to claim the name. We therefore need to change the dist name used to package heat. We have some other projects publishing using an "openstack-" prefix, so I propose using the name "openstack-heat". This will not change the imports or anything else about how the code works, just how it is packaged. Add a grenade plugin for upgrading between the packages with different names so that we can clean up the old version of the code and avoid discovering the heat plugins multiple times in different locations. Change-Id: I59b55cffd9e648f842eb286b936f09c5b55a76db Signed-off-by: Doug Hellmann <doug@doughellmann.com> - Merge "Revert "Add openstack/placement as a required project for heat-grenade*"" - Merge "Increment versioning with pbr instruction" - Revert "Add openstack/placement as a required project for heat-grenade*" This reverts commit 4e8a3ca6b2041c170f4073499ee0315616396af2. This is handled generically with this change and its dependency: https://review.openstack.org/#/c/606853/ Change-Id: Ic79cc07a2d11772c1892b2db8c4c02cc7be4081c - Add heat-status upgrade check command framework This adds basic framework for heat-status upgrade check commands. For now it has only "check_placeholder" check implemented. Real checks can be added to this tool in the future. Change-Id: I83629184b49a6cf91928df702db23156433d99f6 Story: 2003657 Task: 26131 - Use OutputDefinition to generate attributes schema We have the OutputDefinition API now, and no longer need to treat templates like CFN JSON blobs to obtain information about outputs. Change-Id: I8dfea1e9855a56fb85d2e3af40996d5f337d7859 - Ensure Functions can be created without stack definition In a number of places we parse templates passing None as the stack definition, in order to be able to extract information from the template in cases where we don't actually need to resolve functions. However, the GetParam and GetFile HOT functions require an actual stack definition to be passed in order to initialise them. So far we haven't encountered any cases where those functions appear where we pass None for the stack definition. Ensure that all Functions can always be created even with no stack definition. Note that the cfn ParamRef class is not affected, as the Ref function always selects the GetResource implementation when the stack definition is None. Change-Id: Ic947dcdc17e9dbc58459a92ba970c292d70578cd - Revert "Change "services" to "service"" This was correct for the RDO installation guide. This reverts commit b90278b503aff2ad8148af38525beb90ffb117d8. Change-Id: I5107e712a903f334f1a3a996b9140e9e2fe0807a - Change "services" to "service" Because project name is service in Installation Guides[1], It is better not to give reference of this name in installation operation Documentation file. [1]https://github.com/openstack/keystone/blob/master/doc/source/install/common/keystone-users.inc Change-Id: Ib51341d023907b392acbdacd47bd1b3027b721a1 - Delete db resources not in template When migrating stacks to convergence, if there are resources in the database that are not in the current_template_id of the stack, they are possibly of no isue, so it would better to delete those resources from db to avoid any future update issues. Change-Id: Ica99cec6765d22d7ee2262e2d402b2e98cb5bd5e Story: #2004071 Task: 27092 - Don't depend on string interning Use '!=' instead of 'is not' to compare strings. In practice, short strings that appear in the source code are interned in CPython, but this is implementation-specific. Change-Id: If3f305c2d647fcd7515cb0a326a30f4eda93acd3 - Increment versioning with pbr instruction With moving away from required milestone releases, the version numbers calculated by PBR on the master branch will not work for those testing upgrades from the last stable release. More details can be found in the mailing list post here: http://lists.openstack.org/pipermail/openstack-dev/2018-October/135706.html This is an empty commit that will cause PBR to increment its calculated version to get around this. PBR will see the following which will cause it to increment the version: Sem-Ver: feature Please merge this patch as soon as possible to support those testing upgrades. Change-Id: Ifbba36f4974efcf891c7f5938acfeef75bb05dd2 Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com> - Merge "Don't quote {posargs} in tox.ini" - Merge "Don't assert that we haven't signalled the calling thread" - Don't quote {posargs} in tox.ini Quotes around {posargs} cause the entire string to be combined into one arg that gets passed to stestr. This prevents passing multiple args (e.g. '--concurrency=16 some-regex') Change-Id: Iccbd680731592abab4980a51aa97c60d6ef95005 - Merge "Imported Translations from Zanata" - Merge "Fail to release a stack_lock in the database is a fatal action." - Imported Translations from Zanata For more information about this automatic import see: https://docs.openstack.org/i18n/latest/reviewing-translation-import.html Change-Id: I7eaf3039b2c1c54a5b37add7f54f05fd11493670 - Don't assert that we haven't signalled the calling thread We use the notify.signal() call to notify the main thread that the stack status has moved to IN_PROGRESS so that it can wait before returning control to the user. Therefore it is expected that if an operation eventually fails (or succeeds), signal() will have been called a long time previously. The only reason it is there is to guard against failures before the resource attains the IN_PROGRESS state, where the persistence of the state has been deferred to coincide with the lock release. Story #2003988 Task: 26931 Change-Id: Ie0519ee78607f71855c2c0ace2cb4ff52c5809b6 - Merge "Set a concurrency limit on ResourceChain" - Set a concurrency limit on ResourceChain This adds a limit to the number of resources that ResourceChain creates in parallel. It reduces memory usage drastically when ResourceChain contains lots of resources. We use the number of active services, which seems to give a reasonable balance between performance and memory consumption. Story: #2003975 Task: #26917 Change-Id: Id20fb89813b32927298b3a7c0abe23da710b04d1 - Consider volume in 'reserved' status as detachment in progress It's possible that a volume can end up in `reserved` state when volume attachment fails. We should wait for it to be available when detaching. Change-Id: Ieb1d12ea9d7c8c76c448387fadfb2e4dbf4cbb2e Story: #2003789 Task: 26497 - Merge "Check for server in attachements when checking for detach complete" - Merge "Fix missing space in help text for caching option." - Merge "Fix some typos in docstrings." - Merge "Add openstack/placement as a required project for heat-grenade*" - Set the octavia noop drivers correctly In 866864f7e5973053edcd9de59caab05790729964 we changed to use noop drivers, we don't seem to set them correctly. Depends-On: https://review.openstack.org/605000 Change-Id: I7ba69fc047334c9c3d2fffe41e586650f9fb372b - Add openstack/placement as a required project for heat-grenade* In order to get openstack/placement installed as part of the heat grenade jobs, we have to list it as a required project. Needed by https://review.openstack.org/604454/ Change-Id: I58fae5f56fdbcb4fc084689bde95f1a2c4bfad0f - adding a note that specifically states the user cannot rename a stack Change-Id: I466d3fb36727aa384571ad7497fd7d0983b59433 - Check for server in attachements when checking for detach complete It's possible that the volume is already attached to another server, which can happen by marking a VolumeAttachment resource unhealthy after removing the volume from the server. Change-Id: Ia77e0462bf732224b491591d22584cbcfa779956 Story: #2003850 Task: 26643 - Merge "Imported Translations from Zanata" - Merge "Fix heat multi-region support with shared keystone" - Fail to release a stack_lock in the database is a fatal action. When a heat-engine thread activity completes, it calls release on its stack_lock object in the database. If that release action fails due to an inability to update the database, that engine process is no longer usable. This code catches that failure, logs it, and terminates that engine process so that a new one can be started. New heat engines will automatically purge stale stack_locks from the database. Also, make sure that if the thread exit does not teardown the process after 5 seconds, the non blockable os level exit call will be invoked. This bug is very timing specific. The DB error needs to exist when the stack_lock release fails Change-Id: I7663b2270bf325cd8e3dd194f2994227fd6f5e8a Story: 2003439 Task: 24635 - Merge "Use no-op drivers for Octavia in gate" - Merge "Do not perform the tenant stack limit check for admin user" - Fix heat multi-region support with shared keystone keystone endpoint region can be different from other services. Let's use the one from keystone middleware authtoken config, if available, when creating keystone client. Change-Id: I190abd946db1cf8712ef669cf72a1134dfcf9376 Story: #2003806 Task: 26549 - Imported Translations from Zanata For more information about this automatic import see: https://docs.openstack.org/i18n/latest/reviewing-translation-import.html Change-Id: I77181b348a2e244d225d2b02593c4a07a5edd7b5 - Use no-op drivers for Octavia in gate Continue to test Heat's interaction with the Octavia API without relying on Octavia's interaction with Nova, Neutron &c. which is known to be unstable. Change-Id: I8765aceee09ada3086abc704b7f73ce240e21847 - Merge "middleware: return HTTPBadRequest when path is invalid" - middleware: return HTTPBadRequest when path is invalid Gracefully handle paths that are invalid UTF-8 by returning HTTP error 400 instead of an Internal Server Error. Change-Id: I3b30e4d64e758eefd85f7a70fc645db69991b3d7 Task: 26379 - Imported Translations from Zanata For more information about this automatic import see: https://docs.openstack.org/i18n/latest/reviewing-translation-import.html Change-Id: I69833a3b65e99e065479089e9b8735d9a748bb33 - Merge "Use openstack-tox-cover template" - Merge "zun: remove the handling of not acceptable error" - Merge "Use oslo_messaging for JsonPayloadSerializer" - Imported Translations from Zanata For more information about this automatic import see: https://docs.openstack.org/i18n/latest/reviewing-translation-import.html Change-Id: Ib8bd83de87be19350be63d0ce6a6501037a49686 - Merge "Remove unnecessary shebangs from service scripts" - Use openstack-tox-cover template Use openstack-tox-cover template, this runs the cover job in the check queue only. Remove jobs and use template instead. Change-Id: I251086151a23c78f99f39d55b4db36d1db5bc9a0 - Use oslo_messaging for JsonPayloadSerializer JsonPayloadSerializer is added in oslo_messaging lib. This patch suggest to replace current one with oslo_messaging. Change-Id: I8880ae628b6121e2aee0d132238f44571f3168e3 - Merge "Add functional test for legacy in-place update" - Merge "aodh/gnocchi: remove constraints on aggregation methods" - Merge "Support py36 for test in tox" - aodh/gnocchi: remove constraints on aggregation methods The current list misses the percentile and all rate based aggregation methods. Rather than adding them, let's remove the constraint and let Gnocchi and Aodh validate the data. Change-Id: I0330c09d72c20d63d08770b52d3071512a418260 - Fix .zuul.yaml * Remove "queue: heat" from check queue, this is not needed. * Use openstack-lower-constraints-jobs template. Change-Id: Idde37f299c1cde86b2169fd174539249c23c9962 - remove master-only job templates from zuul config Some of the project templates brought in via https://review.openstack.org/#/c/576193/ should have been left in the project-config repo because they only run on the master branch. This patch removes those templates. Change-Id: I727a9ac03781f86845d1af0b2a2ae672665b30db Signed-off-by: Doug Hellmann <doug@doughellmann.com> - Merge "Unit tests: Fix mock errors with too few side effects" - Merge "Fix bug to doc: rally_on_gates" - Merge "Update resource definitions after legacy in-place update" - Merge "Ignore spurious nested stack locks in convergence" - Add functional test for legacy in-place update in-place update of failed stacks with old resource references deleted in last update. Change-Id: I45fbca77c84744e1778c6f2494674233b9193921 Story: #2003612 Task: 25740 - Merge "add python 3.6 unit test job" - Merge "switch documentation job to new PTI" - Update resource definitions after legacy in-place update Since we remove a resource's definition after it is deleted, we must also update a resource's definition after it is updated. Otherwise, a resource that depended on the now-deleted resource prior to its last in-place update could remain in the template, leaving the stack inoperable since it is unable to calculate its own dependencies. We already update the template in the backup stack when updating resources in-place, and in the main stack when creating replacement resources. Also update the main template when resources are updated in-place, not just when they are replaced. Change-Id: I97ddc6a970ffd27bfd40bb8f6bec8d1cbf28a815 Story: #2003612 Task: 24946 - Trivial: Update pypi url to new url Pypi url changed from [1] to [2] [1] https://pypi.python.org/pypi/<package> [2] https://pypi.org/project/<package> Co-Authored-By: Van Hung Pham <hungpv@vn.fujitsu.com> Change-Id: I07e2ecb62076610867568765f25c90e2ccbbd895 - Ignore spurious nested stack locks in convergence Several operations (e.g. stack check) are yet to be converted to convergence-style workflows, and still create locks. While we try to always remove dead locks, it's possible that if one of these gets left behind we won't notice, since convergence doesn't actually use stack locks for most regular operations. When doing _check_status_complete() on a nested stack resource, we wait for the operation to release the nested stack's lock before reporting the resource completed, to ensure that for legacy operations the nested stack is ready to perform another action on as soon as the resource is complete. For convergence stacks this is an unnecessary DB call, and it can lead to resources never completing if a stray lock happens to be left in the database. Only check the nested stack's stack lock for operations where we are not taking a resource lock. This corresponds exactly to legacy-style operations. Change-Id: I4eb20ad82cc3e9434da34500fafa3880567d0959 Story: #1727142 Task: 24939 - Fix bug to doc: rally_on_gates Change-Id: Ia3360c5bc6ada59cdaa30997e1ce39ac9c063006 Story: 2003179 Task: 23325 - Imported Translations from Zanata For more information about this automatic import see: https://docs.openstack.org/i18n/latest/reviewing-translation-import.html Change-Id: I39bcb14e7aaf8db3acf50678785dab51abddbb72 - Support py36 for test in tox Ubuntu Artful and Fedora 26 already ship Python 3.6 by default (And Bionic and F28 have been released since then.). Considering better support of python3 with system, we should enable py36 for test environment. You can find some discussion in: http://lists.openstack.org/pipermail/openstack-dev/2018-June/131193.html Depends-On: https://review.openstack.org/#/c/597267 Change-Id: I0187e862c0b74dbb836d36e7062ccc9e47a51cc2 - add python 3.6 unit test job This is a mechanically generated patch to add a unit test job running under Python 3.6 as part of the python3-first goal. See the python3-first goal document for details: https://governance.openstack.org/tc/goals/stein/python3-first.html Change-Id: I772dbaa9d90cdbdfa1db97a43549937a3e9e3c7d Story: #2002586 Task: #24298 - switch documentation job to new PTI This is a mechanically generated patch to switch the documentation jobs to use the new PTI versions of the jobs as part of the python3-first goal. See the python3-first goal document for details: https://governance.openstack.org/tc/goals/stein/python3-first.html Change-Id: I33e99114889052d6e610a305c5fb659822d5b0d4 Story: #2002586 Task: #24298 - import zuul job settings from project-config This is a mechanically generated patch to complete step 1 of moving the zuul job settings out of project-config and into each project repository. Because there will be a separate patch on each branch, the branch specifiers for branch-specific jobs have been removed. Because this patch is generated by a script, there may be some cosmetic changes to the layout of the YAML file(s) as the contents are normalized. See the python3-first goal document for details: https://governance.openstack.org/tc/goals/stein/python3-first.html Change-Id: I4c5227d740cdb0bc9e08ccdce27b0ca73fb4ded3 Story: #2002586 Task: #24298 - Rename mox_resource_id to mock_resource_id As we no longer use mox library it is misleading to keep names starging with mox so let's rename this prefix to mock_. Change-Id: Ia376b9dcb7bf964df3c30f6757a6061c1850ee95 Signed-off-by: Krzysztof Opasiak <k.opasiak@samsung.com> - Merge "Add UDP to supported protocols for Octavia" - Merge "Ignore conditions when reparsing ResourceDefinition" - Ignore conditions when reparsing ResourceDefinition In the legacy path we reparse a resource definition to refer to a different StackDefinition when copying a resource definition into the existing template (as the resource is updated) or into the backup stack's template. However, the resources 'condition' may reference conditionals that are not defined in the template it is being copied into (e.g. during a stack update that creates a new condition), and in this case the destination template will become unusable. Since the fact that we care about the resource definition at all indicates that the condition was enabled, just ignore the condition after reparsing. This is consistent with what we do for 'if' macros, which is to resolve the condition part during reparsing. Change-Id: I59a374435f6275badc8124efbd7b7db2e36e2de5 Story: #2003558 Task: 24847 - Do not perform the tenant stack limit check for admin user Removing stack creation limit for admin since admin can view all stacks, and stacks created by other tenants would have been counted in the limit check. Change-Id: Ie2e9251245e7e16309661154e17724e5984c21e6 Story: 2003487 Task: 24756 - Add UDP to supported protocols for Octavia Change-Id: I6da312a75f2d998c9dda4f874ab139022429ed2f - Imported Translations from Zanata For more information about this automatic import see: https://docs.openstack.org/i18n/latest/reviewing-translation-import.html Change-Id: Ic2f60e65a30d94f8fa7c667e4661ff398506e3ba - Unit tests: Fix mock errors with too few side effects When a mock hasn't been provided with a long-enough list of side-effects for the number of times it is called, it raises StopIteration (instead of something sensible like AssertionError). Prior to Python 3.7, this exception could bubble up until it just stopped a task. (In Python 3.7 it will eventually be caught and turned into a RuntimeError.) To ensure that any errors of this sort are not handled silently, and to enable us to test for them prior to using Python 3.7, catch any StopIteration errors coming from plugin-provided non-generator functions and convert them to RuntimeError exceptions. This reveals many errors in the unit tests, many introduced in the process of converting from mox to mock, which are also fixed by this patch. Change-Id: I5a1eff6b704dff7c17edcbbe58cdbc380ae6abc9 Story: #2003412 Task: 24553 - Merge "Revert octavia pool member delete fix" - Merge "Update reno for stable/rocky" - Merge "Fix errors with issubclass() in Python 3.7" - Update reno for stable/rocky Change-Id: Ia5db11a684ab2164b8a11a9a49f55d451e01cc0b - Merge "Remove branch-specific templates from .zuul.yaml" - Merge "cors: update default configuration" - Merge "Robust handling of parent and child process deaths" - Merge "Refactor deferral of stack state persistence" - Merge "Don't change INIT_COMPLETE resources to CHECK_COMPLETE" - Merge "Drop ceilometerclient requirement" - cors: update default configuration Set_defaults has been added into oslo_middleware. So we use it to override the configuration defaults. Change-Id: Ic9c6278e43a66da9f94f556430944fe9b42fb1b7 - Drop ceilometerclient requirement ceilometerclient is dead in queens cycle, lets drop the integration and remove the requirement. Change-Id: I5267746e63fd27ad0ce8e36fb843efd4988d6eb7 - Don't change INIT_COMPLETE resources to CHECK_COMPLETE We assume that only resources that are in INIT_COMPLETE need to be created (as opposed to updated). If we run a check on a resource that hasn't been created yet, we shouldn't change the status to CHECK_COMPLETE because thereafter we will try to update it. Changing the status to CHECK_FAILED (as we do if the resource supports handle_check()) is OK because the resource will get replaced on the next update anyway. Change-Id: Iaadf3b57b0e8edc76d338c017dbd05bf2a37698f Story: #2003062 Task: 23102 - Store resources convergence-style in stack check/suspend/resume If there are resources in a template that don't exist in the database at the time of a stack check (or suspend or resume) operation, running the stack check will cause them to be stored in the database. Since these operations have not been converted to convergence (story 1727142), they do not set the current_template_id as a convergence update would. If this occurs then the stack will be unrecoverable. To avoid this, when convergence is enabled store any missing resources in the same manner that a convergence update would, prior to running the stack check/suspend/resume. Just in case, make sure the stack doesn't get stuck if we do end up in the wrong state, by not trying to load a template with None as an ID. Change-Id: Iedba67c5de39dc2d58938da5505dda5dd147c130 Story: #2003062 Task: 23101 - Merge "Handle exceptions in initial convergence traversal setup" - Merge "Ignore RESOLVE translation errors when deleting resources" - Merge "Eliminate client races in legacy operations" - Merge "Clean up test requirements" - Merge "Always start convergence worker service" - Fix errors with issubclass() in Python 3.7 The cfn Ref instrinsic function is implemented as a Python function rather than a class, so calling issubclass() on it fails in Python 3.7. Change-Id: I3f826cd8f024b591d7c3b41ad1e3f960558d3f9b - Merge "Add StoryBoard migration docs and releasenote" - Remove branch-specific templates from .zuul.yaml I jumped the gun a bit here, and it turns out that all branch-specific jobs will be maintained in openstack-infra/project-config, and not moved to project repos. Change-Id: Ia54bd7d29717e2fb025c9547b852c4ec00cbcbd2 - Remove unnecessary shebangs from service scripts These are just python files. Also removes internal .gitignore in heat_integrationtests Change-Id: Ibea3253cd89f905e61b655861274c61ab48a7ea0 - Merge "Add docs of update/replace *_FAILED resource for developer" - Refactor deferral of stack state persistence When we hold a StackLock, we defer any persistence of COMPLETE or FAILED states in state_set() until we release the lock, to avoid a race on the client side. The logic for doing this was scattered about and needed to be updated together, which has caused bugs in the past. Collect all of the logic into a single implementation, for better documentation and so that nothing can fall through the cracks. Change-Id: I6757d911a63708a6c6356f70c24ccf1d1b5ec076 - Unit tests: Fix slow Manila test We were retrying the create due to the client_plugin being mocked out, which resulted in the test taking >60s instead of ~0.3s. Change-Id: I14d0a88b94f7e56e03683021c9c29a55dac9e516 Story: #1745010 Task: 23347 - Handle exceptions in initial convergence traversal setup If an exception occurs while doing the initial setup of a convergence traversal (including sending the check_resource messages for leaf nodes), mark the stack as failed. If we don't do this then any error or exception in this method will cause the stack to hang IN_PROGRESS. Change-Id: Ib8231321e823634a3dc23cff9a1c7d560f64fd6e Story: #2003125 Task: 23245 - Always start convergence worker service The purpose of the cfg.CONF.convergence_engine option is to determine whether new stacks will be created using convergence or not. For some reason, however, the worker service was not started when the option was disabled, meaning that previously-created convergence stacks could no longer be updated (there was a hack in place to allow them to be deleted). Always start the worker service and continue operating any existing convergence stacks in convergence mode, regardless of the current setting for new stacks. Change-Id: Ic3574e4d15ac48b2bc8e0e8101c81d24f40f0606 Related-Bug: #1508324 - Eliminate client races in legacy operations Wait for the legacy stack to move to the IN_PROGRESS state before returning from the API call in the stack update, suspend, resume, check, and restore operations. For the stack delete operation, do the same provided that we can acquire the stack lock immediately, and thus don't need to wait for existing operations to be cancelled before we can change the state to IN_PROGRESS. In other cases there is still a race. Change-Id: Id94d009d69342f311a00ed3859f4ca8ac6b0af09 Story: #1669608 Task: 23175 - Eliminate client race condition in convergence delete Previously when doing a delete in convergence, we spawned a new thread to start the delete. This was to ensure the request returned without waiting for potentially slow operations like deleting snapshots and stopping existing workers (which could have caused RPC timeouts). The result, however, was that the stack was not guaranteed to be DELETE_IN_PROGRESS by the time the request returned. In the case where a previous delete had failed, a client request to show the stack issued soon after the delete had returned would likely show the stack status as DELETE_FAILED still. Only a careful examination of the updated_at timestamp would reveal that this corresponded to the previous delete and not the one just issued. In the case of a nested stack, this could leave the parent stack effectively undeletable. (Since the updated_at time is not modified on delete in the legacy path, we never checked it when deleting a nested stack.) To prevent this, change the order of operations so that the stack is first put into the DELETE_IN_PROGRESS state before the delete_stack call returns. Only after the state is stored, spawn a thread to complete the operation. Since there is no stack lock in convergence, this gives us the flexibility to cancel other in-progress workers after we've already written to the Stack itself to start a new traversal. The previous patch in the series means that snapshots are now also deleted after the stack is marked as DELETE_IN_PROGRESS. This is consistent with the legacy path. Change-Id: Ib767ce8b39293c2279bf570d8399c49799cbaa70 Story: #1669608 Task: 23174 - Merge "Delete snapshots using contemporary resources" - Merge "Ignore RESOLVE translation errors when translating before_props" - Revert octavia pool member delete fix In 705d2a0b7b78813fb84636ac262030bc7dd6be08 we added a hack to handle traslation error when deleting a pool member resource with the pool already deleted in octavia. After the previous patch in this series we can remove this hack. Change-Id: I459cd5e2c3dc8be23e299a2b8360a32b52d8e071 Story: #1747836 Task: 23279 - Ignore RESOLVE translation errors when deleting resources Sometimes when deleting resources, we use parent resources in api calls. If the parent resource is deleted then the children could be assumed to be deleted. So ignore those translations errors and return unresolved values when accessing properties. Change-Id: I2392b5c46920ac6da518fe33bdcba3e7a9fe17a8 Story: #1747836 Task: 23278 - Ignore RESOLVE translation errors when translating before_props When translating name to id, if we don't find the resource it's better to ignore the error and return the non-translated value for before properties. Change-Id: Ic25c35098cff8b68e184a336b21f2e8df70bb1ee Story: #2003015 Task: 23103 - zun: remove the handling of not acceptable error This is for addressing outstanding comments in I4ef36a3c4a805b3e041fcb9456c297e59865485c . The handling of not acceptable error is for being compatible with legacy deployment but this requirement is an expectation so we remove it. Change-Id: I52c6dbc89ef9a00680bd9dc5e2e1ee5f2ce49f10 - Merge "zun: add property 'networks' to container" - Merge "Enhancements to CIDR and IP address constraints." - Merge "Convert bytes_used to int before comparison" - zun: add property 'networks' to container Add a new property 'networks' to resource OS::Zun::Container. This property is an ordered list of nics to be added to this container, with information about connected networks, fixed ips, port etc. This property can be updated without replacement. Story: 2003106 Task: 23222 Change-Id: I4b8c0257b83e97444dd8ff6ce88e240d12278ec2 - Merge "zun: use containers.update for renaming" - Clean up test requirements remove os-testr and testrepository in favor of stestr, and remove qpid-python (does not seem to be imported anywhere). Also adjust docs to describe running tests with stestr instead of testrepository. Change-Id: I12088ea2bf2475963db58d6e8e83fd6abe6f9b3f - Merge "Hidden resources in Rocky" - Convert bytes_used to int before comparison Change-Id: I7fab8c0b3fbda60f68e0b1fef8a2befd2183433d - Merge "Add API doc for files_container support" - Merge "Option for retrieving child templates and env files from swift" - Merge "Add Blazar client plugin to Heat" - Merge "Adds doc8 check to pep8" - Merge "Add a new OS::Heat::Delay resource type" - Robust handling of parent and child process deaths Ensure that heat api child processes terminate when their parent process dies unexpectedly. Ensure that heat api child processes do not hang during clean shutdown. Basically porting the same pipe handler code that exists in oslo_service. This allows the child processes to detect their parent process has died unexpectedly (ie: kill -9) When a child process does a graceful shutdown it was observed to sometimes never compelete. This code waits up to a second before resorting to a more forceful shutdown of the child process. Change-Id: I8404f9727f41f37a9addb5541ae4d45fa1baba67 Story: 2003075 Task: 23126 - Hidden resources in Rocky Resource type ``OS::Magnum::Bay`` is now hidden, please use ``OS::Magnum::Cluster`` instead. Resource type ``OS::Magnum::BayModele`` is now hidden, please use ``OS::Magnum::ClusterTemplate`` instead. Resource type ``OS::Nova::FloatingIP`` is now hidden, please use ``OS::Neutron::FloatingIP`` instead. Resource type ``OS::Nova::FloatingIPAssociation`` is now hidden, please use ``OS::Neutron::FloatingIPAssociation`` instead. Change-Id: I2ea1400cfe1f4eca24f2fb7b4b0ec27ca5e9538d - Adds doc8 check to pep8 This patch adds doc8 check for .rst files to pep8. Files that fail doc8 check are also fixed. Ignore D001 of doc8 check. This requires lot of changes and should be done in a separated patch. Change-Id: I7732abc55ec27026efbf56663ba02ff27e8ec847 - Add StoryBoard migration docs and releasenote Heat curent bug/bluepint reports are migrated from Launchpad to StoryBoard [1]. If you like to create a new story(a bug or a bluepint), please file it under [2]. For more detail, please reference [3]. This change applied to all heat projects. This patch Add documents and guideline link to StoryBoard. Also add releasenote to showing migration message. [1] https://storyboard.openstack.org/#!/project_group/82 [2] https://storyboard.openstack.org/#!/project/989 [3] https://etherpad.openstack.org/p/Heat-StoryBoard-Migration-Info Change-Id: I7c7dab88d269f92c66b3f68352a6913aa244fdfc - Add API doc for files_container support Also adds releasenote for this new feature. Change-Id: I6da965770f2a0583277165cc6f39ae57caf43d93 Story: #1755453 Task: 22714 - Option for retrieving child templates and env files from swift This provides an option to specify a swift container for stack actions and all child templates and env files will be fetched from the container, if available. However, files coming in the 'files' map from the client will have precedence, if the same is also present in swift. Change-Id: Ifa21fbcb41fcb77827997cce2d5e9266ba849b17 Story: #1755453 Task: 17353 - Merge "OS::Nova::Server: Extend addresses attr to include network" - Enhancements to CIDR and IP address constraints. Currently the constraints do not reject an ipaddress for ipv4 which have fewer than 3 dots such as 'a' or 'a.b' or 'a.b.c'. This enhancement provides an extra check that an ipv4 address has syntax: 'a.b.c.d' This also applies to CIDR Change-Id: Ia7ec8bf107abd169b6b6a91d0b8bb913fc3cc7b9 Story: 2002552 Task: 22114 - Merge "Improve resource types list rendering" - Merge "Ignore NotFound error in prepare_for_replace" - Improve resource types list rendering https://docs.openstack.org/heat/latest/template_guide/openstack.html looks a bit messy right now, this tries to make some improvements: * Reduce the TOC depth * Improve version messages * Indent property description * Fix map/list properties rendering Change-Id: I1571d1af9eba2619f8db64c54ad0b141b354a524 - Merge "Retry resource creation on conflict" - Delete snapshots using contemporary resources When deleting a snapshot, we used the current resources in the stack to call delete_snapshot() on. However, there is no guarantee that the resources that existed at the time the snapshot was created were of the same type as any current resources of the same name. Use resources created using the template in the snapshot to do the snapshot deletion. This also solves the problem addressed in df1708b1a83f21f249aa08827fa88a25fc62c9e5, whereby snapshots had to be deleted before the stack deletion was started in convergence because otherwise the 'latest' template contained no resources. That allows us to once again move the snapshot deletion after the start of the stack deletion, which is consistent with when it happens in the legacy path. Amongst other things, this ensures that any failures can be reported correctly. Change-Id: I1d239e9fcda30fec4795a82eba20c3fb11e9e72a - Merge "Define constants for Neutron client resource types" - zun: use containers.update for renaming The old way of renaming is duplicated [1]. This patch uses the new way to rename a container. In addition, this patch also bump the version of python-zunclient to 2.0.0 [1] https://review.openstack.org/#/c/557595/ Change-Id: I4ef36a3c4a805b3e041fcb9456c297e59865485c - Add a new OS::Heat::Delay resource type This allows users to insert a delay at some point in the workflow while creating or deleting the stack. Unlike OS::Heat::TestResource, the delay can be randomised so that when used in parallel across a scaling group it can be used to work around thundering herd problems. Change-Id: I49deb9560df9963120445c5900f79a1c5ca09cb5 - Merge "Docs: Eliminate warnings in docs generation" - Merge "OS::Neutron::Port: Add network attribute" - Ignore NotFound error in prepare_for_replace If the port is already deleted in neutron, we don't need to do anything. Change-Id: I9275610931a174919076330ec443f54f68cc1371 Story: #2003015 Task: 23035 - Merge "Fix server_group create issue" - OS::Nova::Server: Extend addresses attr to include network This change adds 'network' to the addresses attribute of the OS::Nova::Server resource. This enables resolving the properties of the neutron network for the server resource. Story: 1766946 Task: 19690 Change-Id: I5bef751982919103ecbefeee64bf0f2cbede0cd0 - OS::Neutron::Port: Add network attribute Adds attribute 'network' to the port resource. Similar to the existing subnets attribute which returns a list of all subnets. The network attribute returns the properties of the neutron network. Story: 1766946 Task: 18792 Change-Id: I6c667a0ff2c15aa27ca0d7943359e7f595630f87 - Merge "Revert "Remove unittests monasca until story/2002978 is released"" - Merge "Bump monascaclient to at less 1.12.0" - Fix server_group create issue Heat uses python-novaclient version 10.3.0 in which 2.64 microversion changes [1] are not included but heat-engine still gets 2.64 compute API version from client (queries nova api service) resulting in failure in creating server group. Fixed this issue by explicitly using 2.15 api version to create server_group. [1] :839da6fec0 (diff-815379030879ca27268b9336fecbe7dc)
Change-Id: I30b9fc4dd96470832f5e7516179021fbe11c63bc Story: #2003017 - Merge "Add diskimage-builder project install" - Revert "Remove unittests monasca until story/2002978 is released" This reverts commit b6197161b5228803f247728a2fa6689ed69635ee. Depends-On: https://review.openstack.org/#/c/583433 Change-Id: I1f21bd57b3df4c0d1778fe8907e402f5de7eaa77 - Bump monascaclient to at less 1.12.0 Bump mocascaclient to avoid issue in story 2002978 Change-Id: Ibe5d6469de8d18f08eeb2bfb48dfdd57b63eb62d - Fix some typos in docstrings. Change-Id: I94709efb56ed00805c5628e9f3e7a4c58ddc5428 - Remove unittests monasca until story/2002978 is released Change-Id: I3399adb0c78fdcac77654560cfafd77eb4e9d009 Story: #2002978 - Try deleting stack if not soft deleted yet If an engine is stopped before a stack is soft deleted but marked DELETE COMPLETE (updated with ian empty template in convergence), the empty stack can't be soft deleted anymore and would always end up in the stack list. Keeps the nested stack behaviour same by returning EnitityNotFound as before. Change-Id: Idc541fe0cd12d03e2d9b3cc11a1e7b0046be9d25 Story: #2002921 Task: 22901 - Merge "Add some steps for "Preparing to create a stack"" - Merge ""service" is not a default project in rdo." - Fix missing space in help text for caching option. There are C-Style string concatenations by juxtaposition in the help texts and one line was missing a trailing space which caused two sentences to run into each other. Change-Id: I8df8a57b4d27de91945a54b6bd1da47cb6ab481f - Merge "Replace create_a_stack doc stack uuid and timestamp" - Merge "Docs: Remove contrib/rackspace section from template guide" - Merge "Refactor condition expression in Resource.load()" - Merge "Fix Octavia health manager" - Merge "Fix stack delete error with none cluster" - Merge "delete_trust failure will not block a stack delete" - Add Blazar client plugin to Heat Add a Blazar client plugin which will be used by a couple of Balazar resources under development. Change-Id: I0f68fc0525db3ba299d77019a102f24b9d3cea87 Task: 19754 Story: 2002085 - Merge "Fix multi region issue for software deployment" - Merge "Destroy mox forever" - Merge "Remove mox usage from test_heat_client" - Fix multi region issue for software deployment Except commit Iec6f3606c9fdf8474f393b0990356f34d38bcf75 we still need this one to make sure the region name can be populated as deployment metadata. Task: 22854 Story: 2002781 Depends-On: I88182a9a9af74e9760b2ec9b500971f06293f0b8 Change-Id: Id87935bdb1484e66f6066361d80e245d08d9ab66 - Merge "Add functional test for version negotiation" - Merge "Use application_url in API version document" - Merge "Fix versions Controller for Py3" - Merge "Check NotFound with pool property in pool member" - Fix Octavia health manager Recently, there is a change[1] in Octavia which adding new validation to forbid using HTTP method, expectation code and URL path for non-HTTP type health monitor. For example, if user is using TCP as the type of health monitor and now Heat is using default values for HTTP_METHOD, EXPECTED_CODES and URL_PATH, which will cause error response from Octavia. And the stack creation will fail. This patch fixes it by removing those default values. [1]e1e1640af5
Task: 22876 Story: 2002904 Change-Id: Id61cde4c1416cc65b4eac90710f564699ada6a75 - Destroy mox forever Change-Id: Ib983af3392760367c4ce6256456de70d1078fbba - Merge "Handle resource plugins without default_client_name" - Remove mox usage from test_heat_client Change-Id: I3b68374e2de6b87a9389f11386eba7f7908c9915 goal: mox-removal - Merge "Update docs bug link to storyboard" - Merge "Refactor resource plugins for microversion usage" - Merge "Change NovaClientPlugin to use MicroversionMixin" - Merge "Restore auth-less version negotiation" - Merge "Fix purging events from stacks converted to convergence" - Merge "Fix an exception message" - Merge "Support region_name for software deployment" - Merge "Allow lazy load of raw_template but log warning" - Merge "fix tox python3 overrides" - Check NotFound with pool property in pool member We use pool property directly when create/update/delete/show pool member. Which will try to translate and raise value error when it's actually pool not found. In cases like pool can not be found, we will raise NotFound exception instead a value error. Story: #1747836 Task: #16037 Change-Id: Ibc8f5d125436f0f8c9ea5a62d5be4e4f8b3c4401 - Merge "Don't re-use resource-properties-data in backup stacks" - Support region_name for software deployment In magnum, we're running into a problem that the heat-container-agent can't get the region name. As a result, it can't get the correct Heat endpoint, which is causing problems. This patch fixes this from heat side. Task: 22661 Story: 2002781 Change-Id: Iec6f3606c9fdf8474f393b0990356f34d38bcf75 - delete_trust failure will not block a stack delete Deleting a tenant that has active stacks would have to issue a stack delete twice per stack to delete those stacks, that was because the delete_trust would fail, but credentials were still cleared.This change just logs the error but does not fail the stack delete. Change-Id: I9d770a91b20d1db137b3fc313c794fcee4a5e4bf Story: 2002619 Task: 22248 - Merge "Handle invalid depends field" - Fix purging events from stacks converted to convergence Stacks with convergence enabled are guaranteed not to have a backup stack, but they may have previously had backup stacks (prior to being converted to convergence) that remain only soft-deleted. In this case they may still fail the foreign key constraints in the database when purging events, so don't bail out of the workaround path when we encounter this situation. This will allow events to be purged from these stacks without having to wait for purge_deleted to purge the backup stacks first. Change-Id: If541d421a0a2af17ab937ec0a27bee6e4e9c1f6b Story: #2002643 Task: 22648 - Merge "Add segments attribute to OS::Neutron::Net" - Merge "Allow updating the segment property of OS::Neutron::Subnet" - Refactor resource plugins for microversion usage This changes the way nova client plugin is used in some nova resource plugins. Change-Id: Ibc41e0387a3056b3fde7b7690bf924e7493a7a7e - Change NovaClientPlugin to use MicroversionMixin We don't need to maintain a list of versions supported in nova client plugin. Supporting a latest microversion means supporting all versions earlier. Change-Id: I5df1ccd3ebf47b0772c903dbae0613b4f8041634 Story; #1764757 Task: 17358 - Merge "Don't depend on string interning" - Merge "Remove mox usage from test_volume" - Merge "Remove mox usage from `aws/test_volume.py`" - Fix an exception message The message contained "<built-in function id>" instead of the row ID. Change-Id: Ifd345a3098342c8ae5b7b64054f6a130dad2a03c - Rename poorly named function _query_all_by_stack() doesn't tell you what the query it returns will query for. It's events. Change-Id: If4b05a375f853873e90803be956c5e6b70470463 - Don't re-use resource-properties-data in backup stacks When purging events we are only able to (efficiently) search for references to resource properties data from events and resources in the same stack. This results in foreign key constraint errors if the resource properties data is referenced from a backup stack. To avoid this, don't reuse resource properties data IDs after moving a resource between the backup and main stacks, but duplicate the data instead. Change-Id: I93329197c99a2dba37b0e1dbd7efe7b2b17bc036 Story: #2002643 Task: 22510 - Purge events that cross-reference with backup stack In the legacy path, we have resource properties data rows that can be referenced by events and resources both in the main stack and the backup stack. In future we will stop doing this. However, to allow purging of pre-existing events, fall back to excluding resource properties data that is referenced by the other stack in the pair. This is somewhat slower than the happy path, and only affects legacy stacks, so only invoke it when we encounter an error. Change-Id: Id0a4409d53d59ff9939b64ae14ac6c4360fb9299 Story: #2002643 Task: 22511 - Ignore errors in purging events There is a known issue with purging the resource properties data for events where the same resource properties data is referenced by a resource or event in the backup stack. It's more important that we are able to store a new event than that we are able to purge old events, so catch any exceptions in purging and log an error, rather than allowing the exception to abort the event creation. Also use a transaction to ensure that we roll back the deletion of the events when we encounter an error. This ensures that any resource properties data *not* referenced by the backup stack is not orphaned because of failures deleting other rows. This will allow us to properly purge the database once the issue is fixed. Change-Id: Iefa706f91382f7d1c47e09e8f67a9fad53a9390b Story: #2002643 Task: 22334 - Handle invalid depends field Raise a proper error when something wrong is passed in a depends_on field. Change-Id: I9d27cc48a7f4c050fcc89ab5d07b19329d228955 Story: 2002544 Task: 22103 - Merge "Import Zuul setting from project-config" - Merge "Improve error reporting for missing nested template" - Fix stack delete error with none cluster Before stack delete, I delete cluster first, and then stack delete failed. Change-Id: I523575668e5f52728116c68761aa221cb27fa132 Story: #2002623 Task: 22253 - Docs: Eliminate warnings in docs generation Fix all of the existing sphinx warnings, and treat warnings as errors in future. Change-Id: I084ef65da1002c47c7d05a68d6f0268b89a36a7a Depends-On: https://review.openstack.org/553639 Depends-On: https://review.openstack.org/559348 - Merge "Unlimited cinder quotas throws exception" - Merge "External_fixed_ips break router creation." - Add segments attribute to OS::Neutron::Net Adds a attribute to the neutron Net resource to get the segments of a network. Story: 2001988 Task: 19610 Change-Id: I7acb3d29a01fa92bebdb2c85094655e27cb16a29 - Allow updating the segment property of OS::Neutron::Subnet Neutron allows updating the segment_id property of a subnet to enable migration from non routed network to a routed network. This changes the Subnet resource to allow updating the segment property so that heat users can migrate ro a routed network. Story: 2001988 Task: 19609 Depends-On: Iffda823a149a1143f46ee9a05e9640b34bf42c51 Change-Id: I6682a67da1b728f65f603ee40a200e9e0ee2e5b3 - Merge "Docs: Autogenerate config documentation" - Merge "Stop using needed_by field in resource" - Merge "Calculate convergence required_by from graph in Stack" - Retry resource creation on conflict If resource creation fails with an HTTP Conflict error, this is presumably due to a race condition with one of the underlying services. In this case, allow Heat to retry the resource creation so that hopefully in most cases the user will never have to deal with the error. Change-Id: I301bb28231fcdc249f86ec1a2f09cb993023785b Story: #1745010 Task: 17344 - Handle resource plugins without default_client_name We want to be able to call is_not_found() or is_conflict() from the client plugin from the Resource class, regardless of what resource plugin is implementing the resource type. This is only possible when the default_client_name is set, which is optional for plugins to do. As a result, when handling an exception we need to check explicitly that there is a default_client name, or (as in Resource.validate_external()) we will get an assertion error. To simplify the avoidance of programming errors, define a _default_client_plugin() method that *always* returns a plugin whose is_not_found() and is_conflict() methods can be called. Change-Id: I1005397187457d9be42ba86096644dd17287d0e7 Story: #1745024 Task: 17677 - Improve error reporting for missing nested template When the template for a TemplateResource was missing from the files dict and not otherwise available, we provided misleading feedback when validating the parent template. First, we log that we are fetching the template from a URL, even though we would not actually attempt to do so if it is a file:/// URL. Move the log to after the allowed scheme check, and log debug to indicate when we cannot find a template file. While an exception would be generated in TemplateResource._generate_schema() at resource initialisation time, it is suppressed and saved for later. An empty template is used to generate the properties schema. The exception is re-raised when validate() is called, but during a template validation we only call validate_template(), not validate(), so the first error we run into will be a mismatch of property names. (If no property values were passed in the parent template, we may not even get an error even though we won't be able to create a stack from the given data.) Also re-raise the stored exception at the beginning of validate_template() so that users will see the true source of the error. Change-Id: I1bc100684e1b84fc9ac54ef523d798b317e4dc51 Story: #1739447 Task: 22219 - Import Zuul setting from project-config All projects are moving their Zuul configs from the openstack-infra/project-config repo to their local repos. Once this change has been backported to all open stable branches, we can remove the centralised config. translation-jobs-queens is omitted because we will set that up on the stable/queens branch specifically (in place of translation-jobs, which runs only on master). Change-Id: Ia182e4dcf1a0b8e3af124b6676b20e6adb55f76b - Merge "tox: Don't delete Python3 .pyc files" - Merge "Docs: include all useful module docs" - Merge "Replace pbr autodoc with sphinxcontrib-apidoc" - Stop using needed_by field in resource During the original prototype development, I missed the fact that the `needed_by` field of a resource is no longer needed for the actual convergence algorithm:c74aac1f07
Since nothing is using this any more, we can avoid an unnecessary DB write to every resource at the start of a stack update. For now, just write an empty list to that field any time we are storing the resource. In future, we can stop reading/writing the field altogether, and in a subsequent release we could drop the column from the DB. Change-Id: I0c9c77f395db1131b16e5bd9d579a092033400b1 - Calculate convergence required_by from graph in Stack In convergence, resources can continue to exist in the database that are no longer part of the latest template. When calling Resource.required_by() for such resources, we still want to get a list of those resources that depend on them. Previously we did this using the `needed_by` field in the resource. Since this is the only actual use of needed_by, get the information from the Stack's convergence graph instead (which is generated from the resources' `requires` field and ignores `needed_by`). This eliminates any risk that the `requires` and `needed_by` fields get out of sync, and allows us to get rid of `needed_by` altogether in the future. Change-Id: I64e1c66817151f39829d5c54b0a740c56ea8edad - Don't depend on string interning Use '==' instead of 'is' to compare strings. In practice, short strings that appear in the source code are interned in CPython, but this is implementation-specific. Change-Id: I582029629afa9f3b8f6890f6b8c668d70de9f836 - Remove mox usage from test_volume Change-Id: I6a3caf249c693c1e5afdef463a7f6bf1d1ee1906 goal: mox-removal - Unlimited cinder quotas throws exception Cinder-quota-resource is supposed to support a value of -1 to indicate an unlimited quota for a particular key (volumes, gigabytes or snapshots). However the implementation had an iterator error (you cannot modify a dict as you are iterating over it) and no unit tests to expose that bug. This commit fixes the issue. Change-Id: Iea28fdc38c51df5e87e6957a6e4797feea9057ee Story: 2002550 Task: 22112 - Add functional test for version negotiation the test attempts to make an unauthenticated request to the Heat API root. Change-Id: Ib14628927efe561744cda683ca4dcf27b0524e20 Story: 2002531 Task: 22077 - Use application_url in API version document this handles the possible vhost in Heat API path correctly (like http://mucloud.com/heat-api) contrary to host_url. Change-Id: I2f4dbb7d98fbdf5314d5771309919c5b25a72f94 Story: 2002531 Task: 22075 - Fix versions Controller for Py3 the webob's Response.body in fact accepts only bytes, so under Python3 an attempt to assign a text value to webob's response.body produces the following exception: TypeError: You cannot set Response.body to a text object (use Response.text) Change-Id: I6352ec0662441dab039f15b35c8383e6d7d3dc78 Story: 2002531 Task: 22076 - Remove mox usage from `aws/test_volume.py` Change-Id: I3ed928fddf2882cb6397bc6fdc52753c1c6beb6b goal: mox-removal - External_fixed_ips break router creation. In the _resolve_subnet method the code iterates over a dictionary while modifying it using the "pop" command. This is incorrect and causes a Runtime error. The fix is to make a copy, iterate over that while manipulating the original dictionary. Python 2.7 can do this using iter(), however python 3 would not have worked. The fix does a copy first which makes the code operate the same in both interpreters, since there is no "six" routine for this. Change-Id: I11468bfbe7fafd5c0d79138a94520dec1aafbc04 Story: 2002549 Task: 22111 - fix tox python3 overrides We want to default to running all tox environments under python 3, so set the basepython value in each environment. We do not want to specify a minor version number, because we do not want to have to update the file every time we upgrade python. We do not want to set the override once in testenv, because that breaks the more specific versions used in default environments like py35 and py36. Change-Id: I6dde50de599eb3b6b001cbc044929c64df044575 Signed-off-by: Doug Hellmann <doug@doughellmann.com> - Merge "Remove mox usage from test_api_cfn_v1" - Merge "OS::Nova::Server: Extend addresses attr to include subnets" - OS::Nova::Server: Extend addresses attr to include subnets This change adds 'subnets' to the addresses attribute of the OS::Nova::Server resource. This enables resolving the subnet properties for the server resource which brings parity with OS::Neutron::Port's 'subnets' attribute. Story: 1766946 Task: 19689 Change-Id: I6927f1e142e83df7258e259bffc86819e4cbe881 - Restore auth-less version negotiation change Change-Id: I097bf70431a999a0f6aa56079ffb5743b50d4d7f inadvertently started to require (keystone) authentication when accessing API root to get the API version document. This breaks standard version negotiation (that usually does not require auth) and also some monitoring / loadbalancing that used to check returns on GET to API root. Change-Id: If8f13def196442e6f28616e88972f28f8ec23d0d Story: 2002531 Task: 22072 - Update comments on skipped tests SoftwareConfigIntegrationTest has been skipped since 20f17ec25b40ad9654c8ee6f6ce3d7294d44e02c. test_server_signal_userdata_format_software_config has not been skipped since e50ce9c6e2763df3abadd74d18f6d9e54a6af1e7. Change-Id: I71c70ff516d2d4cbd0aef2a3833c6f7bd4cc3984 - Merge "Remove mox usage from test_vpc (part 2)" - Merge "Remove mox usage from test_vpc (part 1)" - Merge "Remove mox from test_neutron_security_group" - Merge "Switch to neutron-* service names" - Merge "Stop testing neutron-lbaas in gate jobs" - Merge "Remove mox usage from `aws/test_security_group.py`" - Switch to neutron-* service names This patch switches gate jobs from old lib/neutron-legacy devstack lib to the new lib/neutron. Change-Id: Icd664cf94e43b4f30e1d5a23a73239f6516dcba0 - Stop testing neutron-lbaas in gate jobs neutron-lbaas has been deprecated since long. Let's remove the service from gate jobs and skip the relevant tests. Change-Id: I8eede46ed2ad014522e00d28b0b7c0b194a0b996 - Merge "Remove mox from openstack_v1/test_stacks" - Update docs bug link to storyboard Change-Id: I9c6cc355b12665036d6faf10773e504a39516056 - Merge "Keep old files in file map for rolling update" - Merge "Don't allow nested or stacks in FAILED state to be migrated" - Remove mox usage from test_api_cfn_v1 Change-Id: I094dea06fc30b924c05be90533769e4850ae803e goal: mox-removal - Remove mox from openstack_v1/test_stacks Change-Id: I71e838bdbff3c40845d2aff4ada8070a39ebe6fa - Merge "Remove install-requirements script" - Merge "Download octavia image in tests" - Handle new oslo.service release It changes the way timers are run during unit tests, let's add an additional mock. Change-Id: Ia3cd8e13992fd561b88752a2f86ce5eb61ebbefd - Merge "Ignore dns domain NotFound when deleting record" - Merge "Reset resource replaced_by field for rollback" - Merge "Make resource requires a set instead of a list" - Merge "Create replacement resources with correct requires" - Merge "Calculate the requires list in check_resource" - Merge "Don't pass input_data to Resource.delete_convergence()" - Merge "Retry on deadlock in purge_deleted" - Merge "Fix gerrit tool query statement" - Merge "Increment resource atomic key when storing attributes" - Merge "Delete internal ports for ERROR-ed nodes" - Merge "Update http links for doc migration" - Delete internal ports for ERROR-ed nodes Precreated ports are never deleted by nova, thus heat must delete the ports that it creates that are internal ports. This is important when baremetal is used via ironic where the MAC address of the port must match the physical node. Neutron does not permit duplicate MAC addresses, hence ports can't be orphaned. Change-Id: I965abefbb10c22a635c35e3a44d227045f33885c Story: #1766763 Task: #18791 - Reset resource replaced_by field for rollback We seem to leave replaced_by field of a resource after rollback (if a stack cancel is issued before the replacement resource is added to the stack). This in turn results in the stack getting stuck in IN_PROGRESS for further updates. Change-Id: I3361ad90e303d24a16a943408fe0f6c406cf785f Co-Authored-By: Zane Bitter <zbitter@redhat.com> Task: #19582 Story: #2001974 - Download octavia image in tests Instead of rebuilding one. Change-Id: I393158a2fac6d4399a7dde6a0f2e482751359072 - Keep old files in file map for rolling update We would need old resource definitions when doing rolling update of certain group resources like RG/ASG. Therefore update the file map with the files from old template. Change-Id: I8f880e5b23c25159ecab1c63b594329d8df33973 Closes-Bug: #1765454 Task: #17360 Story: #1765454 - Don't allow nested or stacks in FAILED state to be migrated Stack and its nested stacks should be in *_COMPLETE state for it to be migrated. Also, we should not allow migration of a nested stack. This also changes stack_get_all_by_owner_id() to not select the backup stacks. This seems to be used only in convergence code path and migrate_convergence_1(). Change-Id: Icd54465d0c593557a12d853ddee4ee8ce6483499 Closes-Bug: #1767962 Story: #1767962 Task: #17363 - Merge "Add retry for resource_purge_deleted call" - Update http links for doc migration This patch is proposed according to the Direction 10 of doc migration(https://etherpad.openstack.org/p/doc-migration-tracking). Change-Id: I2cf863151579357ca0a3227467fa9758463508b1 - Merge "Change non-apache job to non-voting" - Merge "Remove obsolete identity-v3-only job" - Fix gerrit tool query statement Change-Id: I7707318e445e55751c1af45d5d08d09ddcb50b66 - Add retry for resource_purge_deleted call Retry when resource_purge_deleted failed with deadlock or other exceptions. Change-Id: Iadab7b45e653fc6f465e257d3f450c4eaf509297 Story: #2001997 Task: #19629 - Change non-apache job to non-voting Change `heat-functional-convg-mysql-lbaasv2-non-apache` to non-voting. Change-Id: I5ddddd0957fd8129b2af71c66fc5578976226c83 - Merge "Remove mox from test-neutron-firewall" - Merge "Log useful information in the API log" - Merge "Trivial:remove unused import and add reasonable path in import" - Merge "Fix debug logs in _stale_resource_needs_retry()" - Merge "Sync support network type for provider network" - Merge "Remove mox usage from `aws/test_network_interface.py`" - Merge "Remove mox usage from `aws/test_instance_network.py`" - Merge "Remove mox usage from `aws/test_waitcondition.py`" - Sync support network type for provider network Now heat only supports 'vlan' and 'flat' network type for OS::Neutron::ProviderNet, this will add all other supported network types: 'vxlan', 'local', 'gre' and 'geneve'. Change-Id: If7cb4fbc83f5aa62a0b73e003ff23ec2b40b2ded Closes-Bug: #1692445 - Merge "Move context middleware earlier in pipeline" - Fix debug logs in _stale_resource_needs_retry() Missing parameters introduced by 22eb2e253822c176a1316a85317ceeec610ed140. Change-Id: I5d49c3ee0d076e2673d0b2bbacf04c2274e56da7 - Merge "Remove mox usage from test_stack" - Merge "Retry resource check if atomic key incremented" - Merge "Add Rocky versions" - Merge "Add catch-all for property errors in implicit dependencies" - Merge "Use stack_id of None for service timer" - Retry on deadlock in purge_deleted Change-Id: I6dc7429b2564ab07dd66bf0cf0c8657ac1928508 Closes-Bug: #1721695 - Merge "Remove mox from test_neutron_loadbalancer 4/4 - all rest tests" - Merge "Merge before/after 'requires' list on update failure" - Merge "Ignore property errors in implicit dependencies" - Merge "Log traversal ID when beginning" - Docs: Autogenerate config documentation The documentation we had of Heat's config options was static and had last been updated for the Ocata release. Ensure the documentation is kept up to date by automatically generating it from the source when the docs are built. Change-Id: I06b3848451541f8f2d98748c06c556b9dc1104c6 - Remove mox usage from `aws/test_security_group.py` goal: mox-removal Change-Id: I5019eea5e7a4f2589272357afa10e92023044895 - Merge "Docs: Make stack domain users docs visible" - Merge "Avoid double-write when updating FAILED rsrc with no change" - Merge "Docs: modernise links" - Merge "Docs: Fix broken external links" - Merge "Docs: use local references for internal links" - Remove mox usage from test_stack Change-Id: I56c27156536465fdfb594b9cc8cdf2a6c631141e goal: mox-removal - Remove mox usage from `aws/test_network_interface.py` Change-Id: I300dbd9f29a4a935073027e611e1d7a5f28ca304 goal: mox-removal - Remove mox usage from `aws/test_instance_network.py` Change-Id: I741f011bbbea8816716e3fd3a01b6d9cadf9022f goal:mox-removal - Remove mox usage from `aws/test_waitcondition.py` goal:mox-removal Change-Id: Ib8a0dbebea9ff125c2172179a2b83a8b64a2bd7a - Merge "Remove mox usage from test_resource" - Merge "Remove mox from test_instance" - Remove obsolete identity-v3-only job Keystone v2 was removed in Queens, so all devstack builds are now identity-v3-only. Change-Id: Icb01161be206576ad9445baca6b2a880fb1ef0e0 - Add Rocky versions * Add Rocky code name * Add new HOT version for Rocky release The new version is "2018-08-31" or "rocky". * Add sem-ver flag Sem-Ver: api-break Change-Id: I261b6c28b8b7ee9e75ca9a895155a656ef82cd0d - Merge "Remove mox usage from test_stack_resources and tools" - Merge "Remove mox usage from test_software_config" - Merge "Remove mox usage from test_engine_service" - Make resource requires a set instead of a list Since we are doing set operations on it, make the 'requires' attribute of a Resource a set and only convert to/from a list when loading/storing. To avoid churn in the database, sort the list when storing it. Change-Id: I137fae8ae630eb235d7811fcba81738d828e6a1e - Create replacement resources with correct requires When we create a replacement resource, do so with the correct requires list that it will ultimately have, instead of a copy of the old resource's requires. This will happen anyway when Resource.create_convergence() is actually called (which is on the other end of an RPC message, so it may not actually happen if another transition starts), but this makes it consistent from the start. Change-Id: Idf75a55be8d75e55c893ec1fb6ee3704f46bdc4f - Calculate the requires list in check_resource Obtain the list of required resources in one place in check_resource, instead of in both Resource.create_convergence() and Resource.update_convergence(). Since these require knowledge about how the dependencies are encoded in the RPC message, they never really belonged in the Resource class anyway. Change-Id: I030c6287acddcd91dfe5fecba72c276fec52829b - Don't pass input_data to Resource.delete_convergence() In the original prototype for convergence, we passed the input_data from the SyncPoint to the resource when calling the equivalent of convergence_delete(), so that we could clear and needed_by references that no longer exist. This is pointless for a few reasons: * It's implemented incorrectly - it *sets* the referenced resources into needed_by instead of clearing them from it. * We don't actually pass any input data - in WorkerService.check_resource() it's always set to an empty dict for cleanup nodes, regardless of what came in on the wire. * We don't store the result to the database unless we're deleting the resource anyway - in which case it doesn't matter. * It turns out that even in the prototype, the whole needed_by mechanism isn't actually used for anything:c74aac1f07
Rather than pretend that we're doing something useful with the input_data here, just set the needed_by to an empty list, which is what was happening anyway. Change-Id: I73f6cf1646584dc4a83497f5a583c2c8973e8aba - Merge before/after 'requires' list on update failure If an update of a resource fails, its 'requires' should be set to a union of the previous and new requires lists. This is because if the resource depends on a resource that has been replaced in this stack update, we can't know if the current resource now depends on the new or old version of the replaced resource if the current resource failed. This is achieved by splitting up the setting of 'requires' and 'current_template_id', and changing them directly in the update() method instead of via a callback. When the resource state is changed to UPDATE_IN_PROGRESS, the new requirements are added to the old ones. When the state is changed to UPDATE_COMPLETE, the new requirements replace the old ones altogether. If the update fails or handle_update() raises UpdateReplace, the union of the requirements is kept. If _needs_update() raises UpdateReplace, the old requirements are kept. The current_template_id is updated when the state is changed to either UPDATE_COMPLETE or UPDATE_FAILED, or when no update is required (_needs_update() returns False). This also saves an extra database write when the update fails. Change-Id: If70d457fba5c64611173e3f9a0ae6b155ec69e06 Closes-Bug: #1663388 - Avoid double-write when updating FAILED rsrc with no change In convergence, when a resource is traversed and left unchanged, we must still update the current template for it in the database. In addition, if the resource was unchanged in the template but already in a FAILED state and we elected not to replace it by returning False from needs_replace_failed(), we must also update the status to COMPLETE. Currently, we do those in two separate writes. This is an unnecessary overhead (albeit for a fairly rare case), and the two writes can be combined into one in the case where both changes are required. Change-Id: I9e2f1e27ce2c119647c9fe228484228d2c15d943 Related-Bug: #1763021 - Retry resource check if atomic key incremented When updating a resource that hasn't changed, we didn't previously retry the write when the atomic_key of the resource didn't match what we expect. In addition to locking a resource to update it, the atomic key is also incremented when modifying metadata and storing cached attribute values. Apparently there is some mechanism that can cause this to happen in the time between when the resource is loaded and when we attempt to update the template ID &c. in the DB. When the resource is not locked and its template ID hasn't changed since we loaded it, we can assume that the update failed due to a mismatched atomic key alone. Handle this case by sending another resource-check RPC message, so that the operation check will be retried with fresh data from the DB. Change-Id: I5afd5602096be54af5da256927fe828366dbd63b Closes-Bug: #1763021 - Merge "Fix lower-constraints" - Do deepcopy when copying templates test_template is defined as a dict, so do a deepcopy instead. Change-Id: I0f7d222d5f95aa8e8ed73ae0d7a100470271e113 Closes-Bug: #1766448 - Merge "Fix indentation in hot_spec.rst" - Remove mox usage from test_stack_resources and tools Remove mox usage from test_stack_resources. Also remove mox usage from `heat/tests/engine/tools.py` goal: mox-removal Change-Id: I97e1cf983db411c9c1cf8f507d92eead03067a93 - Remove mox usage from test_software_config Change-Id: I122ab7cdfad43ab792b69086ed41733b868a9b32 goal: mox-removal - Remove mox usage from test_engine_service goal: remove_mox Change-Id: I53cc8591b8bc74fed2c609796421ef80b3c0ef56 - Fix nova fakes for server listing With python-novaclient 10.2.0 after eff607ccef91d09052d58f6798f68d67404f51ce server listing is done in multiple requests. Change-Id: Ib02278fca2a17ce2f26d5e28a5ac0971bd80657a Closes-Bug: #1766254 - Merge "Correct behaviour of update_replace property in test resource" - Merge "Configure hidden tag for tempest test" - Merge "Remove mox usage from test_docker_container" - Merge "Fix broken test in DockerContainerTest" - Merge "Fixing Senlin incompatibility with openstacksdk 0.11.x" - Merge "Fixing unicode issue when to_dict is called on py2.7 env" - Merge "Add MicroversionMixin for microversion support" - Log traversal ID when beginning At the beginning of a convergence traversal, log the traversal ID along with the dependency graph for the traversal. This could be useful in debugging. Also, log it at the DEBUG, not INFO level. Change-Id: Ic7c567b6f949bdec9b3cface4fa07748fbe585eb - Remove install-requirements script this script is not used anywhere anymore, and it will not work with recent pip 10.x release due to changes in the pip API. Initially this script was meant to install only a limited set of requirements to run heat_integrationtests without having to install whole Heat. Since the split of heat_tempest_plugin the tests left in the heat_integrationtests are meant to be run only in a DevStack-like env, so all the requirements are already available. Change-Id: I94c82fe100dfb6557cc960ef1198f1195780f28f - Merge "Remove mox from test_neutron_network_gateway" - Merge "Remove mox from test_neutron_loadbalancer 3/4 - PoolMemberTest" - Remove mox usage from test_resource Change-Id: Ie55ee1e692053b438251b9aacb85db290edd308e goal: mox-removal - Merge "Add environment_files parameter in CREATE_STACK request" - Merge "Remove mox usage from test_api_ec2token" - Docs: modernise links Switch to https, replace outdated URLs with canonical ones. Change-Id: I7c474dbf0b21f922b6625e31d6ae7df1e198b8de Co-Authored-By: chenghuiyu <yuchenghui@unionpay.com> Co-Authored-By: Max Pavlukhin <pavlukhinm@gmail.com> Co-Authored-By: yfzhao <dsware@126.com> Co-Authored-By: ricolin <rico.lin@easystack.cn> - Docs: Fix broken external links Change-Id: Iad15069ccee42c1d409915f79e0d8af8e124fe48 Closes-Bug: #1710084 - Docs: use local references for internal links Some internal links in the documentation are broken after the re-organisation because they used absolute URLs. Use :ref:, or at least :doc: instead and let Sphinx take care of keeping the links up to date. Also add a redirect for docs we know to have moved, since they may be linked from other places externally. Change-Id: I107f40e23a757052da7f5179338f88369a4a72cb Partial-Bug: #1710084 - Docs: include all useful module docs The list of modules to exclude from autodoc was set some time back, and is pretty arbitrary looking. Lots of modules that would be useful to a Heat developer to see documented (e.g. all of heat.common, heat.db) were not documented, while some other fairly useless ones were. Change-Id: Ie3aa5bbe8deb8d07eef5fcc352e349a1fa7c68bc - Replace pbr autodoc with sphinxcontrib-apidoc This fixes local building of the documentation using tox, and allows the gate to stop relying on pbr and move completely to the new docs PTI. http://lists.openstack.org/pipermail/openstack-dev/2018-March/128594.html Change-Id: I6ef3e102f5a169262efbbaeec61452f0a2443fb4 Depends-On: https://review.openstack.org/560577 - Docs: Reorganise landing page Consolidate into three sections based on the audience: operator, user, and developer. Don't include API autodoc inline, just link to it. Change-Id: I3c301d66dd05698fca3582761cb0177d66431483 - Fix lower-constraints In the course of switching over to lower-constraints, the lower constraints got out of sync with the requirements. This change fixes the constraints using the script mentioned in http://lists.openstack.org/pipermail/openstack-dev/2018-April/129056.html Change-Id: Iae317fa745862ebde4115ca8e77a26c6c9be20ac - Remove mox usage from test_api_ec2token Change-Id: Ia62a2bf62d88d0641406b183ae912e8cfc1010f4 goal: mox-removal - Increment resource atomic key when storing attributes When we store resource attribute values in the database, we increment the atomic_key in the DB. This means we should also increment the _atomic_key member of the Resource object, so that any future writes to the database will expect the correct key. In practice, this doesn't matter because there's no RPC API call that stores attributes and then writes to the resource in the database for other reasons. (Attributes are updated if they are missing on show-stack with outputs, which is otherwise a read-only call; they're also updated at the very end of a check-resource call in convergence.) But by maintaining a consistent model this could help to prevent future bugs. Change-Id: I71ed5907d33f2293c04ad3bace639b1d114e9a77 Related-Bug: #1763021 - Merge "remove mox usage from test_subscription" - Replace create_a_stack doc stack uuid and timestamp Change-Id: Ib9521ed557a04b7558ef9b08698744169fc82b96 Signed-off-by: Yuanbin.Chen <cybing4@gmail.com> - Fixing Senlin incompatibility with openstacksdk 0.11.x When running Heat with OpenstackSDK 0.11.3 (or older) no clustering version is being passed on, causing the call to Senlin to fail with the following error message. > Unknown version in URI from With this patch we always set the api version for clustering. Change-Id: I61876b599373c5794bb9edb2088b15d21f7584e2 Closes-Bug: #1760932 - Merge "Remove mox from openstack_v1/test_resources" - Merge "Remove mox from openstack_v1/test_actions" - Merge "Remove mox from openstack_v1/test_events" - Fix indentation in hot_spec.rst Current indentation is wrongly interpreted as quotation. This patch fixes that issue. Change-Id: I0372e2d19036930d3d6795d363a0bc71e968fabc - Fixing unicode issue when to_dict is called on py2.7 env When using non-unicode old style user id such as Gāo Unicode error popup on py2.7 environment Fixing it on common/context.py Change-Id: I95e49f359410049ff5b254cd1b8ee16402c8719d Closes-Bug: #1761629 - Refactor condition expression in Resource.load() Even I get confused by this code, and I wrote it. Change-Id: I055b7a51de6f1e8c380c52d43233d3d0c3abb0ea - Remove mox from test_sqlalchemy_api Change-Id: Ided47ef8ce6d8ed689eef297535c3b7bb6f65481 - Merge "Remove mox from test_scheduler.WrapperTaskTest" - Merge "Remove mox from test_scheduler.TaskTest" - Merge "remove mox usage from test_queue" - Merge "Remove mox usage from test_server_tags" - Remove mox from test_provider_template Change-Id: I1439a44ce16a01b65863a3843ecb03445ee96368 goal: mox-removal - Remove mox from test_user Change-Id: Ie6f07c5c8f19e4b16b474559cfcf1c9a50bef281 goal: remove-mox - Remove mox from test_hot Change-Id: Ib188f26d4f494f6dc6f5e10e82f70f8e51b69252 goal: mox-removal - Remove mox from test_s3 Change-Id: I41bdf5bda3e181ad53e1a0a96d0c2d79c61c9a1d goal: mox-removal - Remove mox from test_lifecycle_plugin_utils Change-Id: I003c70ad251639921523b7fa1b453340a3faa233 goal: mox-removal - Remove mox from test_event Change-Id: I86b8661ea6a3c42016d930d2e75c3e6d8bd8a2da goal: mox-removal - Remove mox from test_stack_user goal: mox-removal Change-Id: Ibd50e07647137d48a949673de4455a01dd9a90a6 - Remove mox from test_urlfetch Change-Id: I0bb2993c458e58fde26714f0a0af866bbeaa84dd goal: mox-removal - Merge "Don't load nested stack to get TemplateResource template" - Merge "Remove mox from test_server" - Merge "Remove mox from test_nova_client" - Remove mox from test_neutron_loadbalancer 4/4 - all rest tests Change-Id: I42c4f6e342a2360769c4c87666774634ea9610c2 goal:mox-removal - Remove mox from test_scheduler.WrapperTaskTest Change-Id: Ife18820b3de84e7e6f1f18a6f76ad412eb61198e - Remove mox from test_scheduler.TaskTest Change-Id: Iff1b1632c66d6727798969a8ca6506d8d0fb2051 - Remove mox from test_neutron_loadbalancer 3/4 - PoolMemberTest Change-Id: I7decb2d5dc85987ad6f4dafe9576c88c1daef867 goal: mox-removal - Remove mox from test_neutron_loadbalancer 2/4 - PoolTest goal: mox_removal Change-Id: Ib30fda8b81224bc2ad9361a4ce323fd6859d4846 - Merge "Remove mox from test_neutron_loadbalancer 1/4 - HealthMonitorTest" - Merge "Remove mox from test_scheduler.DependencyTaskGroupTest" - Merge "Remove mox usage from test_nokey" - Merge "Avoid loading nested stack in CloudFormation Stack resource" - Merge "Eliminate nested stack loading in InstanceGroup/ASG scaling" - Remove mox usage from test_vpc (part 2) Change-Id: I7c02da9858f9d328eff2e8810eec32e54267416e goal: mox-removal - Remove mox usage from test_vpc (part 1) Change-Id: I216b2db603ebc8b593626bbed6872ad11769d42c goal: mox-removal - remove mox usage from test_subscription Change-Id: If7611327835261642d1ae42df014660f8163564b goal: mox-removal - remove mox usage from test_queue Change-Id: Iabdacfc9caea330ac7c68a992e5fed4f918ebe6b goal: mox-removal - Remove mox usage from test_server_tags Change-Id: Ia646d50f53e57922814e8c86f119c665a8566fb1 goal: mox-removal - Remove mox usage from test_docker_container Change-Id: I331dfd937efad0dafed3aaa29b2d5a3d9741a4a6 goal: mox-removal - Fix broken test in DockerContainerTest Change-Id: I5b72684ff913c936943a37310fbea2b9684ef180 - Imported Translations from Zanata For more information about this automatic import see: https://docs.openstack.org/i18n/latest/reviewing-translation-import.html Change-Id: If6a8ee1373916d778ebe769e2b904d68f564a40e - Remove mox from openstack_v1/test_events Change-Id: I7122ef89ff5c10ace73d2bc3964ae5f1feb304b9 - Merge "Remove mox from test_neutron_provider_net.py" - Merge "Move openstackdocstheme to extensions in api-ref" - Merge "Remove mox from test_extraroute" - tox: Don't delete Python3 .pyc files Python3 is smarter than Python2 about when to make use of .pyc files, so there's no need to remove them before running unit tests just in case the developer has switched branches.[1] To save recompiling before every test run, disable this behaviour for Python3. Also, use a consistent find command in all of the environments where we do this. [1] https://www.python.org/dev/peps/pep-3147/#case-3-pycache-foo-magic-pyc-with-no-source Change-Id: Iabaa4a697329e7538fab93541f41fc208e591382 - Remove mox from test_instance Change-Id: Ie89ccc7fff15a2b1d7c5a2b3655d5bffd0279720 - Merge "Updated from global requirements" - Merge "Resolve race in providing deployment data to Swift" - Updated from global requirements Change-Id: If19195b9d1c28c45d62636ed5078b0e7bc5caefa - Merge "add lower-constraints job" - Add MicroversionMixin for microversion support This generic mixin overloads plugin client() method to return a client with max microversion unless a specific version is requested. This provides two abstract methods - get_max_microversion() - is_version_supported() for client plugins to override when supporting microversions. Change-Id: I8873ab6d815671b6647b08578d1406dd874269f6 - Merge "Use wait_random_exponential from tenacity 4.4.0" - Remove mox from openstack_v1/test_resources Change-Id: Idc165f515b1c8b9990d74b9f8d099c48620f26cb - Merge "Persist external resources on update" - Merge "Remove mox from test_neutron_metering" - Merge "Dashboard creator: fix bugs query" - Move openstackdocstheme to extensions in api-ref Move openstackdocstheme to extensions. According to the guide below: https://docs.openstack.org/openstackdocstheme/latest/ Change-Id: I09acab1026c621d29ba5076345f8facdf7cfd91f - Remove mox from openstack_v1/test_actions Change-Id: I5fb89640849dd3ba2692952c895429574ac52311 - Dashboard creator: fix bugs query Gerrit used to allow regexps in topic: queries, but now topic: is only for exact matches. intopic: can be used to search within topics. Change-Id: Icc5c80ead124b4383198915984543a7987b1bf2e - Use wait_random_exponential from tenacity 4.4.0 Now that we depend on tenacity >=4.4.0, we can use the library's version of the wait_random_exponential wait strategy in place of our own. Change-Id: I13d3222808a98ef7e333f58df931c8f950ac1221 Depends-On: https://review.openstack.org/556309 - Updated from global requirements Change-Id: If185ea0cc3ac8ab9c93da6e3602838a1653435ed - Merge "Updated from global requirements" - Resolve race in providing deployment data to Swift While we are careful to ensure there are no race conditions in updating the software deployment metadata in the Heat database, there was no such protection in place for the actual update of the metadata provided to the server via Swift. In legacy Heat, this race could only occur when software deployments were created from the REST API or from separate stacks deploying software to the same server, because resources in the same stack were handled by a single thread making synchronous calls to create the software deployments. However, with convergence, resources in the same stack not only can be processed concurrently but will tend to be synchronised if they have the same dependencies (since they'll be triggered by consecutive asynchronous RPC messages). This handles the race by retrying if a concurrent update is detected by the time the data has been provided to Swift. Note that this is only guaranteed to result in the latest data being available on the assumption that a later write to Swift cannot overwrite an earlier one that has been accepted. This is probably *not* guaranteed by Swift in the presence of clock skew. Nonetheless, this should drastically reduce the failure rate. Similar caveats would probably apply to Zaqar, although Zaqar nominally provides FIFO queues (in practice, it likely depends on the backend used). However, with Zaqar clients receive all updates, not just the latest one, and os-collect-config stores the deployments contained in every message. Thus, Zaqar is not affected even assuming out-of-order arrival. Change-Id: Ic9a447f27e8c51f91f47f93b0fd1e9710341ec38 Closes-Bug: #1731032 - Merge "Generate user passwords with special characters" - Merge "Fix entropy problems with OS::Random::String" - Remove mox from test_nova_client Change-Id: I68b0539da33bd438b2314a0eb5fdc6354b7e3152 - add lower-constraints job Create a tox environment for running the unit tests against the lower bounds of the dependencies. Create a lower-constraints.txt to be used to enforce the lower bounds in those tests. Add openstack-tox-lower-constraints job to the zuul configuration. See http://lists.openstack.org/pipermail/openstack-dev/2018-March/128352.html for more details. Change-Id: I72fd5f8c87133713cd272de210b7e957880f48f9 Depends-On: https://review.openstack.org/555034 Signed-off-by: Doug Hellmann <doug@doughellmann.com> - Merge "Remove mox from test_neutron_vpnservice" - Updated from global requirements Change-Id: Ide51dccddd1ff2b384ec9ce1cfe93c1375408623 - Remove mox from test_server Change-Id: I3d832351fe050c496f0dd9b2c1a9f12623637c30 - Generate user passwords with special characters When we create user accounts in order to allow signalling back to Heat, generate the passwords using a mix of upper- and lower-case letters, digits and special characters. This should pass most password_regex filters that users are likely to configure in Keystone, which was not true of the previous method (using only hex characters from a UUID). Change-Id: I7702d6ab550e4f1f53c4cea9f67ed0402afbd66d Closes-Bug: #1666129 Closes-Bug: #1444429 - Merge "Remove mox from test_neutron_floating_ip" - Fix entropy problems with OS::Random::String When generating a random string, once we had selected from the various required pools, we continued by selecting a pool at random and then selecting a character from that pool at random. This did not take into account the differing sizes of the available pools, nor the fact that the same character could appear in multiple pools, which resulted in a non-uniform probability distribution of characters. Since users mostly make use of this feature to generate default passwords for services they are deploying, this would result in the generated passwords having slightly less entropy than expected (and pathological cases were possible). Rectify this by always selecting non-constrained characters from a single combined pool, and by ensuring that each character appears only once in any pool we're selecting from. Since we also want to use this method to generate passwords for OpenStack Users, the new implementation is in a separate module in heat.common rather than mixed in with the resource's logic. Also, use a StringIO object to collect the characters rather than repeatedly appending to a string. Change-Id: Ia7b63e72c1e3c0649290caf4fea8a32f7f89560b Closes-Bug: #1757300 Related-Bug: #1666129 Related-Bug: #1444429 - Configure hidden tag for tempest test Change-Id: I09d4a6556d749c9d034163cdc8fb7ff24aa6a659 Partial-Bug: #1757117 - Merge "Remove mox usage from test_scaling_group" - Merge "Remove mox usage from test_heat_autoscaling_policy" - Remove mox usage from test_nokey mox-removal Change-Id: I75f08137495afcb89ab294ee5ef0f91116a529dd - Remove mox from test_neutron_metering Change-Id: Iee7fef53442e905a67f538438dfa56fc4d228432 - Updated from global requirements Change-Id: Icea60c74a8f6498ddcdd929fbc4ebfae0fcf021b - Remove mox from test_neutron_security_group Change-Id: Ib039f6785efd64935f3e0b0ddab8fed0fed3cbd0 - Persist external resources on update We were missing a call to update the template for convergence purpose when updating an external resource. Change-Id: I37e29aaf4110faa00c777c76640be485224720ca Closes-Bug: #1756269 - Merge "Fix some reST field lists in docstrings" - Docs: Make stack domain users docs visible This document was effectively hidden because of a misspelling in the index. Change-Id: If1bb64f042f38fadfb71dfd58aa854211ffe3d8a - Remove mox from test-neutron-firewall Change-Id: If5da8140d84c545df57a62f4bebeebb30f139dce - Create doc/requirements.txt For compliance with the Project Testing Interface as described in: https://governance.openstack.org/tc/reference/project-testing-interface.html Refer to: http://lists.openstack.org/pipermail/openstack-dev/2017-November/124815.html Change-Id: Ifdfdaacad8574fcfcac9c1b52b0bed6fed4f8411 - Updated from global requirements Change-Id: Ifc7d18e59788ce3c949a2292187292dde035979c - Merge "Turn grenade-heat-multinode to voting" - Remove mox from test_neutron_network_gateway Change-Id: Ifd29191a833c15e3ca66085791ac2e2ed36da1d8 - Turn grenade-heat-multinode to voting Change-Id: Ibbe2404576025f95b35ac50c8f6a9596ff9ab701 - Merge "Remove mox usage from test_gnocchi_alarm" - Remove mox from test_neutron_loadbalancer 1/4 - HealthMonitorTest Change-Id: I21e5750bd49832577f9fed848339386e21ee402c goal: mox-removal - Remove mox from test_scheduler.DependencyTaskGroupTest Change-Id: Iaaecc95f2224714c57f6c855ef84bb3a8eb42776 - Merge "Imported Translations from Zanata" - Use mock in test_auth_password.py In heat, some unit tests still use mox3, which will make the test speed more slower, so I decide to remove it. Change-Id: I68f016053de61ec274636dfe9f83ae57b1b9a84b - Remove mox usage from test_gnocchi_alarm Change-Id: I93de5ffa5b1d9536786e5cb497006abe40129b43 goal: mox_removal - Docs: Remove contrib/rackspace section from template guide We removed the contrib/rackspace directory in 13a5b43d6565af49cabf7323613f4e7924160ce9, so remove the now-empty section in the template guide for the resource plugins it formerly contained. Change-Id: Id2bf166990c27a2ac0be38e5bdd19548e74d8f60 - Remove mox from test_neutron_provider_net.py Change-Id: I056ea0b412ad05a18ab6e3d42661a7af642f2ae2 - Imported Translations from Zanata For more information about this automatic import see: https://docs.openstack.org/i18n/latest/reviewing-translation-import.html Change-Id: Ic5ecbacc7c030c09eef8fb1802619f6057899afc - Updated from global requirements Change-Id: I19eb6a9a21b6a1f9a3a6fd42bff3c9355c6b9bcc - Fix some reST field lists in docstrings Probably the most common format for documenting arguments is reST field lists [1]. This change updates some docstrings to comply with the field lists syntax. [1] http://sphinx-doc.org/domains.html#info-field-lists Change-Id: Ifeedd870f57a4b79f04c5fb9d25d369c9d7ea235 - Remove mox from test_neutron_vpnservice Change-Id: I97892e205e2d31f99e511e83c4ff608543dc7531 - Merge "Remove usage of mox from test_eip" - Merge "Remove mox usage from test_floatingip" - Merge "Remove mox from test_waitcondition" - Merge "Remove mox from test_stack_resource" - Remove mox usage from test_scaling_group These seem to be all reduncant ReplayAll and VerifyAll calls. Change-Id: I8f97297697b5d1d6878cd1758aa59384050190f0 goal: mox_removal - Remove mox usage from test_heat_autoscaling_policy Change-Id: I6d0c905a0e8a9b42ca5964403ad0ac74175e24f1 goal: mox_removal - Correct behaviour of update_replace property in test resource When the update_replace property is set to True, we should replace the resource on every update where the 'value' property is modified. This was the original semantics of the resource, before it was erroneously changed in 113b0a49486967062c6757321027127f6e39c9ff. Change-Id: Icb5c3f3078a748359ee954b51f996ef08e9bc0ec Closes-Bug: #1548356 - Remove mox from test_extraroute Change-Id: Ib1414067987a1a64302e21ba8a646f5171530484 - Merge "Fix for validating VPN service, Ipsec site connection status" - Remove mox from test_neutron_floating_ip Change-Id: I40e38d1f5517ea3c078a37d17299543c11e1ee5a - Remove mox from test_waitcondition Change-Id: Ia0622b42c6a02b3a5036dc6af0efce18eeca3636 - Remove mox from test_stack_resource Change-Id: I1fa0ea4e50d9a420d38e5724bc5767e50278b90c - Fix for validating VPN service, Ipsec site connection status In case of vpn, ipsec site connection resource creation, Stack creation is not waiting for resource status and returning as stack COMPLETE status. Following fix validate for create complete. Change-Id: I99de7759230ca815c12cfae0e90f42ce851be910 Closes-Bug: 1654785 - Merge "Fix sahara job type validation issue" - Merge "Fix races in conditionals tests" - Merge "Remove mox usage from test_template" - Merge "Remove mox from test_common_policy" - Merge "Remove mox usage from test_stack_update" - Merge "Run the specified set of tests in grenade-multinode job" - Merge "Updated from global requirements" - Updated from global requirements Change-Id: Ic60b21343b547b3bb7f43fd7fa878e8250a235f6 - Fix doc build Sphinx 1.7 removed the Directive from the compat module. Let's use the docutils version. Change-Id: I22972feea961f0eb2996d605ff98bbe1f5df2711 - Remove usage of mox from test_eip Change-Id: I5f7f27a77b81883877c2783fde34a791a9e5e801 goal: remove_mox - Remove mox usage from test_floatingip goal: mox_removal Change-Id: Iaae32caf0574eddefc235c3f26d043556c3bab3b - Run the specified set of tests in grenade-multinode job This patch defines the specified set of tests & runs these during upgrade in grenade-multinode job. Change-Id: I99fa1717f4bf46afc8dd989a3aae129e5c4ab9d7 - Remove mox usage from test_template mox-removal Change-Id: I441c9e26455a85fb8853c6e68b368c4f9f26ddad - Remove mox from test_common_policy mox-removal Change-Id: Ie3b3e16afafc4a9ac9bc9c1a438e51cf46a3caa4 - Remove mox usage from test_stack_update Those UnsetStubs calls were unused. mox-removal Change-Id: I4382266160ab566fec61234bf53f522406031488 - Merge "Remove deprecated nova api calls from EIP/EIPAssociation" - Merge "Replace deprecated nova calls for floatingip" - Imported Translations from Zanata For more information about this automatic import see: https://docs.openstack.org/i18n/latest/reviewing-translation-import.html Change-Id: I087233a36d7fd09d8c1efb3371839aaede1af67c - Merge "Return nested parameters for resource group." - Return nested parameters for resource group. This refactors the building of schema from parameter validation to use a new method (which doesn't keep stacks in memory), and use that new method for providing proper schema for resource group when the size is 0. Change-Id: Id3020e8f3fd94e2cef413d5eb9de9d1cd16ddeaa Closes-Bug: #1751074 Closes-Bug: #1626025 - Merge "Delete redundant code" - Merge "Change transport_url in grenade-multinode job" - Merge "Unskip StackCancelTest" - Imported Translations from Zanata For more information about this automatic import see: https://docs.openstack.org/i18n/latest/reviewing-translation-import.html Change-Id: I2161224dfcb1a78ea3b0bd37b1c75998a5b72011 - Merge "Always use string ID for WaitConditionHandle signals" - Change transport_url in grenade-multinode job * Change transport_url to test Heat's upgrade strategy [1] in gate. [1] https://review.openstack.org/475853/ Change-Id: I08770d2ae09891d7983345616186cff7c26df4ce - Merge "Update .stestr.conf with the group_regex" - Imported Translations from Zanata For more information about this automatic import see: https://docs.openstack.org/i18n/latest/reviewing-translation-import.html Change-Id: I5a276068a01d3073fec5bdcfce65d09510ed651a - Update .stestr.conf with the group_regex Tempest has switched to stestr. Update the group_regex in .stestr.conf Change-Id: I6f12f95367748828977550ede5043d2f6b7e9970 Closes-Bug: #1750960 - Merge "Allow iteration over files dict before template stored" - Merge "Fix race conditions in test_create_update" - Merge "Handle non-stored stack in resource group" - Merge "Fix convergence unit tests for py36" - Merge "Fix typo in contributing/blueprint.rst doc" - Clarify resource deprecation process The deprecation process is unclear in that it does not specify the goals of hiding (rather than removing) obsolete resource types. This has led to resources being hidden without removing their implementation, leaving existing stacks potentially inoperable. State clearly the purpose of hidden resource types, and options available to help users migrate. Change-Id: I7362c094122066f98a5c95d4c5da16cb5a5c1108 - Handle non-stored stack in resource group When creating a size 0 resource group with batch create policy, we could get into a situation where the resource was created but the stack wasn't. This fixes that, and still handle the problem in attribute resolution for backward compatibility. Change-Id: I8e139215281dc57fe4dd3ff856eef3ef88e20c6b Closes-Bug: #1750270 - Fix convergence unit tests for py36 Change-Id: I90d68a9b35a1f23195b7153eba83f2bcd0ae6ef6 Closes-Bug: #1750252 - Merge "Updated from global requirements" - Fix race conditions in test_create_update If two updates happen in the same second, stack updated_time does not change. This adds 'action_wait_secs' to have a different updated_time. Change-Id: Ie9d1c7fe83abfc7c9431e2d341392296cfd1f8df Closes-Bug: #1745276 Related-Bug: #1583630 - Merge "Fix README.rst for in-tree functional tests" - Merge "Fix some typos" - Merge "heat_docker: Require docker instead of docker-py" - Updated from global requirements Change-Id: Idde7159da81afbfc1ee58fa1c13f1dc214c3f24f - Imported Translations from Zanata For more information about this automatic import see: https://docs.openstack.org/i18n/latest/reviewing-translation-import.html Change-Id: I0c41dc2d5712251293c29c1b511538ef2c42c9b7 - Merge "Unit tests: Fix broken Monasca client test" - Merge "Remove db api and tables for CloudWatch" - Unit tests: Fix broken Monasca client test The context is held by a weak reference in the ClientPlugin, so don't let it go out of scope in the caller. This test was added by 36e53d3c2b973c45a4dc7af7af39b0110437de48. Change-Id: I6a313970f1c0604e94d008eb12aeaa94da1d25c4 Closes-Bug: #1750024 - Merge "Pass mistral execution argument by name" - Merge "Don't load nested stack to get ResourceGroup blacklist" - Merge "Get Instance/AutoScalingGroup attributes from nested stack outputs" - Pass mistral execution argument by name A recent released change added a positional argument where we don't expect it. It works around it. Closes-Bug: #1749645 Change-Id: I4e9f4bb726c3461d17b2e22679377e262cd13462 - Imported Translations from Zanata For more information about this automatic import see: https://docs.openstack.org/i18n/latest/reviewing-translation-import.html Change-Id: I17da47a66472ef8a5be3005ee010378913dd35d8 - Ignore dns domain NotFound when deleting record when the domain is deleted, all its records are deleted as well. Not ignoring such NotFound can lead to undeletable stacks if they have a Designate::Record resource that references a deleted Designate Domain. Change-Id: If56f84b0b86f4d8136a1f83683a4a9eee5d89759 Closes-Bug: #1748953 - Merge "Remove contrib/rackspace" - Merge "include sample config file in docs" - Merge "Add Heat review dashboard config files" - Merge "Update reno for stable/queens" - Imported Translations from Zanata For more information about this automatic import see: https://docs.openstack.org/i18n/latest/reviewing-translation-import.html Change-Id: I6e51ed62e5ba07d5943cc973115ec96ba719c983 - Update reno for stable/queens Change-Id: I404a43d4e9e5d8fb44d3e0b924af5cc8cf6f51d6 - Always use string ID for WaitConditionHandle signals Using an integer as the ID - which is what we currently do for OS::Heat::WaitConditionHandles without an ID explicitly supplied by the signaller - means that we can implicitly create an entry that will collide with an explicitly specified string ID when we serialise to and then deserialise from JSON to store the metadata in the database. These types of collisions are silent and avoid being logged (despite the fact that we attempt to catch and log collisions), which makes any issues very, very difficult to debug. Change-Id: Ia613e79dc9757c4594b84ed2105bfc1357d1e037 Related-Bug: #1738653 - Merge "Support tenacity exponential backoff retry on resource sync" - Sync releasenote for Queens Sync up releasenotes with some information that users or ops should be aware of. Change-Id: I7553fd848f1ec1f0d7b90c15dd5762ede4a69a55 - Merge "Retry on DB deadlock when updating resource" - Merge "Get ResourceGroup/Chain attributes from nested stack outputs" - Merge "Handle data inconsistency in grouputils" - Fix some typos s/X-Openstack-Reqeuest-Id/X-Openstack-Request-Id/ Change-Id: Ib35b941dbeedfd178a9ef89d4ef38d23e8c0e373 Closes-bug: #1739324 - Merge "zun: add property mount to container" - Merge "Replace random with SystemRandom for RandomString" - Merge "Change note for version/release from releasenotes" - Remove contrib/rackspace These resources are unmaintained and (due to bug 1747775) untested. Change-Id: I932368902abc9a32a4964c064abb49f16aadc1f3 - Prioritise resource deletion over creation Because of quotas, there are times when creating a resource and then deleting another resource may fail where doing it in the reverse order would work, even though the resources are independent of one another. When enqueueing 'check_resource' messages, send those for cleanup nodes prior to those for update nodes. This means that all things being equal (i.e. no dependency relationship), deletions will be started first. It doesn't guarantee success when quotas allow, since only a dependency relationship will cause Heat to wait for the deletion to complete before starting creation, but it is a risk-free way to give us a better chance of succeeding. Change-Id: I9727d906cd0ad8c4bf9c5e632a47af6d7aad0c72 Partial-Bug: #1713900 - Merge "Remove old import path in barbican" - Merge "Unit tests: test ResourceChain attributes better" - Merge "Unit tests: refactor nested stack attributes tests" - Merge "Don't observe reality if name property is None" - Merge "Supply outputs for reference IDs in StackResources" - Fix races in conditionals tests In convergence, resources have a tendency to appear and disappear in their own time, not atomically with a change to the stack state. Therefore when testing conditionals (i.e. looking for resources to appear/disappear) we must be careful to take into account the states of the resources we find to avoid race conditions. Change-Id: I488921c6c4c3324912ded494db5ab9605becce9b Closes-Bug: #1737796 - Merge "Update openstacksdk construction" - Handle data inconsistency in grouputils In convergence, retrieving the template for a stack will return the latest template submitted by the user and accepted to start a traversal. If this template removes resources from the stack, they may continue to appear as available when getting a list of current resources even if a definition for them doesn't appear in the current template. This occasionally caused a KeyError when trying to match up data from the nested stack to the nested stack's template in grouputils using the GroupInspector. Fix this by accounting for the fact that the stack and template may contain different sets of resources. Change-Id: I86d80ab766667194f4e6e3b6112d6b1b7a3733d8 Closes-Bug: #1747365 - Merge "Include outputs in rolling update of ResourceGroup" - Merge "Fix coverage run with tox -ecover" - Merge "Fix hot_spec.rst in Template Guide" - Merge "Log when we bail out without checking a resource" - Add Heat review dashboard config files Add main review dashboard config file, and doc for how to generate it. Change-Id: I47860f9840dfa84041709e38ac361f8756171eb7 - Change note for version/release from releasenotes Release notes are version independent, so remove version/release values. We've found that most projects now require the service package to be installed in order to build release notes, and this is entirely due to the current convention of pulling in the version information. Release notes should not need installation in order to build, so this unnecessary version setting needs to be removed. This patch propose to change the note, so people won't try to use it. Change-Id: I16110bb3951bd5bf9e644bbcd7b35b1f610ab909 - Merge "Update test image" - Fix coverage run with tox -ecover We removed .testr.conf in 93746a9b2895d44fd105ba26ae77ddffbe2db167 and tox -ecover fails with error. Change-Id: I31b00afffe38ab5c7591ba54c959349716df7c48 - Fix hot_spec.rst in Template Guide Add missing space and words in doc/source/template_guide/hot_spec.rst Change-Id: Ic0e5c827d04b78c337c6562df73b5bd1ccf2a977 - Unskip StackCancelTest python-heatclient is bumped, time to resume this test. Change-Id: Ieebe6a6cb99ece42fd06f1bb6626ffb37dec1f6d - Merge "Fix some docstrings for octavia resources" - Merge "Add scenario002 tripleo jobs to experimental pipeline" - Remove old import path in barbican Direct import of barbicanclient is being deprecated for `barbicanclient.containers`, we should use `barbicanclient.v1.containers` instead. Change-Id: I340099ecf6649e7a443712a448a1504ec2e6d8ea - Merge "zun: add hostname, hints and security_groups" - Merge "zun: check delete complete for Zun container" - Merge "Imported Translations from Zanata" - Merge "Move CooldownMixin tests to group test modules" - Merge "Remove use of CooldownMixin with scaling policy" - Imported Translations from Zanata For more information about this automatic import see: https://docs.openstack.org/i18n/latest/reviewing-translation-import.html Change-Id: Ib51b2cebb1c4ecef3fe8cf1f610677ef0f8ceeb6 - Fix README.rst for in-tree functional tests Change-Id: I9343bf68226187ef33caa919fb0e362cdb9abfcb - Merge "Remove stack watch service" - Merge "Zuul: Remove project name" - Merge "Updated from global requirements" - Imported Translations from Zanata For more information about this automatic import see: https://docs.openstack.org/i18n/latest/reviewing-translation-import.html Change-Id: Idbd22aaad0cf01d6cbb2e1db1bc4d01573de1c45 - Update openstacksdk construction The queens release of python-openstacksdk breaks the sdk and senlin code that's in heat right now. This shifts us to the new world order. python-senlinclient is actually just a thin wrapper around python-openstacksdk. Instead of reworking the constructor in the Senlin plugin, just remove use of python-senlinclient and use the openstacksdk client plugin instead. Change-Id: Idf0acebf7b3774db26e335b3f3229227bfe68502 - Zuul: Remove project name Zuul no longer requires the project-name for in-repo configuration. Omitting it makes forking or renaming projects easier. Change-Id: I43b8a36e6ba8457c9594dc4ad579167f95d91d39 - Fix senlin/sdk client plugin unit tests New python-openstacksdk has broken the client plugins. This also change to use `status_code` instead of `http_status` of HttpException when checking for bad request. Closes-Bug: #1746956 Change-Id: Ib1f295a6dea014df3237acf0efd6b84464eac9c1 - Fix some docstrings for octavia resources There is no concept of service extensions, supported by default. Change-Id: Icd4f4e054417aa4ee4ccdd995c50948f75eb697e - include sample config file in docs Leverage oslo.config to rebuild our sample config file in docs. Change-Id: I359e63fad43a5fcde4fe30a97cbed8f4f71961c5 - Remove heat-api-cloudwatch from man pages list Change-Id: I603a3be18ff4cd9b3c7db8a4728d0ac864d0b089 - Merge "Cleanup remaning doc for CloudWatch API" - Updated from global requirements Change-Id: Ie8e8e85121a47477cf90022b3ccfc4b56efef992 - Merge "Return stack.outputs when stack in DELETE_FAILED" - Merge "Remove unused policy cp in devstack" - Merge "Exclude updated networks - match actual interface state" - Cleanup remaning doc for CloudWatch API Change-Id: I0423dba4eb2a9ddfd7419869ac5943638786a282 Partial-Bug: #1743707 - zun: add property mount to container Change-Id: I5e62d599fff87dd4b9cd26267350e9b4b816b15b - Log useful information in the API log heat-api doesn't routinely log anything above DEBUG level, and then most of the information comes from random middleware like version_negotiation. Log the actual request at INFO level, and remove useless information like the address in memory of the Controller object from the debug logs. Change-Id: I5bb115abd31b4050e608e913058b6955b4bddc5a - Move context middleware earlier in pipeline Until the context middleware has been processed, logs from any other middleware will be logged with the request ID of the *previous* request. Currently this is most logs in the API log files, which makes the request ID worse than useless for determining what is going on. To ensure all logs get the correct request ID, move the context middleware close to the front of the pipeline, right after the request_id middleware that generates the request ID and the auth middleware (the data from which is needed to populate the context). Also, explicitly include the request_id middleware in the cfn-compatibility APIs' pipelines. Unfortunately, any failures in the auth middleware will still be logged with the wrong request ID. This appears to be unavoidable, since we can't create the context until after we have authenticated the request. Change-Id: I097bf70431a999a0f6aa56079ffb5743b50d4d7f - Merge "Remove OS::Heat::HARestarter" - Merge "Remove use of heat_watch_server_url" - Merge "Fix for None base_url for Monasca client" - Merge "Fix resource.<n> attribute caching in ResourceGroup/Chain" - Merge "Remove CloudWatch API" - Merge "Fix attribute caching in Heat AutoScalingGroup" - Merge "Updated from global requirements" - Replace random with SystemRandom for RandomString it might be theoretically possible to infer the state of standard Python's RNG in a long-running heat-engine service from multiple created RandomString resources. Let's use the random.SystemRandom (and os.urandom) for OS::Heat::RandomString instead. Change-Id: Iac5c03176fc8bae95ada883621196bd9cb453be3 Closes-Bug: #1745931 - Fix for None base_url for Monasca client Heat engine tries to create client object using token, but the token in Monsca client will be reauthenticated. It can't be reauthenticated, because it doesn't have base_url in kwargs (Monasca client), so it raises an error "AttributeError: 'NoneType' object has no attribute 'find'". Using session instead of token fixes the issue. Change-Id: I154c0360946043f6dbb12ef756aa01d674f6de8f Closes-Bug: #1713050 - Merge "Fix senlinclient plugin" - Imported Translations from Zanata For more information about this automatic import see: https://docs.openstack.org/i18n/latest/reviewing-translation-import.html Change-Id: I33a844ad7928d5ec1bf3d2dd76d25797c8dd02d9 - Fix senlinclient plugin New senlinclient release have broken the plugin. Depends-On: Idf9ada2da132092c579f2aa432b5d128531b2737 Change-Id: I7eb100b0eab75f239c75f1bd015536b24b50ca44 Closes-Bug: #1746156 - Merge "Updated the template guide." - Update test image update test image to use `Fedora-Cloud-Base-27-1.6.x86_64` Change-Id: I162b3a760031c5144525e96512db7b82ff02bc65 - Remove OS::Heat::HARestarter The OS::Heat::HARestarter has been deprecated since Kilo. Now is the time to eliminate support and hide it from the documentation. This replaces it with a placeholder resource (like OS::Heat::None) and marks it as hidden. Change-Id: I56cd1f2d0b3323399ef02c3a0a05d79cc69af956 - Merge "Add octavia L7Rule Resource" - Merge "Add octavia L7Policy Resource" - Move CooldownMixin tests to group test modules Change-Id: I521250fd77fb6e7adce72a1b91e5e7462438a884 - Remove use of CooldownMixin with scaling policy Removes the multiple use of the mixin, so that we don't set metadata for both policy and group. Setting the cooldown metadata only for group would suffice. Change-Id: I241a32b52e0708264c80c3eca313a97534927415 Related-Bug: #1555748 - Remove db api and tables for CloudWatch Change-Id: Ic728a994865a923b7cc56d9c6b5df0c0d38626e1 Closes-Bug: #1743707 - Remove stack watch service This removes the rpc api and related code. Change-Id: Ib89bcc3ff6a542f49467e2ad6c7e2a716a0dc2b4 Partial-Bug: #1743707 - Remove use of heat_watch_server_url This deprecates the config option for removal and removes all its usage. Change-Id: If7dce02d1432d95cb1c9a3c6120e1c1287196836 Partial-Bug: #1743707 - Remove CloudWatch API This patch removes the API, the next set of patches in the series would remove stack watch service and related WatchRule implementation. Change-Id: I8b0472be862907298c8da51f435b5d8b19610ec3 Partial-Bug: #1743707 - Merge "Fix attribute caching in SoftwareDeploymentGroup" - Merge "zun: delete container by using 'stop' option" - zun: add hostname, hints and security_groups Add three properties to the container resource: hostname, hints and security_groups. The 'hostname' is the hostname of the container. The 'hints' is a set of key-value pair passed to the scheduler. The 'security_groups' is a list of neutron security groups. Change-Id: I518e37c9907350e364fa9a16b25013e68cc60fa9 - Merge "Restore functional test jobs" - Updated from global requirements Change-Id: Ibcc294bc9848d1c6cde5c232eaf315814675f492 - Merge "Remove ReloadOnSighupTest" - Merge "Remove integration and configuration tests" - Merge "Remove potential co-gating integration tests" - Merge "Remove use of auth cache directory" - Allow iteration over files dict before template stored Due to a programming error, the files dict of a template would appear to contain no items when iterated over prior to them being stored in the database. (Individual items could still be accessed directly by name.) Since this is the mechanism by which dictionaries are serialised (directly or e.g. via items()), this resulted in an empty files dict when trying to send the data over ReST. (The same is likely true of RPC, but we don't generally send the files dict over RPC any more.) Since validation occurs before the template is stored, this meant that we could not validate a remote stack over ReST. Change-Id: Ie7433557321c54075ad531def0247fad735d07d9 Closes-Bug: #1739447 - Restore functional test jobs These were disabled in I55e204b72cd5a2876576852b10ad09f24d04beb4 to reduce the gate failure rate while we eliminated duplicate tests in heat and heat-tempest-plugin. Change-Id: Ifd2aececfa5b1b0f2c8959ad500d4d3197427d19 - Remove ReloadOnSighupTest Most jobs now run with apache+uwsgi, so the test has no significance. Also we don't test it anymore. Depends-On: I022077c92bc10e908c7fe549ed555ad0194e0704 Change-Id: I6d6e025f60867e5128704f54e5e793762f6c1b8a - Remove integration and configuration tests This patch propose to use heat tempest plugin for Integration tests (with no co-gating potential) and configuration tests. Also we will remove test_autoscaling_lb from tempest plugin as well, since it's no longer been used. Remove senario base, since we now move all senario tests to plugin. Change-Id: Ice6a0e1fe8ce2e1de5253c674d11949b0f8a6e31 - Remove potential co-gating integration tests This patch propose to use heat tempest plugin for potential co-gating integration tests. These tests are consider as potential co-gating. And we now already maintain them in heat tempest plugin. This also disables all except two of the gate/test jobs, to enable us to land this series and stop the carnage. The jobs will be re-enabled once all of the duplicated tests have been eliminated. Change-Id: I55e204b72cd5a2876576852b10ad09f24d04beb4 - Add scenario002 tripleo jobs to experimental pipeline Run tripleo-ci jobs with scenario002 on experimental pipeline of Heat (with "check experimental"). Services that are deployed in scenario002 are in tripleo-heat-templates repo: ci/environments/scenario002-multinode-containers.yaml Change-Id: Ifd6507694d98a7dfc03bb342d60aa2bbf0eac778 - Add some steps for "Preparing to create a stack" The example template need an special image. We need register it before launch stack. Change-Id: I225896b24b9946d3c80dc84b63f810a0cd9ae9af - Merge "docs: fix user-data end user guide link" - docs: fix user-data end user guide link The user-data doc has moved out of the openstack-manuals repo and into the nova docs, so this updates the link. Change-Id: I6fac7128f8c099deaea31b3834cab0e472bc10fe Closes-Bug: #1714017 - Merge "Imported Translations from Zanata" - Imported Translations from Zanata For more information about this automatic import see: https://docs.openstack.org/i18n/latest/reviewing-translation-import.html Change-Id: I84726eb52b30a162af610eb94b7694ab55709381 - Merge "fix a typo in heat-manage.rst: s/covergence/convergence/" - Merge "Updated from global requirements" - Merge "Allow octavia specific client config options" - Remove RH1 OVB jobs from configuration Remove OVB RH1 jobs from configuration for this repo Related-Bug: #1744763 Change-Id: I6adf035c3803f95c84a4db053c79687728622e98 - Merge "Remove tests for Interop Tests for add-on trademark program" - Updated from global requirements Change-Id: Id0bcf7dc76433784fc8a21fdc78161373095aff8 - Merge "Allow run tests for both tempest plugin and regression tests" - Add octavia L7Rule Resource This also adds releasenotes for the newly added resources. Change-Id: Ibaa618d14ea06f6ed2b2fc81d8d3ef270ee16a36 Closes-bug: #1737567 - Add octavia L7Policy Resource Change-Id: I05a5d97633ebbe8473c0f582061fa6b1ca6a1358 Partial-Bug: #1737567 - Allow octavia specific client config options Allows for using specific endpoint_type for octavia service. This was missed in d0787393ce1ba073a317c3dccff8251068a707ac Change-Id: I9efc1ba306d11b40f7ca4d609a9282c64debdca3 Partial-Bug: #1737567 - Merge "Enable octavia services for gate testing" - fix a typo in heat-manage.rst: s/covergence/convergence/ Change-Id: If5da5974162209b7f8f513f65d2d2050ba2c4c59 - Merge "Document RBAC Policy allowed values" - Exclude updated networks - match actual interface state Use update_networks_matching_iface_port() to populate old_nets with implicit values from interfaces before matching new_nets properties. The network is not updated if added properties match what is currently on the interface. Without this chanage a stack created with... networks: - network: net0 ... then updated with... networks: - network: net0 subnet: subnet0 ... would result in a port replace, which is not necessary if the implicit subnet used when the stack was initially created was also 'subnet0'. The port being replaced will result in neutron re-running ipam and the ip_address associated would change. Closes-Bug: #1696483 Closes-Bug: #1702279 Depends-On: Ic4bbbcc3cee329a4de8f0eac710128175c4fc397 Change-Id: Iab75ec49b962617943017dcaf1b04b89f91a982e Implements: blueprint tripleo-routed-networks-deployment - Don't load nested stack to get TemplateResource template In the worst-case scenario if we can't find the template for an existing TemplateResource, we fetch it from the extant nested stack. Rather than do this by loading the nested stack in-memory, fetch the template via RPC. Change-Id: Ide96ef861ef4cbb6afd9ab09e7b51b69e03e5520 Partial-Bug: #1731349 - Avoid loading nested stack in CloudFormation Stack resource We can obtain the reference ID of an AWS::CloudFormation::Stack resource much more cheaply by not calling nested(). Change-Id: I212aaae5c68b90b72c7d693f658c83693cc07bf4 Partial-Bug: #1731349 - Eliminate nested stack loading in InstanceGroup/ASG scaling Obtain the current group capacity, and the reference IDs for updating load balancers (if any) over RPC from a GroupInspector and the nested stack outputs. Fall back to grouputils (which loads the nested stack in memory) only if the output containing the reference IDs is not available. Change-Id: Ic6909e42edf709ba1cd99ba71f2d2f06570a615f Closes-Bug: #1731349 - Don't load nested stack to get ResourceGroup blacklist In ResourceGroup, we (unwisely in retrospect) allow blacklisting of resources by reference ID (i.e. physical resource ID in most cases). In order to both determine whether a removal policy entry was an existing resource name and (if not) to find the resource by physical ID, we would always load the nested stack into memory. To avoid this, unconditionally create an output in the nested stack that returns a mapping of all the resource reference IDs and use it for finding a resource with a matching reference ID. Only if the output is not available (because the nested stack hasn't been updated since prior to the output being defined), fall back to the old way of doing it. Avoid looking up IDs for names that already appear in the blacklist (because they were blacklisted in a previous update and have been removed from the stack.) Since this is more expensive in time (though hopefully cheaper in space), update the blacklist only once on resource creation and whenever it actually changes during an update. This is accomplished by moving the update to a separate function that is called explicitly, rather than making it a side-effect of getting the current blacklist (which occurs up to 4 times in each create/update). Change-Id: Ic32d7d99bad06f92b3d2919d58cd1f9128bcfe83 Partial-Bug: #1731349 - Get Instance/AutoScalingGroup attributes from nested stack outputs Since Pike we've generated the templates for various nested stack resource types with outputs that produce the attribute values that are referenced in the parent stack, and a previous patch did the same for resource IDs. Use these output values when available to calculate the attribute values. This is efficient, because the all of the outputs are fetched together and cached, and avoids using the grouputils functions that cause the nested stack to get loaded into memory in the same process. Fall back to the existing implementation (using grouputils) if the required output isn't available (generally this would be due to the nested stack being created on an earlier version of Heat that did not include the outputs in the generated template, and not updated since). The InstanceGroup and AutoScalingGroup implementations are more complex than those of ResourceGroup and ResourceChain because they exclude failed resources at runtime, and also must have a stable sort order for list-type outputs that cannot be determined at template generation time. Therefore the underlying output value is always a dict, and the attribute value is calculated from it. Change-Id: Ie4ad85068fdc16b8038a09df709ed7eb86434cfa Partial-Bug: #1731349 - Get ResourceGroup/Chain attributes from nested stack outputs Since Pike we've generated the templates for various nested stack resource types with outputs that produce the attribute values that are referenced in the parent stack, and a previous patch did the same for resource IDs. Use these output values when available to calculate the attribute values. This is efficient, because the all of the outputs are fetched together and cached, and avoids using the grouputils functions that cause the nested stack to get loaded into memory in the same process. Fall back to the existing implementation (using grouputils) if the required output isn't available (generally this would be due to the nested stack being created on an earlier version of Heat that did not include the outputs in the generated template, and not updated since). Change-Id: I9146e1d99f494213daef7d61ae75c92a4ef981a9 Partial-Bug: #1731349 - Unit tests: test ResourceChain attributes better This change adds more comprehensive tests for get_attr from a ResourceChain, based on a subset of the ResourceGroup tests. Change-Id: I02b5eb652e10865e1924266b8945c4d045405f28 - Unit tests: refactor nested stack attributes tests Change-Id: I55c793a402f76816a9e4af8ed5b86fd9916ca2eb - Supply outputs for reference IDs in StackResources Previously, in 1fff9516398e40f126a84a5fd6d2ad261838129b, we added outputs to the generated templates of various nested stack resources that referenced any attributes required to calculate the parent resource's attributes. That was necessary because attribute values are not cached unless they are referenced somewhere. It isn't necessary for reference IDs (i.e. the results of get_resource) because those are always available. However, providing reference IDs as outputs as well will allow us to get all attributes of the parent resource from outputs of the nested stack. Unlike ResourceGroup/Chain, in InstanceGroup and AutoscalingGroup the order and content of the member in list-type outputs cannot be determined at template generation time (failed resources are excluded, and the sort order depends on creation time). Therefore, always use dicts instead of lists as outputs, both for the new reference ID outputs in AutoscalingGroup and existing attribute outputs. For ResourceGroup/Chain, create both dicts and lists as appropriate, so that each attribute value can be obtained directly from an output. Change-Id: I3139246a5acb17df9018e7c7a2cf88f740b3f8be Partial-Bug: #1731349 - Include outputs in rolling update of ResourceGroup The template generated by ResourceGroup should include outputs that reference any needed resource attributes. However, previously it only did so when rolling updates were not enabled by an update_policy. Change-Id: Ice7a92fe8d2b14e2470d9089a8e191c8385994f1 Closes-Bug: #1741981 Partial-Bug: #1731349 Related-Bug: #1660831 - Fix resource.<n> attribute caching in ResourceGroup/Chain When generating the nested template output, we need to take into account both forms of syntax: get_attr: [<group_name>, resource.<index>.<attr_name>, ...] or get_attr: [<group_name>, resource.<index>, <attr_name>, ...] Previously we would generate an output for the former case but not the latter, so the attribute wouldn't get cached when that syntax was used. Change-Id: I2a059d5cc42d794ca71caaa1661c32eb76c733fc Closes-Bug: #1742185 Partial-Bug: #1731349 Related-Bug: #1660831 - Fix attribute caching in Heat AutoScalingGroup Attributes for an individual resource in an OS::Heat::AutoScalingGroup can be retrieved using the attribute name 'resource.<index>'. Because we were treating the index as a resource name when generating the template, we would never generate an output in the nested template that referenced the resource's attribute, so it wouldn't get cached. Since we can't know the indices ahead of time, the only way to cache these is to do it for all resources in the group - which is the same data cached by the equivalent 'outputs' attribute. We also need to take into account both possible forms of syntax: [<asg_name>, resource.<index>.<attr_name>, ...] or: [<asg_name>, resource.<index>, <attr_name>, ...] Change-Id: I1d2898cdd4759b1bb9de1896a40056685e728f44 Closes-Bug: #1737047 Partial-Bug: #1731349 Related-Bug: #1660831 - Fix attribute caching in SoftwareDeploymentGroup When the parent template references the 'deploy_stdouts', 'deploy_stderrs', or 'deploy_status_codes' (all plural) attributes of SoftwareDeploymentGroup, ensure that outputs are generated that reference the corresponding 'deploy_stdout', 'deploy_stderr', or 'deploy_status_code' (all singular) attributes of the child template's SoftwareDeployment resources. Change-Id: Iff7101dbaa57602be68e3c7447971dacdd747880 Closes-Bug: #1741552 Partial-Bug: #1731349 Related-Bug: #1660831 - Remove tests for Interop Tests for add-on trademark program Remove tests which been planned to added to add-on trademark program. We now maintain these tests in heat tempest plugin. Change-Id: I458d39f988019c4f1d62c7e39a325a2082a98561 - Allow run tests for both tempest plugin and regression tests Regression tests will be maintained in-tree. See categries list: https://etherpad.openstack.org/p/heat-integration-test-categories We will seperate tests to tempest plugin or in-tree base on conditions of each tests. So we need to rework on framework for integration tests. This patch propose following changes: * This partially reverts commit fff6518e69d19c7946942d1bd7c68e2b76c15966 * Allow running heat tempest plugin and heat regression tests. * iniset configs to both heat_integrationtests.conf and tempest.conf Change-Id: Ief31dc961bc108e2863119598dfb16581a38e9cf Depends-On: I5e9325766ce166e62c731330c462c030cb1e11fc Co-Authored-By: Zane Bitter <zbitter@redhat.com> - Merge "Fix nested template genearation when attribute path is not string" - Merge "Don't load nested stack in batched ResourceGroup" - Fix nested template genearation when attribute path is not string When determining attributes to cache, we seem to be assuming the paths to be strings. If there are resource references in outputs it would be None during validation. As we've the same code snippet in all group resources ex. ResourceGroup, ResourceChain and heat AutoScalingGroup, this fixes it in all those places. Change-Id: I36d623b183d01632041113f7bff63bf255f5d53c Closes-Bug: #1719603 - Merge "Raise HTTPNotFound if not in create or adopt" - Merge "Add octavia HealthMonitor Resource" - Merge "Add octavia Loadbalancer Resource" - Merge "Add octavia Listener Resource" - Merge "Avoid loading nested stack in some grouputils functions" - Merge "Updated from global requirements" - Merge "Revert "Register heat tempest plugin"" - Updated from global requirements Change-Id: I58a11158b99467d46eb237c3bcf0178ed31eb9ad - Merge "Add octavia Pool Resource" - zun: check delete complete for Zun container Zun container is deleted asynchronously so it is desirable to implement check_delete_complete in the resource. Change-Id: I5fe59b16e231cc9467e0739738d0000efcdda784 - zun: delete container by using 'stop' option This option will stop the container first before deleting the container. Unlike the old 'force' option, this option doesn't require admin privilege thus enabling non-admin users to delete the container. Using 'stop' option requires a newer api_version so we add support for setting api_version as well. Change-Id: I34a0331afd7452e193c166838e8ca8b2dc268fb1 - Updated from global requirements Change-Id: Ie0254ca0137048986e3800fff9eaa863461c58aa - Merge "Cache the TemplateResource reference ID like an attribute" - Merge "Eliminate errors getting TemplateResource OS::stack_id" - Merge "Updated from global requirements" - Merge "Imported Translations from Zanata" - Merge "Unit tests: fix absurdly slow tests" - Merge "Include value in string Property type error" - Updated from global requirements Change-Id: I8e3e26ad738c78fee20dfe0625310d6039e38033 - Imported Translations from Zanata For more information about this automatic import see: https://docs.openstack.org/i18n/latest/reviewing-translation-import.html Change-Id: I690468f849255887ab2a95712c3e01559dedabae - Merge "Refactor CooldownMixin" - heat_docker: Require docker instead of docker-py docker-py was renamed to docker. Use docker with the version mentioned in global-requirements. Also docker.Client was renamed to docker.APIClient (see [1]). [1] https://docker-py.readthedocs.io/en/stable/change-log.html#breaking-changes Change-Id: I041727c1817c55be5265cc77d6ab9b5db5bf5dcf - Merge "Add octavia PoolMember Resource" - Imported Translations from Zanata For more information about this automatic import see: https://docs.openstack.org/i18n/latest/reviewing-translation-import.html Change-Id: I61e7ef141855174f15e5a579af9c412b9865b467 - Log when we bail out without checking a resource There's nothing more frustrating than trying to debug when a message appears to get lost without rhyme or reason. When we're not going to process a resource (because it has already been replaced), log what is happening so that we can see it when debugging. Change-Id: I89d4b51205798d3c930f0132c81ff987d46382c7 - Include value in string Property type error When a property value does not match the defined type of the property, we include the offending value in the error message for all property types except 'string'. This brings string properties into line with the other types, for easier debugging. Change-Id: I34ac7bad414403707f2e8a8f2dfbf804fa9bfaa7 Related-Bug: #1742646 - Cache the TemplateResource reference ID like an attribute TemplateResources can have their reference ID defined by an OS::stack_id output in the template. However getting it - or just checking that it even exists - is very expensive, and it happens virtually every time we load the stack in memory for any reason. Since when the output is present the reference ID is also available as an attribute, simply cache it in the database as if it were an attribute regardless of whether the output exists. Once it is in the cache, subsequent accesses will be cheap just like they are for attributes. Keep previous performance optimisations in place as well, since attributes are not stored in the database in the legacy path (so legacy stacks don't benefit from this change), and since the cache does get invalidated e.g. on every stack update. Change-Id: Ib9cd0aa40d377ec227754e386e02f185fd871909 Closes-Bug: #1742847 Related-Bug: #1719333 - Eliminate errors getting TemplateResource OS::stack_id We want to avoid making an RPC call to get the OS::stack_id output in order to calculate the reference ID of a TemplateResource whenever possible - which should be most of the time given the limited use of this feature in the wild - since this happens virtually every time we load the stack. Previously, we were doing this by trying to load the template from the parent stack's environment, and combine it with the stored parameters (using live parameter values caused failures). However, this resulted in failures in the case that the template in the parent stack's environment differed from the one that was last used to update the resource if the parameter schema had been modified in an incompatible way. It transpires that we already determine what outputs are available at object instantiation time in order to populate the attributes_schema. So we can just use the attributes schema to infer the existence or otherwise of the output without doing any other calculation in get_reference_id() that could cause errors. This method doesn't fail-safe, in the sense that if there are any errors then we end up with an empty schema and won't fall back to making the RPC call. However, if there are any errors then probably all bets are off for the reference ID anyway. In common with the old method, it also has the drawback of using the latest template rather than the existing one (although again, once you do an update removing the OS::stack_id output then all bets are off in terms of continuing to see that value as the reference ID). Unlike the previous method, this doesn't take into account disabled outputs (since calculating conditions requires access to the parameters, which were the cause of the problem). Also, the code that calculates the attributes schema doesn't use a proper API but hackily dives into the raw template data, and fixing it would raise all of the same issues that we've had with trying to load the template properly here. So we are kicking the can down the road a bit or, alternatively, consolidating our issues into a single place, depending on your point of view. Change-Id: Ic63fe290249dba5bc00abbde1f4df608181d6a9c Closes-Bug: #1742646 Related-Bug: #1719333 - Merge "Avoid always loading nested stack on update" - Merge "Translate networks and subnet in cluster template" - Merge "Deprecate OS::Aodh::Alarm" - Merge "Updated from global requirements" - Unit tests: fix absurdly slow tests Eliminate the 3 slowest unit tests, which collectively block one of the test processes for a minute and a half, by stubbing out the sleep() calls in tenacity. ServerInternalPortTest.test_prepare_ports_for_replace_detach_failed 75s -> 0.3s SoftwareConfigServiceTest.test_push_metadata_software_deployments_retry 11s -> 0.2s ServerInternalPortTest.test_restore_ports_after_rollback_attach_failed 4.6s -> 0.3s Change-Id: I4ee0c7eb249c4e8aa6ab6da1e4f945e3a629451b - Merge "Use stestr for unit tests" - Add project tags to heat orchestration Enables heat orchestration to accept meta-data for project tags Depends-On: I486b2969ae0aa2638842d842fb8b0955cc086d25 Change-Id: If9125519e35f9f95ea8343cb07c377de9ccf5edf Partially-Implements: bp project-tags - Updated from global requirements Change-Id: I46ba44885117f8f10b2b83f8c0bab7cae8c88b91 - Merge "Imported Translations from Zanata" - Don't load nested stack in batched ResourceGroup Previously whenever we did a batched operation on a ResourceGroup, we loaded the nested stack into memory in the local engine in order to determine the current capacity. Change to using the GroupInspector class to calculate the capacity using only RPC. Change-Id: Ie4c6791bf70df01a66e49cb8ef104e8155c90443 Partial-Bug: #1731349 - Avoid loading nested stack in some grouputils functions We want to avoid loading a nested stack in memory wherever possible, since that is known to cause memory high-water-mark issues. The grouputils functions are among the worst offenders at doing this. Some of the data that they return is easily obtained from an RPC call to list_stack_resources, so swap out the implementations using nested(). Rather than simply add more utility functions, a GroupInspector class is created that can cache the data returned. In future this will allow groups that need to access multiple functions from the grouputils to do so without making multiple RPC calls. (Previously, the data was cached in the group's nested Stack.) Change-Id: Icd35d91bce30ee36d9592b0516767ef273a9f34d Partial-Bug: #1731349 - Avoid always loading nested stack on update Previously, when calling StackResource._validate_nested_resources() (which we do whenever we create or update a nested stack), we would load the nested stack into memory to validate the number of resources in the nested stack, unless the max_resources_per_stack config option was set to -1. This meant we would load the nested stack into memory in the same engine as the parent on every update. To reduce the memory high-water mark, fetch the information we need over RPC from another engine instead. To ensure this is only called once, move the call into the validate code. (Previously it was called again in the create/update itself.) Change-Id: I78d12ecc8240c697e26893ae2d7172b60883fb93 Partial-Bug: #1731349 - Merge "Ensure the whole ResourceDefinition is validated" - Merge "Use appropriate exception in StackResource.get_output()" - Merge "Define resource/output definition sections with constants" - Merge "Allow paths as lists in function.validate()" - Remove deprecated nova api calls from EIP/EIPAssociation This removes the add_floatingip/remove_floatingip nova calls from these resources. Change-Id: Ic2672982d7f1e0644268e35a9d79a33ec2f922cd - Replace deprecated nova calls for floatingip This replaces the deprecated nova api calls for add_floatingip/remove_floatingip. Nova api microversion > 2.43 does not support it. Change-Id: I3e6c9f56b84934a937a2275acd58fd9891a00e75 - Imported Translations from Zanata For more information about this automatic import see: https://docs.openstack.org/i18n/latest/reviewing-translation-import.html Change-Id: I11e01a93defc9811fe4d892e1a0ce15443a409cf - Merge "Use correct raw string format" - Merge "Match on subnet update_networks_matching_iface_port" - Merge "Add removal_policies_mode to ResourceGroup" - Merge "Downgrade WARNING-level log" - Merge "Don't try resource-level locks when stack locked" - Merge "Re-factor and improve update_networks_matching_iface_port" - Merge "Add octavia client plugin and constraints" - Add removal_policies_mode to ResourceGroup This enables choice over the current behavior which is to always append to the resource data blacklist, or overwrite it (which is sometimes needed e.g if you decide you want to reuse that group index) The default behavior is unchanged, but the new behavior can be selected via the "update" value. Change-Id: I1157627b07d98dd079657c320ad783a3ba5bce81 Closes-Bug: #1741053 - "service" is not a default project in rdo. I deploy openstack by rdo.The default project named "service" is not found. Change-Id: Ic532c378fe5552a1b2272ed6fee04fced06d3344 - Fix bypass list length check when all values are None The patch https://review.openstack.org/#/c/527001/ bypassed the None list but did not consider the case where all the for_each lists are getting their values from attributes and therefore are None. This patch set fix it by also skipping the check if all the values are None -- as len(value_lens) would be 0 in that case. Closes-bug: #1732934 Change-Id: Iba574bbd3877f9ca1ca755c384b3ddb47aec711c - Merge "Refactor unit tests for server network" - Revert "Register heat tempest plugin" This reverts commit 00a1814d5e456143518e4806c049cd1bf61ca48b. Change-Id: If0def06223164609adc3f8ce69c672cfcb93de3f - Enable octavia services for gate testing Change-Id: Iaad4c59ce39fe8ca11d136a7c554d773693862cc - Add octavia HealthMonitor Resource Change-Id: I1e779c4c257f0a57c2f6fdafa5f6fabb72a4b967 Partial-Bug: #1737567 - Add octavia Loadbalancer Resource Change-Id: I05f105d290696e1dedc5c9135a5ee36a1271a7ad Partial-Bug: #1737567 - Add octavia Listener Resource Change-Id: I55e7682f6d2cb7f42c683ee4a6abb7c2106a3d86 Partial-Bug: #1737567 - Add octavia Pool Resource Change-Id: Iaaa18636d799148728d0858c3bd3bab877d23079 Partial-Bug: #1737567 - Add octavia PoolMember Resource Change-Id: I29a66c016714270faf1f2e8d98f1d496e7704e57 Partial-Bug: #1737567 - Remove unused policy cp in devstack We won't have policy.json file by default anymore, so we don't need this `cp` command. Implements: bp policy-in-code Change-Id: I84c99e38c34dc41cc33126291563ea90038ce107 - Merge "Imported Translations from Zanata" - Remove policy.json from setup.cfg Change-Id: Iec915afdc290f87f74597353c7e4a455bc28e791 - Merge "Bypass list length check for None values in repeat function" - Merge "[policy in code] part 7 add sample and releasenote" - Merge "[policy in code] part 6(cfn, cloudwatch)" - Imported Translations from Zanata For more information about this automatic import see: https://docs.openstack.org/i18n/latest/reviewing-translation-import.html Change-Id: I75180b26ee4450828330755baebbfaf5dfefaa44 - Imported Translations from Zanata For more information about this automatic import see: https://docs.openstack.org/i18n/latest/reviewing-translation-import.html Change-Id: I684319eae9c06e666f266bce8cfdc00229fa7894 - Add octavia client plugin and constraints Change-Id: I0fa34d19781c49ede8743defe72bbdc63f375896 Partial-Bug: #1737567 - Match on subnet update_networks_matching_iface_port * Add subnet to interface managed keys when updating nets matching interface port * Add tests to verify that matching on "poor" nets with only subnet works. Change-Id: I3c2a251088fafef78c69aa4a758137907f414a60 Related-Bug: #1696483 Related-Bug: #1702279 - Re-factor and improve update_networks_matching_iface_port * Implement _find_best_match() using subset intersection. * Reduce number of expensive external neutron calls to get net_id by moving this outside the inner loop. Unit test updated, we now do a better job at matching and populate more data when updating. Partial-Bug: #1696483 Partial-Bug: #1702279 Change-Id: I2157d35e1c58f3960d5b6aa86199d93bb5b88401 - Refactor unit tests for server network * Consolidate unit test fake_interface to one global. Instead of three implementations of the fake_interface. Use one shared globally defined implementation. * Use create_old_net() instead of just writing the dict. * Adds the subnet_id property in fixed_ips of fake interfaces. * Use 'network' instead of deprecated 'uuid' Related-Bug: #1696483 Change-Id: Ifa7728402e1189348b1c4fcc0bbbd1eec8c0deaf - Fix a typo in tests s/mult_subnet_template/multi_subnet_template/ Change-Id: I15392f335da728b2080368ff63bda8b21dc0639b - Use appropriate exception in StackResource.get_output() Don't raise InvalidTemplateAttribute in StackResource.get_output() when an output does not exist - it's not the case that get_output() is only used for fetching attributes. Instead, raise NotFound from get_output(), and translate that to InvalidTemplateAttribute in the caller when we are actually fetching an attribute. Change-Id: I4f883e4b583a965785d0a595c8c33b47dc94118c Related-Bug: #1719333 - Updated from global requirements Change-Id: I5b5b16189d6618759f7caef8e7bd61213a45f2e3 - Merge "ForceDelete Instance" - Merge "Enable to specify network for Trove Cluster" - Merge "Avoid RPC call in TemplateResource.get_reference_id()" - Enable to specify network for Trove Cluster we should enable heat to specify network for trove cluster after [1] merged [1] https://review.openstack.org/#/c/179443/ Change-Id: I161b7cc1c4824f6aa4a4667bf2d909a2ead81cb4 - Use stestr for unit tests Change-Id: Ifaf3991e1c9400683e9c98185b86cb39ac3d318d - Imported Translations from Zanata For more information about this automatic import see: https://docs.openstack.org/i18n/latest/reviewing-translation-import.html Change-Id: I9149a09b534c33f52b2073318e2a3bafdc05e65a - Merge "Remove setting of version/release from releasenotes" - Remove setting of version/release from releasenotes Release notes are version independent, so remove version/release values. We've found that projects now require the service package to be installed in order to build release notes, and this is entirely due to the current convention of pulling in the version information. Release notes should not need installation in order to build, so this unnecessary version setting needs to be removed. This is needed for new release notes publishing, see I56909152975f731a9d2c21b2825b972195e48ee8 and the discussion starting at http://lists.openstack.org/pipermail/openstack-dev/2017-November/124480.html . Closes-Bug: #1737955 Change-Id: I981c5a6476b40bf9f7fcd35ae0c638e77912f060 - Bypass list length check for None values in repeat function Attributes may return "None" before the resource is created, so the length check for for_each lists in repeat function will fail. Bybass the length check for the None list Change-Id: Ida81060e62340b3ea37ba0fdd7443dc88ba6c5aa Closes-bug: #1732934 - Merge "Remove unused variable" - Don't observe reality if name property is None We use `physical_resource_name` for name property in some resources when name not provided during create, so we shouldn't add name in resource_data if it's None in property (might just the cases that we using `physical_resource_name`). Closes-Bug: #1737696 Change-Id: I7b6bcfdd748fdb8664245c4b65ec7170c2ad4a94 - Merge "Don't log PolicyNotRegistered when check resource type" - [policy in code] part 7 add sample and releasenote Since we're moving all policy into code and documenting it there we should generate those docs automatically, so they are less likely to be out-of-date. Also add releasenote for ops to aware that Heat is now using policies in code and their Orchestration service needs to avoid depends on policy.json file if file not exist, since there is no such file by default after this patch. Partially-Implements: bp policy-in-code Change-Id: I25fc5a110b1fe515918e042f220c23ac9a7e811f - [policy in code] part 6(cfn, cloudwatch) Add cloudformation and cloudwatch policy in code rules. Remove policy.json. We don't keep any default policy rules in policy.json from now. Still they can create policy.json file and add any rules they try to override. Partially-Implements: bp policy-in-code Change-Id: I610115dc1974b2182ce673bb086a1da15b022de3 - Merge "[policy in code] part 5 (software-*)" - Merge "Update Zuul branch limit" - Merge "Updated from global requirements" - Merge "Protect prepare_update_replace() with resource lock" - Allow reserved status for attach check This issue was due to Nova changes it's volume attach workflow (https://review.openstack.org/#/c/330285) which allows `reserved` status. Closes-Bug: #1737343 Change-Id: I13f1906f53a89586bb51c0b70fa768a9500dfa26 - Don't log PolicyNotRegistered when check resource type Silence PolicyNotRegistered exception when checking resource type. We do not consider PolicyNotRegistered as an error when checking resource type, so we shouldn't log the exception as well. Depends-On: I13f1906f53a89586bb51c0b70fa768a9500dfa26 Change-Id: I111ea7ffa43a46d2a6fe360fb0c20d672df5771c - Protect prepare_update_replace() with resource lock When we consolidated resource locking with resource state transitions in a7376f7494b310e9367ebe5dcb43b432a4053023, prepare_update_replace() moved outside of the locked section, so that other update operations could potentially conflict with it. This change ensures that we call it while the lock is still held (if we have transitioned the resource to UPDATE_IN_PROGRESS, and thus acquired the lock), or that we acquire the lock before calling it (if UpdateReplace is raised before attempting to update the resource). To avoid unnecessary locking and unlocking, we only take the lock if the Resource plugin has overridden the default handler method (either restore_prev_rsrc() or prepare_for_replace()), since the defaults do nothing. Change-Id: Ie32836ed643e440143bde8b83aeb4d6159acd034 Closes-Bug: #1727127 - Updated from global requirements Change-Id: I774c2015a728eaa1911475923bad585d3bf6cec2 - Merge "Updated from global requirements" - Merge "Build heat.conf reproducibly" - Merge "Fix non-destructive upgrade for deprecated res types in convergence" - [policy in code] part 5 (software-*) Add software_deployments rules, software_configs rules. Partially-Implements: bp policy-in-code Change-Id: If0c98ffcfceae395ab2443356aea3904edaf7b4e - Merge "Removed unnecessary setUp() call in tests" - Merge "[policy in code] part 4" - Merge "Generate correct name for Sahara cluster" - Merge "Speed up event listing with nested stacks" - Build heat.conf reproducibly The hostname of the build machine leaks in heat.conf in the host directive, making the whole heat package not reproducible. If you don't know about reproducibility, please read here: https://wiki.debian.org/ReproducibleBuilds/About This patch therefore sets sample_default to avoid this problem. Change-Id: Ibfe81f89df75417eab7437817aabd69bfb2985cd Closes-Bug: #1736761 - Merge "Eager-load properties for nested resource list" - Merge "Ignore resources with non-existent template" - Merge "[policy in code] part3 (resource types)" - Merge "Fix reset_stack_status" - Merge "Remove broken heat.resource_type custom constraint" - Merge "[policy in code] part 2 (stacks)" - Updated from global requirements Change-Id: I9e2c03c7a265eebfa24ab3d3e72cbf993bd6fe3c - Merge "Remove cloudwatch service from devstack plugin" - Merge "Fixes heat-keystone-setup-domain authentication failures with v3" - Fix reset_stack_status Fix reset_stack_status on legacy, by using a different session for each stack reset, and handling lock duplicate errors. Closes-Bug: #1735755 Change-Id: I6bcd7448052e86ec3e4eb4c49ef3139c20d4f919 - Update Zuul branch limit Now mitaka and newton already mark as EOL, so we should remove any branch limit that relative to those branches. Also since we have dependent zuul jobs for each branch, we remove ocata branch limit from master. Change-Id: Ide2ddbe4d50faa0356992b55d99c1b1af8a44a35 - Translate networks and subnet in cluster template Translate 'external_network', 'fixed_network', 'fixed_subnet' in cluster template from name to id. Closes-Bug: #1719493 Change-Id: I78670cb4d394597b36c43d53f0f8094cb8bd285a - Merge "Remove unused attribute 'RESOURCE_TYPE' for tests" - [policy in code] part 4 Add service rule, resource rules, actions rules, build_info rules, events rules. Partially-Implements: bp policy-in-code Change-Id: I497f4d02b5ea8399265dedc548214e4eca6b6a35 - [policy in code] part3 (resource types) Allow use policy in code to resource type's rule. Also add test for override the in-code resource type rule in json file. Partially-Implements: bp policy-in-code Change-Id: Id6c21732e66de6c421427ded98de52f5da0a4db2 - [policy in code] part 2 (stacks) Allow use policy in code to stacks's rule. Also convert check_is_admin to use new mechanism. Partially-Implements: bp policy-in-code Change-Id: I398ed162790294d0d4453f7f12c77b38e95a5580 - Add environment_files parameter in CREATE_STACK request Actually, stack creation API needs environment_files parameter, so I add environment_files in documentation. Change-Id: Id1c5ea9917c0ab5ff8b6db982749d30a070f1d7d - Fix sahara job type validation issue Sahara Job Type api doesn't have 'get' function. Heat sahara client should handle 'job_types' seperately. And update the tests accordingly. Closes-Bug: 1734421 Change-Id: Ia8a50cd964dd537a17f4b4c92835901e40d512a8 - Merge "Updated from global requirements" - Merge "Move grenade multinode job out of experimental" - Merge "Use heat-tempest-plugin in grenade job" - Updated from global requirements Change-Id: I6af141926f6b21b384c72512828b950a55ac1e9e - Merge "Use heat-tempest-plugin for integration tests" - Ignore resources with non-existent template When listing resources with nested depth, we get all resources for the root stack and cache them in the context. By the time we iterate through the resources for a nested stack, if the current_template_id of resources do not match the stack template, we try to load the template. However, it's possible that the template does not exist anymore. It would be good to ignore those resources. Change-Id: I838320539838ed74f490bca8601cde96eaf7c7ee Closes-Bug: #1734815 - Merge "Add more logging for software deployment updates" - Avoid RPC call in TemplateResource.get_reference_id() Most TemplateResources probably don't have an OS::stack_id output defined in the template, so it's unfortunate that we have to make an RPC call to check that every time we retrieve the resource's reference_id, which we have to do for most stack operations. To cut down on unnecessary RPC calls, check the template first locally to see if there is an OS::stack_id output present. Change-Id: Ia32ed6bca453b391371f544ad0a07d49dc0616e3 Related-Bug: #1719333 - Merge "Remove SSLMiddleware from oslo.config namespace" - Merge "Zuul: add file extension to playbook path" - Remove SSLMiddleware from oslo.config namespace Change [1] removed SSLMiddleware from code but missed removal of oslo-config namespace, this patch cleans it up. [1] https://review.openstack.org/#/c/506900/ Change-Id: Ibeeece25a13706e3ce0b58dbb9746090d881b909 - Merge "Remove SSLMiddleware" - Use heat-tempest-plugin in grenade job Change-Id: I3a3744fa116a2d424b2afd95eb89058ad78d7f0b - Use heat-tempest-plugin for integration tests Note: We still have to change grenade jobs to use the plugin, before removing the plugin from tree. Change-Id: Ie63f25e724ed3fbd42adcf5bb2d625c2fec67df1 - Zuul: add file extension to playbook path Zuul now supports including the file extension on the playbook path and omitting the extension is now deprecrated. Add `.yaml` back for playbook path. Change-Id: I4a2ae5f28740e41f5b77c05954e7ea4562016be9 - Merge "Increase interface detach polling period" - ForceDelete Instance If nova config 'reclaim_instance_interval', a instance will be soft-deleted, that will cause a error, that volumes or other attached with the instance will cann't be deleted for its 'in-use' status. Now force_delete the instance, the error no longer occurs. Closes-Bug: #1564265 Change-Id: I4d1dd33b23d5882481f4d2c06107c3fb8c08cb93 - Merge "Ignore errors in OS::stack_id output" - Merge "[policy in code] Part 1 Base framework" - Move grenade multinode job out of experimental Change-Id: I5399ededd2938499b0dd155f55a0066c34c486ad - Add more logging for software deployment updates Include the server ID in existing logs, and add additional logs that will let us know when updates to software deployments have happened and whether they have succeeded or not. This should save considerable time investigating in the future if there's ever any question of whether a deployment has gone missing in Heat itself or somewhere on the target server. Change-Id: I33d7aee5974d03b7e41beab9fef4ad643cb0c469 - Merge "Updated from global requirements" - Merge "zuul: add tripleo ovb-ha in experimental pipeline" - Updated from global requirements Change-Id: Ic62299753a51c9f9e81f13284a679663b0871d35 - Merge "Updated from global requirements" - [policy in code] Part 1 Base framework This adds the basic framework for registering and using default policy rules. Rules should be defined and returned from a module in heat/policies/, and then added to the list in heat/policies/__init__.py. new policy wrapers `registered_identified_stack` and `registered_policy_enforce` has been added for policy enforcement of registered rules with same parameter as `identified_stack` and `policy_enforce` besides set `is_registered_policy` flag to true. This flag will decide to use new policy framework or not. Now we can use `tox -e genpolicy` to check and generate policy file. Change-Id: I7a232b3ea7ce0f69a5b7ffa278ceace7a76b666f Partially-Implements: bp policy-in-code - Speed up event listing with nested stacks When handling the command "openstack stack event list --nested-depth=n", we obtain the list of nested stacks by querying the database for all resources that share the root stack ID. However, since all we're getting is the stack IDs, there's no need to query all of the fields and construct a versioned object for each resource. Just get the set of stack IDs. Change-Id: I12155433e2ac1af919aa4b5e780fb965cd5885d8 Related-Bug: #1588561 - Eager-load properties for nested resource list When doing "openstack stack resource list --nested-depth=n", we were lazy-loading the resources' properties data. This is expensive, especially when there are a large number of resources. Eager-load the data, as we always use it to show the resources. For consistency, always eager-load the resource 'data' (even in resource_get), because the Resource versioned object accesses it unconditionally. Change-Id: Idb871fddf77bf24828878c315e19e200c28841be Related-Bug: #1665503 - Check swift deployments before update Before making a software deployments update in the Swift backend, check the data content to operate as a transaction. Change-Id: I58d3af3d1de481b3d314313c9f39255504792634 Closes-Bug: #1731032 - zuul: add tripleo ovb-ha in experimental pipeline It's useful for Heat project to have a TripleO job in experimental pipeline, in case some patches need to be tested against TripleO OVB HA job. Change-Id: Ibbcbbcc2e1429ee894588bf76449b6ed7e3f5392 - Retry on DB deadlock when updating resource Updating a row in the DB using UPDATE...WHERE... seems to be prone to deadlocks. We already fixed added retries for updates of the Stack table (bug 1578615); do the same with the Resource table. Change-Id: I3928f1acc3c354711a27523215784a8cf8767581 Closes-Bug: #1732969 - Refactor CooldownMixin The present way of comparing timestamp in metadata + cooldown with current timestamp can be wrong, if the policy cooldown is updated in between scaling activity. Set metadata with the expected cooldown end timestmap rather than the current timestamp when scaling is finished and compare it with current timestamp. Change-Id: I50ddba4c8da3c6ec8b34ebb4768017cb1a446f5f - Updated from global requirements Change-Id: Ie17c468e5e437bfdcdf5b44307666b9cb19c9228 - Merge "Add functional tests for stack cancel" - Merge "Cleanup ROLLBACK_COMPLETE stack after the test finishes" - Merge "Remove non-updatable attrs from SFC update test" - Use correct raw string format You're not supposed to just randomly put backslashes in front of other characters that don't need escaping unless it's a raw string. This is deprecated in Python 3.6. Change-Id: I7a69ba6dd10412bb065f2b6ba405840d0a98663a - Ensure the whole ResourceDefinition is validated We used to validate intrinsic functions in the ResourceDefinition by just passing it to function.validate(). That worked when it also acted as a CloudFormation template snippet but, since that behaviour was removed by 2c6fc7bcd6b280a85488b0f32f8db7c7835f8b5c, that call now does nothing. This replaces it with a validate() method on the ResourceDefinition itself. This also improves path reporting for validation errors in deletion or update policies. Change-Id: I58c788002136ddf9bded309366e2072823b3ac77 Closes-Bug: #1732798 - Define resource/output definition sections with constants It was unclear what the valid arguments to Template.get_section_name() were (especially since the function is mis-named for what it actually does in HOT). Define the arguments as constants and don't pass string literals any more. Be consistent in how we define paths, standardising on the method in Resource.validate_template(). Change-Id: Ifd073d9889ff60502f78aaa54532cec2b7814d93 - Allow paths as lists in function.validate() Previously when calling function.validate() we passed the path to the function in the template (used for debugging purposes) as a string. This change allows us to pass it as a list of path components, as we do elsewhere, so that higher-level code that catches StackValidationFailed can deal with its components separately. Change-Id: I017aa6f7511b8478ef8273522ab8087684ae71c6 - Merge "Updated from global requirements" - Merge "Added tags attribute to the template parameter" - Updated from global requirements Change-Id: Ib8cb68bc949d2397c9803acceeb3b02e63a56ae9 - Merge "Refactor the format_validate_parameter fuction to reduce complexity" - Updated from global requirements Change-Id: I7352331708d9f2c16921606df28f2fdb8bc895e7 - Merge "Modify the docstring of method 'check_is_admin'" - Merge "Return Environment from validate_template" - Merge "Don't return the sensitive information to user" - Return Environment from validate_template This is to enable preview of the merged environment without merging the environment on the client side. Related-Bug: #1635409 Change-Id: I7ec3af729a65164230153021f438bf226cc5e858 - Merge "Fix AodhAlarmTest to use gnocchi" - Deprecate OS::Aodh::Alarm Theshold Alarm which depends on ceilometer api has been deprecated in Aodh since Ocata. Change-Id: I4aa1d63031874ec8df618e8da2e484929e70df1a Partial-Bug: #1728490 - Merge "Set `public` as update allowed for cluster template" - Merge "Check for existing interfaces before adding implicit one" - Merge "Use argument default None for 'last_evaluated'" - Generate correct name for Sahara cluster Sahara has certain requirements for cluster names. The autogenerated name may previously have ended up invalid, if a resource name contained an incompatible character (for example underscore). Also adding some missed constraints to Sahara cluster name, borrowed from Sahara template name constraints. Change-Id: If1358a104beb831d04d4012e64cd1afedce5a7dd Closes-Bug: #1718989 Related-Bug: #1472998 - Added tags attribute to the template parameter In some scenarios, it is required to categorize the parameter. Associating tags with the parameter, it is will be easy to categorize the parameter and add simple validations to users. This patch introduces a new attribute 'tags' with the parameters, which is a list of strings. Change-Id: I20fc95d606b0b8a08d3b46bf33f4860bff49c74f - Merge "Imported Translations from Zanata" - Updated from global requirements Change-Id: I0ee3956cbfd27b3a570a15c77b4cc59cbebcac41 - Imported Translations from Zanata For more information about this automatic import see: https://docs.openstack.org/i18n/latest/reviewing-translation-import.html Change-Id: I855c22bde9208b44eb7328bb488ca9ffb4aa7d0e - Merge "Migrate grenade jobs to zuul v3" - Merge "Migrate functional jobs to Zuul v3" - Merge "noauth: only set token_info if set" - Add functional tests for stack cancel This adds tests for user initiated stack cancel operations w/ or w/o rollback. Change-Id: I8ff787dc02dd76f99e9ece073b2f700c1922104d Depends-On: Ib107c82f341f4d271859ca7681d65f7ce4c5d0b1 Related-Bug: #1709041 - Merge "Add REST api support for cancel without rollback" - Don't return the sensitive information to user We return back the sensitive information to user when some exceptions happened, for example, when DBError happened, we return the whole sql statement to user, it's not safe. This patch changes to return the message if the exception is the HeatException, otherwise the message won't be revealed to user. Change-Id: I6e01b1003a39106274e79c3b413917a30b5651b6 Closes-Bug: #1708122 - Merge "Fix logic error in sqlarchemy api" - Add REST api support for cancel without rollback We already have REST api support for cancelling a UPDATE_IN_PROGRESS stack with rollback. This adds a new action 'cancel_without_rollback' to allow for canceling a create/update in_progress stack without rollback. APIImpact Change-Id: I6c6ffa0502ab8745cfb2f9c5ef263f1e02dfc4ca Closes-Bug: #1709041 - Merge "Do deep redirects from old docs.openstack.org URLs" - Fix logic error in sqlarchemy api Change-Id: Ic3074969b8a75f7b38aa3ba9a37b23dcf133aa8e - Merge "Replace pycrypto with cryptography" - Merge "Set stack.thread_group_mgr for cancel_update" - Remove non-updatable attrs from SFC update test ingress, egress and service_function_parameters were part of the test_resource_handle_update(), testing the update of a networking-sfc Port Pair. However, these attributes are not updatable from the API definition and so are being removed from this test. Change-Id: I88b197255eeae86331a9e52e872e6828ad93d6f6 - Merge "Retrigger new traversals after resource cancellation" - Merge "Create replacement resource atomically" - Updated from global requirements Change-Id: I05dce6e00bcb86b58c4e0b1a0b36adbbf8bb497b - Define constants for Neutron client resource types Use named constants to pass as the 'resource' parameter to the find_resourceid_by_name_or_id() method of the Neutron client plugin. Apart from making everything more explicit and less error-prone, this also provides clarity on the meaning of obtuse names like 'policy' (QoS, not any other kind of policy) or 'pool' (load balancer, not subnet). Change-Id: If9b5528835775d303db477fafabf3ed6bd224bd0 - Document RBAC Policy allowed values For entirely spurious reasons, RBAC Policies were validated using a custom validate() method that is invisible to the user instead of using constraints to validate values in a way consistent with all other resource types, and that would appear in the API documentation. This rectifies that error and improves the documentation and error reporting. Change-Id: I0cde2fe5dd5c57c6f5a367d0405bcffad01678fa - noauth: only set token_info if set Only add the token_info to the keystone headers if it is set. This resolves an issue which breaks the catalog for some previous deployments using noauth middleware (the old TripleO undercloud deploy architecture). Change-Id: I74d0f71149014418db3af226b419eb7a9cb5bb1e Closes-bug: #1730280 - Downgrade WARNING-level log It's expected that when a resource completes and the traversal it was processing is no longer active, we won't update the stack state in the database. Therefore don't log at WARNING level for this routine event. Change-Id: Ic901e46c251400c4eeb90a4d9ad4a97d61fb1af8 - Don't try resource-level locks when stack locked The actions SUSPEND, RESUME, SNAPSHOT, CHECK, and also DELETE when abandoning rather than deleting a stack, have not (yet) been converted to the convergence workflow and still do a legacy-style in-memory traversal with the Stack lock held, even on stacks with convergence enabled. Since a7376f7494b310e9367ebe5dcb43b432a4053023 we have been attempting to get a lock on the Resource to do these operations, even though there is no engine ID set (which caused a WARNING-level log). For these operations, respect existing Resource locks when convergence is enabled, but don't try to acquire the lock. Change-Id: I6f232380398a7caf9664717debfe39e3422e70d8 Related-Bug: #1727142 Related-Bug: #1662585 - Migrate grenade jobs to zuul v3 Needed-By: Ie080d74e54850f2d892b23ebd5e0412b0d139199 Change-Id: I02edcec5b464fc294dd87e1a4e1ef8e0d50592bc - Set stack.thread_group_mgr for cancel_update In change I84d2b34d65b3ce7d8d858de106dac531aff509b7, we changed to call self._converge_create_or_update() in a sub-thread. However, thread_group_mgr is not set for cancel_update (with rollback), which in turn calls converge_stack. This also enables test_cancel_update_server_with_port, as bug #1607714 seems to be fixed now. Change-Id: Ie674fd556418f6aa8e79654458cbe43648851db2 Closes-Bug: #1713952 - Retrigger new traversals after resource cancellation If we force-cancel a resource check operation (i.e. by killing the thread) then there was a short window where another traversal could have started and been waiting on the not-yet-cancelled resource. That traversal would then hang forever, as there was nothing to retrigger it. This change ensures we retrigger the latest traversal on the resource after cancellation. Change-Id: Iae27c9cc5c0895b52aef2f2c72686dc48ec5983c Closes-Bug: #1727007 - Create replacement resource atomically Use a single transaction to create the replacement resource and set it as the replaced_by link in the old resource. Also, ensure that no other traversal has taken a lock on the old resource before we modify it. If we end up bailing out and not creating a replacement or sending an RPC message to check it, make sure we retrigger any new traversal. Change-Id: I23db4f06a4060f3d26a78f7b26700de426f355e3 Closes-Bug: #1727128 - Migrate functional jobs to Zuul v3 Migrate all functional jobs and gate to zuul v3. We will have to migrate all other legacy jobs too later. Migrating guideline: https://docs.openstack.org/infra/manual/zuulv3.html Change-Id: I6757ba558d2a89a6b1d1ab4694106f0b84b26d4f - Merge "Remove redundant null engine id check" - Do deep redirects from old docs.openstack.org URLs The docs.openstack.org site was rearranged so that the old docs at /developer/heat/ are now at /heat/latest/. The base .htaccess file in openstack-manuals redirects everything in the old hierarchy to the root of the new one unless there is a .htaccess file present (see I0af0e32d9c646d61f4c72599cdc4613299aa5ec9). Add a .htaccess file so that old links redirect to the equivalent page in the new location. Change-Id: I74de09f4e25c2eda050053d3b8646586c9f24c04 Closes-Bug: #1722820 - Fix non-destructive upgrade for deprecated res types in convergence When a user updates from a deprecated resource type to an equivalent-but-differently-named one (e.g. from OS::Heat::SoftwareDeployments to OS::Heat::SoftwareDeploymentGroup), Heat is supposed to change the type without replacing the resource as it would normally do when a resource type changes. This was broken in convergence, because since 45073226752c58d640ea5a59b7e532c022a4939b the new Resource object we create during the check_resource operation (using the new type's plugin) is no longer automatically initialised with data from the database as resources in the legacy path are. Move the substitution checking to the Resource.load() method, so that it now returns an instance of the new plugin where allowed. In the actual update_convergence() method then we need only check that the resource class is the one we'd expect from the new template, and replace the resource if not. We do have a test that is designed to check that this is working, but in it we didn't compare the physical IDs of the resource that is potentially getting replaced, but rather the physical IDs of some other resource that can't possibly get modified (surprise! it doesn't change). Change-Id: I75778abc303525a71d0a918f7192f00a43c21284 Closes-Bug: #1729439 - Merge "Allow configuring payload of noauth middleware" - Refactor the format_validate_parameter fuction to reduce complexity Adding a if condition to format_validate_parameter function, is failing the pep validation, because of complexity is going beyond the 20 limit. Separating the constraints from the base makes it, simpler. Change-Id: Iaa4cdfceea67a17af1e17512cc28eb54dd52907d - Merge "Move FakeKeystoneClient to engine.clients" - Fix AodhAlarmTest to use gnocchi This changes the test to use AodhGnocchiAggregationByMetricsAlarm and add measures using gnocchi api. Change-Id: I0b2fe154b93d9882b703e95a0b09f139697ceb9a Closes-Bug: #1727637 - Imported Translations from Zanata For more information about this automatic import see: https://docs.openstack.org/i18n/latest/reviewing-translation-import.html Change-Id: I4ac22fd5b4da9c42bd5ff5cfbbd8d208699eca2e - Imported Translations from Zanata For more information about this automatic import see: https://docs.openstack.org/i18n/latest/reviewing-translation-import.html Change-Id: I2ddbf8fd432989a123eb19fd94ad0a74095e722f - Merge "Defer exceptions in calculating node_data()" - Skip AodhAlarmTest It seems ceilometer api has been removed[1]. As part od zuulv3 transition we are moving jobs to project repo[2] and change to enable gnocchi devstack plugin at the gate has to wait till the movement is complete. We probably have no other option but to disable the test for the time being. [1] https://review.openstack.org/#/c/512286/ [2] https://review.openstack.org/#/c/509098/ Change-Id: I547b26ea78ad70742ae6fcf9a0eae4cd4af60100 Related-Bug: #1727637 - Cleanup ROLLBACK_COMPLETE stack after the test finishes Change-Id: Ia457aa7025c464a9dec78e95757d85c977307560 Closes-Bug: #1727632 - Use argument default None for 'last_evaluated' Change-Id: Idafabfe94be66f836c4cbf9bdda4a4f2377d3ee6 Closes-Bug: #1727536 - Remove cloudwatch service from devstack plugin Change-Id: Ie18d80e8faaec984b127ab3abeeb66d6a3aa1b6c - Move FakeKeystoneClient to engine.clients Add the ability to consume FakeKeystoneClient class as a client module. This avoids extra dependencies on the Python mox/mox3 libraries at runtime and allows the end user to wire in the custom class by simply making the following config setting: keystone_backend = heat.engine.clients.os.keystone.fake_keystoneclient.FakeKeystoneClient This use case for this is the TripleO undercloud installer which uses a heat-all process to transform heat templates into Ansible playbooks. Previously we leveraged a "fake_keystone" service as multiple applications and binaries required keystone dependencies. (Both Heat, os-collect-config, and some openstackclients wanted a fully version of keystone running) Now that we've streamlined these dependencies allowing heat to use a fake KeystoneClient library shim instead of the full fake_keystone process is much lighter and avoids the security concerns of the extra socket. This patch allows us to do all of this without installing extra test time dependencies on mox/mox3. Change-Id: I03f1789957ba157871fd13164592116d7fcdabe4 Closes-bug: #1723639 - Merge "Do not validate property network of sahara cluster" - Check for existing interfaces before adding implicit one We seem to be adding an implicit interface/port for the server irrespective of the fact that there are existing non-updated interfaces already there. Change-Id: Ifce2301c4d47c8462f2280955c665bdad72dd417 Closes-Bug: #1724843 - Use stack_id of None for service timer In order to keep the engine service alive, we add a timer that periodically does nothing. Calls to add_timer() require a stack_id, and currently we pass cfg.CONF.periodic_interval. This is highly misleading, because the value you pass for the stack_id has no effect on the interval. The cause was a copy-paste error in 07884448fed64badcf8922c01c1ea044462584b7, when the code changed from calling ThreadGroup.add_timer() to ThreadGroupManager.add_timer(). Use None as the stack ID instead. Change-Id: Ia24a0d3ae9a0295fc811eb5300656399f426408b - Merge "Unit tests: Remove deprecated oslo_db test case classes" - Merge "Unit tests: Don't leave RPC server running" - Merge "Unit tests: ensure all threads complete" - Merge "Don't attempt to update tmpl ID when resource in progress" - Replace pycrypto with cryptography Pycrypto is no longer maintained, and heat currently uses Pycrypto. This patch rewrites functions using Pycrypto and replaces them with cryptography functions. See [1], [2] for reference. [1] http://lists.openstack.org/pipermail/openstack-dev/2017-March/113568.html [2] http://paste.openstack.org/show/617715/ Change-Id: I27e7208e19726c5325bad2874e43f5d841779008 Co-Authored-By: Omar Tleimat <otleimat@ucla.edu> - Do not validate property network of sahara cluster enable to use network which is resource creating in template Change-Id: I2f2014cd1c05bce5f248fe618eecbc4476914662 Closes-Bug: #1726334 - Merge "Imported Translations from Zanata" - Merge "Retrigger new traversals after resource timeout" - Merge "Make scheduler.Timeout exception hashable" - Merge "Add integration tests for simultaneous updates" - Imported Translations from Zanata For more information about this automatic import see: https://docs.openstack.org/i18n/latest/reviewing-translation-import.html Change-Id: Iacb2ec1ca9ce4177d0af24e1dd3efecf31e26c00 - Merge "Add attributes schema for OS::Keystone::Project" - Merge "Reoder barbican container import" - Merge "Raise NotFound() when group member does not exist" - Merge "Replace SCREEN_LOGDIR with LOGDIR" - Unit tests: Remove deprecated oslo_db test case classes Migrate to the new, modern, fixtures-based test framework in oslo_db. This eliminates about 35 deprecation messages when running the unit tests. Change-Id: I502c8f9e0718558ae93ff5a568c1ef9ee17ca7f7 - Unit tests: Don't leave RPC server running Several tests started an RPC server (sometimes unnecessarily) and left it running after the test had concluded. This resulted in messy logs ("Exception in thread Thread-1 (most likely raised during interpreter shutdown)" and so on) in the console when the test worker exited. Avoid these by either not starting the RPC server or ensuring it is stopped before the end of the test. Change-Id: I8a506c6b83801cc90ad2268875f848b6d73fdea6 - Unit tests: ensure all threads complete A number of tests used to leave greenthreads that they had spawned running beyond the end of the test. This fixes those tests to not start unnecessary threads and/or to wait for all threads to complete before they finish. Change-Id: Ic0465dc25235ea3b6ec56646df1cb57878faf748 - Don't attempt to update tmpl ID when resource in progress If we attempt to do a convergence update on a resource and find it already locked by another traversal, don't try to update the resource's current template ID or requirements data. Doing so will usually fail with the same exception, but it is unnecessary and leaves ERROR-level messages in the log. However, there is a race which could result in the call succeeding (i.e. if the other task releases the lock just after we fail to get it), and that could result in the resource not being updated at all. Change-Id: I6bde1f9359cd52c99cca092e8abc660bac8b3065 Closes-Bug: #1722371 - Retrigger new traversals after resource timeout If a resource times out, we still need to check whether there is a new traversal underway that we need to retrigger, otherwise the new traversal will never complete. Change-Id: I4ac7ac88797b7fb14046b5668649b2277ee55517 Closes-Bug: #1721654 - Make scheduler.Timeout exception hashable The python standard library in Python 3.6.3 and earlier has a bug with handling unhashable exceptions: https://bugs.python.org/issue28603 Although oslo_log will catch the error, make scheduler.Timeout hashable so that all exceptions will be printable. Prior to 640abe0c12e63c207fcf67592838f112a29f5b43 we just used __cmp__(), but that isn't used in Python 3. Defining __eq__(), which is required for the total_ordering decorator, makes the class unhashable in Python 3. Change-Id: Idde65b2d41490ab8318b5a8b95ea74e9b96b4e5c Related-Bug: #1724366 Related-Bug: #1721654 - Add integration tests for simultaneous updates Convergence allows us to start a stack update while a previous update (or create) is still in progress. If a resource is found to be locked by a previous graph traversal when the new traversal comes to update it, the earlier traversal is responsible for retriggering the resource after it completes. This patch adds functional tests to confirm that this works when the long-running resource update ends in success or failure (as opposed to timing out or being cancelled). Change-Id: I5cb0cfd6bb05a94cd32709b5cda8454df6e81a61 - Allow configuring payload of noauth middleware This adds the ability to return a catalog in the noauth middleware response. Change-Id: I5413ae4eadd5affdd1ae01678b61eafbe50b7919 Related-Bug: #1724263 - Raise HTTPNotFound if not in create or adopt We should raise exception when stack not found, if it's not waiting for create or adopt (which should be the only two possible reason that stack may not have been created yet). Change-Id: I6c13ba8ab1b0e312a0932c40f7e9e2b7f92822c3 - Remove SSLMiddleware We deprecate SSLMiddleware in 6.0.0, and announce to remove it at 8.0.0. Now will be a good time to remove it out. Change-Id: Iaf9be21ffcec9018ad91a5122ed7f2477a7e4a52 - Hidden OS::Designate::Record and OS::Designate::Domain We deprecated the resources OS::Designate::Record and OS::Designate::Domain in ocata, so hidden them now. Change-Id: I8f0529602f8d2ca50508c4f5ec00d255d078b8a9 - Correct the filter attribute of db model 'ResourcePropertiesData' Make sure to delete the ResourcePropertiesData row if the resource 'attr_data_id' has been set already. Change-Id: I2df5dfc72d051b6619c769c1260ff24b016bc294 Closes-Bug: #1708378 - Imported Translations from Zanata For more information about this automatic import see: https://docs.openstack.org/i18n/latest/reviewing-translation-import.html Change-Id: Ib058e69f51da0e600e98a7e52b777ca41c86b82a - Merge "Eager load resource_properties_data in resource" - Merge "Fix regex for updated_time in PATCH update api test" - Raise NotFound() when group member does not exist Raise NotFound() exception with proper message when fetching atrribitutes for a member, that is not in the nested stack. Change-Id: I8969893781439ef5e87ea9f815d7f22e23be3e78 Closes-Bug: #1723315 - Merge "Refactor FloatingIP add_dependencies() method" - Modify the docstring of method 'check_is_admin' Modify the docstring of 'check_is_admin', by default the rule will check whether or not roles contains 'admin' role and is admin project. Change-Id: I60903152dead93e1421a0a32cff7d5c7e3646579 - Merge "Add release note for StackDefinition API" - Merge "Updated from global requirements" - Updated from global requirements Change-Id: I51d159a446bc99fd98e1f27df9f55f7977fc4bcd - Correct example for list_concat-unique function Change-Id: I330a265a16b0d7a2ac9f94e859f7b81f1e193dc5 - Fix regex for updated_time in PATCH update api test Change-Id: I2d7a6a3613ed218ca561dc30840e3824d5ce1bd2 Related-Bug: #1721692 - Merge "Allow convergence-specific Tempest tests" - Remove use of auth cache directory Change-Id: Ie573f27d47e7e88cde27712a12aa8b40b5afda8b Closes-Bug: #1720988 - Merge "Add action_wait_secs for update" - Remove unused attribute 'RESOURCE_TYPE' for tests Change-Id: Iddd367212a4df06a34696ed53b171f16882b70d6 - Merge "Avoid cloudwatch endpoint retrieve when disabled" - Merge "API Tests: Ensure updated_time changes in stack patch update" - Add action_wait_secs for update Add action_wait_secs for TestResource in api tests, so that consecutive updates would have different updated_time. Change-Id: I08d2ac6625acaa0828b18b5e805c81031bb87c01 Related-Bug: #1721692 - Imported Translations from Zanata For more information about this automatic import see: https://docs.openstack.org/i18n/latest/reviewing-translation-import.html Change-Id: I05160666e877e2c7540251a71e69b7307ae6f582 - Remove redundant null engine id check if `lock != self.LOCK_NONE` is true here then `self._calling_engine_id is None` will alway be true. Change-Id: I9a9158c4b135f0890807f63784468c069d320c52 - Merge "Fix get_watch_server_url for ipv6 addresses" - API Tests: Ensure updated_time changes in stack patch update Check that the patch update has actually run and that we're not simply looking at the UPDATE_COMPLETE status from the previous test, otherwise the outputs may not have been updated before we run the next test. Change-Id: Ia45a303e74e19e7d1887c4e242a8ee5655208e59 Closes-Bug: #1721692 - Allow convergence-specific Tempest tests Add a "convergence_engine_enabled" configuration option to the functional tests' Tempest plugin. The option is enabled by default, but can be disabled by setting the DISABLE_CONVERGENCE environment variable to "true" when running prepare_test_env.sh, so that if convergence is disabled in devstack it will also be disabled in Tempest. This will allow us to write functional tests for convergence-specific (or non-convergence-specific) behaviours. Change-Id: If3a37de75467d50af10582215e16611e59a4ad06 - Fix get_watch_server_url for ipv6 addresses https://review.openstack.org/#/c/508021 fixed it to work in uwsgi mode but did broke it to work with ipv6 as it contains ':' as part of the host. This patch fixes how the URL is parsed in order to work properly with ipv6. Change-Id: I3965ee83e1f743e25cbdcbbec516be0ed6a30afc Closes-Bug: #1721045 - Merge "Use a namedtuple for convergence graph nodes" - Merge "Don't use Dependencies repr in tests" - Merge "Fix error for invalid auth_encryption_key" - Merge "Make private flavor tenant access works" - Skip test_notifications.NotificationTest.* functional tests Change-Id: I4cb4b03adef132c11ca664676b6e03ad3efe60d2 Related-Bug: #1721202 - Merge "Adds 5 backport db migration place holder for Pike" - Merge "Don't load resource data for list_outputs" - Merge "Use show_output in TemplateResource.get_reference_id()" - Merge "Speed up show-stack with outputs in legacy path" - Increase interface detach polling period We increase max interval to cover all waiting time from retries in Nova detach. With Nova's current design (with virt driver) it will do retry at time(in seconds): 2, 6, 12, 20, 30, 42, 56 And with new time interval (set max waiting time to 12s) in Heat: 0.5, 1.5, 3.5, 7.5, 15.5, 27.5, 39.5, 51.5, 63.5 So we can cover and check till nova actually stop retry. Related-Bug: #1585858 Change-Id: I912ab9ab8b1b13b13cec479a1b7d8b2c083c2d09 - Add release note for StackDefinition API Change-Id: I00f71f9b47690d9e9fcbd8aeae2289de34d87c60 Implements: blueprint stack-definition - Merge "Make sure we can get watch server url in uwsgi mode" - Make private flavor tenant access works Make private flavor tenant access works. Change-Id: Ibcb82d3518320534e9e0397e9d0651c3b602d439 Closes-Bug: #1719805 - Fix translate tenants in flavor Closes-Bug: #1719762 Change-Id: Ibb5a1d176de30bb82bb874f0d66f181251fa5b11 - Use a namedtuple for convergence graph nodes The node key in the convergence graph is a (resource id, update/!cleanup) tuple. Sometimes it would be convenient to access the members by name, so convert to a namedtuple. Change-Id: Id8c159b0137df091e96f1f8d2312395d4a5664ee - Don't use Dependencies repr in tests Change-Id: I92fb9928bf839568bb3e4cfb547ddbb579b991cf - Increase interface detach polling period The gate has started failing due to interface detaches timing out. Examining the logs, it looks like Nova's interface detach retry takes about 6s to run one attempt. Heat, on the other hand, does 10 retries at 0.5s intervals. So if Nova has to retry then Heat will fail. Increase Heat's polling interval to make it more likely that if Nova succeeds, Heat will see it. Change-Id: Ie74980a3f806b8c17e4e494ae979725b0078f135 Closes-Bug: #1585858 - Ignore errors in OS::stack_id output If a provider stack contained an OS::stack_id output and there was an error in the output, we would raise TemplateOutputError when trying to calculate the reference ID of the facade resource. Since we do that in many API calls, such an error could render the stack effectively unusable. If we encounter such an error, log it and fall back to the default reference ID. Change-Id: I1bc921fe74c54eb0999541ef36afc42b9c19e9bc Partial-Bug: #1712280 Related-Bug: #1719333 - Use show_output in TemplateResource.get_reference_id() TemplateResource is unique in that it can return a custom reference ID defined as an output in the nested template. This was being fetched by the StackResource.get_output() method, which also has the effect of retrieving *all* of the outputs of the nested stack and caching them in memory in case something else were to reference any of them. Unfortunately when calculating the resource data for a stack (which we must always do when e.g. showing the outputs), we always include the reference IDs of all resources, regardless of whether they are referenced by get_resource in the data we are looking to populate. (In fact, we have no way from the Template API to distinguish where get_resource is used on a particular resource, only where there are dependencies on it.) This is no problem under the assumption that getting the reference ID is quick, but that assumption does not hold for TemplateResource. The show_output RPC call only retrieves a single output (as opposed to show_stack, used in StackResource.get_output(), which calculates all of them). Fall back to that call in TemplateResource.get_reference_id() if the outputs are not already cached to avoid unnecessary overhead in the common case. Attribute values are now always fetched before the reference ID, so that we won't end up making two RPC calls in the case where we also need to read other outputs. Change-Id: I66da13c0bb024749de4ae3f0c4b06ebb485cee37 Closes-Bug: #1719333 - Don't load resource data for list_outputs This RPC call only generates a list of the outputs defined in the template, not their values, so don't load the resource data needed to calculate the output values (which can be very slow). Also, explicitly pass resolve_value=False instead of relying on the default argument (which is different for format_stack_output() and format_stack_outputs()), to reduce future confusion. Change-Id: I79aae94b6552d465db6707cd4a40cd53ff18455b Closes-Bug: #1719340 - Speed up show-stack with outputs in legacy path When we show a stack including the outputs, we calculate all of the resource attributes that are referenced anywhere in the stack. In convergence, these are either already cached (and therefore fast) or need to be cached (and therefore the initial slowness will pay off in future). This isn't the case in the legacy path though, since we are not doing caching of attributes in the database in that path. So this is unnecessarily calculating all of the referenced attribute values, which are potentially very slow to get. For legacy stacks, only calculate the attribute values needed to show the outputs. Change-Id: I35800c7f87b58daf05cbabd05bcbcd75d0c0fadb Partial-Bug: #1719333 - Merge "Improve update_wait_condition_handle docstring" - Replace SCREEN_LOGDIR with LOGDIR SCREEN_LOGDIR already been deprecated. This patch move to use LOGDIR instead. Change-Id: Ie2323412708efd523d934690b5516d635b008fb2 - Merge "Remove the existing snapshots from the backend" - Add attributes schema for OS::Keystone::Project Add `attributes_schema` and `_resolve_attribute` for OS::Keystone::Project. This patch allow get_attr works with OS::Keystone::Project resource. Partial-Bug: #1648004 Change-Id: Ia561aa44e6939a0488207d60af8d7efdbca05f36 - Make sure we can get watch server url in uwsgi mode We get the 'watch' server url by translating the 'cfn' server url if the 'watch' server url is not configurated, but in uwsgi mode, the code logic is incorrect, this patch changes the logic to make sure that we can get the watch server url in both modes. Change-Id: Iebf2480f97c11474412d74726eba9cf5c8ed2193 Closes-Bug: #1716815 - Merge "Add default configuration files to data_files" - Merge "Update URL in README.rst" - Defer exceptions in calculating node_data() When generating the node_data() for a resource, catch and store any exceptions (other than InvalidTemplateAttribute) encountered while getting attributes. Re-raise the exception at the point where we try to read the attribute value, including where we try to serialise the NodeData object to store in the database. In convergence, we generate and immediately serialise the NodeData, so this should result in no substantial change in behaviour there. In other situations (e.g. when we're just loading the data to show the stack), this prevents an error in attribute calculation from aborting the whole operation. The exception will still be raised if (and only if) the erroneous attribute is accessed, but may be handled more appropriately. For example, errors in calculating output values are handled by reporting an error only for that particular output. Change-Id: Idc97aee87405cc13e83be3373078b52e725850ea Co-Authored-By: Zane Bitter <zbitter@redhat.com> Closes-Bug: #1712280 - Updated from global requirements Change-Id: I5104389fbdfe6bb4c4527f35793a0f17db72b255 - Fix unit tests with oslo_messaging 5.32.0 The 5.32.0 release of oslo_messaging included the following change to the Fake driver: https://git.openstack.org/cgit/openstack/oslo.messaging/commit/?id=d1dac1c11d357aa8391de7e62f4d003eb820948d Which means that FakeExchanges are now shared between all FakeExchangeManager instances. This broke the unit tests in Heat. This patch effectively reverts the change in oslo_messaging by isolating FakeExchanges to individual FakeExchangeManger instances once more. Change-Id: I6c53f477b17247958be7bd6056136bc9f7f50865 Closes-Bug: #1718780 - Add default configuration files to data_files In order to make it simpler to use the default configuration files when deploying services from source, the files are added to pbr's data_files section so that the files are included in the built wheels and therefore deployed with the code. Packaging and deployment tools can then more easily use the default files if they wish to. This pattern is already established with similar files for neutron, designate and glance as has been mentioned in the related bug report. Change-Id: I1f3c9b2e27d44f28f0e649e2c96a1c3dc1da9ea6 Closes-Bug: #1718356 - Add catch-all for property errors in implicit dependencies The previous patch ensures that we ignore errors getting properties in all extant add_dependencies() methods for calculating implicit dependencies. To guard against similar errors in future, also ignore and log any uncaught ValueError or TypeError exceptions encountered during implicit dependency calculation. Change-Id: I2cac0add975e36a9c52b9cbb50f0660882322754 Related-Bug: #1708209 - Ignore property errors in implicit dependencies Implicit dependencies are calculated before resources are validated. Therefore, any error getting a property that occurs during processing on implicit dependencies will be raised without any of the context of which resource is in error. By ignoring such property errors in implicit dependency calculations, we can defer the error handling until the actual resource validation and thus give the user an error message they can actually use. Change-Id: If7e0e8fd46b1b06a29cdd89743635ed32a9392e7 Closes-Bug: #1708209 - Refactor FloatingIP add_dependencies() method This method is approaching the complexity limits in the pep8 gate, so refactor it to allow us to continue making changes. Change-Id: Ifdb500df121e60431023c43ff0633b790c4052fe - Update incorrect timezone description Change-Id: I6bb45335478b40d2548597979d4754bcd3bfe4fe - Improve update_wait_condition_handle docstring UpdateWaitConditionHandle docstring tells it is identical to WaitConditionHandle without specifying that it inherits from aws_wch.WaitConditionHandle. This is confusing as Heat::WaitConditionHandle is called via curl_cli while Heat::UpdateWaitConditionHandle should be called via cfn-signal. Change-Id: I48d6cc8ebbc754c89befb0281b9097e964c21a49 - Remove the existing snapshots from the backend We only have to remove the existing snapshots for resources when stack delete. Change-Id: Ia195f3c3380fe71e0888c8291209dd4562318951 Closes-Bug: #1716612 - Avoid cloudwatch endpoint retrieve when disabled If openstack-api-cloudwatch service is disabled in a deployment these lines of code will provoke a traceback when trying to get the cloudwatch end-point. Change-Id: I7257f79af764cddc16423826143b0d8babfe54e7 Partial-Bug: #1715083 - Adds 5 backport db migration place holder for Pike Db migration version from 81 to 85 are made as place holder of db backporting for stable/pike. Change-Id: Iddd91173f4fa77f9ef8cf2a60f629d551f6dcdd5 - Return stack.outputs when stack in DELETE_FAILED Users can not view outputs of stacks in DELETE_FAILED. But it's useful for magnum to get cluster info from stack.outputs and for users to debug failed stacks. Change-Id: Ib3927c61660b1b2162d5b34809b48401062a6f02 Close-Bug: #1712274 - Fix error for invalid auth_encryption_key cryptography module throws an InvalidToken exception that does not have any error message. Change-Id: Ied520cd9ff60032fad1ae13e4a81dad6847d82ab Closes-Bug: #1711047 - Delete redundant code In function update_task, the following codes have already judged action is in (self.UPDATE, self.ROLLBACK, self.RESTORE), therefore the deleted code is not necessary. if action not in (self.UPDATE, self.ROLLBACK, self.RESTORE): LOG.error("Unexpected action %s passed to update!", action) self.state_set(self.UPDATE, self.FAILED, "Invalid action %s" % action) return Change-Id: I85f4aaf4c294923358172896cf8efcb5af238957 - Reoder barbican container import Use non-deprecated path first, so there are no warning of use deprecated path. We will remove old path soon. Change-Id: I4472d6a57e77662bf639b4a8ece6a75da8e1c375 - Update URL in README.rst Change-Id: Ib2d036d71f9d4ce7185e2a40d69ab4f9314cebb5 - Updated the template guide. The current template guide does not show required/optional flag for subproperties, it means that users need to look at the source code to clarify if the subproperties is required or not. Updated a sphinx extension doc/ext/resource.py to display required/optional flag for subproperties. Change-Id: Ie9e23d633d749aa7ad087330520cd3b83294794a Closes-Bug: #1694748 - Add diskimage-builder project install Disk-image-create file has been removed from 'diskimage-builder/bin/'. see: https://github.com/openstack/diskimage-builder/tree/master/bin So,We nedd install diskimage-builder project before execute disk-image-create command. Change-Id: Idd052154ac3603a70a4b23eb17f4c84a8b321358 - Remove broken heat.resource_type custom constraint This reverts the commits f5c32ad8fda3d7dcee6e4f9447621cde6d0f8a0 and 14fdf72b000c82a80abb2587189dd7c6c7dfa0a0e. The constraint never worked and the stuff to pass the template to constraints (which was broken because we actually passed a ResourceDefinition instead) is a pain. Just get rid of it rather than fix it. Change-Id: I4e1e787ad94ac1951f472ea066a9b1c9d3e03611 Closes-Bug: #1661403 - Removed unnecessary setUp() call in tests TrivialFix Change-Id: Id74e7a6734b99eaee99d1e2587ea2b2e497a3446 - Eager load resource_properties_data in resource Eager load resource_properties_data in resources in the typical resource-loading scenarios where properties data will be accessed. Thus, we can save an extra db query per resource when loading all the resources in a stack, for instance. Fall back to lazy loading properties data in other scenarios. Also, the resource object doesn't need to store a copy of its ResourcePropertiesData object in self.rsrc_prop_data, so don't. Change-Id: Ib7684af3fe06f818628fd21f1216de5047872948 Closes-Bug: #1665503 - Allow lazy load of raw_template but log warning To avoid hard-to-debug bugs, rather than represent a db_stack's raw_template as None if it hasn't been eagerly loaded, lazy load it. In the lazy loading case, log a warning so developers are aware there is a non-eager loading query that needs to be fixed. Change-Id: Ic06a4bd7afbd4838ca97be302d75dede5402ae6e - Set `public` as update allowed for cluster template Considering with cluster dependency with cluster template. `public` is the only update-able property when cluster depends on that cluster tempate. Therefore, we allow update on that property. Change-Id: I04a45b6a6b70de2cc4886f5d4ada170f959c309a - Support tenacity exponential backoff retry on resource sync Change to use tenacity as the retry library for SyncPoints. Use exponential backoff retry waiting time. The amount of jitter per potential conflict increases 'exponentially' (*cough* geometrically) with each retry. The number of expected conflicts (which drops over time) is updated at each attempt. This allows us to discover the right rate for attempting commits across all resources that are in contention, while actually reducing the delay between retries for any particular resource as the number of outstanding resources drops. Change-Id: I7d5a546a695480df309f22688b239572aa0f897a Co-Authored-By: Zane Bitter <zbitter@redhat.com> Closes-Bug: #1591469 - Trivial:remove unused import and add reasonable path in import Change-Id: Ida320e376a84a36de62a5e97b4107e1f948cd934 - Fix typo in contributing/blueprint.rst doc Change-Id: If5f2aafa3247dc00fc77280bfef0dfc14b6b2cfe Closes-Bug: #1689080 - Fixes heat-keystone-setup-domain authentication failures with v3 With keystone v3 configured, attempting to create a heat domain and heat user fails with authentication errors Change-Id: Ieb1cb7a62b32e842d80020b4ab2870ef1394ba86 Closes-Bug: #1667795 - Remove unused variable TrivialFix Change-Id: I81e3a62c890f0ad043dd604443f56766ba6c11b4 - Add docs of update/replace *_FAILED resource for developer After we implemented the blueprint https://blueprints.launchpad.net/heat/+spec/custom-update-replace-for-failed-resources it is good to add docs for developer about this. Change-Id: I36ebe184497ecac0b8477d6778a0121dcb8ea406
This commit is contained in:
parent
bfe4d973fd
commit
377fca91ab
2
heat
2
heat
|
@ -1 +1 @@
|
|||
Subproject commit e58d1f84f61be0f5d910447fa6d2513b0ee4a364
|
||||
Subproject commit 6f0f14a3a04fb780312076335283f32806929179
|
Loading…
Reference in New Issue