3042 Commits

Author SHA1 Message Date
Renat Akhmerov
15ea7c06f6 BP mistral-actions-design (ad-hoc action)
* Created AdHocAction that works as a transforming wrapper
  for any other action (uses action specification dictionary)
* Added tests for AdHocAction
* Moved and refactored tests for email and echo actions from
  old package

TODO:
* Integrate AdHocAction with action factory (it is a special built-in
  action class and requires some special action on action factory side)
* Refactoring executor to use new action infrastructure
* Remove old action infrastructure
* Action plugin architecture

Partially implements: blueprint mistral-actions-design

Change-Id: Ieaf479047bfcaf3da61d40f28f108d198af6d16f
2014-04-14 19:24:37 +07:00
Renat Akhmerov
0d5a04d7cc BP mistral-actions-design (Mistral HTTP action)
* Created MistralHTTPAction
* More unit tests
* Refactoring

TODO:
* Ad-hoc per-workflow namespaces and actions
* Refactoring executor to use new action infrastructure
* Remove old action infrastructure
* Action plugin architecture

Partially implements: blueprint mistral-actions-design

Change-Id: I6856d6587243872a67fa42efff0bf3f88ec19d82
2014-04-14 19:24:37 +07:00
Renat Akhmerov
b0e94b00a8 BP mistral-actions-design (action creation)
* Method create_action in action_factory.py
* Method get_action_class in action_factory.py
* Additional tests for action_factory.py
* Removing namespace and action name params from action constructors

TODO:
* Ad-hoc per-workflow namespaces and actions
* Mistral HTTP standard action
* Refactoring executor to use new action infrastructure
* Remove old action infrastructure
* Action plugin architecture

Partially implements: blueprint mistral-actions-design
Partially implements: blueprint mistral-change-rest-api-to-http

Change-Id: Ia90c66e9e06a577d06d3db687098cdb6d741afaa
2014-04-14 19:24:37 +07:00
Renat Akhmerov
32720f2c72 BP mistral-actions-design (add new actions package)
* Base class for actions (Action)
* Class for representing action namespaces (NameSpace)
* Module action_factory.py for registering action classes, accessing
  them and creating action instances
* Unit tests for action_factory.py
* Standard actions (echo, http, email)

TODO:
* Ad-hoc per-workflow namespaces and actions
* Mistral HTTP standard action
* Refactoring executor to use new action infrastructure
* Remove old action infrastructure
* Action plugin architecture

Partially implements: blueprint mistral-actions-design
Partially implements: blueprint mistral-change-rest-api-to-http

Change-Id: I8e79b1f8173809584f0156e666e1675f5bd7bb37
2014-04-14 19:24:27 +07:00
Renat Akhmerov
ebeb68b6dc BP mistral-actions-design
* Renaming task 'input' property to 'parameters' according to BP
* Fixing auth token generation related test
* Small refactoring in expressions
* Formatting changes

Partially implements: blueprint mistral-actions-design

Change-Id: I8659145d022ac4a2895c9d06f85f0961bf0b4f50
2014-04-14 15:21:27 +07:00
Nikolay Mahotkin
75fc00796a Add SSH Action
* We need an another way to communicate with VM,
   so SSH is this way. Also it gives a possibility
   to check if VM operation system is up by sending
   a simple ssh command to it.
 * Added SSH Action
 * Only password access, forwarding public/private keys
   will be in the future
 * It is tested on real VM. However, it needs to be tested
   by somebody else

Implements blueprint mistral-ssh-action

Change-Id: I7c51453afb6a926bbce01f4a9065a3c313a79fff
2014-04-11 13:41:35 +04:00
Winson Chan
a736f03bf8 Remove local engine
The local engine is replaced in process executor using the oslo.messaging
fake transport.  The local engine module is removed here along with related
unit tests. Some unit tests are rewritten to use the ScalableEngine. An
EngineTestCase is included in the base module of tests to simplify engine
testing.

Change-Id: I1a49a53eac87a209660c493fd7c3fe3e914fd092
Implements: blueprint mistral-inproc-executor
2014-04-07 10:46:03 -07:00
Jenkins
7f8530aeda Merge "Fix repeatable task scheduling" 2014-04-07 17:07:26 +00:00
Nikolay Mahotkin
6342ffb6ab Fix repeatable task scheduling
* Fixed repeatable task scheduling. See bug description for
   more information.
 * Small fix in data flow test

 Fixes bug: #1301866

Change-Id: I40e653ac242df0f5deee8b72ac77a7c04e4fda2a
2014-04-04 19:04:17 +04:00
Nikolay Mahotkin
e1bf0a1479 Add resolving inline expressions
* We allow to resolve something like:
   - /{$.project_id}/servers
   - {$.auth_token}

Change-Id: Ic2a5a32b42c168bac021ff1e96d1dae8bac6647e
2014-04-04 13:37:17 +04:00
Jenkins
697f67d7e7 Merge "Refactor the local engine to use an in process executor" 2014-04-04 08:58:56 +00:00
Jenkins
f8e7327678 Merge "Fix minor issues" 2014-04-03 05:23:32 +00:00
Jenkins
1a24ebc4e0 Merge "Add extracting action output" 2014-04-03 05:23:32 +00:00
Jenkins
7905fa5be5 Merge "Cosmetic change." 2014-04-03 03:41:29 +00:00
Manas Kelshikar
8b0bfa5061 Cosmetic change.
Pick up cosmetic changes from https://review.openstack.org/#/c/80127/10
* rename exec_flow_context to task_runtime_context.
* Fix comments to start with cap letter and end with a period.

Change-Id: Ic74f045f8d401053a983b1f6356a8d3562b23f2e
2014-04-02 11:43:23 -07:00
Jenkins
a1754e9e24 Merge "Fix keystone trust client" 2014-04-02 17:09:50 +00:00
TimurNurlygayanov
0b743fdd74 Fixed issue with deprecated exception
Change-Id: Ic2f1fec236f193fa4b07ebfd96e878d86cc6ad37
Closes-Bug: #1298524
2014-04-02 13:06:35 +04:00
Nikolay Mahotkin
24f1c7d455 Fix minor issues
* Action factory creates rest_action with either
   parameters or body accordingly to request method
   (for GET - parameters, for POST/PUT - body)
 * Fix apply context method:
   * All data which is not resolved by YAQL is ignored
 * Rest action send body in form json.dumps(data)
   (it is better for nested dictionary)
 * Improved some unit tests

 * With this patch Mistral is able to connect to Nova
   and create instance

Change-Id: Ibc73a7dc2de4339acbb9310755c2c53028815b38
2014-04-02 12:11:01 +04:00
Nikolay Mahotkin
36a84e299e Fix keystone trust client
* The issue popped up after release of python-keystoneclient 0.7.
   The issue is caused by new keystoneclient not accepting
   project_id and trust_id simultaneously as constructor parameters.

Change-Id: Id0b7b582e9b794dace0f440df582ff9502b781c5
2014-04-02 12:04:05 +04:00
Nikolay Mahotkin
94fde896b5 Add extracting action output
* In case if we define the specific output
   in action, we should evaluate all given YAQL
   expressions and return calculated dict,
   otherwise - return raw result

Change-Id: I8f4a91c7fb0de743793c7cd66b892e038e25332f
2014-03-31 12:18:40 +04:00
Winson Chan
e9a0f14db2 Refactor the local engine to use an in process executor
Refactored launch script to start the API and executor on the same process
so the transport for a oslo.messaging fake driver can be shared. A transport
attribute is added to the abstract engine as a class attribute to reference
the transport object.

Change-Id: I4b8684ebded9eb993749f2503380e078087edd7d
Implements: blueprint mistral-inproc-executor
2014-03-30 18:05:54 -07:00
Manas Kelshikar
025343c050 Implements: blueprint mistral-std-repeat-action
In this implementation there is only a provision to repeat a specific action.
- Features like configurable iterations, delay, break-on are implemented.
- All final data is published to the context and there is no per iteration
  persistence.
- The iteration context which is specific to the scope is saved in the
  exec_flow_context property of a task.

Change-Id: I1e9ff7847f0046e11d3b32eec65f664c2b09d3b1
2014-03-28 10:33:28 -07:00
Jenkins
6424670a58 Merge "Add keystone auth_token in context" 2014-03-27 00:33:47 +00:00
Jenkins
5673db7b2f Merge "Correct fake action test name" 2014-03-27 00:32:29 +00:00
Jenkins
ebf9bb0bc0 Merge "Remove unneeded declarations in unit tests" 2014-03-27 00:31:30 +00:00
Nikolay Mahotkin
9fd4c69ac1 Correct fake action test name
Change-Id: Ibb2322139fd8d7f3365d3522afde622def910fe9
2014-03-26 17:26:24 +04:00
Nikolay Mahotkin
7709cb0f0b Remove unneeded declarations in unit tests
Change-Id: I75e3a60ef7720011aa2bd3610bbc60d14140bdaa
2014-03-26 17:23:54 +04:00
Nikolay Mahotkin
64ea892672 Add keystone auth_token in context
* Add creating keystone auth_token in context
   when task is scheduled for execution. Auth token
   is created from workbook trust_id in case Auth_enable
 * Added transforming values of task_spec and service_spec
   with YAQL and context.
 * This feature affects some unit tests, so they are fixed
 * Added unit tests

Partially implements: blueprint mistral-dataflow

Change-Id: I15794da427bfc7babbc4e3d58a90200c10a19a72
2014-03-24 14:24:18 +04:00
Nikolay Mahotkin
23073881aa Fix keystone config group name
* keystone_authtoken is renamed to keystone due to
   unexisting name. It isn't able to create workbook
   with auth_enable=True without this fix

Change-Id: I3d865e31b87dbb18e62ee3d6fa4ca5054ff5c8fd
2014-03-17 17:21:01 +04:00
Jenkins
da7b742aa7 Merge "Add script to allow update dependencies in all envs" 2014-03-17 05:04:58 +00:00
Nikolay Mahotkin
f00803f467 Add script to allow update dependencies in all envs
Change-Id: I75aa921607fb214d55205cc380749180e0991024
2014-03-14 14:41:48 +04:00
Renat Akhmerov
91df3876ed Fixing ordering bugs in local engine tests
Change-Id: I4429895f6a342ee796b5b21291bd9de1da38d09e
2014-03-14 17:14:21 +07:00
Renat Akhmerov
427db9d84b Fixing ordering bugs in workbook model and tests
* When parsing yaml we can't rely on order of properties
 because it's against yaml specification
* In tests we often make assumptions about list items order

Change-Id: I536bc31ba7bee68e2904a23af73004f2e0ba2261
2014-03-14 14:47:44 +07:00
Renat Akhmerov
d90a727c6b Fixing executor launch script
* Executor launch script should now start engine because
 it calls engine.convey_task_result() directly. It's a
 temporary solution until engine is a standalone process.
 When it's done executor will be sending a message back to
 engine instead of making a local call.

Change-Id: I4666ca5f1a9cb093e8b7c9e640048f0176955595
2014-03-13 22:19:20 +07:00
Jenkins
06aaa949fd Merge "Rename 'events' to 'triggers'" 2014-03-13 14:48:14 +00:00
Nikolay Mahotkin
383e7775f6 Fix getting task on-* properties
Fixes: Bug #1288227

Change-Id: I46680abe82c86e6b40b7f0274ad40f8c6cc250ef
2014-03-12 19:05:40 +04:00
Nikolay Mahotkin
7e01958c95 Rename 'events' to 'triggers'
* "events" are renamed to "triggers"
 * triggers moved out of workflow section

Implements: blueprint mistral-rename-event-to-trigger

Change-Id: I5b76e9c063a3ce9eec5d485bb5db5e22ac7981d4
2014-03-12 15:04:45 +04:00
Nikolay Mahotkin
0884cd0ff0 Implement new object-model specification
* Implemented new object model
 * Implemented spec validation
 * Old DSL parser was deleted
 * All code is replaced by new workbook spec
 * Also tests are fixed
 * events -> triggers
 * Unit tests

Implements: blueprint mistral-dsl-model
Partially implements: blueprint mistral-dsl-validation
Partially implements: blueprint mistral-rename-event-to-trigger

Change-Id: I782b7983ddc724375229587f467c55ad9341478b
2014-03-12 14:58:54 +04:00
Jenkins
c3ecd49d26 Merge "Working on Data Flow (step 5)" 2014-03-11 06:45:13 +00:00
Winson Chan
791f6f14f7 Use oslo.messaging for AMQP communications
Ported task executor to oslo.messaging. The executor module is
replaced with RPC server/client named Executor and ExecutorClient
respectively to handle task execution. The old executor module is
deleted.  The engine is modified to use the ExecutorClient to send
task requests over the transport to the Executor. The launcher is
modified to start the new Executor.

Change-Id: Ibce01813e51c2220c45e05bb820b4729027446a3
Implements: blueprint mistral-oslo-messaging
2014-03-10 10:27:28 -07:00
Renat Akhmerov
70b9291e2d Working on Data Flow (step 5)
* Testing subsequent tasks

TODO:
* More tests
* Actions refactoring

Change-Id: Ib057e76479cac065cc430e4b5af60db314c3a3f2
2014-03-10 15:33:04 +00:00
Renat Akhmerov
69a6b9bf28 Working on Data Flow (step 4)
* Publishing variables in execution context
* Additional tests (publishing and context manipulations)

TODO:
* More tests

Change-Id: I8f1e6d7a7a1251b507e33a0cc11bb1afbee3a2c2
2014-03-08 07:51:24 +00:00
Renat Akhmerov
31ed10aa18 Working on Data Flow (step 3)
* Refactored tests hierarcy (moved get_resource to the base class)
* Fixed logic in abstract engine (transaction scoope was wrong)
* Added stub methods for context manipulations (not everywhere yet)
* Refactored actions
* Created ECHO action to be able to emulate required output
* Added simple data flow test (two dependent tasks)

TODO:
* More tests
* Refactor actions according to the latest discussions

Change-Id: I5ba5e330110889014eeca53501ddae54dc9a1236
2014-03-07 16:53:02 +07:00
Jenkins
69f6aa6477 Merge "Make engine configurable, make debugger show local variables" 2014-03-05 08:04:57 +00:00
Jenkins
f1847fe5c1 Merge "Partially fixed the pylint errors" 2014-03-04 12:01:59 +00:00
Dmitri Zimine
b0fa4605ce Make engine configurable, make debugger show local variables
To make it easy to set Mistral for debugging.
1) Make engine configurable.
2) Enable debugger via command line. Passing --use-debugger will make PyDev or PyCharm show
 local variables in debugging session.

Also, a tiny fix in action_factory.

Change-Id: Id054ae5156649b1f6bbba7131e54bf21d71c659a
2014-03-03 10:21:03 -08:00
Timur Nurlygayanov
ba90868050 Partially fixed the pylint errors
Fixed several errors:
   E1003
   E1101

It is impossible to fix all E1101 errors: looks like we have
problems with outdated DataBase models.

Change-Id: I46863c553f033e5f9e4ae9cb9c20e297c099a114
Partial-Fixes: bug #1281164
2014-03-03 13:26:05 +04:00
Nikolay Mahotkin
368c1d9880 Fix throwing exception when 'output' block is not defined
* Deleted exception when 'output' block in REST_API task
   is not defined. In this case result is the whole object
   returned from extract_state_result

Fixes bug: #1280117

Change-Id: I8143283133994d39e34e9c82493b2cff67fa0d59
2014-02-26 15:52:56 +04:00
Timur Nurlygayanov
e1cfad9f0b Fixed critical pylint warnings
Fixed the following pylint warnings:
   W0231
   W0621
   W0621

Change-Id: I6d49a19f843ab2b63740199ef387e7e878fa030f
Partial-Fixes: bug #1281164
2014-02-25 23:25:42 +04:00
Renat Akhmerov
b3c23b7e1b Working on Data Flow (step 2)
* Added ExpressionEvaluator interface and YAQL implementation
* Removed yaql_utils, evaluator is now a more flexible replacement
* Fixed the places where yaql_utils was used
* Adjusted test cases

Change-Id: I7de70e678c211df34788763047b8361de0d3902a
2014-02-25 12:27:43 +07:00