3653 Commits

Author SHA1 Message Date
Winson Chan
791f6f14f7 Use oslo.messaging for AMQP communications
Ported task executor to oslo.messaging. The executor module is
replaced with RPC server/client named Executor and ExecutorClient
respectively to handle task execution. The old executor module is
deleted.  The engine is modified to use the ExecutorClient to send
task requests over the transport to the Executor. The launcher is
modified to start the new Executor.

Change-Id: Ibce01813e51c2220c45e05bb820b4729027446a3
Implements: blueprint mistral-oslo-messaging
2014-03-10 10:27:28 -07:00
Renat Akhmerov
70b9291e2d Working on Data Flow (step 5)
* Testing subsequent tasks

TODO:
* More tests
* Actions refactoring

Change-Id: Ib057e76479cac065cc430e4b5af60db314c3a3f2
2014-03-10 15:33:04 +00:00
Renat Akhmerov
69a6b9bf28 Working on Data Flow (step 4)
* Publishing variables in execution context
* Additional tests (publishing and context manipulations)

TODO:
* More tests

Change-Id: I8f1e6d7a7a1251b507e33a0cc11bb1afbee3a2c2
2014-03-08 07:51:24 +00:00
Renat Akhmerov
31ed10aa18 Working on Data Flow (step 3)
* Refactored tests hierarcy (moved get_resource to the base class)
* Fixed logic in abstract engine (transaction scoope was wrong)
* Added stub methods for context manipulations (not everywhere yet)
* Refactored actions
* Created ECHO action to be able to emulate required output
* Added simple data flow test (two dependent tasks)

TODO:
* More tests
* Refactor actions according to the latest discussions

Change-Id: I5ba5e330110889014eeca53501ddae54dc9a1236
2014-03-07 16:53:02 +07:00
Jenkins
69f6aa6477 Merge "Make engine configurable, make debugger show local variables" 2014-03-05 08:04:57 +00:00
Jenkins
f1847fe5c1 Merge "Partially fixed the pylint errors" 2014-03-04 12:01:59 +00:00
Dmitri Zimine
b0fa4605ce Make engine configurable, make debugger show local variables
To make it easy to set Mistral for debugging.
1) Make engine configurable.
2) Enable debugger via command line. Passing --use-debugger will make PyDev or PyCharm show
 local variables in debugging session.

Also, a tiny fix in action_factory.

Change-Id: Id054ae5156649b1f6bbba7131e54bf21d71c659a
2014-03-03 10:21:03 -08:00
Timur Nurlygayanov
ba90868050 Partially fixed the pylint errors
Fixed several errors:
   E1003
   E1101

It is impossible to fix all E1101 errors: looks like we have
problems with outdated DataBase models.

Change-Id: I46863c553f033e5f9e4ae9cb9c20e297c099a114
Partial-Fixes: bug #1281164
2014-03-03 13:26:05 +04:00
Nikolay Mahotkin
368c1d9880 Fix throwing exception when 'output' block is not defined
* Deleted exception when 'output' block in REST_API task
   is not defined. In this case result is the whole object
   returned from extract_state_result

Fixes bug: #1280117

Change-Id: I8143283133994d39e34e9c82493b2cff67fa0d59
2014-02-26 15:52:56 +04:00
Timur Nurlygayanov
e1cfad9f0b Fixed critical pylint warnings
Fixed the following pylint warnings:
   W0231
   W0621
   W0621

Change-Id: I6d49a19f843ab2b63740199ef387e7e878fa030f
Partial-Fixes: bug #1281164
2014-02-25 23:25:42 +04:00
Renat Akhmerov
b3c23b7e1b Working on Data Flow (step 2)
* Added ExpressionEvaluator interface and YAQL implementation
* Removed yaql_utils, evaluator is now a more flexible replacement
* Fixed the places where yaql_utils was used
* Adjusted test cases

Change-Id: I7de70e678c211df34788763047b8361de0d3902a
2014-02-25 12:27:43 +07:00
Renat Akhmerov
58407b6f94 Working on Data Flow (step 1)
* Refactored API layer so that we can work with 'context' as with a
 json object in underlying layers (DB, engine, etc.) rather than
 a string
* Added "context" parameter in all required places
* Added necessary Data Flow related properties to DB models
* Refactored and fixed a series of tests
* Minor formatting changes

TODO:
* Calculation of task incoming context
* Abstract interface for expression evaluator
* Data Flow related tests

Partially implements blueprint: mistral-dataflow

Change-Id: Ie7f94d79265e9861f7ad15c76ff6d788ec62b683
2014-02-25 12:27:43 +07:00
Nikolay Mahotkin
8a73da20fd Add scheduling specific task on sucess/error
* Fix get task properties, get task in DSL
 * Evaluating task on success/error/finish properties in engine:
   - evaluate YAQL conditions
   - schedule tasks
 * Unit tests for handling success/error/finish

 * For real work with context there is need to store
   it somewhere

Implements: blueprint schedule-tasks-after-completion

Change-Id: Ia9c34e755055ff0273dcd79a314858dbfbcc2fa9
2014-02-20 21:12:10 +04:00
Jenkins
77a70c36e2 Merge "Send email action, part 2" 2014-02-17 17:14:11 +00:00
Dmitri Zimine
e971459eb7 Send email action, part 2
Added send_email to action types and action factory.
Now it works end-to-end.

NOTE:this renames DSL from action.parameters.responce
to action.output. Other projects will need to adjust.

Tested manually with mistral-extra/examples/webhooks/cmd/run.py,
It works, almost:  except a bug causing exception on parsing out results.
In discussion, but a fix should go in separate commit.

Also, small fixes:
action_factory:31 - used to blow up when action has no 'parameters'
dsl.py:32 - return details on parsing errors (error, line, column)
api/config.py - deleted: not used

Change-Id: Ie7079eb9589c23cc6704f8e8739af67a635f1c77
2014-02-17 08:09:51 -08:00
Renat Akhmerov
dbfeedd9d7 Rename "target_task" to "task"
* Renamed all occurrences of "target_task" to "task"
* Fixed a typo in executors controller.

Change-Id: I6b412090de224282c7eef2abceb989446d8b8df9
2014-02-14 18:19:42 +07:00
dzimine
9bca04c79e Send email action, step 1
Basic send-email functionality and unit tests
See comments in test_send_email_action on how to run with real smtpd.

Implements: blueprint mistral-std-send-email-action
Change-Id: Ib1d460d26ce54f4ab85d34b18947d858ab471dfd
2014-02-11 01:11:42 -08:00
Jenkins
842d75713a Merge "Fixing access to task "parameters" property in DSL" 2014-01-31 00:23:10 +00:00
Jenkins
d8082084bc Merge "Add negative tests to api" 2014-01-30 08:42:28 +00:00
dzimine
1c15945f86 Add negative tests to api
1. Add couple of negative tests, find the way, more to go.

2. Add assertNotFound to the base test class for convenience.

3. MistralException - added default message to enable default instantiation - make mock.assertRaise fails when exception has no default constructor

Change-Id: I708637e3995597a819571a90f3292d6e34deab4e
2014-01-29 19:01:27 -08:00
Renat Akhmerov
430b5ec2e4 Fixing access to task "parameters" property in DSL
Change-Id: I046430ffb77abf2d462c89050e9e28e67fa72816
2014-01-29 16:48:00 -08:00
Nikolay Mahotkin
ca37071c5d Fix getting task on-* properties in DSL
* Unit test

Change-Id: I949db952920da0606993f3dc260fb84ef2c4fdaf
2014-01-28 20:35:01 +04:00
Jenkins
aae48e19ec Merge "Fix task keys properties in DSL parser" 2014-01-28 13:49:18 +00:00
Jenkins
96611e8576 Merge "Add YAQL expression evaluation" 2014-01-28 13:11:28 +00:00
Jenkins
3a9c3a499e Merge "Modified Rest action for process 'input' property" 2014-01-28 13:05:52 +00:00
Jenkins
3af480bb27 Merge "Add sync task execution" 2014-01-28 13:05:22 +00:00
Nikolay Mahotkin
d32552b9fa Fix task keys properties in DSL parser
Change-Id: I5624fc6e35d100c63e79ed96c832bdc8eda67cbb
2014-01-28 17:01:22 +04:00
Jenkins
7d55b2befb Merge "Fixing and refactoring authentication" 2014-01-28 13:00:48 +00:00
Jenkins
4637e46423 Merge "Deleting client and demo app from main Mistral repo" 2014-01-28 13:00:45 +00:00
Nikolay Mahotkin
4ebc881b0c Add YAQL expression evaluation
* YAQL evaluate
 * Unit tests

Change-Id: I0853a08234c75853875b42cd8e89819e9b1e431f
2014-01-28 15:21:58 +04:00
Nikolay Mahotkin
1020554f1c Modified Rest action for process 'input' property
* For HTTP GET task input is converted to request url parameters
 * For HTTP POST/PUT - into request body

Change-Id: Ibc863ffb99ffdd0099f9ba2f43178e02dc1dd0c2
2014-01-28 13:57:40 +04:00
Nikolay Mahotkin
380689f223 Add sync task execution
* Non-Mistral actions gets a JSON response
   from which result can be extracted with
   YAQL-expression
 * Unit test
 * Feature is added to both engines

Change-Id: Iae6aa031e6b5700b3a08ee184f5eae291458a8c4
2014-01-28 13:57:32 +04:00
Renat Akhmerov
44e38c97b7 Fixing and refactoring authentication
* Moved pecan configuration to oslo config
* Refactored functional base test
* Got rig of thread local related code in mistral/context.py
* Added new exception ApplicationContextNotFoundException
* Fixed example configuration file
* Made minor cosmetic changes (blank lines, naming)

Change-Id: I1899ce2562a34ebafa20c5735bdf4f0c80dd0175
2014-01-27 12:11:28 -05:00
Renat Akhmerov
378c33ec95 Deleting client and demo app from main Mistral repo
Change-Id: Ib4e1deb0bcad53b55478b00f7f8a203186af305f
2014-01-27 11:13:55 -05:00
Jenkins
05b0989327 Merge "Add integration tests" 2014-01-27 14:21:10 +00:00
TimurNurlygayanov
d25e1de5cc Fixed issue with tarballs.
Closes-Bug: #1271068

Change-Id: I6f3035b19655313bb481650af69d9681cae02cb7
2014-01-24 20:04:46 +04:00
Sergey Murashov
6aff5394dc Add integration tests
Change-Id: I0873da3f0ddc73722f3d8506fde463da9d39cea9
2014-01-24 12:20:53 +04:00
Jenkins
4f7ba4402d Merge "Divide RestAction on two separated actions in DSL" 2014-01-20 15:48:33 +00:00
Jenkins
c69cfbd4f5 Merge "Adding Data Flow related code to REST API" 2014-01-17 19:20:55 +00:00
Nikolay Mahotkin
050f6220f6 Divide RestAction on two separated actions in DSL
* in case MISTRAL_REST_ACTION we add Mistral-specific
   headers, in case REST_ACTION - we are not.
 * new class for this type of action is unnecessary,
   all of these are Rest, only properties can be edited
   in factories methods

Change-Id: I431e5009a531da6bea191e7253d43b4cd58c46fb
2014-01-17 17:01:42 +04:00
Nikolay Mahotkin
12d7056def Add new access methods in DSL parser
* Access to new task properties:
   input, onSuccess, OnError; any task property
 * 'dependsOn' is renamed to 'requires' and returns dict object
   which contains conditions of task transition.

  DSL can be defined either:
     task3:
       requires: [task1, task2]
   or:
     task3:
       requires:
         task1: condition1  (or empty)
         task2: condition2  (or empty)

Change-Id: Ic3ab5754443100e101f04260d9054b0a604fcd6d
2014-01-17 13:06:27 +04:00
Jenkins
08113db18b Merge "Refactoring local and scalable engines" 2014-01-16 21:14:12 +00:00
Renat Akhmerov
baba497d99 Refactoring local and scalable engines
* Moved iethods start_workflow_execution and convey_task_result into
 abstract engine
* Refactored existing local engine unit tests
* Added simple unit tests for scalable engine
* Minor cosmetic changes (imports, blank lines)

Change-Id: Ic781b134068f52ed5827b945fb57b202239fe9c4
2014-01-16 07:06:30 -08:00
Renat Akhmerov
b940d50c87 Adding Data Flow related code to REST API
* Added "context" property for executions
* Refactored all REST API test to use @mock.patch

Change-Id: Ia0939e7099ae4386c1c3a761ac3892533eb3aed4
2014-01-15 18:01:10 -08:00
Renat Akhmerov
385d123e5d Fixing typo in an exception message in Mistral client
Closes-Bug: #1264146
Change-Id: I69170e7529f7ec85c8dade71f212adb34085304e
2014-01-14 15:55:12 -08:00
Nikolay Mahotkin
ea1fbeeb6b Step 2 refactoring Mistral engines
* Combine common parts of single and scalable engines

Change-Id: Iaa8708a831ca22a79d3ecd59c960be09a904e57a
2014-01-14 16:11:47 +04:00
Nikolay Mahotkin
0934507ed2 Step 1 refactoring Mistral engines
* Action definition was combined in actions module
 * workflow from both single and scalable engines were deleted
 * task from single engine was deleted
 * unit test 'test_workflow' was deleted due to deleting workflow
 * minor fixes to single engine

 Next step will be extracting and combining common parts
 of scalable and single engine.

Change-Id: I15fbbb7f9d5e453c17625423eeb1bf841a19db18
2014-01-14 15:57:28 +04:00
Nikolay Mahotkin
348f3eea19 Fix exceptions output when creating object in DB
Fixes: bug #1268501

Change-Id: I0fe7f61b72afc9cd7a5c3b65b343c1d438e07632
2014-01-13 13:59:08 +04:00
Sergey Murashov
673b3976ad Add SQLAlchemy in requirements.txt
Change-Id: Ieb7849cadfa9a14423bea74fd280a52aeb406f0b
2014-01-10 14:06:11 +04:00
Jenkins
d8ece7f8c2 Merge "Fix DB API import in scheduler.py" 2014-01-09 10:28:00 +00:00