From 174dd39807a22df125708b8b94571963ec5c1400 Mon Sep 17 00:00:00 2001 From: Nikolay Mahotkin Date: Wed, 26 Aug 2015 11:13:14 +0300 Subject: [PATCH] Mistral documentation: main features Implements blueprint mistral-docs-features Change-Id: I26902db03b6db0b68491aa0c3374efebb8ce840a --- doc/source/guides/using_yaql.rst | 4 + doc/source/main_features.rst | 254 +++++++++++++++++++++++++++++-- 2 files changed, 249 insertions(+), 9 deletions(-) create mode 100644 doc/source/guides/using_yaql.rst diff --git a/doc/source/guides/using_yaql.rst b/doc/source/guides/using_yaql.rst new file mode 100644 index 000000000..1426ab04a --- /dev/null +++ b/doc/source/guides/using_yaql.rst @@ -0,0 +1,4 @@ +How to use YAQL in Mistral +========================== + +TBD \ No newline at end of file diff --git a/doc/source/main_features.rst b/doc/source/main_features.rst index 5b6c26de8..11050b05e 100644 --- a/doc/source/main_features.rst +++ b/doc/source/main_features.rst @@ -1,10 +1,246 @@ -Mistral Features -================ +Mistral Main Features +===================== + + +Task results / Dataflow +----------------------- + +Mistral supports transferring data from one task to another. In other words, if *taskA* produces +some result data, and then *taskB* (which is next after *taskA*) can use that result data. For referring +to the result data and also probably for selecting, filtering actions of the data in Mistral is used specific +query language called *YAQL*. For the details how to use *YAQL*, please see :doc:`How to use YAQL in Mistral `. +From Mistral standpoint, feature of referring any task produced result in subsequent task was called **DataFlow**. + + +Task affinity +------------- + +Task affinity is a feature which could be useful for executing particular +tasks on specific Mistral executors. In fact, there are 2 cases: + +1. Need to execute the task on single executor. +2. Need to execute the task on one of executor in executors group which has one name. + +For enabling task affinity feature, edit edit section "executor" host property +in configuration file:: + + [executor] + host = my_favorite_executor + +Then start (restart) executor. Use target task property to specify +this executor in Workflow DSL:: + + ... Workflow YAML ... + task1: + ... + target: my_favorite_executor + ... Workflow YAML ... + +Task policies +------------- + +Any Mistral task regardless of its workflow type can optionally have configured policies. +Policies itself control the flow of the task - for example, policy can delay task execution before task starts +or after task completes. + +YAML example +^^^^^^^^^^^^ +:: + + my_task: + action: my_action + pause-before: true + wait-before: 2 + wait-after: 4 + timeout: 30 + retry: + count: 10 + delay: 20 + break-on: <% $.my_var = true %> + +There are different types of policies in Mistral. + +1. **pause-before** + + Defines whether Mistral Engine should put the workflow on pause or not before starting a task. + +2. **wait-before** + + Defines a delay in seconds that Mistral Engine should wait before starting a task. + +3. **wait-after** + + Defines a delay in seconds that Mistral Engine should wait after a task has completed before starting next tasks defined in *'on-success'*, *'on-error'* or *'on-complete'*. + +4. **timeout** + + Defines a period of time in seconds after which a task will be failed automatically by engine if hasn't completed. + +5. **retry** + + Defines a pattern how task should be repeated. + +* *count* - Defines a maximum number of times that a task can be repeated. +* *delay* - Defines a delay in seconds between subsequent task iterations. +* *break-on* - Defines a YAQL expression that will break iteration loop if it evaluates to *'true'*. If it fires then the task is considered error. +* *continue-on* - Defines a YAQL expression that will continue iteration loop if it evaluates to *'true'*. If it fires then the task is considered successful. + + Retry policy can also be configured on a single line as:: + + task1: + action: my_action + retry: count=10 delay=5 break-on=<% $.foo = 'bar' %> + + All parameter values for any policy can be defined as YAQL expressions. + +Join +---- + +Join flow control allows to synchronize multiple parallel workflow branches and aggregate their data. + +**Full Join (join: all)**. + +YAML example +^^^^^^^^^^^^ +:: + + register_vm_in_load_balancer: + ... + on-success: + - wait_for_all_registrations + + register_vm_in_dns: + ... + on-success: + - wait_for_all_registrations + + try_to_do_something_without_registration: + ... + on-error: + - wait_for_all_registrations + + wait_for_all_registrations: + join: all + action: send_email + +When a task has property *"join"* assigned with value *"all"* the task will run only +if all upstream tasks (ones that lead to this task) are completed and corresponding +conditions have triggered. Task A is considered an upstream task of Task B if Task A +has Task B mentioned in any of its *"on-success"*, *"on-error"* and *"on-complete"* clauses +regardless of YAQL guard expressions. + +**Partial Join (join: 2)** + +YAML example +^^^^^^^^^^^^ +:: + + register_vm_in_load_balancer: + ... + on-success: + - wait_for_all_registrations + + register_vm_in_dns: + ... + on-success: + - wait_for_all_registrations + + register_vm_in_zabbix: + ... + on-success: + - wait_for_all_registrations + + wait_for_two_registrations: + join: 2 + action: send_email + +When a task has property *"join"* assigned with a numeric value then the task +will run once at least this number of upstream tasks are completed and +corresponding conditions have triggered. In the example about task +"wait_for_two_registrations" will run if two any of "register_vm_xxx" tasks complete. + +**Discriminator (join: one)** + +Discriminator is a special case of Partial Join when *"join"* property has value 1. +In this case instead of 1 it is possible to specify special string value *"one"* +which is introduced for symmetry with *"all"*. However, it's up to the user whether to use *"1"* or *"one"*. + + +Processing Collections (with-items) +----------------------------------- + +YAML example +^^^^^^^^^^^^ +:: + + --- + version: '2.0' + + create_vms: + description: Creating multiple virtual servers using "with-items". + input: + - vm_names + - image_ref + - flavor_ref + output: + vm_ids: <% $.vm_ids %> + + tasks: + create_servers: + with-items: vm_name in <% $.vm_names %> + action: nova.servers_create name=<% $.vm_name %> image=<% $.image_ref %> flavor=<% $.flavor_ref %> + publish: + vm_ids: <% $.create_servers.id %> + on-success: + - wait_for_servers + + wait_for_servers: + with-items: vm_id in <% $.vm_ids %> + action: nova.servers_find id=<% $.vm_id %> status='ACTIVE' + retry: + delay: 5 + count: <% $.vm_names.len() * 10 %> + +Workflow *"create_vms"* in this example creates as many virtual servers as we +provide in *"vm_names"* input parameter. E.g., if it is specified *vm_names=["vm1", "vm2"]* +then it'll create servers with these names based on same image and flavor. +It is possible because of using *"with-items"* keyword that makes an action +or a workflow associated with a task run multiple times. Value of *"with-items"* +task property contains an expression in the form: ** in <% YAQL_expression %>**. + +The most common form is:: + + with-items: + - var1 in <% YAQL_expression_1 %> + - var2 in <% YAQL_expression_2 %> + ... + - varN in <% YAQL_expression_N %> + +where collections expressed as YAQL_expression_1, YAQL_expression_2, +YAQL_expression_N must have equal sizes. When a task gets started Mistral +will iterate over all collections in parallel, i.e. number of iterations will +be equal to length of any collections. + +Note that in case of using *"with-items"* task result accessible in workflow +context as <% $.task_name %> will be a list containing results of corresponding +action/workflow calls. If at least one action/workflow call has failed then +the whole task will get into *ERROR* state. It's also possible to apply retry +policy for tasks with *"with-items"* property. In this case retry policy will +be relaunching all action/workflow calls according to *"with-items"* +configuration. Other policies can also be used the same way as with regular non *"with-items"* tasks. + +Execution expiration policy +--------------------------- + +For Mistral used in production, it is often hardly to control the number of workflow executions. The number of +workflow executions is significantly growing for the long time of Mistral running. The purpose of this feature to +delete old workflow executions which has been already completed. The criteria is the time when a workflow execution was +updated last time. + +**By default this feature is disabled.** + +In order to configure this feature, please open and edit configuration file and specify time in minutes:: + + [execution_expiration_policy] + older_than = 10080 # Workflow executions older than 1 week will be deleted automatically. -TBD: - - Task results / Dataflow - - Task affinity - - Task policies - - Loops (with-items) - - Service coordinator - - Execution expiration policy \ No newline at end of file