Mistral documentation: main features
Implements blueprint mistral-docs-features Change-Id: I26902db03b6db0b68491aa0c3374efebb8ce840a
This commit is contained in:
parent
eb4f86998c
commit
174dd39807
|
@ -0,0 +1,4 @@
|
||||||
|
How to use YAQL in Mistral
|
||||||
|
==========================
|
||||||
|
|
||||||
|
TBD
|
|
@ -1,10 +1,246 @@
|
||||||
Mistral Features
|
Mistral Main Features
|
||||||
================
|
=====================
|
||||||
|
|
||||||
|
|
||||||
|
Task results / Dataflow
|
||||||
|
-----------------------
|
||||||
|
|
||||||
|
Mistral supports transferring data from one task to another. In other words, if *taskA* produces
|
||||||
|
some result data, and then *taskB* (which is next after *taskA*) can use that result data. For referring
|
||||||
|
to the result data and also probably for selecting, filtering actions of the data in Mistral is used specific
|
||||||
|
query language called *YAQL*. For the details how to use *YAQL*, please see :doc:`How to use YAQL in Mistral </guides/using_yaql>`.
|
||||||
|
From Mistral standpoint, feature of referring any task produced result in subsequent task was called **DataFlow**.
|
||||||
|
|
||||||
|
|
||||||
|
Task affinity
|
||||||
|
-------------
|
||||||
|
|
||||||
|
Task affinity is a feature which could be useful for executing particular
|
||||||
|
tasks on specific Mistral executors. In fact, there are 2 cases:
|
||||||
|
|
||||||
|
1. Need to execute the task on single executor.
|
||||||
|
2. Need to execute the task on one of executor in executors group which has one name.
|
||||||
|
|
||||||
|
For enabling task affinity feature, edit edit section "executor" host property
|
||||||
|
in configuration file::
|
||||||
|
|
||||||
|
[executor]
|
||||||
|
host = my_favorite_executor
|
||||||
|
|
||||||
|
Then start (restart) executor. Use target task property to specify
|
||||||
|
this executor in Workflow DSL::
|
||||||
|
|
||||||
|
... Workflow YAML ...
|
||||||
|
task1:
|
||||||
|
...
|
||||||
|
target: my_favorite_executor
|
||||||
|
... Workflow YAML ...
|
||||||
|
|
||||||
|
Task policies
|
||||||
|
-------------
|
||||||
|
|
||||||
|
Any Mistral task regardless of its workflow type can optionally have configured policies.
|
||||||
|
Policies itself control the flow of the task - for example, policy can delay task execution before task starts
|
||||||
|
or after task completes.
|
||||||
|
|
||||||
|
YAML example
|
||||||
|
^^^^^^^^^^^^
|
||||||
|
::
|
||||||
|
|
||||||
|
my_task:
|
||||||
|
action: my_action
|
||||||
|
pause-before: true
|
||||||
|
wait-before: 2
|
||||||
|
wait-after: 4
|
||||||
|
timeout: 30
|
||||||
|
retry:
|
||||||
|
count: 10
|
||||||
|
delay: 20
|
||||||
|
break-on: <% $.my_var = true %>
|
||||||
|
|
||||||
|
There are different types of policies in Mistral.
|
||||||
|
|
||||||
|
1. **pause-before**
|
||||||
|
|
||||||
|
Defines whether Mistral Engine should put the workflow on pause or not before starting a task.
|
||||||
|
|
||||||
|
2. **wait-before**
|
||||||
|
|
||||||
|
Defines a delay in seconds that Mistral Engine should wait before starting a task.
|
||||||
|
|
||||||
|
3. **wait-after**
|
||||||
|
|
||||||
|
Defines a delay in seconds that Mistral Engine should wait after a task has completed before starting next tasks defined in *'on-success'*, *'on-error'* or *'on-complete'*.
|
||||||
|
|
||||||
|
4. **timeout**
|
||||||
|
|
||||||
|
Defines a period of time in seconds after which a task will be failed automatically by engine if hasn't completed.
|
||||||
|
|
||||||
|
5. **retry**
|
||||||
|
|
||||||
|
Defines a pattern how task should be repeated.
|
||||||
|
|
||||||
|
* *count* - Defines a maximum number of times that a task can be repeated.
|
||||||
|
* *delay* - Defines a delay in seconds between subsequent task iterations.
|
||||||
|
* *break-on* - Defines a YAQL expression that will break iteration loop if it evaluates to *'true'*. If it fires then the task is considered error.
|
||||||
|
* *continue-on* - Defines a YAQL expression that will continue iteration loop if it evaluates to *'true'*. If it fires then the task is considered successful.
|
||||||
|
|
||||||
|
Retry policy can also be configured on a single line as::
|
||||||
|
|
||||||
|
task1:
|
||||||
|
action: my_action
|
||||||
|
retry: count=10 delay=5 break-on=<% $.foo = 'bar' %>
|
||||||
|
|
||||||
|
All parameter values for any policy can be defined as YAQL expressions.
|
||||||
|
|
||||||
|
Join
|
||||||
|
----
|
||||||
|
|
||||||
|
Join flow control allows to synchronize multiple parallel workflow branches and aggregate their data.
|
||||||
|
|
||||||
|
**Full Join (join: all)**.
|
||||||
|
|
||||||
|
YAML example
|
||||||
|
^^^^^^^^^^^^
|
||||||
|
::
|
||||||
|
|
||||||
|
register_vm_in_load_balancer:
|
||||||
|
...
|
||||||
|
on-success:
|
||||||
|
- wait_for_all_registrations
|
||||||
|
|
||||||
|
register_vm_in_dns:
|
||||||
|
...
|
||||||
|
on-success:
|
||||||
|
- wait_for_all_registrations
|
||||||
|
|
||||||
|
try_to_do_something_without_registration:
|
||||||
|
...
|
||||||
|
on-error:
|
||||||
|
- wait_for_all_registrations
|
||||||
|
|
||||||
|
wait_for_all_registrations:
|
||||||
|
join: all
|
||||||
|
action: send_email
|
||||||
|
|
||||||
|
When a task has property *"join"* assigned with value *"all"* the task will run only
|
||||||
|
if all upstream tasks (ones that lead to this task) are completed and corresponding
|
||||||
|
conditions have triggered. Task A is considered an upstream task of Task B if Task A
|
||||||
|
has Task B mentioned in any of its *"on-success"*, *"on-error"* and *"on-complete"* clauses
|
||||||
|
regardless of YAQL guard expressions.
|
||||||
|
|
||||||
|
**Partial Join (join: 2)**
|
||||||
|
|
||||||
|
YAML example
|
||||||
|
^^^^^^^^^^^^
|
||||||
|
::
|
||||||
|
|
||||||
|
register_vm_in_load_balancer:
|
||||||
|
...
|
||||||
|
on-success:
|
||||||
|
- wait_for_all_registrations
|
||||||
|
|
||||||
|
register_vm_in_dns:
|
||||||
|
...
|
||||||
|
on-success:
|
||||||
|
- wait_for_all_registrations
|
||||||
|
|
||||||
|
register_vm_in_zabbix:
|
||||||
|
...
|
||||||
|
on-success:
|
||||||
|
- wait_for_all_registrations
|
||||||
|
|
||||||
|
wait_for_two_registrations:
|
||||||
|
join: 2
|
||||||
|
action: send_email
|
||||||
|
|
||||||
|
When a task has property *"join"* assigned with a numeric value then the task
|
||||||
|
will run once at least this number of upstream tasks are completed and
|
||||||
|
corresponding conditions have triggered. In the example about task
|
||||||
|
"wait_for_two_registrations" will run if two any of "register_vm_xxx" tasks complete.
|
||||||
|
|
||||||
|
**Discriminator (join: one)**
|
||||||
|
|
||||||
|
Discriminator is a special case of Partial Join when *"join"* property has value 1.
|
||||||
|
In this case instead of 1 it is possible to specify special string value *"one"*
|
||||||
|
which is introduced for symmetry with *"all"*. However, it's up to the user whether to use *"1"* or *"one"*.
|
||||||
|
|
||||||
|
|
||||||
|
Processing Collections (with-items)
|
||||||
|
-----------------------------------
|
||||||
|
|
||||||
|
YAML example
|
||||||
|
^^^^^^^^^^^^
|
||||||
|
::
|
||||||
|
|
||||||
|
---
|
||||||
|
version: '2.0'
|
||||||
|
|
||||||
|
create_vms:
|
||||||
|
description: Creating multiple virtual servers using "with-items".
|
||||||
|
input:
|
||||||
|
- vm_names
|
||||||
|
- image_ref
|
||||||
|
- flavor_ref
|
||||||
|
output:
|
||||||
|
vm_ids: <% $.vm_ids %>
|
||||||
|
|
||||||
|
tasks:
|
||||||
|
create_servers:
|
||||||
|
with-items: vm_name in <% $.vm_names %>
|
||||||
|
action: nova.servers_create name=<% $.vm_name %> image=<% $.image_ref %> flavor=<% $.flavor_ref %>
|
||||||
|
publish:
|
||||||
|
vm_ids: <% $.create_servers.id %>
|
||||||
|
on-success:
|
||||||
|
- wait_for_servers
|
||||||
|
|
||||||
|
wait_for_servers:
|
||||||
|
with-items: vm_id in <% $.vm_ids %>
|
||||||
|
action: nova.servers_find id=<% $.vm_id %> status='ACTIVE'
|
||||||
|
retry:
|
||||||
|
delay: 5
|
||||||
|
count: <% $.vm_names.len() * 10 %>
|
||||||
|
|
||||||
|
Workflow *"create_vms"* in this example creates as many virtual servers as we
|
||||||
|
provide in *"vm_names"* input parameter. E.g., if it is specified *vm_names=["vm1", "vm2"]*
|
||||||
|
then it'll create servers with these names based on same image and flavor.
|
||||||
|
It is possible because of using *"with-items"* keyword that makes an action
|
||||||
|
or a workflow associated with a task run multiple times. Value of *"with-items"*
|
||||||
|
task property contains an expression in the form: **<variable_name> in <% YAQL_expression %>**.
|
||||||
|
|
||||||
|
The most common form is::
|
||||||
|
|
||||||
|
with-items:
|
||||||
|
- var1 in <% YAQL_expression_1 %>
|
||||||
|
- var2 in <% YAQL_expression_2 %>
|
||||||
|
...
|
||||||
|
- varN in <% YAQL_expression_N %>
|
||||||
|
|
||||||
|
where collections expressed as YAQL_expression_1, YAQL_expression_2,
|
||||||
|
YAQL_expression_N must have equal sizes. When a task gets started Mistral
|
||||||
|
will iterate over all collections in parallel, i.e. number of iterations will
|
||||||
|
be equal to length of any collections.
|
||||||
|
|
||||||
|
Note that in case of using *"with-items"* task result accessible in workflow
|
||||||
|
context as <% $.task_name %> will be a list containing results of corresponding
|
||||||
|
action/workflow calls. If at least one action/workflow call has failed then
|
||||||
|
the whole task will get into *ERROR* state. It's also possible to apply retry
|
||||||
|
policy for tasks with *"with-items"* property. In this case retry policy will
|
||||||
|
be relaunching all action/workflow calls according to *"with-items"*
|
||||||
|
configuration. Other policies can also be used the same way as with regular non *"with-items"* tasks.
|
||||||
|
|
||||||
|
Execution expiration policy
|
||||||
|
---------------------------
|
||||||
|
|
||||||
|
For Mistral used in production, it is often hardly to control the number of workflow executions. The number of
|
||||||
|
workflow executions is significantly growing for the long time of Mistral running. The purpose of this feature to
|
||||||
|
delete old workflow executions which has been already completed. The criteria is the time when a workflow execution was
|
||||||
|
updated last time.
|
||||||
|
|
||||||
|
**By default this feature is disabled.**
|
||||||
|
|
||||||
|
In order to configure this feature, please open and edit configuration file and specify time in minutes::
|
||||||
|
|
||||||
|
[execution_expiration_policy]
|
||||||
|
older_than = 10080 # Workflow executions older than 1 week will be deleted automatically.
|
||||||
|
|
||||||
TBD:
|
|
||||||
- Task results / Dataflow
|
|
||||||
- Task affinity
|
|
||||||
- Task policies
|
|
||||||
- Loops (with-items)
|
|
||||||
- Service coordinator
|
|
||||||
- Execution expiration policy
|
|
Loading…
Reference in New Issue