diff --git a/doc/source/index.rst b/doc/source/index.rst index 2a0ec6453..60616a23e 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -41,7 +41,7 @@ User guide .. toctree:: :maxdepth: 2 - user/dsl_v2 + user/wf_lang_v2 **CLI** diff --git a/doc/source/quickstart.rst b/doc/source/quickstart.rst index 032b1c31c..0c3246af3 100644 --- a/doc/source/quickstart.rst +++ b/doc/source/quickstart.rst @@ -64,7 +64,7 @@ This simple workflow iterates through a list of names in ``task1`` (using then stores the word "Done" as a result of the second task (`task2`). To learn more about the Mistral Workflows and what you can do, read the -:doc:`Mistral Workflow Language specification ` +:doc:`Mistral Workflow Language specification ` Upload the workflow ------------------- diff --git a/doc/source/terminology/workbooks.rst b/doc/source/terminology/workbooks.rst index bf004fa15..2a12b9f0e 100644 --- a/doc/source/terminology/workbooks.rst +++ b/doc/source/terminology/workbooks.rst @@ -80,4 +80,4 @@ the original workbook. * **actions** - Dictionary containing ad-hoc action definitions. *Optional*. For more details about Mistral Workflow Language itself, please see -:doc:`Mistral Workflow Language specification ` +:doc:`Mistral Workflow Language specification ` diff --git a/doc/source/terminology/workflows.rst b/doc/source/terminology/workflows.rst index c68843011..da08013ee 100644 --- a/doc/source/terminology/workflows.rst +++ b/doc/source/terminology/workflows.rst @@ -137,4 +137,4 @@ YAML example       requires: [create_vm, associate_ip] For more details about Mistral Workflow Language itself, please see -:doc:`Mistral Workflow Language specification ` +:doc:`Mistral Workflow Language specification ` diff --git a/doc/source/user/dsl_v2.rst b/doc/source/user/wf_lang_v2.rst similarity index 92% rename from doc/source/user/dsl_v2.rst rename to doc/source/user/wf_lang_v2.rst index 59156e042..3d30e0a38 100644 --- a/doc/source/user/dsl_v2.rst +++ b/doc/source/user/wf_lang_v2.rst @@ -30,12 +30,18 @@ Prerequisites ------------- Mistral Workflow Language supports -`YAQL `__ and +`YAQL `__ and `Jinja2 `__ expression languages to reference workflow context variables and thereby implements passing data between workflow tasks. It's also referred to as Data Flow mechanism. YAQL is a simple but powerful query language that allows to extract needed information from JSON -structured data. It is allowed to use YAQL in the following sections of +structured data. Although Jinja2 is primarily a templating technology, Mistral +also uses it for evaluating expression so user have a choice between YAQL and +Jinja2. It's also possible to combine both expression languages within one +workflow definition. The only limitation is that it's impossible to use both +types of expression within one line. As long as there are YAQL and Jinja2 +expressions on different lines of the workflow definition text, it is valid. +It is allowed to use YAQL/Jinja2 in the following sections of Mistral Workflow Language: - Workflow `'output' attribute <#common-workflow-attributes>`__ @@ -126,9 +132,9 @@ Common workflow attributes - **description** - Arbitrary text containing workflow description. *Optional*. - **input** - List defining required input parameter names and optionally their default values in a form "my_param: 123". *Optional*. -- **output** - Any data structure arbitrarily containing +- **output** - Any data structure arbitrarily containing YAQL/Jinja2 expressions that defines workflow output. May be nested. *Optional*. -- **output-on-error** - Any data structure arbitrarily containing YAQL +- **output-on-error** - Any data structure arbitrarily containing YAQL/Jinja2 expressions that defines output of a workflow to be returned if it goes into error state. May be nested. *Optional*. - **task-defaults** - Default settings for some of task attributes @@ -183,21 +189,21 @@ it's also easy to plug new actions into Mistral. Common task attributes '''''''''''''''''''''' -All Mistral tasks regardless of workflow type have the following common +All Mistral tasks, regardless of workflow type, have the following common attributes: - **description** - Arbitrary text containing task description. *Optional*. -- **action** - Name of the action associated with the task.Can be static +- **action** - Name of the action associated with the task. Can be a static value or an expression (for example, "{{ _.action_name }}"). *Mutually exclusive with* **workflow**. If neither action nor workflow are - provided then the action 'std.noop' will be used. -- **workflow** - Name of the workflow associated with the task. Can be static + provided then the action 'std.noop' will be used that does nothing. +- **workflow** - Name of the workflow associated with the task. Can be a static value or an expression (for example, "{{ _.subworkflow_name }}"). *Mutually exclusive with* **action**. -- **input** - Actual input parameter values of the task. *Optional*. - Value of each parameter is a JSON-compliant type such as number, - string etc, dictionary or list. It can also be an expression to +- **input** - Actual input parameter values of the task's action or workflow. + *Optional*. Value of each parameter is a JSON-compliant type such as number, + string etc, dictionary or list. It can also be a YAQL/Jinja2 expression to retrieve value from task context or any of the mentioned types containing inline expressions (for example, string "<% $.movie_name %> is a cool movie!") Can be an expression that evaluates to @@ -229,62 +235,65 @@ attributes: during action execution. If set to 'true' task may be run twice. *Optional*. By default set to 'false'. -Workflow +workflow '''''''' -Synchronously starts a sub-workflow with the given name. +If a task has the attribute 'workflow' it synchronously starts a sub-workflow +with the given name. -Example static workflow call: +Example of a static sub-workflow name: .. code-block:: mistral my_task:   workflow: name_of_my_workflow -Example dynamic workflow selection: +Example of a dynamic sub-workflow name: .. code-block:: mistral --- version: '2.0' - name: weather_data_processing + framework: + input: + - magic_workflow_name: show_weather - workflows: - framework: - input: - - magic_workflow_name: show_weather + tasks: + weather_data: + action: std.echo + input: + output: + location: wherever + temperature: "22C" + publish: + weather_data: <% task().result %> + on-success: + - do_magic - tasks: - weather_data: - action: std.echo - input: - output: - location: wherever - temperature: "22C" - publish: - weather_data: <% task(weather_data).result %> - on-success: - - do_magic + do_magic: + # Reference workflow by parameter. + workflow: <% $.magic_workflow_name %> + # Expand dictionary to input parameters. + input: <% $.weather_data %> - do_magic: - # reference workflow by parameter - workflow: <% $.magic_workflow_name %> - # expand dictionary to input parameters - input: <% $.weather_data %> + show_weather: + input: + - location + - temperature - show_weather: - input: - - location - - temperature + tasks: + write_data: + action: std.echo + input: + output: "<% $.location %>: <% $.temperature %>" - tasks: - write_data: - action: std.echo - input: - output: "<% $.location %>: <% $.temperature %>" +In this example, we defined two workflows in one YAML snippet and the workflow +'framework' may call the workflow 'show_weather' if 'framework' receives the +corresponding workflow name through the input parameter 'magic_workflow_name'. +In this case it is set by default so a user doesn't need to pass anything +explicitly. - -Note: Typical use for the dynamic workflow selection is when parts of a +Note: Typical use for the dynamic sub-workflow selection is when parts of a workflow can be customized. E.g. collect some weather data and then execute some custom workflow on it. @@ -333,7 +342,7 @@ has completed before starting next tasks defined in *on-success*, **timeout** Defines a period of time in seconds after which a task will be failed -automatically by engine if hasn't completed. +automatically by engine if it hasn't completed. **concurrency** @@ -368,7 +377,7 @@ Retry policy can also be configured on a single line as:   action: my_action   retry: count=10 delay=5 break-on=<% $.foo = 'bar' %> -All parameter values for any policy can be defined as expressions. +All parameter values for any policy can be defined as YAQL/Jinja2 expressions. Input syntax '''''''''''' @@ -512,6 +521,12 @@ Direct workflow task attributes - **on-complete** - List of tasks which will run after the task has completed regardless of whether it is successful or not. *Optional*. +Note: All of the above clauses cannot contain task names evaluated as +YAQL/Jinja expressions. They have to be static values. However, task +transitions can be conditional, based on expressions. See +`Transitions with expressions <#transitions-with-expressions>`__ for more +details. + It is important to understand the semantics of **on-success**, **on-error** and **on-complete** around handling action errors. @@ -525,6 +540,26 @@ stop the exception from bubbling up to an upper layer. So **on-complete** should only be understood as a language construction that allows to define some clean up actions. +Transitions with expressions +'''''''''''''''''''''''''''' + +Task transitions can be determined by success/error/completeness of the +previous tasks and also by additional guard expressions that can access any +data produced by upstream tasks and as workflow input. So in the example above +task 'create_vm' could also have a YAQL expression on transition to task +'send_success_email' as follows: + +.. code-block:: mistral + + create_vm: +  ... +  on-success: +    - send_success_email: <% $.vm_id != null %> + +And this would tell Mistral to run 'send_success_email' task only if 'vm_id' +variable published by task 'create_vm' is not empty. Expressions can also be +applied to 'on-error' and 'on-complete'. + Engine Commands ''''''''''''''' @@ -554,7 +589,7 @@ YAML example create_server: action: nova.servers_create name=<% $.vm_name %> publish: - vm_id: <% task(create_server).result.id %> + vm_id: <% task().result.id %> on-complete: - fail: <% not $.vm_id %> @@ -578,26 +613,6 @@ never be called. ``taskB`` would not be called if ``succeed`` was used in this example either, but if ``pause`` was used ``taskB`` would be called after the workflow is resumed. -Transitions with YAQL expressions -''''''''''''''''''''''''''''''''' - -Task transitions can be determined by success/error/completeness of the -previous tasks and also by additional guard expressions that can access any -data produced by upstream tasks. So in the example above task 'create_vm' could -also have a YAQL expression on transition to task 'send_success_email' as -follows: - -.. code-block:: mistral - - create_vm: -  ... -  on-success: -    - send_success_email: <% $.vm_id != null %> - -And this would tell Mistral to run 'send_success_email' task only if 'vm_id' -variable published by task 'create_vm' is not empty. Expressions can also be -applied to 'on-error' and 'on-complete'. - Fork '''' @@ -792,18 +807,19 @@ provide in "vm_names" input parameter. E.g., if we specify vm_names=["vm1", "vm2"] then it'll create servers with these names based on same image and flavor. It is possible because of using "with-items" keyword that makes an action or a workflow associated with a task run multiple times. -Value of "with-items" task property contains an expression in the form: in -<% YAQL_expression %>. +Value of "with-items" task property contains an expression in the form: 'my_var' in +<% YAQL_expression %>. Similar for Jinja2 expression: 'my_var' in +{{ Jinja2_expression }}. The most common form is: .. code-block:: mistral with-items: -   - var1 in <% YAQL_expression_1 %> -   - var2 in <% YAQL_expression_2 %> +   - var1 in <% YAQL_expression_1 %> # or: var1 in <% Jinja2_expression_1 %> +   - var2 in <% YAQL_expression_2 %> # or: var2 in <% Jinja2_expression_2 %>   ... -   - varN in <% YAQL_expression_N %> +   - varN in <% YAQL_expression_N %> # or: varN in <% Jinja2_expression_N %> where collections expressed as YAQL_expression_1, YAQL_expression_2, YAQL_expression_N must have equal sizes. When a task gets started Mistral will @@ -1120,7 +1136,7 @@ Attributes - **input** - List of declared action parameters which should be specified as corresponding task input. This attribute is optional and used only for documenting purposes. Mistral now does not enforce actual input parameters to - exactly correspond to this list. Based parameters will be calculated based on + exactly correspond to this list. Base parameters will be calculated based on provided actual parameters with using expressions so what's used in expressions implicitly define real input parameters. Dictionary of actual input parameters (expression context) is referenced as '$.' in YAQL and as @@ -1387,9 +1403,13 @@ returns the following fields of task executions: * **spec** - task execution spec dict (loaded from Mistral Workflow Language). * **state** - task execution state. * **state_info** - task execution state info. -* **result** - task execution result. +* **result** - task execution result. In case of a non 'with-items' task it's + simply a result of the task's action/sub-workflow execution. For a + 'with-items' task it will be a list of results of corresponding + action/sub-workflow execution. * **published** - task execution published variables. + Execution info ^^^^^^^^^^^^^^ @@ -1462,5 +1482,5 @@ Workflow definition: Environment ^^^^^^^^^^^ -Environment info is available by **env()**. It is passed when user submit +Environment info is available by **env()**. It is passed when user submits workflow execution. It contains variables specified by user.