openstack/mistral
Code Issues Proposed changes

Docs improvements: task timeout, global context, Docker and jinja

Browse Source 
A not obvious point for users is the task does not retry after a
timeout is triggered. Added clarification in the retry section.

The documentation contains enough example with Jinja usage. Improved
only the create_vm workflow definition.

Added global publishing to the doc. It is brash copy-paste from
https://specs.openstack.org/openstack/mistral-specs/specs/pike/approved/advanced_publishing.html without mention of atomic publish.

Move Docker guides to the installation section.

Change-Id: I149b2e1dff7f86bd356f4dd2f758659469e6a4a8
Closes-Bug: #1789384
Closes-Bug: #1690156
Closes-Bug: #1779244
Signed-off-by: Vitalii Solodilov <mcdkr@yandex.ru>
This commit is contained in:
Vitalii Solodilov 2018-09-11 10:33:59 +04:00 committed by Renat Akhmerov
parent 604d905c24
commit 09cd21d561
3 changed files with 317 additions and 213 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

183
doc/source/install/installation_guide.rst
View File

@ -276,33 +276,178 @@ Running Mistral By Systemd
Mistral And Docker
------------------
Please first refer
`installation steps for docker <https://docs.docker.com/installation/>`_.
To build the image from the mistral source, change directory to the root
directory of the Mistral git repository and run::
$ docker build -t <Name of image> .
Docker containers provide an easy way to quickly deploy independent or
networked Mistral instances in seconds. This guide describes the process
to launch an all-in-one Mistral container.
In case you want pre-built image, you can download it from `openstack tarballs
source <https://tarballs.openstack.org/mistral/images/mistral-docker.tar.gz>`_.
To load this image to docker registry, please run following command::
Docker Installation
-------------------
$ docker load -i '<path of mistral-docker.tar.gz>'
The following links contain instructions to install latest Docker software:
The Mistral Docker image is configured to store the database in the user's home
directory. For persistence of these data, you may want to keep this directory
outside of the container. This may be done by the following steps::
* `Docker Engine <https://docs.docker.com/engine/installation/>`_
* `Docker Compose <https://docs.docker.com/compose/install/>`_
$ sudo mkdir '<user-defined-directory>'
$ docker run -it -v \
'<user-defined-directory>':/home/mistral <Name of image>
More about docker: https://www.docker.com/
Build the Mistral Image Manually
--------------------------------
**NOTE:** This docker image uses **SQLite** database. So, it cannot be used for
production environment. If you want to use this for production environment,
then put customized mistral.conf to '<user-defined-directory>'.
Execute the following command from the repository top-level directory::
docker build -t mistral -f tools/docker/Dockerfile .
The Mistral Docker image has one build parameter:
+-------------------------+-------------+--------------------------------------+
|Name |Default value| Description |
+=========================+=============+======================================+
|`BUILD_TEST_DEPENDENCIES`|false |If the `BUILD_TEST_DEPENDENCIES` |
| | |equals `true`, the Mistral test |
| | |dependencies will be installed inside |
| | |the Docker image |
+-------------------------+-------------+----------------------+---------------+
Running Mistral using Docker Compose
------------------------------------
To launch Mistral in the single node configuration::
docker-compose -f tools/docker/docker-compose/infrastructure.yaml \
-f tools/docker/docker-compose/mistral-single-node.yaml \
-p mistral up -d
To launch Mistral in the multi node configuration::
docker-compose -f tools/docker/docker-compose/infrastructure.yaml \
-f tools/docker/docker-compose/mistral-multi-node.yaml \
-p mistral up -d
The infrastructure docker-compose file contains examples of RabbitMQ,
PostgreSQL and MySQL configurations. Feel free to modify the docker-compose
files as needed.
The docker-compose Mistral configurations also include the CloudFlow container.
It is available at `link <http://localhost:8000/>`_
The `--build` option can be used when it is necessary to rebuild the image,
for example::
docker-compose -f tools/docker/docker-compose/infrastructure.yaml \
-f tools/docker/docker-compose/mistral-single-node.yaml \
-p mistral up -d --build
Running the Mistral client from the Docker Compose container
------------------------------------------------------------
To run the mistral client against the server in the container using the client
present in the container::
docker run -it mistral_mistral mistral workflow-list
Configuring Mistral
-------------------
The Docker image contains the minimal set of Mistral configuration parameters
by default:
+--------------------+------------------+--------------------------------------+
|Name |Default value | Description |
+====================+==================+======================================+
|`MESSAGE_BROKER_URL`|rabbit://guest:gu\|The message broker URL |
| |est@rabbitmq:5672 | |
+--------------------+------------------+----------------------+---------------+
|`DATABASE_URL` |sqlite:///mistral\|The database URL |
| |.db | |
+--------------------+------------------+----------------------+---------------+
|`UPGRADE_DB` |false |If the `UPGRADE_DB` equals `true`, |
| | |a database upgrade will be launched |
| | |before Mistral main process |
+--------------------+------------------+----------------------+---------------+
|`MISTRAL_SERVER` |all |Specifies which mistral server to |
| | |start by the launch script. |
+--------------------+------------------+----------------------+---------------+
|`LOG_DEBUG` |false |If set to true, the logging level will|
| | |be set to DEBUG instead of the default|
| | |INFO level. |
+--------------------+------------------+----------------------+---------------+
|`RUN_TESTS` |false |If the `UPGRADE_DB` equals `true`, |
| | |the Mistral unit tests will be |
| | |launched inside container |
+--------------------+------------------+----------------------+---------------+
The `/etc/mistral/mistral.conf` configuration file can be mounted to the Mistral
Docker container by uncommenting and editing the `volumes` sections in the
Mistral docker-compose files.
Launch tests inside Container
-----------------------------
Build mistral::
docker build -t mistral -f tools/docker/Dockerfile \
--build-arg BUILD_TEST_DEPENDENCIES=true .
Run tests using SQLite::
docker run -it -e RUN_TESTS=true mistral
or PostgreSQL::
docker run -it \
-e DATABASE_URL=postgresql://postgres:postgres@localhost:5432/postgres \
-e RUN_TESTS=true mistral
Keycloak integration
--------------------
If you set AUTH_ENABLE to True value in the mistral.env file then Mistral will
enable Keycloak integration by default. Keycloak will be deployed with
mistral/mistral credentials. You should uncomment the volume line in the
`infrastructure.yaml` for the CloudFlow.
Next step you login in the administrative console using the
http://localhost:8080/auth/admin URL. Create a oauth client, you can
specify only a name, for example mistral.
Specify valid redirect URL: http://localhost:8000/* and turn on the
"Implicit Flow Enabled" in the your client page. Save your changes.
Add the following line to your /etc/hosts file::
127.0.0.1 keycloak
Export the following environments variable for mistral cli::
export MISTRAL_AUTH_TYPE=keycloak-oidc
export OS_AUTH_URL=http://keycloak:8080/auth
export OS_TENANT_NAME=master
export OS_USERNAME=mistral
export OS_PASSWORD=mistral
export OS_MISTRAL_URL=http://localhost:8989/v2
export OPENID_CLIENT_ID=mistral
export OPENID_CLIENT_SECRET=
export MISTRALCLIENT_INSECURE=True
Check your configuration::
mistral workflow-list
Or open a cloud flow page in a browser::
http://localhost:8000
Using Mistral Client with Docker
--------------------------------
The Mistral API will be accessible from the host machine on the default
port 8989. Install `python-mistralclient` on the host machine to
execute mistral commands.
Mistral Client Installation
---------------------------

171
doc/source/user/wf_lang_v2.rst
View File

@ -73,38 +73,43 @@ a various number of ways and that can do some job interesting to the end user.
Each workflow consists of tasks (at least one) describing what exact steps
should be made during workflow execution.
You should use '<% $.x %>' in YAQL or '{{ _.x }}' in Jinja expressions to get
access to the x variable in a data context of workflow execution.
YAML example
^^^^^^^^^^^^
.. code-block:: mistral
---
version: '2.0'
version: '2.0'
create_vm:
  description: Simple workflow example
  type: direct
description: Simple workflow example
input:
    - vm_name
    - image_ref
    - flavor_ref
  output:
    vm_id: <% $.vm_id %>
- vm_name
- image_ref
- flavor_ref
output:
vm_id: "{{ _.vm_id }}"
vm_status: <% $.vm_status %>
tasks:
    create_server:
      action: nova.servers_create name=<% $.vm_name %> image=<% $.image_ref %> flavor=<% $.flavor_ref %>
      publish:
        vm_id: <% task(create_server).result.id %>
      on-success:
        - wait_for_instance
create_server:
action: nova.servers_create name=<% $.vm_name %> image=<% $.image_ref %> flavor=<% $.flavor_ref %>
publish:
vm_id: <% task().result.id %>
on-success:
- wait_for_instance
    wait_for_instance:
      action: nova.servers_find id=<% $.vm_id %> status='ACTIVE'
      retry:
        delay: 5
        count: 15
wait_for_instance:
action: nova.servers_find id={{ _.vm_id }} status='ACTIVE'
retry:
delay: 5
count: 15
publish:
vm_status: "{{ task().result.status }}"
This example workflow simply sends a command to OpenStack Compute
service Nova to start creating a virtual machine and wait till it's
@ -219,7 +224,34 @@ attributes:
context. Any JSON-compatible data structure optionally containing
expression to select precisely what needs to be published.
Published variables will be accessible for downstream tasks via using
expressions. *Optional*.
expressions. **NOTE!** Mistral saves variables into a storage (context)
which is associated only with a branch. For example, the expression
“$.my_var” in the declaration of A1 will always evaluate to 1, for B1
it will always evaluate to 2. This does’t depend on the order in which
A and B will run. This is because we have two branches (A -> A1 and B
-> B1) for which the variable “my_var” has its own different version.
*Optional*.
.. code-block:: mistral
version: '2.0'
wf:
tasks:
A:
action: std.noop
publish:
my_var: 1
on-success: A1
A1:
action: my_action param1=<% $.my_var %>
B:
action: std.noop
publish:
my_var: 2
on-success: B1
B1:
action: my_action param1=<% $.my_var %>
- **publish-on-error** - Same as **publish** but evaluated in case of
task execution failures. *Optional*
- **with-items** - If configured, it allows to run action or workflow
@ -394,6 +426,10 @@ stop and the task will be in ERROR. *continue-on* should be used if the action
will usually return *SUCCESS*, but the action has other results that can be
used to signal whether to continue the loop or not.
**NOTE**: Retry task policy doesn't work after the timeout policy is
triggered. You should use the *on-error* in case of direct workflow or task
rerun to re-execute a task.
Input syntax
''''''''''''
@ -529,6 +565,8 @@ YAML example
Direct workflow task attributes
'''''''''''''''''''''''''''''''
Mistral supports the following task transitions:
- **on-success** - List of tasks which will run after the task has
completed successfully. *Optional*.
- **on-error** - List of tasks which will run after the task has
@ -536,6 +574,92 @@ Direct workflow task attributes
- **on-complete** - List of tasks which will run after the task has
completed regardless of whether it is successful or not. *Optional*.
You can define the task transitions in two ways:
The first is just a list of tasks. You can find the example of workflow
above. The second way is:
.. code-block:: mistral
*transition*:
publish:
global:
some_global_variable: some_value
branch:
some_branch_variable: some_value
next:
- *next_task*
The publish defined under *transitions* can optionally define scopes to be
able to publish into different scopes: ‘branch’ and ‘global’.
Specifying variables under ‘branch’ will make Mistral publish into a branch
workflow context just like ‘publish’ and ‘publish-on-error’. Specifying
variables under ‘global’ will make Mistral publish into a global workflow
context. You can use “$.” in YAQL and “_.” in Jinja to access to a global
variable but branch variables can shadow them if they are
published in the current branch. To prevent it, you may use the YAQL/Jinja
function “global()” to explicitly access variables in workflow global context.
If ‘publish’ is defined in ‘on-complete’ and also in ‘on-success’ and/or
‘on-error’ then the result of publishing will be a merge of what
‘on-complete’ publishes with what ‘on-success’ or ‘on-error’ publishes
depending on the task status. If ‘on-complete’ publishes variables that
are also published by ‘on-success’ or ‘on-error’ then latter take precedence.
In other words, ‘on-complete’ in this case is considered a default which can
be overridden by more specific ‘on-XXX’ clause.
The keyword ‘next’ defined under *transitions* optionally contains list of
tasks which will run after the current task finished.
Example of writing and reading global variables
'''''''''''''''''''''''''''''''''''''''''''''''
.. code-block:: mistral
---
version: '2.0'
wf:
tasks:
A:
action: std.noop
on-success:
publish:
branch:
my_var: "branch value"
global:
my_var: "global value"
next: A1
A1:
# $.my_var will always evaluate to "branch value" because A1 belongs
# to the same branch as A and runs after A. When using "$" to access
# context variables branch values have higher priority.
# In order to access global context reliably we need to use YAQL/Jinja
# function 'global'. So global(my_var) will always evaluate to
# 'global value'.
action: my_action1 param1=<% $.my_var %> param2=<% global(my_var) %>
B:
# $.my_var will evaluate to "global value" if task A completes
# before task B and "null", if not. It's because A and B are
# parallel and 'publish' in A doesn't apply to B, only
# 'publish-global' does. In this example global(my_var) has the same
# meaning as $.my_var because there's no ambiguity from what context
# we should take variable 'my_var'.
action: my_action2 param1=<% $.my_var %> param2=<% global(my_var) %>
**NOTE!** It’s important to note that this is an unprotected way of modifying
data because race conditions are possible when writing different values for
same variables in the global context from parallel branches. In other words,
if we have branches A and B and there are tasks in these branches that first
read global variable X, then increment it and write the new value Mistral
won’t provide any guarantee that the result value after finishing tasks A
and B will be X + 2. In some cases it can be X + 1 because the following may
happen: task A read X, Task B read X, Task B incremented X, Task B wrote
X + 1, Task A incremented X (the old one, not incremented by B), Task A
wrote X + 1.
Note: All of the above clauses cannot contain task names evaluated as
YAQL/Jinja expressions. They have to be static values. However, task
transitions can be conditional, based on expressions. See
@ -1599,3 +1723,10 @@ Environment
Environment info is available by **env()**. It is passed when user submits
workflow execution. It contains variables specified by user.
Global
^^^^^^
Global variables are available by **global(variable_name)**. If the
variable doesn't exist than None will be returned.

176
tools/docker/DOCKER_README.rst
View File

@ -1,174 +1,2 @@
Using Mistral with Docker
=========================
Docker containers provide an easy way to quickly deploy independent or
networked Mistral instances in seconds. This guide describes the process
to launch an all-in-one Mistral container.
Docker Installation
-------------------
The following links contain instructions to install latest Docker software:
* `Docker Engine <https://docs.docker.com/engine/installation/>`_
* `Docker Compose <https://docs.docker.com/compose/install/>`_
Build the Mistral Image Manually
--------------------------------
Execute the following command from the repository top-level directory::
docker build -t mistral -f tools/docker/Dockerfile .
The Mistral Docker image has one build parameter:
+-------------------------+-------------+--------------------------------------+
|Name |Default value| Description |
+=========================+=============+======================================+
|`BUILD_TEST_DEPENDENCIES`|false |If the `BUILD_TEST_DEPENDENCIES` |
| | |equals `true`, the Mistral test |
| | |dependencies will be installed inside |
| | |the Docker image |
+-------------------------+-------------+----------------------+---------------+
Running Mistral using Docker Compose
------------------------------------
To launch Mistral in the single node configuration::
docker-compose -f tools/docker/docker-compose/infrastructure.yaml \
-f tools/docker/docker-compose/mistral-single-node.yaml \
-p mistral up -d
To launch Mistral in the multi node configuration::
docker-compose -f tools/docker/docker-compose/infrastructure.yaml \
-f tools/docker/docker-compose/mistral-multi-node.yaml \
-p mistral up -d
The infrastructure docker-compose file contains examples of RabbitMQ,
PostgreSQL and MySQL configurations. Feel free to modify the docker-compose
files as needed.
The docker-compose Mistral configurations also include the CloudFlow container.
It is available at `link <http://localhost:8000/>`_
The `--build` option can be used when it is necessary to rebuild the image,
for example::
docker-compose -f tools/docker/docker-compose/infrastructure.yaml \
-f tools/docker/docker-compose/mistral-single-node.yaml \
-p mistral up -d --build
Running the Mistral client from the Docker Compose container
------------------------------------------------------------
To run the mistral client against the server in the container using the client
present in the container::
docker run -it mistral_mistral mistral workflow-list
Configuring Mistral
-------------------
The Docker image contains the minimal set of Mistral configuration parameters
by default:
+--------------------+------------------+--------------------------------------+
|Name |Default value | Description |
+====================+==================+======================================+
|`MESSAGE_BROKER_URL`|rabbit://guest:gu\|The message broker URL |
| |est@rabbitmq:5672 | |
+--------------------+------------------+----------------------+---------------+
|`DATABASE_URL` |sqlite:///mistral\|The database URL |
| |.db | |
+--------------------+------------------+----------------------+---------------+
|`UPGRADE_DB` |false |If the `UPGRADE_DB` equals `true`, |
| | |a database upgrade will be launched |
| | |before Mistral main process |
+--------------------+------------------+----------------------+---------------+
|`MISTRAL_SERVER` |all |Specifies which mistral server to |
| | |start by the launch script. |
+--------------------+------------------+----------------------+---------------+
|`LOG_DEBUG` |false |If set to true, the logging level will|
| | |be set to DEBUG instead of the default|
| | |INFO level. |
+--------------------+------------------+----------------------+---------------+
|`RUN_TESTS` |false |If the `UPGRADE_DB` equals `true`, |
| | |the Mistral unit tests will be |
| | |launched inside container |
+--------------------+------------------+----------------------+---------------+
The `/etc/mistral/mistral.conf` configuration file can be mounted to the Mistral
Docker container by uncommenting and editing the `volumes` sections in the Mistral
docker-compose files.
Launch tests inside Container
-----------------------------
Build mistral::
docker build -t mistral -f tools/docker/Dockerfile \
--build-arg BUILD_TEST_DEPENDENCIES=true .
Run tests using SQLite::
docker run -it -e RUN_TESTS=true mistral
or PostgreSQL::
docker run -it \
-e DATABASE_URL=postgresql://postgres:postgres@localhost:5432/postgres \
-e RUN_TESTS=true mistral
Keycloak integration
--------------------
If you set AUTH_ENABLE to True value in the mistral.env file then Mistral will
enable Keycloak integration by default. Keycloak will be deployed with
mistral/mistral credentials. You should uncomment the volume line in the
`infrastructure.yaml` for the CloudFlow.
Next step you login in the administrative console using the
http://localhost:8080/auth/admin URL. Create a oauth client, you can
specify only a name, for example mistral.
Specify valid redirect URL: http://localhost:8000/* and turn on the
"Implicit Flow Enabled" in the your client page. Save your changes.
Add the following line to your /etc/hosts file::
127.0.0.1 keycloak
Export the following environments variable for mistral cli::
export MISTRAL_AUTH_TYPE=keycloak-oidc
export OS_AUTH_URL=http://keycloak:8080/auth
export OS_TENANT_NAME=master
export OS_USERNAME=mistral
export OS_PASSWORD=mistral
export OS_MISTRAL_URL=http://localhost:8989/v2
export OPENID_CLIENT_ID=mistral
export OPENID_CLIENT_SECRET=
export MISTRALCLIENT_INSECURE=True
Check your configuration::
mistral workflow-list
Or open a cloud flow page in a browser::
http://localhost:8000
Using Mistral Client
--------------------
The Mistral API will be accessible from the host machine on the default
port 8989. Install `python-mistralclient` on the host machine to
execute mistral commands.
You can find the latest documentation of Mistral integration with Docker
`here <https://docs.openstack.org/mistral/latest/install/installation_guide.html#mistral-and-docker>`_.