diff --git a/doc/source/install/installation_guide.rst b/doc/source/install/installation_guide.rst index ffae5295b..dda967c03 100644 --- a/doc/source/install/installation_guide.rst +++ b/doc/source/install/installation_guide.rst @@ -276,33 +276,178 @@ Running Mistral By Systemd Mistral And Docker ------------------ -Please first refer -`installation steps for docker `_. -To build the image from the mistral source, change directory to the root -directory of the Mistral git repository and run:: - $ docker build -t . +Docker containers provide an easy way to quickly deploy independent or +networked Mistral instances in seconds. This guide describes the process +to launch an all-in-one Mistral container. -In case you want pre-built image, you can download it from `openstack tarballs -source `_. -To load this image to docker registry, please run following command:: +Docker Installation +------------------- - $ docker load -i '' +The following links contain instructions to install latest Docker software: -The Mistral Docker image is configured to store the database in the user's home -directory. For persistence of these data, you may want to keep this directory -outside of the container. This may be done by the following steps:: +* `Docker Engine `_ +* `Docker Compose `_ - $ sudo mkdir '' - $ docker run -it -v \ - '':/home/mistral -More about docker: https://www.docker.com/ +Build the Mistral Image Manually +-------------------------------- -**NOTE:** This docker image uses **SQLite** database. So, it cannot be used for -production environment. If you want to use this for production environment, -then put customized mistral.conf to ''. +Execute the following command from the repository top-level directory:: + + docker build -t mistral -f tools/docker/Dockerfile . + +The Mistral Docker image has one build parameter: + ++-------------------------+-------------+--------------------------------------+ +|Name |Default value| Description | ++=========================+=============+======================================+ +|`BUILD_TEST_DEPENDENCIES`|false |If the `BUILD_TEST_DEPENDENCIES` | +| | |equals `true`, the Mistral test | +| | |dependencies will be installed inside | +| | |the Docker image | ++-------------------------+-------------+----------------------+---------------+ + + +Running Mistral using Docker Compose +------------------------------------ + +To launch Mistral in the single node configuration:: + + docker-compose -f tools/docker/docker-compose/infrastructure.yaml \ + -f tools/docker/docker-compose/mistral-single-node.yaml \ + -p mistral up -d + +To launch Mistral in the multi node configuration:: + + docker-compose -f tools/docker/docker-compose/infrastructure.yaml \ + -f tools/docker/docker-compose/mistral-multi-node.yaml \ + -p mistral up -d + +The infrastructure docker-compose file contains examples of RabbitMQ, +PostgreSQL and MySQL configurations. Feel free to modify the docker-compose +files as needed. + +The docker-compose Mistral configurations also include the CloudFlow container. +It is available at `link `_ + +The `--build` option can be used when it is necessary to rebuild the image, +for example:: + + docker-compose -f tools/docker/docker-compose/infrastructure.yaml \ + -f tools/docker/docker-compose/mistral-single-node.yaml \ + -p mistral up -d --build + +Running the Mistral client from the Docker Compose container +------------------------------------------------------------ + +To run the mistral client against the server in the container using the client +present in the container:: + + docker run -it mistral_mistral mistral workflow-list + +Configuring Mistral +------------------- + +The Docker image contains the minimal set of Mistral configuration parameters +by default: + ++--------------------+------------------+--------------------------------------+ +|Name |Default value | Description | ++====================+==================+======================================+ +|`MESSAGE_BROKER_URL`|rabbit://guest:gu\|The message broker URL | +| |est@rabbitmq:5672 | | ++--------------------+------------------+----------------------+---------------+ +|`DATABASE_URL` |sqlite:///mistral\|The database URL | +| |.db | | ++--------------------+------------------+----------------------+---------------+ +|`UPGRADE_DB` |false |If the `UPGRADE_DB` equals `true`, | +| | |a database upgrade will be launched | +| | |before Mistral main process | ++--------------------+------------------+----------------------+---------------+ +|`MISTRAL_SERVER` |all |Specifies which mistral server to | +| | |start by the launch script. | ++--------------------+------------------+----------------------+---------------+ +|`LOG_DEBUG` |false |If set to true, the logging level will| +| | |be set to DEBUG instead of the default| +| | |INFO level. | ++--------------------+------------------+----------------------+---------------+ +|`RUN_TESTS` |false |If the `UPGRADE_DB` equals `true`, | +| | |the Mistral unit tests will be | +| | |launched inside container | ++--------------------+------------------+----------------------+---------------+ + +The `/etc/mistral/mistral.conf` configuration file can be mounted to the Mistral +Docker container by uncommenting and editing the `volumes` sections in the +Mistral docker-compose files. + + +Launch tests inside Container +----------------------------- + +Build mistral:: + + docker build -t mistral -f tools/docker/Dockerfile \ + --build-arg BUILD_TEST_DEPENDENCIES=true . + +Run tests using SQLite:: + + docker run -it -e RUN_TESTS=true mistral + +or PostgreSQL:: + + docker run -it \ + -e DATABASE_URL=postgresql://postgres:postgres@localhost:5432/postgres \ + -e RUN_TESTS=true mistral + + +Keycloak integration +-------------------- + +If you set AUTH_ENABLE to True value in the mistral.env file then Mistral will +enable Keycloak integration by default. Keycloak will be deployed with +mistral/mistral credentials. You should uncomment the volume line in the +`infrastructure.yaml` for the CloudFlow. + +Next step you login in the administrative console using the +http://localhost:8080/auth/admin URL. Create a oauth client, you can +specify only a name, for example mistral. + +Specify valid redirect URL: http://localhost:8000/* and turn on the +"Implicit Flow Enabled" in the your client page. Save your changes. + +Add the following line to your /etc/hosts file:: + + 127.0.0.1 keycloak + +Export the following environments variable for mistral cli:: + + export MISTRAL_AUTH_TYPE=keycloak-oidc + export OS_AUTH_URL=http://keycloak:8080/auth + export OS_TENANT_NAME=master + export OS_USERNAME=mistral + export OS_PASSWORD=mistral + export OS_MISTRAL_URL=http://localhost:8989/v2 + export OPENID_CLIENT_ID=mistral + export OPENID_CLIENT_SECRET= + export MISTRALCLIENT_INSECURE=True + +Check your configuration:: + + mistral workflow-list + +Or open a cloud flow page in a browser:: + + http://localhost:8000 + + +Using Mistral Client with Docker +-------------------------------- + +The Mistral API will be accessible from the host machine on the default +port 8989. Install `python-mistralclient` on the host machine to +execute mistral commands. Mistral Client Installation --------------------------- diff --git a/doc/source/user/wf_lang_v2.rst b/doc/source/user/wf_lang_v2.rst index 73b6f2157..6d2838ab6 100644 --- a/doc/source/user/wf_lang_v2.rst +++ b/doc/source/user/wf_lang_v2.rst @@ -73,38 +73,43 @@ a various number of ways and that can do some job interesting to the end user. Each workflow consists of tasks (at least one) describing what exact steps should be made during workflow execution. +You should use '<% $.x %>' in YAQL or '{{ _.x }}' in Jinja expressions to get +access to the x variable in a data context of workflow execution. + YAML example ^^^^^^^^^^^^ .. code-block:: mistral --- - version: '2.0' + version: '2.0' create_vm: -   description: Simple workflow example -   type: direct + description: Simple workflow example input: -     - vm_name -     - image_ref -     - flavor_ref -   output: -     vm_id: <% $.vm_id %> + - vm_name + - image_ref + - flavor_ref + output: + vm_id: "{{ _.vm_id }}" + vm_status: <% $.vm_status %> tasks: -     create_server: -       action: nova.servers_create name=<% $.vm_name %> image=<% $.image_ref %> flavor=<% $.flavor_ref %> -       publish: -         vm_id: <% task(create_server).result.id %> -       on-success: -         - wait_for_instance + create_server: + action: nova.servers_create name=<% $.vm_name %> image=<% $.image_ref %> flavor=<% $.flavor_ref %> + publish: + vm_id: <% task().result.id %> + on-success: + - wait_for_instance -     wait_for_instance: -       action: nova.servers_find id=<% $.vm_id %> status='ACTIVE' -       retry: -         delay: 5 -         count: 15 + wait_for_instance: + action: nova.servers_find id={{ _.vm_id }} status='ACTIVE' + retry: + delay: 5 + count: 15 + publish: + vm_status: "{{ task().result.status }}" This example workflow simply sends a command to OpenStack Compute service Nova to start creating a virtual machine and wait till it's @@ -219,7 +224,34 @@ attributes: context. Any JSON-compatible data structure optionally containing expression to select precisely what needs to be published. Published variables will be accessible for downstream tasks via using - expressions. *Optional*. + expressions. **NOTE!** Mistral saves variables into a storage (context) + which is associated only with a branch. For example, the expression + “$.my_var” in the declaration of A1 will always evaluate to 1, for B1 + it will always evaluate to 2. This does’t depend on the order in which + A and B will run. This is because we have two branches (A -> A1 and B + -> B1) for which the variable “my_var” has its own different version. + *Optional*. + +.. code-block:: mistral + + version: '2.0' + wf: + tasks: + A: + action: std.noop + publish: + my_var: 1 + on-success: A1 + A1: + action: my_action param1=<% $.my_var %> + B: + action: std.noop + publish: + my_var: 2 + on-success: B1 + B1: + action: my_action param1=<% $.my_var %> + - **publish-on-error** - Same as **publish** but evaluated in case of task execution failures. *Optional* - **with-items** - If configured, it allows to run action or workflow @@ -394,6 +426,10 @@ stop and the task will be in ERROR. *continue-on* should be used if the action will usually return *SUCCESS*, but the action has other results that can be used to signal whether to continue the loop or not. +**NOTE**: Retry task policy doesn't work after the timeout policy is +triggered. You should use the *on-error* in case of direct workflow or task +rerun to re-execute a task. + Input syntax '''''''''''' @@ -529,6 +565,8 @@ YAML example Direct workflow task attributes ''''''''''''''''''''''''''''''' +Mistral supports the following task transitions: + - **on-success** - List of tasks which will run after the task has completed successfully. *Optional*. - **on-error** - List of tasks which will run after the task has @@ -536,6 +574,92 @@ Direct workflow task attributes - **on-complete** - List of tasks which will run after the task has completed regardless of whether it is successful or not. *Optional*. +You can define the task transitions in two ways: + +The first is just a list of tasks. You can find the example of workflow +above. The second way is: + +.. code-block:: mistral + + *transition*: + publish: + global: + some_global_variable: some_value + branch: + some_branch_variable: some_value + next: + - *next_task* + +The publish defined under *transitions* can optionally define scopes to be +able to publish into different scopes: ‘branch’ and ‘global’. +Specifying variables under ‘branch’ will make Mistral publish into a branch +workflow context just like ‘publish’ and ‘publish-on-error’. Specifying +variables under ‘global’ will make Mistral publish into a global workflow +context. You can use “$.” in YAQL and “_.” in Jinja to access to a global +variable but branch variables can shadow them if they are +published in the current branch. To prevent it, you may use the YAQL/Jinja +function “global()” to explicitly access variables in workflow global context. + +If ‘publish’ is defined in ‘on-complete’ and also in ‘on-success’ and/or +‘on-error’ then the result of publishing will be a merge of what +‘on-complete’ publishes with what ‘on-success’ or ‘on-error’ publishes +depending on the task status. If ‘on-complete’ publishes variables that +are also published by ‘on-success’ or ‘on-error’ then latter take precedence. +In other words, ‘on-complete’ in this case is considered a default which can +be overridden by more specific ‘on-XXX’ clause. + +The keyword ‘next’ defined under *transitions* optionally contains list of +tasks which will run after the current task finished. + +Example of writing and reading global variables +''''''''''''''''''''''''''''''''''''''''''''''' + +.. code-block:: mistral + + --- + version: '2.0' + + wf: + tasks: + A: + action: std.noop + on-success: + publish: + branch: + my_var: "branch value" + global: + my_var: "global value" + next: A1 + + A1: + # $.my_var will always evaluate to "branch value" because A1 belongs + # to the same branch as A and runs after A. When using "$" to access + # context variables branch values have higher priority. + # In order to access global context reliably we need to use YAQL/Jinja + # function 'global'. So global(my_var) will always evaluate to + # 'global value'. + action: my_action1 param1=<% $.my_var %> param2=<% global(my_var) %> + + B: + # $.my_var will evaluate to "global value" if task A completes + # before task B and "null", if not. It's because A and B are + # parallel and 'publish' in A doesn't apply to B, only + # 'publish-global' does. In this example global(my_var) has the same + # meaning as $.my_var because there's no ambiguity from what context + # we should take variable 'my_var'. + action: my_action2 param1=<% $.my_var %> param2=<% global(my_var) %> + +**NOTE!** It’s important to note that this is an unprotected way of modifying +data because race conditions are possible when writing different values for +same variables in the global context from parallel branches. In other words, +if we have branches A and B and there are tasks in these branches that first +read global variable X, then increment it and write the new value Mistral +won’t provide any guarantee that the result value after finishing tasks A +and B will be X + 2. In some cases it can be X + 1 because the following may +happen: task A read X, Task B read X, Task B incremented X, Task B wrote +X + 1, Task A incremented X (the old one, not incremented by B), Task A +wrote X + 1. + Note: All of the above clauses cannot contain task names evaluated as YAQL/Jinja expressions. They have to be static values. However, task transitions can be conditional, based on expressions. See @@ -1599,3 +1723,10 @@ Environment Environment info is available by **env()**. It is passed when user submits workflow execution. It contains variables specified by user. + + +Global +^^^^^^ + +Global variables are available by **global(variable_name)**. If the +variable doesn't exist than None will be returned. diff --git a/tools/docker/DOCKER_README.rst b/tools/docker/DOCKER_README.rst index 97359f053..0ffd732c0 100644 --- a/tools/docker/DOCKER_README.rst +++ b/tools/docker/DOCKER_README.rst @@ -1,174 +1,2 @@ -Using Mistral with Docker -========================= - -Docker containers provide an easy way to quickly deploy independent or -networked Mistral instances in seconds. This guide describes the process -to launch an all-in-one Mistral container. - - -Docker Installation -------------------- - -The following links contain instructions to install latest Docker software: - -* `Docker Engine `_ -* `Docker Compose `_ - - -Build the Mistral Image Manually --------------------------------- - -Execute the following command from the repository top-level directory:: - - docker build -t mistral -f tools/docker/Dockerfile . - -The Mistral Docker image has one build parameter: - -+-------------------------+-------------+--------------------------------------+ -|Name |Default value| Description | -+=========================+=============+======================================+ -|`BUILD_TEST_DEPENDENCIES`|false |If the `BUILD_TEST_DEPENDENCIES` | -| | |equals `true`, the Mistral test | -| | |dependencies will be installed inside | -| | |the Docker image | -+-------------------------+-------------+----------------------+---------------+ - - -Running Mistral using Docker Compose ------------------------------------- - -To launch Mistral in the single node configuration:: - - docker-compose -f tools/docker/docker-compose/infrastructure.yaml \ - -f tools/docker/docker-compose/mistral-single-node.yaml \ - -p mistral up -d - -To launch Mistral in the multi node configuration:: - - docker-compose -f tools/docker/docker-compose/infrastructure.yaml \ - -f tools/docker/docker-compose/mistral-multi-node.yaml \ - -p mistral up -d - -The infrastructure docker-compose file contains examples of RabbitMQ, -PostgreSQL and MySQL configurations. Feel free to modify the docker-compose -files as needed. - -The docker-compose Mistral configurations also include the CloudFlow container. -It is available at `link `_ - -The `--build` option can be used when it is necessary to rebuild the image, -for example:: - - docker-compose -f tools/docker/docker-compose/infrastructure.yaml \ - -f tools/docker/docker-compose/mistral-single-node.yaml \ - -p mistral up -d --build - -Running the Mistral client from the Docker Compose container ------------------------------------------------------------- - -To run the mistral client against the server in the container using the client -present in the container:: - - docker run -it mistral_mistral mistral workflow-list - -Configuring Mistral -------------------- - -The Docker image contains the minimal set of Mistral configuration parameters -by default: - -+--------------------+------------------+--------------------------------------+ -|Name |Default value | Description | -+====================+==================+======================================+ -|`MESSAGE_BROKER_URL`|rabbit://guest:gu\|The message broker URL | -| |est@rabbitmq:5672 | | -+--------------------+------------------+----------------------+---------------+ -|`DATABASE_URL` |sqlite:///mistral\|The database URL | -| |.db | | -+--------------------+------------------+----------------------+---------------+ -|`UPGRADE_DB` |false |If the `UPGRADE_DB` equals `true`, | -| | |a database upgrade will be launched | -| | |before Mistral main process | -+--------------------+------------------+----------------------+---------------+ -|`MISTRAL_SERVER` |all |Specifies which mistral server to | -| | |start by the launch script. | -+--------------------+------------------+----------------------+---------------+ -|`LOG_DEBUG` |false |If set to true, the logging level will| -| | |be set to DEBUG instead of the default| -| | |INFO level. | -+--------------------+------------------+----------------------+---------------+ -|`RUN_TESTS` |false |If the `UPGRADE_DB` equals `true`, | -| | |the Mistral unit tests will be | -| | |launched inside container | -+--------------------+------------------+----------------------+---------------+ - -The `/etc/mistral/mistral.conf` configuration file can be mounted to the Mistral -Docker container by uncommenting and editing the `volumes` sections in the Mistral -docker-compose files. - - -Launch tests inside Container ------------------------------ - -Build mistral:: - - docker build -t mistral -f tools/docker/Dockerfile \ - --build-arg BUILD_TEST_DEPENDENCIES=true . - -Run tests using SQLite:: - - docker run -it -e RUN_TESTS=true mistral - -or PostgreSQL:: - - docker run -it \ - -e DATABASE_URL=postgresql://postgres:postgres@localhost:5432/postgres \ - -e RUN_TESTS=true mistral - - -Keycloak integration --------------------- - -If you set AUTH_ENABLE to True value in the mistral.env file then Mistral will -enable Keycloak integration by default. Keycloak will be deployed with -mistral/mistral credentials. You should uncomment the volume line in the -`infrastructure.yaml` for the CloudFlow. - -Next step you login in the administrative console using the -http://localhost:8080/auth/admin URL. Create a oauth client, you can -specify only a name, for example mistral. - -Specify valid redirect URL: http://localhost:8000/* and turn on the -"Implicit Flow Enabled" in the your client page. Save your changes. - -Add the following line to your /etc/hosts file:: - - 127.0.0.1 keycloak - -Export the following environments variable for mistral cli:: - - export MISTRAL_AUTH_TYPE=keycloak-oidc - export OS_AUTH_URL=http://keycloak:8080/auth - export OS_TENANT_NAME=master - export OS_USERNAME=mistral - export OS_PASSWORD=mistral - export OS_MISTRAL_URL=http://localhost:8989/v2 - export OPENID_CLIENT_ID=mistral - export OPENID_CLIENT_SECRET= - export MISTRALCLIENT_INSECURE=True - -Check your configuration:: - - mistral workflow-list - -Or open a cloud flow page in a browser:: - - http://localhost:8000 - - -Using Mistral Client --------------------- - -The Mistral API will be accessible from the host machine on the default -port 8989. Install `python-mistralclient` on the host machine to -execute mistral commands. \ No newline at end of file +You can find the latest documentation of Mistral integration with Docker +`here `_. \ No newline at end of file