Workflow Service for OpenStack.
Go to file
Andrey Kurilin 38388a0a20 Restore rally job
Due to various changes in OpenStack and Rally CI, mistral-rally-task job
became broken.

Unfortunately, it took much time for Rally team to finish some
stuff(sorry for that) and migrate jobs for native Zuul V3 format.

This patch restores mistral-rally-task job which inherits from the job
which Rally team runs constantly on all inner patches.

Change-Id: Ia7177af9b3e1aaacf555f5c472c4f0471b796930
2018-03-07 13:04:21 +02:00
api-ref/source Remove the invalid toctree 2018-01-31 04:16:42 +00:00
devstack Implement policy in code - event trigger (11) 2017-11-27 02:47:50 +00:00
doc/source Merge "Fix doc format" 2018-03-02 14:28:23 +00:00
etc Implement policy in code - event trigger (11) 2017-11-27 02:47:50 +00:00
functionaltests Remove intree mistral tempest plugin 2017-12-26 07:49:14 +00:00
mistral Fix errors of parameter note 2018-02-27 13:14:29 +08:00
playbooks Restore rally job 2018-03-07 13:04:21 +02:00
rally-jobs Restore rally job 2018-03-07 13:04:21 +02:00
releasenotes Merge "Cache action definitions" 2018-02-15 10:14:19 +00:00
tools Remove ceilometer actions from mistral 2018-01-11 14:28:16 -05:00
.coveragerc Exclude .tox folder from coverage report 2016-12-19 03:33:29 +00:00
.gitignore Implement policy in code - docs and reno (end) 2017-11-27 02:47:56 +00:00
.gitreview Update .gitreview file for project rename 2015-06-12 23:12:30 +00:00
.testr.conf Allow users to set the test run concurrency 2017-03-07 15:02:10 +00:00
.zuul.yaml Restore rally job 2018-03-07 13:04:21 +02:00
CONTRIBUTING.rst Merge "Chnage the mailing list URL" 2017-07-24 15:23:07 +00:00
docker_image_build.sh Fixes the Mistral Docker image 2016-05-27 13:26:05 +02:00
HACKING.rst Fix the pep8 commands failed 2017-07-27 22:15:12 +08:00
LICENSE Adding license and authors file 2013-12-30 13:11:49 +07:00
README.rst Remove achieved goals from the lis of annual goals 2018-02-05 18:09:57 +07:00
requirements.txt Updated from global requirements 2018-02-16 02:14:20 +00:00
run_functional_tests.sh Removed mistral/tests/functional 2016-01-29 11:04:03 +09:00
run_tests.sh Merge "Migrate the jobs to native Zuul v3 format" 2018-01-24 19:57:34 +00:00
setup.cfg Implement notification of execution events 2018-02-24 07:25:55 +00:00
setup.py Updated from global requirements 2017-03-03 10:17:44 +00:00
test-requirements.txt Updated from global requirements 2018-01-25 05:20:37 +00:00
tox.ini Remove intree mistral tempest plugin 2017-12-26 07:49:14 +00:00

Team and repository tags

image

Mistral

Workflow Service for OpenStack cloud. This project aims to provide a mechanism to define tasks and workflows without writing code, manage and execute them in the cloud environment.

Installation

The following are the steps to install Mistral on debian-based systems.

To install Mistral, you have to install the following prerequisites:

$ apt-get install python-dev python-setuptools libffi-dev \
  libxslt1-dev libxml2-dev libyaml-dev libssl-dev

Mistral can be used without authentication at all or it can work with OpenStack.

In case of OpenStack, it works only with Keystone v3, make sure Keystone v3 is installed.

Install Mistral

First of all, clone the repo and go to the repo directory:

$ git clone https://git.openstack.org/openstack/mistral.git
$ cd mistral

Devstack installation

Information about how to install Mistral with devstack can be found here.

Configuring Mistral

Mistral configuration is needed for getting it work correctly with and without an OpenStack environment.

  1. Install and configure a database which can be MySQL or PostgreSQL (SQLite can't be used in production.). Here are the steps to connect Mistral to a MySQL database.

    • Make sure you have installed mysql-server package on your Mistral machine.

    • Install MySQL driver for python:

      $ pip install mysql-python

      or, if you work in virtualenv, run:

      $ tox -evenv -- pip install mysql-python

      NOTE: If you're using Python 3 then you need to install mysqlclient instead of mysql-python.

    • Create the database and grant privileges:

      $ mysql -u root -p
      mysql> CREATE DATABASE mistral;
      mysql> USE mistral
      mysql> GRANT ALL PRIVILEGES ON mistral.* TO 'mistral'@'localhost' \
             IDENTIFIED BY 'MISTRAL_DBPASS';
      mysql> GRANT ALL PRIVILEGES ON mistral.* TO 'mistral'@'%' IDENTIFIED BY 'MISTRAL_DBPASS';
  2. Generate mistral.conf file:

    $ oslo-config-generator --config-file tools/config/config-generator.mistral.conf \
      --output-file etc/mistral.conf.sample
  3. Copy service configuration files:

    $ sudo mkdir /etc/mistral
    $ sudo chown `whoami` /etc/mistral
    $ cp etc/event_definitions.yml.sample /etc/mistral/event_definitions.yml
    $ cp etc/logging.conf.sample /etc/mistral/logging.conf
    $ cp etc/policy.json /etc/mistral/policy.json
    $ cp etc/wf_trace_logging.conf.sample /etc/mistral/wf_trace_logging.conf
    $ cp etc/mistral.conf.sample /etc/mistral/mistral.conf
  4. Edit file /etc/mistral/mistral.conf according to your setup. Pay attention to the following sections and options:

    [oslo_messaging_rabbit]
    rabbit_host = <RABBIT_HOST>
    rabbit_userid = <RABBIT_USERID>
    rabbit_password = <RABBIT_PASSWORD>
    
    [database]
    # Use the following line if *PostgreSQL* is used
    # connection = postgresql://<DB_USER>:<DB_PASSWORD>@localhost:5432/mistral
    connection = mysql://<DB_USER>:<DB_PASSWORD>@localhost:3306/mistral
  5. If you are not using OpenStack, add the following entry to the /etc/mistral/mistral.conf file and skip the following steps:

    [pecan]
    auth_enable = False
  6. Provide valid keystone auth properties:

    [keystone_authtoken]
    auth_uri = http://keystone-host:port/v3
    auth_url = http://keystone-host:port
    auth_type = password
    username = <user>
    password = <password>
    user_domain_name = <usually 'Default'>
    project_name = <project name>
    project_domain_name = <usually 'Default'>
  7. Register Mistral service and Mistral endpoints on Keystone:

    $ MISTRAL_URL="http://[host]:[port]/v2"
    $ openstack service create --name mistral workflowv2
    $ openstack endpoint create mistral public $MISTRAL_URL
    $ openstack endpoint create mistral internal $MISTRAL_URL
    $ openstack endpoint create mistral admin $MISTRAL_URL
  8. Update the mistral/actions/openstack/mapping.json file which contains all available OpenStack actions, according to the specific client versions of OpenStack projects in your deployment. Please find more detailed information in the tools/get_action_list.py script.

Before the First Run

After local installation you will find the commands mistral-server and mistral-db-manage available in your environment. The mistral-db-manage command can be used for migrating database schema versions. If Mistral is not installed in system then this script can be found at mistral/db/sqlalchemy/migration/cli.py, it can be executed using Python command line.

To update the database schema to the latest revision, type:

$ mistral-db-manage --config-file <path_to_config> upgrade head
To populate the database with standard actions and workflows, type::

$ mistral-db-manage --config-file <path_to_config> populate

For more detailed information about mistral-db-manage script please check file mistral/db/sqlalchemy/migration/alembic_migrations/README.md.

Running Mistral API server

To run Mistral API server:

$ tox -evenv -- python mistral/cmd/launch.py --server api --config-file <path_to_config>

Running Mistral Engines

To run Mistral Engine:

$ tox -evenv -- python mistral/cmd/launch.py --server engine --config-file <path_to_config>

Running Mistral Task Executors

To run Mistral Task Executor instance:

$ tox -evenv -- python mistral/cmd/launch.py --server executor --config-file <path_to_config>

Note that at least one Engine instance and one Executor instance should be running in order for workflow tasks to be processed by Mistral.

If you want to run some tasks on specific executor, the task affinity feature can be used to send these tasks directly to a specific executor. You can edit the following property in your mistral configuration file for this purpose:

[executor]
host = my_favorite_executor

After changing this option, you will need to start (restart) the executor. Use the target property of a task to specify the executor:

... Workflow YAML ...
task1:
  ...
  target: my_favorite_executor
... Workflow YAML ...

Running Multiple Mistral Servers Under the Same Process

To run more than one server (API, Engine, or Task Executor) on the same process:

$ tox -evenv -- python mistral/cmd/launch.py --server api,engine --config-file <path_to_config>

The value for the --server option can be a comma-delimited list. The valid options are all (which is the default if not specified) or any combination of api, engine, and executor.

It's important to note that the fake transport for the rpc_backend defined in the configuration file should only be used if all Mistral servers are launched on the same process. Otherwise, messages do not get delivered because the fake transport is using an in-process queue.

Project Goals 2018

  1. Complete Mistral documentation.

    Mistral documentation should be more usable. It requires focused work to make it well structured, eliminate gaps in API/Mistral Workflow Language specifications, add more examples and tutorials.

    Definition of done: All capabilities are covered, all documentation topics are written using the same style and structure principles. The obvious sub-goal of this goal is to establish these principles.

  2. Finish Mistral multi-node mode.

    Mistral needs to be proven to work reliably in multi-node mode. In order to achieve it we need to make a number of engine, executor and RPC changes and configure a CI gate to run stress tests on multi-node Mistral.

    Definition of done: CI gate supports MySQL, all critically important functionality (join, with-items, parallel workflows, sequential workflows) is covered by tests.

Project Resources