openstack/mistral
Code Issues Proposed changes

Add Jinja evaluator

Browse Source 
Allows to use Jinja instead of or along with YAQL for expression
evaluation.

 * Improved error reporting on API endpoints. Previously, Mistral API
   tend to mute important logs related to errors during YAML parsing
   or expression evaluation. The messages were shown in the http
   response, but would not appear in logs.

 * Renamed yaql_utils to evaluation_utils and added few more tests to
   ensure evaluation functions can be safely reused between Jinja and
   YAQL evaluators.

 * Updated action_v2 example to reflect similarities between YAQL and
   Jinja syntax.

Change-Id: Ie3cf8b4a6c068948d6dc051b12a02474689cf8a8
Implements: blueprint mistral-jinga-templates
This commit is contained in:
Kirill Izotov 2016-09-22 19:30:55 +07:00 committed by Dougal Matthews
parent 26f7d62bbf
commit 362c2295e8
28 changed files with 1283 additions and 379 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

43
doc/source/dsl/dsl_v2.rst
View File

@ -30,8 +30,8 @@ will be described in details below:
Prerequisites
-------------
Mistral DSL takes advantage of
`YAQL <https://pypi.python.org/pypi/yaql/1.0.0>`__ expression language to
Mistral DSL supports `YAQL <https://pypi.python.org/pypi/yaql/1.0.0>`__ and
`Jinja2 <http://jinja.pocoo.org/docs/dev/>`__ expression languages to
reference workflow context variables and thereby implements passing data
between workflow tasks. It's also referred to as Data Flow mechanism.
YAQL is a simple but powerful query language that allows to extract
@ -51,11 +51,12 @@ in the following sections of DSL:
Mistral DSL is fully based on YAML and knowledge of YAML is a plus for
better understanding of the material in this specification. It also
takes advantage of YAQL query language to define expressions in workflow
takes advantage of supported query languages to define expressions in workflow
and action definitions.
- Yet Another Markup Language (YAML): http://yaml.org
- Yet Another Query Language (YAQL): https://pypi.python.org/pypi/yaql/1.0.0
- Jinja 2: http://jinja.pocoo.org/docs/dev/
Workflows
---------
@ -124,7 +125,7 @@ Common Workflow Attributes
- **description** - Arbitrary text containing workflow description. *Optional*.
- **input** - List defining required input parameter names and
optionally their default values in a form "my_param: 123". *Optional*.
- **output** - Any data structure arbitrarily containing YAQL
- **output** - Any data structure arbitrarily containing
expressions that defines workflow output. May be nested. *Optional*.
- **task-defaults** - Default settings for some of task attributes
defined at workflow level. *Optional*. Corresponding attribute
@ -188,15 +189,15 @@ attributes:
*Mutually exclusive with* **action**.
- **input** - Actual input parameter values of the task. *Optional*.
Value of each parameter is a JSON-compliant type such as number,
string etc, dictionary or list. It can also be a YAQL expression to
string etc, dictionary or list. It can also be an expression to
retrieve value from task context or any of the mentioned types
containing inline YAQL expressions (for example, string "<%
containing inline expressions (for example, string "<%
$.movie_name %> is a cool movie!")
- **publish** - Dictionary of variables to publish to the workflow
context. Any JSON-compatible data structure optionally containing
YAQL expression to select precisely what needs to be published.
expression to select precisely what needs to be published.
Published variables will be accessible for downstream tasks via using
YAQL expressions. *Optional*.
expressions. *Optional*.
- **with-items** - If configured, it allows to run action or workflow
associated with a task multiple times on a provided list of items.
See `Processing collections using
@ -278,10 +279,10 @@ Defines a pattern how task should be repeated in case of an error.
repeated.
- **delay** - Defines a delay in seconds between subsequent task
iterations.
- **break-on** - Defines a YAQL expression that will break iteration
- **break-on** - Defines an expression that will break iteration
loop if it evaluates to 'true'. If it fires then the task is
considered error.
- **continue-on** - Defines a YAQL expression that will continue iteration
- **continue-on** - Defines an expression that will continue iteration
loop if it evaluates to 'true'. If it fires then the task is
considered successful. If it evaluates to 'false' then policy will break the iteration.
@ -293,7 +294,7 @@ Retry policy can also be configured on a single line as:
  action: my_action
  retry: count=10 delay=5 break-on=<% $.foo = 'bar' %>
All parameter values for any policy can be defined as YAQL expressions.
All parameter values for any policy can be defined as expressions.
Simplified Input Syntax
'''''''''''''''''''''''
@ -407,7 +408,7 @@ Transitions with YAQL expressions
'''''''''''''''''''''''''''''''''
Task transitions can be determined by success/error/completeness of the
previous tasks and also by additional YAQL guard expressions that can
previous tasks and also by additional guard expressions that can
access any data produced by upstream tasks. So in the example above task
'create_vm' could also have a YAQL expression on transition to task
'send_success_email' as follows:
@ -420,8 +421,8 @@ access any data produced by upstream tasks. So in the example above task
   - send_success_email: <% $.vm_id != null %>
And this would tell Mistral to run 'send_success_email' task only if
'vm_id' variable published by task 'create_vm' is not empty. YAQL
expressions can also be applied to 'on-error' and 'on-complete'.
'vm_id' variable published by task 'create_vm' is not empty.
Expressions can also be applied to 'on-error' and 'on-complete'.
Fork
''''
@ -475,7 +476,7 @@ run only if all upstream tasks (ones that lead to this task) are
completed and corresponding conditions have triggered. Task A is
considered an upstream task of Task B if Task A has Task B mentioned in
any of its "on-success", "on-error" and "on-complete" clauses regardless
of YAQL guard expressions.
of guard expressions.
Partial Join (join: 2)
@ -945,14 +946,14 @@ Attributes
used only for documenting purposes. Mistral now does not enforce
actual input parameters to exactly correspond to this list. Based
parameters will be calculated based on provided actual parameters
with using YAQL expressions so what's used in expressions implicitly
with using expressions so what's used in expressions implicitly
define real input parameters. Dictionary of actual input parameters
is referenced in YAQL as '$.'. Redundant parameters will be simply
ignored.
(expression context) is referenced as '$.' in YAQL and as '_.' in Jinja.
Redundant parameters will be simply ignored.
- **output** - Any data structure defining how to calculate output of
this action based on output of base action. It can optionally have
YAQL expressions to access properties of base action output
referenced in YAQL as '$.'.
expressions to access properties of base action output through expression
context.
Workbooks
---------
@ -1045,7 +1046,7 @@ Attributes
Predefined Values/Functions in execution data context
-----------------------------------------------------
Using YAQL it is possible to use some predefined values in Mistral DSL.
Using expressions it is possible to use some predefined values in Mistral DSL.
- **OpenStack context**
- **Task result**

19
mistral/exceptions.py
View File

@ -1,5 +1,6 @@
# Copyright 2013 - Mirantis, Inc.
# Copyright 2015 - StackStorm, Inc.
# Copyright 2016 - Brocade Communications Systems, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@ -112,8 +113,15 @@ class DSLParsingException(MistralException):
http_code = 400
class YaqlGrammarException(DSLParsingException):
class ExpressionGrammarException(DSLParsingException):
http_code = 400
class JinjaGrammarException(ExpressionGrammarException):
message = "Invalid grammar of Jinja expression"
class YaqlGrammarException(ExpressionGrammarException):
message = "Invalid grammar of YAQL expression"
@ -124,8 +132,15 @@ class InvalidModelException(DSLParsingException):
# Various common exceptions and errors.
class YaqlEvaluationException(MistralException):
class EvaluationException(MistralException):
http_code = 400
class JinjaEvaluationException(EvaluationException):
message = "Can not evaluate Jinja expression"
class YaqlEvaluationException(EvaluationException):
message = "Can not evaluate YAQL expression"

103
mistral/expressions/__init__.py Normal file
View File

@ -0,0 +1,103 @@
# Copyright 2013 - Mirantis, Inc.
# Copyright 2015 - StackStorm, Inc.
# Copyright 2016 - Brocade Communications Systems, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import copy
from oslo_log import log as logging
import six
from stevedore import extension
from mistral import exceptions as exc
LOG = logging.getLogger(__name__)
_mgr = extension.ExtensionManager(
namespace='mistral.expression.evaluators',
invoke_on_load=False
)
_evaluators = []
patterns = {}
for name in sorted(_mgr.names()):
evaluator = _mgr[name].plugin
_evaluators.append((name, evaluator))
patterns[name] = evaluator.find_expression_pattern.pattern
def validate(expression):
LOG.debug("Validating expression [expression='%s']", expression)
if not isinstance(expression, six.string_types):
return
expression_found = None
for name, evaluator in _evaluators:
if evaluator.is_expression(expression):
if expression_found:
raise exc.ExpressionGrammarException(
"The line already contains an expression of type '%s'. "
"Mixing expression types in a single line is not allowed."
% expression_found)
try:
evaluator.validate(expression)
except Exception:
raise
else:
expression_found = name
def evaluate(expression, context):
for name, evaluator in _evaluators:
# Check if the passed value is expression so we don't need to do this
# every time on a caller side.
if (isinstance(expression, six.string_types) and
evaluator.is_expression(expression)):
return evaluator.evaluate(expression, context)
return expression
def _evaluate_item(item, context):
if isinstance(item, six.string_types):
try:
return evaluate(item, context)
except AttributeError as e:
LOG.debug("Expression %s is not evaluated, [context=%s]: %s"
% (item, context, e))
return item
else:
return evaluate_recursively(item, context)
def evaluate_recursively(data, context):
data = copy.deepcopy(data)
if not context:
return data
if isinstance(data, dict):
for key in data:
data[key] = _evaluate_item(data[key], context)
elif isinstance(data, list):
for index, item in enumerate(data):
data[index] = _evaluate_item(item, context)
elif isinstance(data, six.string_types):
return _evaluate_item(data, context)
return data

55
mistral/expressions/base_expression.py Normal file
View File

@ -0,0 +1,55 @@
# Copyright 2013 - Mirantis, Inc.
# Copyright 2015 - StackStorm, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import abc
class Evaluator(object):
"""Expression evaluator interface.
Having this interface gives the flexibility to change the actual expression
language used in Mistral DSL for conditions, output calculation etc.
"""
@classmethod
@abc.abstractmethod
def validate(cls, expression):
"""Parse and validates the expression.
:param expression: Expression string
:return: True if expression is valid
"""
pass
@classmethod
@abc.abstractmethod
def evaluate(cls, expression, context):
"""Evaluates the expression against the given data context.
:param expression: Expression string
:param context: Data context
:return: Expression result
"""
pass
@classmethod
@abc.abstractmethod
def is_expression(cls, expression):
"""Check expression string and decide whether it is expression or not.
:param expression: Expression string
:return: True if string is expression
"""
pass

142
mistral/expressions/jinja_expression.py Normal file
View File

@ -0,0 +1,142 @@
# Copyright 2016 - Brocade Communications Systems, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import re
import jinja2
from jinja2 import parser as jinja_parse
from oslo_log import log as logging
import six
from mistral import exceptions as exc
from mistral.expressions.base_expression import Evaluator
from mistral.utils import expression_utils
LOG = logging.getLogger(__name__)
JINJA_REGEXP = '({{(.*)?}})'
JINJA_BLOCK_REGEXP = '({%(.*)?%})'
_environment = jinja2.Environment(
undefined=jinja2.StrictUndefined,
trim_blocks=True,
lstrip_blocks=True
)
_filters = expression_utils.get_custom_functions()
for name in _filters:
_environment.filters[name] = _filters[name]
class JinjaEvaluator(Evaluator):
_env = _environment.overlay()
@classmethod
def validate(cls, expression):
LOG.debug(
"Validating Jinja expression [expression='%s']", expression)
if not isinstance(expression, six.string_types):
raise exc.JinjaEvaluationException("Unsupported type '%s'." %
type(expression))
try:
parser = jinja_parse.Parser(cls._env, expression, state='variable')
parser.parse_expression()
except jinja2.exceptions.TemplateError as e:
raise exc.JinjaGrammarException("Syntax error '%s'." %
str(e))
@classmethod
def evaluate(cls, expression, data_context):
LOG.debug(
"Evaluating Jinja expression [expression='%s', context=%s]"
% (expression, data_context)
)
opts = {
'undefined_to_none': False
}
ctx = expression_utils.get_jinja_context(data_context)
try:
result = cls._env.compile_expression(expression, **opts)(**ctx)
# For StrictUndefined values, UndefinedError only gets raised when
# the value is accessed, not when it gets created. The simplest way
# to access it is to try and cast it to string.
str(result)
except jinja2.exceptions.UndefinedError as e:
raise exc.JinjaEvaluationException("Undefined error '%s'." %
str(e))
LOG.debug("Jinja expression result: %s" % result)
return result
@classmethod
def is_expression(cls, s):
# The class should only be called from within InlineJinjaEvaluator. The
# return value prevents the class from being accidentally added as
# Extension
return False
class InlineJinjaEvaluator(Evaluator):
# The regular expression for Jinja variables and blocks
find_expression_pattern = re.compile(JINJA_REGEXP)
find_block_pattern = re.compile(JINJA_BLOCK_REGEXP)
_env = _environment.overlay()
@classmethod
def validate(cls, expression):
LOG.debug(
"Validating Jinja expression [expression='%s']", expression)
if not isinstance(expression, six.string_types):
raise exc.JinjaEvaluationException("Unsupported type '%s'." %
type(expression))
try:
cls._env.parse(expression)
except jinja2.exceptions.TemplateError as e:
raise exc.JinjaGrammarException("Syntax error '%s'." %
str(e))
@classmethod
def evaluate(cls, expression, data_context):
LOG.debug(
"Evaluating Jinja expression [expression='%s', context=%s]"
% (expression, data_context)
)
patterns = cls.find_expression_pattern.findall(expression)
if patterns[0][0] == expression:
result = JinjaEvaluator.evaluate(patterns[0][1], data_context)
else:
ctx = expression_utils.get_jinja_context(data_context)
result = cls._env.from_string(expression).render(**ctx)
LOG.debug("Jinja expression result: %s" % result)
return result
@classmethod
def is_expression(cls, s):
return (cls.find_expression_pattern.search(s) or
cls.find_block_pattern.search(s))

107
mistral/expressions.py → mistral/expressions/yaql_expression.py
View File

@ -1,5 +1,6 @@
# Copyright 2013 - Mirantis, Inc.
# Copyright 2015 - StackStorm, Inc.
# Copyright 2016 - Brocade Communications Systems, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@ -13,8 +14,6 @@
# See the License for the specific language governing permissions and
# limitations under the License.
import abc
import copy
import inspect
import re
@ -24,50 +23,13 @@ from yaql.language import exceptions as yaql_exc
from yaql.language import factory
from mistral import exceptions as exc
from mistral.utils import yaql_utils
from mistral.expressions.base_expression import Evaluator
from mistral.utils import expression_utils
LOG = logging.getLogger(__name__)
YAQL_ENGINE = factory.YaqlFactory().create()
class Evaluator(object):
"""Expression evaluator interface.
Having this interface gives the flexibility to change the actual expression
language used in Mistral DSL for conditions, output calculation etc.
"""
@classmethod
@abc.abstractmethod
def validate(cls, expression):
"""Parse and validates the expression.
:param expression: Expression string
:return: True if expression is valid
"""
pass
@classmethod
@abc.abstractmethod
def evaluate(cls, expression, context):
"""Evaluates the expression against the given data context.
:param expression: Expression string
:param context: Data context
:return: Expression result
"""
pass
@classmethod
@abc.abstractmethod
def is_expression(cls, expression):
"""Check expression string and decide whether it is expression or not.
:param expression: Expression string
:return: True if string is expression
"""
pass
INLINE_YAQL_REGEXP = '<%.*?%>'
class YAQLEvaluator(Evaluator):
@ -87,7 +49,7 @@ class YAQLEvaluator(Evaluator):
try:
result = YAQL_ENGINE(expression).evaluate(
context=yaql_utils.get_yaql_context(data_context)
context=expression_utils.get_yaql_context(data_context)
)
except (yaql_exc.YaqlException, KeyError, ValueError, TypeError) as e:
raise exc.YaqlEvaluationException(
@ -101,12 +63,9 @@ class YAQLEvaluator(Evaluator):
@classmethod
def is_expression(cls, s):
# TODO(rakhmerov): It should be generalized since it may not be YAQL.
# Treat any string as a YAQL expression.
return isinstance(s, six.string_types)
INLINE_YAQL_REGEXP = '<%.*?%>'
# The class should not be used outside of InlineYAQLEvaluator since by
# convention, YAQL expression should always be wrapped in '<% %>'.
return False
class InlineYAQLEvaluator(YAQLEvaluator):
@ -155,56 +114,8 @@ class InlineYAQLEvaluator(YAQLEvaluator):
@classmethod
def is_expression(cls, s):
return s
return cls.find_expression_pattern.search(s)
@classmethod
def find_inline_expressions(cls, s):
return cls.find_expression_pattern.findall(s)
# TODO(rakhmerov): Make it configurable.
_EVALUATOR = InlineYAQLEvaluator
def validate(expression):
return _EVALUATOR.validate(expression)
def evaluate(expression, context):
# Check if the passed value is expression so we don't need to do this
# every time on a caller side.
if (not isinstance(expression, six.string_types) or
not _EVALUATOR.is_expression(expression)):
return expression
return _EVALUATOR.evaluate(expression, context)
def _evaluate_item(item, context):
if isinstance(item, six.string_types):
try:
return evaluate(item, context)
except AttributeError as e:
LOG.debug("Expression %s is not evaluated, [context=%s]: %s"
% (item, context, e))
return item
else:
return evaluate_recursively(item, context)
def evaluate_recursively(data, context):
data = copy.deepcopy(data)
if not context:
return data
if isinstance(data, dict):
for key in data:
data[key] = _evaluate_item(data[key], context)
elif isinstance(data, list):
for index, item in enumerate(data):
data[index] = _evaluate_item(item, context)
elif isinstance(data, six.string_types):
return _evaluate_item(data, context)
return data

21
mistral/tests/resources/action_jinja.yaml Normal file
View File

@ -0,0 +1,21 @@
---
version: "2.0"
greeting:
description: "This action says 'Hello'"
tags: [hello]
base: std.echo
base-input:
output: 'Hello, {{ _.name }}'
input:
- name
output:
string: '{{ _ }}'
farewell:
base: std.echo
base-input:
output: 'Bye!'
output:
info: '{{ _ }}'

4
mistral/tests/resources/action_v2.yaml
View File

@ -10,12 +10,12 @@ greeting:
input:
- name
output:
string: <% $.output %>
string: <% $ %>
farewell:
base: std.echo
base-input:
output: 'Bye!'
output:
info: <% $.output %>
info: <% $ %>

34
mistral/tests/resources/wf_jinja.yaml Normal file
View File

@ -0,0 +1,34 @@
---
version: '2.0'
wf:
type: direct
tasks:
hello:
action: std.echo output="Hello"
wait-before: 1
publish:
result: '{{ task("hello").result }}'
wf1:
type: reverse
input:
- farewell
tasks:
addressee:
action: std.echo output="John"
publish:
name: '{{ task("addressee").result }}'
goodbye:
action: std.echo output="{{ _.farewell }}, {{ _.name }}"
requires: [addressee]
wf2:
type: direct
tasks:
hello:
action: std.echo output="Hello"

397
mistral/tests/unit/expressions/test_jinja_expression.py Normal file
View File

@ -0,0 +1,397 @@
# Copyright 2016 - Brocade Communications Systems, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import mock
from mistral import exceptions as exc
from mistral.expressions import jinja_expression as expr
from mistral.tests.unit import base
from mistral import utils
DATA = {
"server": {
"id": "03ea824a-aa24-4105-9131-66c48ae54acf",
"name": "cloud-fedora",
"status": "ACTIVE"
},
"status": "OK"
}
SERVERS = {
"servers": [
{'name': 'centos'},
{'name': 'ubuntu'},
{'name': 'fedora'}
]
}
class JinjaEvaluatorTest(base.BaseTest):
def setUp(self):
super(JinjaEvaluatorTest, self).setUp()
self._evaluator = expr.JinjaEvaluator()
def test_expression_result(self):
res = self._evaluator.evaluate('_.server', DATA)
self.assertEqual({
'id': '03ea824a-aa24-4105-9131-66c48ae54acf',
'name': 'cloud-fedora',
'status': 'ACTIVE'
}, res)
res = self._evaluator.evaluate('_.server.id', DATA)
self.assertEqual('03ea824a-aa24-4105-9131-66c48ae54acf', res)
res = self._evaluator.evaluate("_.server.status == 'ACTIVE'", DATA)
self.assertTrue(res)
def test_wrong_expression(self):
res = self._evaluator.evaluate("_.status == 'Invalid value'", DATA)
self.assertFalse(res)
# One thing to note about Jinja is that by default it would not raise
# an exception on KeyError inside the expression, it will consider
# value to be None. Same with NameError, it won't return an original
# expression (which by itself seems confusing). Jinja allows us to
# change behavior in both cases by switching to StrictUndefined, but
# either one or the other will surely suffer.
self.assertRaises(
exc.JinjaEvaluationException,
self._evaluator.evaluate,
'_.wrong_key',
DATA
)
self.assertRaises(
exc.JinjaEvaluationException,
self._evaluator.evaluate,
'invalid_expression_string',
DATA
)
def test_select_result(self):
res = self._evaluator.evaluate(
'_.servers|selectattr("name", "equalto", "ubuntu")',
SERVERS
)
item = list(res)[0]
self.assertEqual({'name': 'ubuntu'}, item)
def test_function_string(self):
self.assertEqual('3', self._evaluator.evaluate('_|string', '3'))
self.assertEqual('3', self._evaluator.evaluate('_|string', 3))
def test_function_len(self):
self.assertEqual(3,
self._evaluator.evaluate('_|length', 'hey'))
data = [{'some': 'thing'}]
self.assertEqual(
1,
self._evaluator.evaluate(
'_|selectattr("some", "equalto", "thing")|list|length',
data
)
)
def test_validate(self):
self._evaluator.validate('abc')
self._evaluator.validate('1')
self._evaluator.validate('1 + 2')
self._evaluator.validate('_.a1')
self._evaluator.validate('_.a1 * _.a2')
def test_validate_failed(self):
self.assertRaises(exc.JinjaGrammarException,
self._evaluator.validate,
'*')
self.assertRaises(exc.JinjaEvaluationException,
self._evaluator.validate,
[1, 2, 3])
self.assertRaises(exc.JinjaEvaluationException,
self._evaluator.validate,
{'a': 1})
def test_function_json_pp(self):
self.assertEqual('"3"', self._evaluator.evaluate('json_pp(_)', '3'))
self.assertEqual('3', self._evaluator.evaluate('json_pp(_)', 3))
self.assertEqual(
'[\n 1,\n 2\n]',
self._evaluator.evaluate('json_pp(_)', [1, 2])
)
self.assertEqual(
'{\n "a": "b"\n}',
self._evaluator.evaluate('json_pp(_)', {'a': 'b'})
)
self.assertEqual(
'"Mistral\nis\nawesome"',
self._evaluator.evaluate(
'json_pp(_)', '\n'.join(['Mistral', 'is', 'awesome'])
)
)
def test_filter_json_pp(self):
self.assertEqual('"3"', self._evaluator.evaluate('_|json_pp', '3'))
self.assertEqual('3', self._evaluator.evaluate('_|json_pp', 3))
self.assertEqual(
'[\n 1,\n 2\n]',
self._evaluator.evaluate('_|json_pp', [1, 2])
)
self.assertEqual(
'{\n "a": "b"\n}',
self._evaluator.evaluate('_|json_pp', {'a': 'b'})
)
self.assertEqual(
'"Mistral\nis\nawesome"',
self._evaluator.evaluate(
'_|json_pp', '\n'.join(['Mistral', 'is', 'awesome'])
)
)
def test_function_uuid(self):
uuid = self._evaluator.evaluate('uuid()', {})
self.assertTrue(utils.is_valid_uuid(uuid))
def test_filter_uuid(self):
uuid = self._evaluator.evaluate('_|uuid', '3')
self.assertTrue(utils.is_valid_uuid(uuid))
def test_function_env(self):
ctx = {'__env': 'some'}
self.assertEqual(ctx['__env'], self._evaluator.evaluate('env()', ctx))
def test_filter_env(self):
ctx = {'__env': 'some'}
self.assertEqual(ctx['__env'], self._evaluator.evaluate('_|env', ctx))
@mock.patch('mistral.db.v2.api.get_task_executions')
@mock.patch('mistral.workflow.data_flow.get_task_execution_result')
def test_filter_task_without_taskexecution(self, task_execution_result,
task_executions):
task = mock.MagicMock(return_value={})
task_executions.return_value = [task]
ctx = {
'__task_execution': None,
'__execution': {
'id': 'some'
}
}
result = self._evaluator.evaluate('_|task("some")', ctx)
self.assertEqual({
'id': task.id,
'name': task.name,
'published': task.published,
'result': task_execution_result(),
'spec': task.spec,
'state': task.state,
'state_info': task.state_info
}, result)
@mock.patch('mistral.db.v2.api.get_task_execution')
@mock.patch('mistral.workflow.data_flow.get_task_execution_result')
def test_filter_task_with_taskexecution(self, task_execution_result,
task_execution):
ctx = {
'__task_execution': {
'id': 'some',
'name': 'some'
}
}
result = self._evaluator.evaluate('_|task("some")', ctx)
self.assertEqual({
'id': task_execution().id,
'name': task_execution().name,
'published': task_execution().published,
'result': task_execution_result(),
'spec': task_execution().spec,
'state': task_execution().state,
'state_info': task_execution().state_info
}, result)
@mock.patch('mistral.db.v2.api.get_task_execution')
@mock.patch('mistral.workflow.data_flow.get_task_execution_result')
def test_function_task(self, task_execution_result, task_execution):
ctx = {
'__task_execution': {
'id': 'some',
'name': 'some'
}
}
result = self._evaluator.evaluate('task("some")', ctx)
self.assertEqual({
'id': task_execution().id,
'name': task_execution().name,
'published': task_execution().published,
'result': task_execution_result(),
'spec': task_execution().spec,
'state': task_execution().state,
'state_info': task_execution().state_info
}, result)
@mock.patch('mistral.db.v2.api.get_workflow_execution')
def test_filter_execution(self, workflow_execution):
wf_ex = mock.MagicMock(return_value={})
workflow_execution.return_value = wf_ex
ctx = {
'__execution': {
'id': 'some'
}
}
result = self._evaluator.evaluate('_|execution', ctx)
self.assertEqual({
'id': wf_ex.id,
'name': wf_ex.name,
'spec': wf_ex.spec,
'input': wf_ex.input,
'params': wf_ex.params
}, result)
@mock.patch('mistral.db.v2.api.get_workflow_execution')
def test_function_execution(self, workflow_execution):
wf_ex = mock.MagicMock(return_value={})
workflow_execution.return_value = wf_ex
ctx = {
'__execution': {
'id': 'some'
}
}
result = self._evaluator.evaluate('execution()', ctx)
self.assertEqual({
'id': wf_ex.id,
'name': wf_ex.name,
'spec': wf_ex.spec,
'input': wf_ex.input,
'params': wf_ex.params
}, result)
class InlineJinjaEvaluatorTest(base.BaseTest):
def setUp(self):
super(InlineJinjaEvaluatorTest, self).setUp()
self._evaluator = expr.InlineJinjaEvaluator()
def test_multiple_placeholders(self):
expr_str = """
Statistics for tenant "{{ _.project_id }}"
Number of virtual machines: {{ _.vm_count }}
Number of active virtual machines: {{ _.active_vm_count }}
Number of networks: {{ _.net_count }}
-- Sincerely, Mistral Team.
"""
result = self._evaluator.evaluate(
expr_str,
{
'project_id': '1-2-3-4',
'vm_count': 28,
'active_vm_count': 0,
'net_count': 1
}
)
expected_result = """
Statistics for tenant "1-2-3-4"
Number of virtual machines: 28
Number of active virtual machines: 0
Number of networks: 1
-- Sincerely, Mistral Team.
"""
self.assertEqual(expected_result, result)
def test_block_placeholders(self):
expr_str = """
Statistics for tenant "{{ _.project_id }}"
Number of virtual machines: {{ _.vm_count }}
{% if _.active_vm_count %}
Number of active virtual machines: {{ _.active_vm_count }}
{% endif %}
Number of networks: {{ _.net_count }}
-- Sincerely, Mistral Team.
"""
result = self._evaluator.evaluate(
expr_str,
{
'project_id': '1-2-3-4',
'vm_count': 28,
'active_vm_count': 0,
'net_count': 1
}
)
expected_result = """
Statistics for tenant "1-2-3-4"
Number of virtual machines: 28
Number of networks: 1
-- Sincerely, Mistral Team.
"""
self.assertEqual(expected_result, result)
def test_single_value_casting(self):
self.assertEqual(3, self._evaluator.evaluate('{{ _ }}', 3))
self.assertEqual('33', self._evaluator.evaluate('{{ _ }}{{ _ }}', 3))
def test_function_string(self):
self.assertEqual('3', self._evaluator.evaluate('{{ _|string }}', '3'))
self.assertEqual('3', self._evaluator.evaluate('{{ _|string }}', 3))
def test_validate(self):
self._evaluator.validate('There is no expression.')
self._evaluator.validate('{{ abc }}')
self._evaluator.validate('{{ 1 }}')
self._evaluator.validate('{{ 1 + 2 }}')
self._evaluator.validate('{{ _.a1 }}')
self._evaluator.validate('{{ _.a1 * _.a2 }}')
self._evaluator.validate('{{ _.a1 }} is {{ _.a2 }}')
self._evaluator.validate('The value is {{ _.a1 }}.')
def test_validate_failed(self):
self.assertRaises(exc.JinjaGrammarException,
self._evaluator.validate,
'The value is {{ * }}.')
self.assertRaises(exc.JinjaEvaluationException,
self._evaluator.validate,
[1, 2, 3])
self.assertRaises(exc.JinjaEvaluationException,
self._evaluator.validate,
{'a': 1})

213
mistral/tests/unit/expressions/test_yaql_expression.py Normal file
View File

@ -0,0 +1,213 @@
# Copyright 2013 - Mirantis, Inc.
# Copyright 2015 - StackStorm, Inc.
# Copyright 2016 - Brocade Communications Systems, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from mistral import exceptions as exc
from mistral.expressions import yaql_expression as expr
from mistral.tests.unit import base
from mistral import utils
DATA = {
"server": {
"id": "03ea824a-aa24-4105-9131-66c48ae54acf",
"name": "cloud-fedora",
"status": "ACTIVE"
},
"status": "OK"
}
SERVERS = {
"servers": [
{'name': 'centos'},
{'name': 'ubuntu'},
{'name': 'fedora'}
]
}
class YaqlEvaluatorTest(base.BaseTest):
def setUp(self):
super(YaqlEvaluatorTest, self).setUp()
self._evaluator = expr.YAQLEvaluator()
def test_expression_result(self):
res = self._evaluator.evaluate('$.server', DATA)
self.assertEqual({
'id': "03ea824a-aa24-4105-9131-66c48ae54acf",
'name': 'cloud-fedora',
'status': 'ACTIVE'
}, res)
res = self._evaluator.evaluate('$.server.id', DATA)
self.assertEqual('03ea824a-aa24-4105-9131-66c48ae54acf', res)
res = self._evaluator.evaluate("$.server.status = 'ACTIVE'", DATA)
self.assertTrue(res)
def test_wrong_expression(self):
res = self._evaluator.evaluate("$.status = 'Invalid value'", DATA)
self.assertFalse(res)
self.assertRaises(
exc.YaqlEvaluationException,
self._evaluator.evaluate,
'$.wrong_key',
DATA
)
expression_str = 'invalid_expression_string'
res = self._evaluator.evaluate(expression_str, DATA)
self.assertEqual(expression_str, res)
def test_select_result(self):
res = self._evaluator.evaluate(
'$.servers.where($.name = ubuntu)',
SERVERS
)
item = list(res)[0]
self.assertEqual({'name': 'ubuntu'}, item)
def test_function_string(self):
self.assertEqual('3', self._evaluator.evaluate('str($)', '3'))
self.assertEqual('3', self._evaluator.evaluate('str($)', 3))
def test_function_len(self):
self.assertEqual(3, self._evaluator.evaluate('len($)', 'hey'))
data = [{'some': 'thing'}]
self.assertEqual(
1,
self._evaluator.evaluate('$.where($.some = thing).len()', data)
)
def test_validate(self):
self._evaluator.validate('abc')
self._evaluator.validate('1')
self._evaluator.validate('1 + 2')
self._evaluator.validate('$.a1')
self._evaluator.validate('$.a1 * $.a2')
def test_validate_failed(self):
self.assertRaises(exc.YaqlGrammarException,
self._evaluator.validate,
'*')
self.assertRaises(exc.YaqlGrammarException,
self._evaluator.validate,
[1, 2, 3])
self.assertRaises(exc.YaqlGrammarException,
self._evaluator.validate,
{'a': 1})
def test_function_json_pp(self):
self.assertEqual('"3"', self._evaluator.evaluate('json_pp($)', '3'))
self.assertEqual('3', self._evaluator.evaluate('json_pp($)', 3))
self.assertEqual(
'[\n 1,\n 2\n]',
self._evaluator.evaluate('json_pp($)', [1, 2])
)
self.assertEqual(
'{\n "a": "b"\n}',
self._evaluator.evaluate('json_pp($)', {'a': 'b'})
)
self.assertEqual(
'"Mistral\nis\nawesome"',
self._evaluator.evaluate(
'json_pp($)', '\n'.join(['Mistral', 'is', 'awesome'])
)
)
def test_function_uuid(self):
uuid = self._evaluator.evaluate('uuid()', {})
self.assertTrue(utils.is_valid_uuid(uuid))
def test_function_env(self):
ctx = {'__env': 'some'}
self.assertEqual(ctx['__env'], self._evaluator.evaluate('env()', ctx))
class InlineYAQLEvaluatorTest(base.BaseTest):
def setUp(self):
super(InlineYAQLEvaluatorTest, self).setUp()
self._evaluator = expr.InlineYAQLEvaluator()
def test_multiple_placeholders(self):
expr_str = """
Statistics for tenant "<% $.project_id %>"
Number of virtual machines: <% $.vm_count %>
Number of active virtual machines: <% $.active_vm_count %>
Number of networks: <% $.net_count %>
-- Sincerely, Mistral Team.
"""
result = self._evaluator.evaluate(
expr_str,
{
'project_id': '1-2-3-4',
'vm_count': 28,
'active_vm_count': 0,
'net_count': 1
}
)
expected_result = """
Statistics for tenant "1-2-3-4"
Number of virtual machines: 28
Number of active virtual machines: 0
Number of networks: 1
-- Sincerely, Mistral Team.
"""
self.assertEqual(expected_result, result)
def test_single_value_casting(self):
self.assertEqual(3, self._evaluator.evaluate('<% $ %>', 3))
self.assertEqual('33', self._evaluator.evaluate('<% $ %><% $ %>', 3))
def test_function_string(self):
self.assertEqual('3', self._evaluator.evaluate('<% str($) %>', '3'))
self.assertEqual('3', self._evaluator.evaluate('<% str($) %>', 3))
def test_validate(self):
self._evaluator.validate('There is no expression.')
self._evaluator.validate('<% abc %>')
self._evaluator.validate('<% 1 %>')
self._evaluator.validate('<% 1 + 2 %>')
self._evaluator.validate('<% $.a1 %>')
self._evaluator.validate('<% $.a1 * $.a2 %>')
self._evaluator.validate('<% $.a1 %> is <% $.a2 %>')
self._evaluator.validate('The value is <% $.a1 %>.')
def test_validate_failed(self):
self.assertRaises(exc.YaqlGrammarException,
self._evaluator.validate,
'The value is <% * %>.')
self.assertRaises(exc.YaqlEvaluationException,
self._evaluator.validate,
[1, 2, 3])
self.assertRaises(exc.YaqlEvaluationException,
self._evaluator.validate,
{'a': 1})

186
mistral/tests/unit/test_expressions.py
View File

@ -1,5 +1,6 @@
# Copyright 2013 - Mirantis, Inc.
# Copyright 2015 - StackStorm, Inc.
# Copyright 2016 - Brocade Communications Systems, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@ -35,168 +36,6 @@ SERVERS = {
}
class YaqlEvaluatorTest(base.BaseTest):
def setUp(self):
super(YaqlEvaluatorTest, self).setUp()
self._evaluator = expr.YAQLEvaluator()
def test_expression_result(self):
res = self._evaluator.evaluate('$.server', DATA)
self.assertEqual({
'id': "03ea824a-aa24-4105-9131-66c48ae54acf",
'name': 'cloud-fedora',
'status': 'ACTIVE'
}, res)
res = self._evaluator.evaluate('$.server.id', DATA)
self.assertEqual('03ea824a-aa24-4105-9131-66c48ae54acf', res)
res = self._evaluator.evaluate("$.server.status = 'ACTIVE'", DATA)
self.assertTrue(res)
def test_wrong_expression(self):
res = self._evaluator.evaluate("$.status = 'Invalid value'", DATA)
self.assertFalse(res)
self.assertRaises(
exc.YaqlEvaluationException,
self._evaluator.evaluate,
'$.wrong_key',
DATA
)
expression_str = 'invalid_expression_string'
res = self._evaluator.evaluate(expression_str, DATA)
self.assertEqual(expression_str, res)
def test_select_result(self):
res = self._evaluator.evaluate(
'$.servers.where($.name = ubuntu)',
SERVERS
)
item = list(res)[0]
self.assertEqual({'name': 'ubuntu'}, item)
def test_function_string(self):
self.assertEqual('3', self._evaluator.evaluate('str($)', '3'))
self.assertEqual('3', self._evaluator.evaluate('str($)', 3))
def test_function_len(self):
self.assertEqual(3, self._evaluator.evaluate('len($)', 'hey'))
data = [{'some': 'thing'}]
self.assertEqual(
1,
self._evaluator.evaluate('$.where($.some = thing).len()', data)
)
def test_validate(self):
self._evaluator.validate('abc')
self._evaluator.validate('1')
self._evaluator.validate('1 + 2')
self._evaluator.validate('$.a1')
self._evaluator.validate('$.a1 * $.a2')
def test_validate_failed(self):
self.assertRaises(exc.YaqlGrammarException,
self._evaluator.validate,
'*')
self.assertRaises(exc.YaqlGrammarException,
self._evaluator.validate,
[1, 2, 3])
self.assertRaises(exc.YaqlGrammarException,
self._evaluator.validate,
{'a': 1})
def test_json_pp(self):
self.assertEqual('"3"', self._evaluator.evaluate('json_pp($)', '3'))
self.assertEqual('3', self._evaluator.evaluate('json_pp($)', 3))
self.assertEqual(
'[\n 1,\n 2\n]',
self._evaluator.evaluate('json_pp($)', [1, 2])
)
self.assertEqual(
'{\n "a": "b"\n}',
self._evaluator.evaluate('json_pp($)', {'a': 'b'})
)
self.assertEqual(
'"Mistral\nis\nawesome"',
self._evaluator.evaluate(
'json_pp($)', '\n'.join(['Mistral', 'is', 'awesome'])
)
)
class InlineYAQLEvaluatorTest(base.BaseTest):
def setUp(self):
super(InlineYAQLEvaluatorTest, self).setUp()
self._evaluator = expr.InlineYAQLEvaluator()
def test_multiple_placeholders(self):
expr_str = """
Statistics for tenant "<% $.project_id %>"
Number of virtual machines: <% $.vm_count %>
Number of active virtual machines: <% $.active_vm_count %>
Number of networks: <% $.net_count %>
-- Sincerely, Mistral Team.
"""
result = self._evaluator.evaluate(
expr_str,
{
'project_id': '1-2-3-4',
'vm_count': 28,
'active_vm_count': 0,
'net_count': 1
}
)
expected_result = """
Statistics for tenant "1-2-3-4"
Number of virtual machines: 28
Number of active virtual machines: 0
Number of networks: 1
-- Sincerely, Mistral Team.
"""
self.assertEqual(expected_result, result)
def test_function_string(self):
self.assertEqual('3', self._evaluator.evaluate('<% str($) %>', '3'))
self.assertEqual('3', self._evaluator.evaluate('<% str($) %>', 3))
def test_validate(self):
self._evaluator.validate('There is no expression.')
self._evaluator.validate('<% abc %>')
self._evaluator.validate('<% 1 %>')
self._evaluator.validate('<% 1 + 2 %>')
self._evaluator.validate('<% $.a1 %>')
self._evaluator.validate('<% $.a1 * $.a2 %>')
self._evaluator.validate('<% $.a1 %> is <% $.a2 %>')
self._evaluator.validate('The value is <% $.a1 %>.')
def test_validate_failed(self):
self.assertRaises(exc.YaqlGrammarException,
self._evaluator.validate,
'The value is <% * %>.')
self.assertRaises(exc.YaqlEvaluationException,
self._evaluator.validate,
[1, 2, 3])
self.assertRaises(exc.YaqlEvaluationException,
self._evaluator.validate,
{'a': 1})
class ExpressionsTest(base.BaseTest):
def test_evaluate_complex_expressions(self):
data = {
@ -327,3 +166,26 @@ class ExpressionsTest(base.BaseTest):
expected = 'mysql://admin:secrete@vm1234.example.com/test'
self.assertEqual(expected, applied['conn'])
def test_validate_jinja_with_yaql_context(self):
self.assertRaises(exc.JinjaGrammarException,
expr.validate,
'{{ $ }}')
def test_validate_mixing_jinja_and_yaql(self):
self.assertRaises(exc.ExpressionGrammarException,
expr.validate,
'<% $.a %>{{ _.a }}')
self.assertRaises(exc.ExpressionGrammarException,
expr.validate,
'{{ _.a }}<% $.a %>')
def test_evaluate_mixing_jinja_and_yaql(self):
actual = expr.evaluate('<% $.a %>{{ _.a }}', {'a': 'b'})
self.assertEqual('<% $.a %>b', actual)
actual = expr.evaluate('{{ _.a }}<% $.a %>', {'a': 'b'})
self.assertEqual('b<% $.a %>', actual)

12
mistral/tests/unit/workbook/v2/test_actions.py
View File

@ -1,4 +1,5 @@
# Copyright 2015 - StackStorm, Inc.
# Copyright 2016 - Brocade Communications Systems, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@ -37,7 +38,10 @@ class ActionSpecValidation(base.WorkbookSpecValidationTestCase):
({'actions': {'a1': {'base': 'std.echo output="foo"'}}}, False),
({'actions': {'a1': {'base': 'std.echo output="<% $.x %>"'}}},
False),
({'actions': {'a1': {'base': 'std.echo output="<% * %>"'}}}, True)
({'actions': {'a1': {'base': 'std.echo output="<% * %>"'}}}, True),
({'actions': {'a1': {'base': 'std.echo output="{{ _.x }}"'}}},
False),
({'actions': {'a1': {'base': 'std.echo output="{{ * }}"'}}}, True)
]
for actions, expect_error in tests:
@ -49,7 +53,9 @@ class ActionSpecValidation(base.WorkbookSpecValidationTestCase):
({'base-input': {}}, True),
({'base-input': None}, True),
({'base-input': {'k1': 'v1', 'k2': '<% $.v2 %>'}}, False),
({'base-input': {'k1': 'v1', 'k2': '<% * %>'}}, True)
({'base-input': {'k1': 'v1', 'k2': '<% * %>'}}, True),
({'base-input': {'k1': 'v1', 'k2': '{{ _.v2 }}'}}, False),
({'base-input': {'k1': 'v1', 'k2': '{{ * }}'}}, True)
]
actions = {
@ -100,6 +106,8 @@ class ActionSpecValidation(base.WorkbookSpecValidationTestCase):
({'output': 'foobar'}, False),
({'output': '<% $.x %>'}, False),
({'output': '<% * %>'}, True),
({'output': '{{ _.x }}'}, False),
({'output': '{{ * }}'}, True),
({'output': ['v1']}, False),
({'output': {'k1': 'v1'}}, False)
]

66
mistral/tests/unit/workbook/v2/test_tasks.py
View File

@ -1,5 +1,6 @@
# Copyright 2015 - Huawei Technologies Co. Ltd
# Copyright 2015 - StackStorm, Inc.
# Copyright 2016 - Brocade Communications Systems, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@ -55,12 +56,18 @@ class TaskSpecValidation(v2_base.WorkflowSpecValidationTestCase):
({'action': 'std.http url=<% $.url %>'}, False),
({'action': 'std.http url=<% $.url %> timeout=<% $.t %>'}, False),
({'action': 'std.http url=<% * %>'}, True),
({'action': 'std.http url={{ _.url }}'}, False),
({'action': 'std.http url={{ _.url }} timeout={{ _.t }}'}, False),
({'action': 'std.http url={{ $ }}'}, True),
({'workflow': 'test.wf'}, False),
({'workflow': 'test.wf k1="v1"'}, False),
({'workflow': 'test.wf k1="v1" k2="v2"'}, False),
({'workflow': 'test.wf k1=<% $.v1 %>'}, False),
({'workflow': 'test.wf k1=<% $.v1 %> k2=<% $.v2 %>'}, False),
({'workflow': 'test.wf k1=<% * %>'}, True),
({'workflow': 'test.wf k1={{ _.v1 }}'}, False),
({'workflow': 'test.wf k1={{ _.v1 }} k2={{ _.v2 }}'}, False),
({'workflow': 'test.wf k1={{ $ }}'}, True),
({'action': 'std.noop', 'workflow': 'test.wf'}, True),
({'action': 123}, True),
({'workflow': 123}, True),
@ -87,7 +94,10 @@ class TaskSpecValidation(v2_base.WorkflowSpecValidationTestCase):
({'input': {'k1': 'v1'}}, False),
({'input': {'k1': '<% $.v1 %>'}}, False),
({'input': {'k1': '<% 1 + 2 %>'}}, False),
({'input': {'k1': '<% * %>'}}, True)
({'input': {'k1': '<% * %>'}}, True),
({'input': {'k1': '{{ _.v1 }}'}}, False),
({'input': {'k1': '{{ 1 + 2 }}'}}, False),
({'input': {'k1': '{{ * }}'}}, True)
]
for task_input, expect_error in tests:
@ -116,7 +126,15 @@ class TaskSpecValidation(v2_base.WorkflowSpecValidationTestCase):
({'with-items': ['x in <% $.y %>', 'i in [1, 2, 3]']}, False),
({'with-items': ['x in <% $.y %>', 'i in <% $.j %>']}, False),
({'with-items': ['x in <% * %>']}, True),
({'with-items': ['x in <% $.y %>', 'i in <% * %>']}, True)
({'with-items': ['x in <% $.y %>', 'i in <% * %>']}, True),
({'with-items': '{{ _.y }}'}, True),
({'with-items': 'x in {{ _.y }}'}, False),
({'with-items': ['x in [1, 2, 3]']}, False),
({'with-items': ['x in {{ _.y }}']}, False),
({'with-items': ['x in {{ _.y }}', 'i in [1, 2, 3]']}, False),
({'with-items': ['x in {{ _.y }}', 'i in {{ _.j }}']}, False),
({'with-items': ['x in {{ * }}']}, True),
({'with-items': ['x in {{ _.y }}', 'i in {{ * }}']}, True)
]
for with_item, expect_error in tests:
@ -136,7 +154,10 @@ class TaskSpecValidation(v2_base.WorkflowSpecValidationTestCase):
({'publish': {'k1': 'v1'}}, False),
({'publish': {'k1': '<% $.v1 %>'}}, False),
({'publish': {'k1': '<% 1 + 2 %>'}}, False),
({'publish': {'k1': '<% * %>'}}, True)
({'publish': {'k1': '<% * %>'}}, True),
({'publish': {'k1': '{{ _.v1 }}'}}, False),
({'publish': {'k1': '{{ 1 + 2 }}'}}, False),
({'publish': {'k1': '{{ * }}'}}, True)
]
for output, expect_error in tests:
@ -164,39 +185,61 @@ class TaskSpecValidation(v2_base.WorkflowSpecValidationTestCase):
({'retry': {'count': '<% * %>', 'delay': 1}}, True),
({'retry': {'count': 3, 'delay': '<% 1 %>'}}, False),
({'retry': {'count': 3, 'delay': '<% * %>'}}, True),
({'retry': {
'continue-on': '{{ 1 }}', 'delay': 2,
'break-on': '{{ 1 }}', 'count': 2
}}, False),
({'retry': {
'count': 3, 'delay': 1, 'continue-on': '{{ 1 }}'
}}, False),
({'retry': {'count': '{{ 3 }}', 'delay': 1}}, False),
({'retry': {'count': '{{ * }}', 'delay': 1}}, True),
({'retry': {'count': 3, 'delay': '{{ 1 }}'}}, False),
({'retry': {'count': 3, 'delay': '{{ * }}'}}, True),
({'retry': {'count': -3, 'delay': 1}}, True),
({'retry': {'count': 3, 'delay': -1}}, True),
({'retry': {'count': '3', 'delay': 1}}, True),
({'retry': {'count': 3, 'delay': '1'}}, True),
({'retry': 'count=3 delay=1 break-on=<% false %>'}, False),
({'retry': 'count=3 delay=1 break-on={{ false }}'}, False),
({'retry': 'count=3 delay=1'}, False),
({'retry': 'coun=3 delay=1'}, True),
({'retry': None}, True),
({'wait-before': 1}, False),
({'wait-before': '<% 1 %>'}, False),
({'wait-before': '<% * %>'}, True),
({'wait-before': '{{ 1 }}'}, False),
({'wait-before': '{{ * }}'}, True),
({'wait-before': -1}, True),
({'wait-before': 1.0}, True),
({'wait-before': '1'}, True),
({'wait-after': 1}, False),
({'wait-after': '<% 1 %>'}, False),
({'wait-after': '<% * %>'}, True),
({'wait-after': '{{ 1 }}'}, False),
({'wait-after': '{{ * }}'}, True),
({'wait-after': -1}, True),
({'wait-after': 1.0}, True),
({'wait-after': '1'}, True),
({'timeout': 300}, False),
({'timeout': '<% 300 %>'}, False),
({'timeout': '<% * %>'}, True),
({'timeout': '{{ 300 }}'}, False),
({'timeout': '{{ * }}'}, True),
({'timeout': -300}, True),
({'timeout': 300.0}, True),
({'timeout': '300'}, True),
({'pause-before': False}, False),
({'pause-before': '<% False %>'}, False),
({'pause-before': '<% * %>'}, True),
({'pause-before': '{{ False }}'}, False),
({'pause-before': '{{ * }}'}, True),
({'pause-before': 'False'}, True),
({'concurrency': 10}, False),
({'concurrency': '<% 10 %>'}, False),
({'concurrency': '<% * %>'}, True),
({'concurrency': '{{ 10 }}'}, False),
({'concurrency': '{{ * }}'}, True),
({'concurrency': -10}, True),
({'concurrency': 10.0}, True),
({'concurrency': '10'}, True)
@ -218,6 +261,10 @@ class TaskSpecValidation(v2_base.WorkflowSpecValidationTestCase):
({'on-success': [{'email': '<% 1 %>'}, 'echo']}, False),
({'on-success': [{'email': '<% $.v1 in $.v2 %>'}]}, False),
({'on-success': [{'email': '<% * %>'}]}, True),
({'on-success': [{'email': '{{ 1 }}'}]}, False),
({'on-success': [{'email': '{{ 1 }}'}, 'echo']}, False),
({'on-success': [{'email': '{{ _.v1 in _.v2 }}'}]}, False),
({'on-success': [{'email': '{{ * }}'}]}, True),
({'on-success': 'email'}, False),
({'on-success': None}, True),
({'on-success': ['']}, True),
@ -229,6 +276,10 @@ class TaskSpecValidation(v2_base.WorkflowSpecValidationTestCase):
({'on-error': [{'email': '<% 1 %>'}, 'echo']}, False),
({'on-error': [{'email': '<% $.v1 in $.v2 %>'}]}, False),
({'on-error': [{'email': '<% * %>'}]}, True),
({'on-error': [{'email': '{{ 1 }}'}]}, False),
({'on-error': [{'email': '{{ 1 }}'}, 'echo']}, False),
({'on-error': [{'email': '{{ _.v1 in _.v2 }}'}]}, False),
({'on-error': [{'email': '{{ * }}'}]}, True),
({'on-error': 'email'}, False),
({'on-error': None}, True),
({'on-error': ['']}, True),
@ -240,6 +291,10 @@ class TaskSpecValidation(v2_base.WorkflowSpecValidationTestCase):
({'on-complete': [{'email': '<% 1 %>'}, 'echo']}, False),
({'on-complete': [{'email': '<% $.v1 in $.v2 %>'}]}, False),
({'on-complete': [{'email': '<% * %>'}]}, True),
({'on-complete': [{'email': '{{ 1 }}'}]}, False),
({'on-complete': [{'email': '{{ 1 }}'}, 'echo']}, False),
({'on-complete': [{'email': '{{ _.v1 in _.v2 }}'}]}, False),
({'on-complete': [{'email': '{{ * }}'}]}, True),
({'on-complete': 'email'}, False),
({'on-complete': None}, True),
({'on-complete': ['']}, True),
@ -322,7 +377,10 @@ class TaskSpecValidation(v2_base.WorkflowSpecValidationTestCase):
({'keep-result': False}, False),
({'keep-result': "<% 'a' in $.val %>"}, False),
({'keep-result': '<% 1 + 2 %>'}, False),
({'keep-result': '<% * %>'}, True)
({'keep-result': '<% * %>'}, True),
({'keep-result': "{{ 'a' in _.val }}"}, False),
({'keep-result': '{{ 1 + 2 }}'}, False),
({'keep-result': '{{ * }}'}, True)
]
for keep_result, expect_error in tests:

30
mistral/tests/unit/workbook/v2/test_workflows.py
View File

@ -1,4 +1,5 @@
# Copyright 2015 - StackStorm, Inc.
# Copyright 2016 - Brocade Communications Systems, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@ -234,6 +235,9 @@ class WorkflowSpecValidation(base.WorkflowSpecValidationTestCase):
({'vars': {'v1': '<% $.input_var1 %>'}}, False),
({'vars': {'v1': '<% 1 + 2 %>'}}, False),
({'vars': {'v1': '<% * %>'}}, True),
({'vars': {'v1': '{{ _.input_var1 }}'}}, False),
({'vars': {'v1': '{{ 1 + 2 }}'}}, False),
({'vars': {'v1': '{{ * }}'}}, True),
({'vars': []}, True),
({'vars': 'whatever'}, True),
({'vars': None}, True),
@ -280,6 +284,10 @@ class WorkflowSpecValidation(base.WorkflowSpecValidationTestCase):
({'on-success': [{'email': '<% 1 %>'}, 'echo']}, False),
({'on-success': [{'email': '<% $.v1 in $.v2 %>'}]}, False),
({'on-success': [{'email': '<% * %>'}]}, True),
({'on-success': [{'email': '{{ 1 }}'}]}, False),
({'on-success': [{'email': '{{ 1 }}'}, 'echo']}, False),
({'on-success': [{'email': '{{ _.v1 in _.v2 }}'}]}, False),
({'on-success': [{'email': '{{ * }}'}]}, True),
({'on-success': 'email'}, False),
({'on-success': None}, True),
({'on-success': ['']}, True),
@ -291,6 +299,10 @@ class WorkflowSpecValidation(base.WorkflowSpecValidationTestCase):
({'on-error': [{'email': '<% 1 %>'}, 'echo']}, False),
({'on-error': [{'email': '<% $.v1 in $.v2 %>'}]}, False),
({'on-error': [{'email': '<% * %>'}]}, True),
({'on-error': [{'email': '{{ 1 }}'}]}, False),
({'on-error': [{'email': '{{ 1 }}'}, 'echo']}, False),
({'on-error': [{'email': '{{ _.v1 in _.v2 }}'}]}, False),
({'on-error': [{'email': '{{ * }}'}]}, True),
({'on-error': 'email'}, False),
({'on-error': None}, True),
({'on-error': ['']}, True),
@ -302,6 +314,10 @@ class WorkflowSpecValidation(base.WorkflowSpecValidationTestCase):
({'on-complete': [{'email': '<% 1 %>'}, 'echo']}, False),
({'on-complete': [{'email': '<% $.v1 in $.v2 %>'}]}, False),
({'on-complete': [{'email': '<% * %>'}]}, True),
({'on-complete': [{'email': '{{ 1 }}'}]}, False),
({'on-complete': [{'email': '{{ 1 }}'}, 'echo']}, False),
({'on-complete': [{'email': '{{ _.v1 in _.v2 }}'}]}, False),
({'on-complete': [{'email': '{{ * }}'}]}, True),
({'on-complete': 'email'}, False),
({'on-complete': None}, True),
({'on-complete': ['']}, True),
@ -321,6 +337,10 @@ class WorkflowSpecValidation(base.WorkflowSpecValidationTestCase):
({'retry': {'count': '<% * %>', 'delay': 1}}, True),
({'retry': {'count': 3, 'delay': '<% 1 %>'}}, False),
({'retry': {'count': 3, 'delay': '<% * %>'}}, True),
({'retry': {'count': '{{ 3 }}', 'delay': 1}}, False),
({'retry': {'count': '{{ * }}', 'delay': 1}}, True),
({'retry': {'count': 3, 'delay': '{{ 1 }}'}}, False),
({'retry': {'count': 3, 'delay': '{{ * }}'}}, True),
({'retry': {'count': -3, 'delay': 1}}, True),
({'retry': {'count': 3, 'delay': -1}}, True),
({'retry': {'count': '3', 'delay': 1}}, True),
@ -329,28 +349,38 @@ class WorkflowSpecValidation(base.WorkflowSpecValidationTestCase):
({'wait-before': 1}, False),
({'wait-before': '<% 1 %>'}, False),
({'wait-before': '<% * %>'}, True),
({'wait-before': '{{ 1 }}'}, False),
({'wait-before': '{{ * }}'}, True),
({'wait-before': -1}, True),
({'wait-before': 1.0}, True),
({'wait-before': '1'}, True),
({'wait-after': 1}, False),
({'wait-after': '<% 1 %>'}, False),
({'wait-after': '<% * %>'}, True),
({'wait-after': '{{ 1 }}'}, False),
({'wait-after': '{{ * }}'}, True),
({'wait-after': -1}, True),
({'wait-after': 1.0}, True),
({'wait-after': '1'}, True),
({'timeout': 300}, False),
({'timeout': '<% 300 %>'}, False),
({'timeout': '<% * %>'}, True),
({'timeout': '{{ 300 }}'}, False),
({'timeout': '{{ * }}'}, True),
({'timeout': -300}, True),
({'timeout': 300.0}, True),
({'timeout': '300'}, True),
({'pause-before': False}, False),
({'pause-before': '<% False %>'}, False),
({'pause-before': '<% * %>'}, True),
({'pause-before': '{{ False }}'}, False),
({'pause-before': '{{ * }}'}, True),
({'pause-before': 'False'}, True),
({'concurrency': 10}, False),
({'concurrency': '<% 10 %>'}, False),
({'concurrency': '<% * %>'}, True),
({'concurrency': '{{ 10 }}'}, False),
({'concurrency': '{{ * }}'}, True),
({'concurrency': -10}, True),
({'concurrency': 10.0}, True),
({'concurrency': '10'}, True)

10
mistral/utils/__init__.py
View File

@ -2,6 +2,7 @@
#
# Copyright 2013 - Mirantis, Inc.
# Copyright 2015 - Huawei Technologies Co. Ltd
# Copyright 2016 - Brocade Communications Systems, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@ -46,6 +47,15 @@ def generate_unicode_uuid():
return six.text_type(str(uuid.uuid4()))
def is_valid_uuid(uuid_string):
try:
val = uuid.UUID(uuid_string, version=4)
except ValueError:
return False
return val.hex == uuid_string.replace('-', '')
def _get_greenlet_local_storage():
greenlet_id = corolocal.get_ident()

67
mistral/utils/yaql_utils.py → mistral/utils/expression_utils.py
View File

@ -1,4 +1,5 @@
# Copyright 2015 - Mirantis, Inc.
# Copyright 2016 - Brocade Communications Systems, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@ -12,6 +13,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
from functools import partial
from oslo_serialization import jsonutils
from stevedore import extension
@ -21,18 +23,17 @@ from mistral.db.v2 import api as db_api
from mistral import utils
ROOT_CONTEXT = None
ROOT_YAQL_CONTEXT = None
def get_yaql_context(data_context):
global ROOT_CONTEXT
global ROOT_YAQL_CONTEXT
if not ROOT_CONTEXT:
ROOT_CONTEXT = yaql.create_context()
if not ROOT_YAQL_CONTEXT:
ROOT_YAQL_CONTEXT = yaql.create_context()
_register_functions(ROOT_CONTEXT)
new_ctx = ROOT_CONTEXT.create_child_context()
_register_yaql_functions(ROOT_YAQL_CONTEXT)
new_ctx = ROOT_YAQL_CONTEXT.create_child_context()
new_ctx['$'] = data_context
if isinstance(data_context, dict):
@ -43,24 +44,50 @@ def get_yaql_context(data_context):
return new_ctx
def _register_custom_functions(yaql_ctx):
"""Register custom YAQL functions
def get_jinja_context(data_context):
new_ctx = {
'_': data_context
}
Custom YAQL functions must be added as entry points in the
'mistral.yaql_functions' namespace
:param yaql_ctx: YAQL context object
_register_jinja_functions(new_ctx)
if isinstance(data_context, dict):
new_ctx['__env'] = data_context.get('__env')
new_ctx['__execution'] = data_context.get('__execution')
new_ctx['__task_execution'] = data_context.get('__task_execution')
return new_ctx
def get_custom_functions():
"""Get custom functions
Retreives the list of custom evaluation functions
"""
functions = dict()
mgr = extension.ExtensionManager(
namespace='mistral.yaql_functions',
namespace='mistral.expression.functions',
invoke_on_load=False
)
for name in mgr.names():
yaql_function = mgr[name].plugin
yaql_ctx.register_function(yaql_function, name=name)
functions[name] = mgr[name].plugin
return functions
def _register_functions(yaql_ctx):
_register_custom_functions(yaql_ctx)
def _register_yaql_functions(yaql_ctx):
functions = get_custom_functions()
for name in functions:
yaql_ctx.register_function(functions[name], name=name)
def _register_jinja_functions(jinja_ctx):
functions = get_custom_functions()
for name in functions:
jinja_ctx[name] = partial(functions[name], jinja_ctx['_'])
# Additional YAQL functions needed by Mistral.
@ -83,9 +110,9 @@ def execution_(context):
}
def json_pp_(data):
def json_pp_(context, data=None):
return jsonutils.dumps(
data,
data or context,
indent=4
).replace("\\n", "\n").replace(" \n", "\n")
@ -128,5 +155,5 @@ def task_(context, task_name):
}
def uuid_(context):
def uuid_(context=None):
return utils.generate_unicode_uuid()

6
mistral/utils/rest_utils.py
View File

@ -1,6 +1,7 @@
# -*- coding: utf-8 -*-
#
# Copyright 2014 - Mirantis, Inc.
# Copyright 2016 - Brocade Communications Systems, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@ -17,6 +18,7 @@
import functools
import json
from oslo_log import log as logging
import pecan
import six
@ -25,6 +27,8 @@ from wsme import exc as wsme_exc
from mistral import exceptions as exc
LOG = logging.getLogger(__name__)
def wrap_wsme_controller_exception(func):
"""Decorator for controllers method.
@ -39,6 +43,7 @@ def wrap_wsme_controller_exception(func):
except (exc.MistralException, exc.MistralError) as e:
pecan.response.translatable_error = e
LOG.error('Error during API call: %s' % str(e))
raise wsme_exc.ClientSideError(
msg=six.text_type(e),
status_code=e.http_code
@ -58,6 +63,7 @@ def wrap_pecan_controller_exception(func):
try:
return func(*args, **kwargs)
except (exc.MistralException, exc.MistralError) as e:
LOG.error('Error during API call: %s' % str(e))
return webob.Response(
status=e.http_code,
content_type='application/json',

11
mistral/workbook/base.py
View File

@ -27,7 +27,7 @@ from mistral.workbook import types
CMD_PTRN = re.compile("^[\w\.]+[^=\(\s\"]*")
INLINE_YAQL = expr.INLINE_YAQL_REGEXP
EXPRESSION = '|'.join([expr.patterns[name] for name in expr.patterns])
_ALL_IN_BRACKETS = "\[.*\]\s*"
_ALL_IN_QUOTES = "\"[^\"]*\"\s*"
_ALL_IN_APOSTROPHES = "'[^']*'\s*"
@ -37,7 +37,7 @@ _FALSE = "false"
_NULL = "null"
ALL = (
_ALL_IN_QUOTES, _ALL_IN_APOSTROPHES, INLINE_YAQL,
_ALL_IN_QUOTES, _ALL_IN_APOSTROPHES, EXPRESSION,
_ALL_IN_BRACKETS, _TRUE, _FALSE, _NULL, _DIGITS
)
@ -194,7 +194,7 @@ class BaseSpec(object):
"""
pass
def validate_yaql_expr(self, dsl_part):
def validate_expr(self, dsl_part):
if isinstance(dsl_part, six.string_types):
expr.validate(dsl_part)
elif isinstance(dsl_part, list):
@ -278,9 +278,10 @@ class BaseSpec(object):
params = {}
for k, v in re.findall(PARAMS_PTRN, cmd_str):
for match in re.findall(PARAMS_PTRN, cmd_str):
k = match[0]
# Remove embracing quotes.
v = v.strip()
v = match[1].strip()
if v[0] == '"' or v[0] == "'":
v = v[1:-1]
else:

37
mistral/workbook/types.py
View File

@ -12,6 +12,9 @@
# See the License for the specific language governing permissions and
# limitations under the License.
from mistral import expressions
NONEMPTY_STRING = {
"type": "string",
"minLength": 1
@ -34,16 +37,18 @@ POSITIVE_NUMBER = {
"minimum": 0.0
}
YAQL = {
"type": "string",
"pattern": "^<%.*?%>\\s*$"
EXPRESSION = {
"oneOf": [{
"type": "string",
"pattern": "^%s\\s*$" % expressions.patterns[name]
} for name in expressions.patterns]
}
YAQL_CONDITION = {
EXPRESSION_CONDITION = {
"type": "object",
"minProperties": 1,
"patternProperties": {
"^\w+$": YAQL
"^\w+$": EXPRESSION
}
}
@ -54,8 +59,7 @@ ANY = {
{"type": "integer"},
{"type": "number"},
{"type": "object"},
{"type": "string"},
YAQL
{"type": "string"}
]
}
@ -67,8 +71,7 @@ ANY_NULLABLE = {
{"type": "integer"},
{"type": "number"},
{"type": "object"},
{"type": "string"},
YAQL
{"type": "string"}
]
}
@ -89,31 +92,31 @@ ONE_KEY_DICT = {
}
}
STRING_OR_YAQL_CONDITION = {
STRING_OR_EXPRESSION_CONDITION = {
"oneOf": [
NONEMPTY_STRING,
YAQL_CONDITION
EXPRESSION_CONDITION
]
}
YAQL_OR_POSITIVE_INTEGER = {
EXPRESSION_OR_POSITIVE_INTEGER = {
"oneOf": [
YAQL,
EXPRESSION,
POSITIVE_INTEGER
]
}
YAQL_OR_BOOLEAN = {
EXPRESSION_OR_BOOLEAN = {
"oneOf": [
YAQL,
EXPRESSION,
{"type": "boolean"}
]
}
UNIQUE_STRING_OR_YAQL_CONDITION_LIST = {
UNIQUE_STRING_OR_EXPRESSION_CONDITION_LIST = {
"type": "array",
"items": STRING_OR_YAQL_CONDITION,
"items": STRING_OR_EXPRESSION_CONDITION,
"uniqueItems": True,
"minItems": 1
}

6
mistral/workbook/v2/actions.py
View File

@ -54,12 +54,12 @@ class ActionSpec(base.BaseSpec):
# Validate YAQL expressions.
inline_params = self._parse_cmd_and_input(self._data.get('base'))[1]
self.validate_yaql_expr(inline_params)
self.validate_expr(inline_params)
self.validate_yaql_expr(self._data.get('base-input', {}))
self.validate_expr(self._data.get('base-input', {}))
if isinstance(self._data.get('output'), six.string_types):
self.validate_yaql_expr(self._data.get('output'))
self.validate_expr(self._data.get('output'))
def get_name(self):
return self._name

20
mistral/workbook/v2/policies.py
View File

@ -19,11 +19,11 @@ from mistral.workbook.v2 import retry_policy
RETRY_SCHEMA = retry_policy.RetrySpec.get_schema(includes=None)
WAIT_BEFORE_SCHEMA = types.YAQL_OR_POSITIVE_INTEGER
WAIT_AFTER_SCHEMA = types.YAQL_OR_POSITIVE_INTEGER
TIMEOUT_SCHEMA = types.YAQL_OR_POSITIVE_INTEGER
PAUSE_BEFORE_SCHEMA = types.YAQL_OR_BOOLEAN
CONCURRENCY_SCHEMA = types.YAQL_OR_POSITIVE_INTEGER
WAIT_BEFORE_SCHEMA = types.EXPRESSION_OR_POSITIVE_INTEGER
WAIT_AFTER_SCHEMA = types.EXPRESSION_OR_POSITIVE_INTEGER
TIMEOUT_SCHEMA = types.EXPRESSION_OR_POSITIVE_INTEGER
PAUSE_BEFORE_SCHEMA = types.EXPRESSION_OR_BOOLEAN
CONCURRENCY_SCHEMA = types.EXPRESSION_OR_POSITIVE_INTEGER
class PoliciesSpec(base.BaseSpec):
@ -59,11 +59,11 @@ class PoliciesSpec(base.BaseSpec):
super(PoliciesSpec, self).validate_schema()
# Validate YAQL expressions.
self.validate_yaql_expr(self._data.get('wait-before', 0))
self.validate_yaql_expr(self._data.get('wait-after', 0))
self.validate_yaql_expr(self._data.get('timeout', 0))
self.validate_yaql_expr(self._data.get('pause-before', False))
self.validate_yaql_expr(self._data.get('concurrency', 0))
self.validate_expr(self._data.get('wait-before', 0))
self.validate_expr(self._data.get('wait-after', 0))
self.validate_expr(self._data.get('timeout', 0))
self.validate_expr(self._data.get('pause-before', False))
self.validate_expr(self._data.get('concurrency', 0))
def get_retry(self):
return self._retry

16
mistral/workbook/v2/retry_policy.py
View File

@ -26,15 +26,15 @@ class RetrySpec(base.BaseSpec):
"properties": {
"count": {
"oneOf": [
types.YAQL,
types.EXPRESSION,
types.POSITIVE_INTEGER
]
},
"break-on": types.YAQL,
"continue-on": types.YAQL,
"break-on": types.EXPRESSION,
"continue-on": types.EXPRESSION,
"delay": {
"oneOf": [
types.YAQL,
types.EXPRESSION,
types.POSITIVE_INTEGER
]
},
@ -74,10 +74,10 @@ class RetrySpec(base.BaseSpec):
super(RetrySpec, self).validate_schema()
# Validate YAQL expressions.
self.validate_yaql_expr(self._data.get('count'))
self.validate_yaql_expr(self._data.get('delay'))
self.validate_yaql_expr(self._data.get('break-on'))
self.validate_yaql_expr(self._data.get('continue-on'))
self.validate_expr(self._data.get('count'))
self.validate_expr(self._data.get('delay'))
self.validate_expr(self._data.get('break-on'))
self.validate_expr(self._data.get('continue-on'))
def get_count(self):
return self._count

4
mistral/workbook/v2/task_defaults.py
View File

@ -32,7 +32,7 @@ class TaskDefaultsSpec(base.BaseSpec):
_on_clause_type = {
"oneOf": [
types.NONEMPTY_STRING,
types.UNIQUE_STRING_OR_YAQL_CONDITION_LIST
types.UNIQUE_STRING_OR_EXPRESSION_CONDITION_LIST
]
}
@ -93,7 +93,7 @@ class TaskDefaultsSpec(base.BaseSpec):
def _validate_transitions(self, on_clause):
val = self._data.get(on_clause, [])
[self.validate_yaql_expr(t)
[self.validate_expr(t)
for t in ([val] if isinstance(val, six.string_types) else val)]
def get_policies(self):

30
mistral/workbook/v2/tasks.py
View File

@ -19,15 +19,15 @@ import re
import six
from mistral import exceptions as exc
from mistral import expressions as expr
from mistral import expressions
from mistral import utils
from mistral.workbook import types
from mistral.workbook.v2 import base
from mistral.workbook.v2 import policies
_expr_ptrns = [expressions.patterns[name] for name in expressions.patterns]
WITH_ITEMS_PTRN = re.compile(
"\s*([\w\d_\-]+)\s*in\s*(\[.+\]|%s)" % expr.INLINE_YAQL_REGEXP
"\s*([\w\d_\-]+)\s*in\s*(\[.+\]|%s)" % '|'.join(_expr_ptrns)
)
RESERVED_TASK_NAMES = [
'noop',
@ -62,8 +62,8 @@ class TaskSpec(base.BaseSpec):
"pause-before": policies.PAUSE_BEFORE_SCHEMA,
"concurrency": policies.CONCURRENCY_SCHEMA,
"target": types.NONEMPTY_STRING,
"keep-result": types.YAQL_OR_BOOLEAN,
"safe-rerun": types.YAQL_OR_BOOLEAN
"keep-result": types.EXPRESSION_OR_BOOLEAN,
"safe-rerun": types.EXPRESSION_OR_BOOLEAN
},
"additionalProperties": False,
"anyOf": [
@ -122,12 +122,12 @@ class TaskSpec(base.BaseSpec):
# Validate YAQL expressions.
if action or workflow:
inline_params = self._parse_cmd_and_input(action or workflow)[1]
self.validate_yaql_expr(inline_params)
self.validate_expr(inline_params)
self.validate_yaql_expr(self._data.get('input', {}))
self.validate_yaql_expr(self._data.get('publish', {}))
self.validate_yaql_expr(self._data.get('keep-result', {}))
self.validate_yaql_expr(self._data.get('safe-rerun', {}))
self.validate_expr(self._data.get('input', {}))
self.validate_expr(self._data.get('publish', {}))
self.validate_expr(self._data.get('keep-result', {}))
self.validate_expr(self._data.get('safe-rerun', {}))
def _transform_with_items(self):
raw = self._data.get('with-items', [])
@ -149,11 +149,13 @@ class TaskSpec(base.BaseSpec):
"%s" % self._data)
raise exc.InvalidModelException(msg)
var_name, array = match.groups()
match_groups = match.groups()
var_name = match_groups[0]
array = match_groups[1]
# Validate YAQL expression that may follow after "in" for the
# with-items syntax "var in {[some, list] | <% $.array %> }".
self.validate_yaql_expr(array)
self.validate_expr(array)
if array.startswith('['):
try:
@ -223,7 +225,7 @@ class DirectWorkflowTaskSpec(TaskSpec):
_on_clause_type = {
"oneOf": [
types.NONEMPTY_STRING,
types.UNIQUE_STRING_OR_YAQL_CONDITION_LIST
types.UNIQUE_STRING_OR_EXPRESSION_CONDITION_LIST
]
}
@ -271,7 +273,7 @@ class DirectWorkflowTaskSpec(TaskSpec):
def _validate_transitions(self, on_clause):
val = self._data.get(on_clause, [])
[self.validate_yaql_expr(t)
[self.validate_expr(t)
for t in ([val] if isinstance(val, six.string_types) else val)]
@staticmethod

6
mistral/workbook/v2/workflows.py
View File

@ -77,9 +77,9 @@ class WorkflowSpec(base.BaseSpec):
"Workflow doesn't have any tasks [data=%s]" % self._data
)
# Validate YAQL expressions.
self.validate_yaql_expr(self._data.get('output', {}))
self.validate_yaql_expr(self._data.get('vars', {}))
# Validate expressions.
self.validate_expr(self._data.get('output', {}))
self.validate_expr(self._data.get('vars', {}))
def validate_semantics(self):
super(WorkflowSpec, self).validate_semantics()

1
requirements.txt
View File

@ -7,6 +7,7 @@ Babel>=2.3.4 # BSD
croniter>=0.3.4 # MIT License
cachetools>=1.1.0 # MIT License
eventlet!=0.18.3,>=0.18.2 # MIT
Jinja2>=2.8 # BSD License (3 clause)
jsonschema!=2.5.0,<3.0.0,>=2.0.0 # MIT
keystonemiddleware!=4.5.0,>=4.2.0 # Apache-2.0
mock>=2.0 # BSD

16
setup.cfg
View File

@ -69,12 +69,16 @@ mistral.actions =
std.javascript = mistral.actions.std_actions:JavaScriptAction
std.sleep = mistral.actions.std_actions:SleepAction
mistral.yaql_functions =
json_pp = mistral.utils.yaql_utils:json_pp_
task = mistral.utils.yaql_utils:task_
execution = mistral.utils.yaql_utils:execution_
env = mistral.utils.yaql_utils:env_
uuid = mistral.utils.yaql_utils:uuid_
mistral.expression.functions =
json_pp = mistral.utils.expression_utils:json_pp_
task = mistral.utils.expression_utils:task_
execution = mistral.utils.expression_utils:execution_
env = mistral.utils.expression_utils:env_
uuid = mistral.utils.expression_utils:uuid_
mistral.expression.evaluators =
yaql = mistral.expressions.yaql_expression:InlineYAQLEvaluator
jinja = mistral.expressions.jinja_expression:InlineJinjaEvaluator
mistral.auth =
keystone = mistral.auth.keystone:KeystoneAuthHandler