openstack/yaql
Code Issues Proposed changes

Small improvements to yaql

Browse Source 
Contexts:
* context module was renamed to contexts
* convention property is now part of ContextBase class.
* Standard Context implementation automatically uses convention
  from parent context
* Context interface was enhanced to add capability to check key presence,
   get data with different default or not try to access parent context
* MultiContext to virtually merge several contexts without of making them related.
  All merged contexts may have parents of their own which are also merged.
  Source contexts are not modified. This is sort of context mix-in

Parser:
* keyword arguments (name => value) now require name to be keyword token.
   So f('abc' => value) will not work. This doesnt affect dict() and similar functions
   as they use different argument format.
* tokens that start with two underscores (__) are no more valid. This is done so that it
  would be possible to have auto-injected Python arguments like "__context" without
  affecting possible kwargs of the same function

Delegates:
* Added ability to call delegate (callable) passed as a context value.
  The syntax is $var(args) (or even $(args)).
* Delegate doesn't have to be a context value but also can be result of expression:
  func(args1)(args2), (f() op g())() and so on
* Delegates are disabled by default for security reasons. "allow_delegates" parameter
  was added to both YaqlFactory classes to enable them
* (expr)(args) will be translated to #call(expr, args). So for this to work #call function
  must be present in context. Standard context doesn't provide this function by default.
  Use delegates=True parameter of create_context method (including legacy mode version)
  to register #call() and lambda() functions
* additional lambda(expression) method that returns delegate to passed expression thus
  making it 2nd order lambda. This delegate may be stored in context using let() method
  (or alternatives) or be called as usual. lambda(expression)(args) is equal to expression(args)

Function specs:
* FunctionDefinition now has "meta" dictionary to store arbitrary extended metadata for the
   function (for example version etc.)
* use @specs.meta('key', 'value') decorator to assign meta value to the function. Several
  decorators may be applied to single function
* Context.__call__() / runner.call() now accept optional function_filter parameter that is a
  predicate (functionDefinition, context -> boolean) to additionally filter function candidates
  based on context values, function metadata and so on.
* returns_context decorator and functionality was removed because it was shawn to be useless
* "__context" and "__engine" parameters where not explicitly declared treated the
  same way as  "context" and "engine"
* added ability to control what type be assigned to parameters without explicit specification
  based on parameter name
* added ability to get function definition with stripped hidden parameters for function wrappers
* refactoring of ParameterDefinition creation code. Code from decorator was moved to a more
  generic set_parameter method of FunctionDefinition to support wider range of scenarios
  (for example to overwrite auto-generated parameter specs)
* FunctionDefinition.__call__ was changed:
  a) order of parameter was changed so the sender could have default value
  b) args now automatically prefixed with sender when provided.
       There is no need to provide it twice anymore
* a helper functions was added to utils to get yaql 0.2 style extension methods specs
  for functions that already registered in context as pure functions or methods

Smart types:
* "context" argument was added to each check() method so that it would be possible to do
  checks using context values. Non of standard smart-types use it, however custom types may do.
* return_context flag was removed from Lambda/Delegate/Super types for the same reasons as in FD
* GenericType helper class was added as a base class for custom non-lazy smart-types simplifying
  their development
* added ability to pass received Lambda (Delegate etc) to another method (or base method version)
  when the later expects it in another format (for example sender had Lambda() while the receiver
  had Lambda(with_context=True))

Standard library:
* "#operator_." method for obj.method() now expects first argument (sender expression) to be object
   (pre-evaluated) rather than Lambda. This doesn't brake anything but allows to override this function
   in child contexts for some more specific types of sender without getting resolution error because of
   several versions of the same function being differ by parameter laziness
* collection.flatten() method was added
* string.matches(regexpString) method was added to complement
   regexp.matches(string) that was before
* string.join(collection) method was added to complement collection.join(string)
   that was before. Also now both functions apply str() function to each element
   of collection
* now int(null) = 0 and float(null) = 0.0

Also bumps requirements to latest OpenStack global-requirements and removes version
number from setup.cfg to use git tag-driven versioning of pbr

Change-Id: I0dd06bf1fb70296157ebb0e9831d2b70d93ca137
This commit is contained in:
Stan Lagun 2015-07-20 22:08:54 +03:00
parent 3828b49ab2
commit 1d1f187c5c
30 changed files with 643 additions and 320 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
requirements.txt
View File

@ -1,5 +1,5 @@
pbr>=0.6,!=0.7,<1.0
Babel>=0.9.6
pbr>=0.11,<2.0
Babel>=1.3
ply<=3.6
six>=1.7.0
six>=1.9.0

1
setup.cfg
View File

@ -1,6 +1,5 @@
[metadata]
name = yaql
version = 1.0.0b3
summary = YAQL - Yet Another Query Language
description-file =
README.rst

16
test-requirements.txt
View File

@ -1,11 +1,11 @@
hacking>=0.5.6,<0.8
hacking>=0.10.0,<0.11
coverage>=3.6
discover
fixtures>=0.3.14
python-subunit
sphinx>=1.1.2
oslosphinx
testrepository>=0.0.17
testscenarios>=0.4,<0.5
testtools>=0.9.32
fixtures>=1.3.1
python-subunit>=0.0.18
sphinx>=1.1.2,!=1.2.0,!=1.3b1,<1.3
oslosphinx>=2.5.0
testrepository>=0.0.18
testscenarios>=0.4
testtools>=1.4.0

3
tox.ini
View File

@ -28,9 +28,10 @@ commands = python setup.py build_sphinx
# H803 skipped on purpose per list discussion.
# E123, E125 skipped as they are invalid PEP-8.
# H404 multi line docstring should start with a summary
# H405 multi line docstring summary not separated with an empty line
## TODO(ruhe) following checks should be fixed
# E721 do not compare types, use 'isinstance()'
show-source = True
ignore = E123,E125,E721,H404,H803
ignore = E123,E125,E721,H404,H405,H803
builtins = _
exclude=.venv,.git,.tox,dist,doc,*openstack/common*,*lib/python*,*egg,build

15
yaql/__init__.py
View File

@ -15,7 +15,7 @@
import os.path
import pkg_resources
from yaql.language import context as yaqlcontext
from yaql.language import contexts
from yaql.language import conventions
from yaql.language import factory
from yaql.language import specs
@ -36,10 +36,10 @@ _cached_engine = None
_default_context = None
def _setup_context(data, context, finalizer):
def _setup_context(data, context, finalizer, convention):
if context is None:
context = yaqlcontext.Context(
convention=conventions.CamelCaseConvention())
context = contexts.Context(
convention=convention or conventions.CamelCaseConvention())
if finalizer is None:
@specs.parameter('iterator', yaqltypes.Iterable())
@ -67,11 +67,12 @@ def create_context(data=utils.NO_VALUE, context=None, system=True,
common=True, boolean=True, strings=True,
math=True, collections=True, queries=True,
regex=True, branching=True,
no_sets=False, finalizer=None):
no_sets=False, finalizer=None, delegates=False,
convention=None):
context = _setup_context(data, context, finalizer)
context = _setup_context(data, context, finalizer, convention)
if system:
std_system.register(context)
std_system.register(context, delegates)
if common:
std_common.register(context)
if boolean:

110
yaql/language/context.py → yaql/language/contexts.py
View File

@ -21,8 +21,11 @@ from yaql.language import utils
class ContextBase(object):
def __init__(self, parent_context):
def __init__(self, parent_context=None, convention=None):
self._parent_context = parent_context
self._convention = convention
if convention is None and parent_context:
self._convention = parent_context.convention
@property
def parent(self):
@ -31,8 +34,11 @@ class ContextBase(object):
def register_function(self, spec, *args, **kwargs):
pass
def get_data(self, name, default=None, ask_parent=True):
return default
def __getitem__(self, name):
return None
return self.get_data(name)
def __setitem__(self, name, value):
pass
@ -40,10 +46,15 @@ class ContextBase(object):
def __delitem__(self, name):
pass
def __contains__(self, item):
return self.get_data(item, utils.NO_VALUE) is not utils.NO_VALUE
def __call__(self, name, engine, sender=utils.NO_VALUE,
data_context=None, return_context=False,
use_convention=False):
raise exceptions.NoFunctionRegisteredException(name)
data_context=None, use_convention=False,
function_filter=None):
return lambda *args, **kwargs: runner.call(
name, self, args, kwargs, engine, sender,
data_context, use_convention, function_filter)
def get_functions(self, name, predicate=None, use_convention=False):
return []
@ -54,14 +65,17 @@ class ContextBase(object):
def create_child_context(self):
return type(self)(self)
@property
def convention(self):
return self._convention
class Context(ContextBase):
def __init__(self, parent_context=None, data=utils.NO_VALUE,
convention=None):
super(Context, self).__init__(parent_context)
super(Context, self).__init__(parent_context, convention)
self._functions = {}
self._data = {}
self._convention = convention
if data is not utils.NO_VALUE:
self['$'] = data
@ -84,6 +98,7 @@ class Context(ContextBase):
self._functions.setdefault(spec.name, list()).append((spec, exclusive))
def get_functions(self, name, predicate=None, use_convention=False):
name = name.rstrip('_')
if use_convention and self._convention is not None:
name = self._convention.convert_function_name(name)
if predicate is None:
@ -93,6 +108,7 @@ class Context(ContextBase):
self._functions.get(name, list()))))
def collect_functions(self, name, predicate=None, use_convention=False):
name = name.rstrip('_')
if use_convention and self._convention is not None:
name = self._convention.convert_function_name(name)
overloads = []
@ -105,20 +121,13 @@ class Context(ContextBase):
for spec, exclusive in layer_overloads:
if exclusive:
p = None
if predicate and not predicate(spec):
if predicate and not predicate(spec, self):
continue
layer.append(spec)
if layer:
overloads.append(layer)
return overloads
def __call__(self, name, engine, sender=utils.NO_VALUE,
data_context=None, return_context=False,
use_convention=False):
return lambda *args, **kwargs: runner.call(
name, self, args, kwargs, engine, sender,
data_context, return_context, use_convention)
@staticmethod
def _normalize_name(name):
if not name.startswith('$'):
@ -130,15 +139,80 @@ class Context(ContextBase):
def __setitem__(self, name, value):
self._data[self._normalize_name(name)] = value
def __getitem__(self, name):
def get_data(self, name, default=None, ask_parent=True):
name = self._normalize_name(name)
if name in self._data:
return self._data[name]
if self.parent:
return self.parent[name]
if self.parent and ask_parent:
return self.parent.get_data(name, default, ask_parent)
return default
def __delitem__(self, name):
self._data.pop(self._normalize_name(name))
def create_child_context(self):
return Context(self, convention=self._convention)
class MultiContext(ContextBase):
def __init__(self, context_list, convention=None):
self._context_list = context_list
if convention is None:
convention = context_list[0].convention
parents = six.moves.filter(
lambda t: t,
six.moves.map(lambda t: t.parent, context_list))
if not parents:
super(MultiContext, self).__init__(None, convention)
elif len(parents) == 1:
super(MultiContext, self).__init__(parents[0], convention)
else:
super(MultiContext, self).__init__(MultiContext(parents),
convention)
def register_function(self, spec, *args, **kwargs):
self._context_list[0].register_function(spec, *args, **kwargs)
def get_data(self, name, default=None, ask_parent=True):
for context in self._context_list:
result = context.get_data(name, utils.NO_VALUE, False)
if result is not utils.NO_VALUE:
return result
if ask_parent and self.parent:
return self.parent.get_data(name, default, ask_parent)
return default
def __setitem__(self, name, value):
self._context_list[0][name] = value
def get_functions(self, name, predicate=None, use_convention=False):
result = []
for context in self._context_list:
result.extend(context.get_functions(
name, predicate, use_convention))
return result
def collect_functions(self, name, predicate=None, use_convention=False):
functions = six.moves.map(
lambda ctx: ctx.collect_functions(name, predicate, use_convention),
self._context_list)
i = 0
result = []
while True:
level = []
has_level = False
for f in functions:
if len(f) > i:
has_level = True
level.extend(f[i])
if not has_level:
return result
i += 1
result.append(level)
def __delitem__(self, name):
for context in self._context_list:
del context[name]
def create_child_context(self):
return Context(self)

8
yaql/language/conventions.py
View File

@ -28,13 +28,13 @@ class PythonConvention(Convention):
if not name or not name[0].isalpha():
return name
return name.rstrip('_')
return name
def convert_parameter_name(self, name):
if not name or not name[0].isalpha():
return name
return name.rstrip('_')
return name
class CamelCaseConvention(Convention):
@ -45,12 +45,12 @@ class CamelCaseConvention(Convention):
if not name or not name[0].isalpha():
return name
return self._to_camel_case(name.strip('_'))
return self._to_camel_case(name)
def convert_parameter_name(self, name):
if not name or not name[0].isalpha():
return name
return self._to_camel_case(name.rstrip('_', ))
return self._to_camel_case(name)
def _to_camel_case(self, name):
return self.regex.sub(lambda m: m.group(1).upper(), name)

15
yaql/language/expressions.py
View File

@ -34,8 +34,7 @@ class Function(Expression):
self.uses_sender = True
def __call__(self, sender, context, engine):
return context(self.name, engine, sender, context,
return_context=True)(*self.args)
return context(self.name, engine, sender, context)(*self.args)
def __str__(self):
return u'{0}({1})'.format(self.name, ', '.join(
@ -105,7 +104,7 @@ class Constant(Expression):
return six.text_type(self.value)
def __call__(self, sender, context, engine):
return self.value, context
return self.value
class KeywordConstant(Constant):
@ -137,8 +136,8 @@ class MappingRuleExpression(Expression):
def __call__(self, sender, context, engine):
return utils.MappingRule(
self.source(sender, context, engine)[0],
self.destination(sender, context, engine)[0]), context
self.source(sender, context, engine),
self.destination(sender, context, engine))
@six.python_2_unicode_compatible
@ -158,12 +157,12 @@ class Statement(Function):
except exceptions.WrappedException as e:
six.reraise(type(e.wrapped), e.wrapped, sys.exc_info()[2])
def evaluate(self, data=utils.NO_VALUE, context=utils.NO_VALUE):
if context is utils.NO_VALUE:
def evaluate(self, data=utils.NO_VALUE, context=None):
if context is None or context is utils.NO_VALUE:
context = yaql.create_context()
if data is not utils.NO_VALUE:
context['$'] = utils.convert_input_data(data)
return self(utils.NO_VALUE, context, self.engine)[0]
return self(utils.NO_VALUE, context, self.engine)
def __str__(self):
return str(self.expression)

11
yaql/language/factory.py
View File

@ -82,8 +82,9 @@ class YaqlEngine(object):
class YaqlFactory(object):
def __init__(self, keyword_operator='=>'):
def __init__(self, keyword_operator='=>', allow_delegates=False):
self._keyword_operator = keyword_operator
self._allow_delegates = allow_delegates
self.operators = self._standard_operators()
if keyword_operator:
self.operators.insert(0, (keyword_operator,
@ -93,6 +94,10 @@ class YaqlFactory(object):
def keyword_operator(self):
return self._keyword_operator
@property
def allow_delegates(self):
return self._allow_delegates
# noinspection PyMethodMayBeStatic
def _standard_operators(self):
return [
@ -218,7 +223,7 @@ class YaqlFactory(object):
# noinspection PyMethodMayBeStatic
def _create_parser(self, lexer_rules, operators):
return parser.Parser(lexer_rules, operators)
return parser.Parser(lexer_rules, operators, self)
def create(self, options=None):
names = self._name_generator()
@ -227,7 +232,7 @@ class YaqlFactory(object):
ply_lexer = lex.lex(object=lexer_rules, reflags=re.UNICODE)
ply_parser = yacc.yacc(
module=self._create_parser(lexer_rules, operators),
debug=False if not options else options.get("yaql.debug", False),
debug=False if not options else options.get('yaql.debug', False),
tabmodule='m' + uuid.uuid4().hex, write_tables=False)
return YaqlEngine(ply_lexer, ply_parser, options, self)

2
yaql/language/lexer.py
View File

@ -104,7 +104,7 @@ class Lexer(object):
def t_KEYWORD_STRING(self, t):
"""
\\b[^\\W\\d]\\w*\\b
(?!__)\\b[^\\W\\d]\\w*\\b
"""
if t.value in self._operators_table:
t.type = self._operators_table[t.value][2]

18
yaql/language/parser.py
View File

@ -20,12 +20,12 @@ from yaql.language import utils
class Parser(object):
def __init__(self, lexer, yaql_operators):
def __init__(self, lexer, yaql_operators, engine):
self.tokens = lexer.tokens
self._aliases = {}
self._generate_operator_funcs(yaql_operators)
self._generate_operator_funcs(yaql_operators, engine)
def _generate_operator_funcs(self, yaql_operators):
def _generate_operator_funcs(self, yaql_operators, engine):
binary_doc = ''
unary_doc = ''
precedence_dict = {}
@ -84,6 +84,18 @@ class Parser(object):
precedence.reverse()
self.precedence = tuple(precedence)
def p_value_call(this, p):
"""
func : value '(' args ')'
"""
arg = ()
if len(p) > 4:
arg = p[3]
p[0] = expressions.Function('#call', p[1], *arg)
if engine.allow_delegates:
self.p_value_call = six.create_bound_method(p_value_call, self)
@staticmethod
def p_value_to_const(p):
"""

27
yaql/language/runner.py
View File

@ -23,15 +23,18 @@ from yaql.language import yaqltypes
def call(name, context, args, kwargs, engine, sender=utils.NO_VALUE,
data_context=None, return_context=False, use_convention=False):
data_context=None, use_convention=False, function_filter=None):
if data_context is None:
data_context = context
if function_filter is None:
function_filter = lambda fd, ctx: True
if sender is utils.NO_VALUE:
predicate = lambda fd: fd.is_function
predicate = lambda fd, ctx: fd.is_function and function_filter(fd, ctx)
else:
predicate = lambda fd: fd.is_method
predicate = lambda fd, ctx: fd.is_method and function_filter(fd, ctx)
all_overloads = context.collect_functions(
name, predicate, use_convention=use_convention)
@ -46,8 +49,8 @@ def call(name, context, args, kwargs, engine, sender=utils.NO_VALUE,
data_context, args, kwargs)
try:
result = delegate()
utils.limit_memory_usage(engine, (1, result[0]))
return result if return_context else result[0]
utils.limit_memory_usage(engine, (1, result))
return result
except StopIteration as e:
six.reraise(
exceptions.WrappedException,
@ -82,7 +85,7 @@ def choose_overload(name, candidates, engine, sender, context, args, kwargs):
elif no_kwargs != c.no_kwargs:
raise_ambiguous()
mapping = c.map_args(args, kwargs)
mapping = c.map_args(args, kwargs, context)
if mapping is None:
continue
pos, kwd = mapping
@ -105,7 +108,7 @@ def choose_overload(name, candidates, engine, sender, context, args, kwargs):
raise_not_found()
arg_evaluator = lambda i, arg: (
arg(utils.NO_VALUE, context, engine)[0]
arg(utils.NO_VALUE, context, engine)
if (i not in lazy_params and isinstance(arg, expressions.Expression)
and not isinstance(arg, expressions.Constant))
else arg
@ -120,7 +123,7 @@ def choose_overload(name, candidates, engine, sender, context, args, kwargs):
for level in candidates2:
for c, mapping in level:
try:
d = c.get_delegate(sender, engine, args, kwargs)
d = c.get_delegate(sender, engine, context, args, kwargs)
except exceptions.ArgumentException:
pass
else:
@ -136,7 +139,7 @@ def choose_overload(name, candidates, engine, sender, context, args, kwargs):
if delegate is None:
raise_not_found()
return lambda: delegate(context)
return lambda: delegate()
def _translate_args(without_kwargs, args, kwargs):
@ -145,13 +148,13 @@ def _translate_args(without_kwargs, args, kwargs):
raise exceptions.ArgumentException(six.next(iter(kwargs)))
return args, {}
pos_args = []
kw_args = dict(kwargs)
kw_args = {}
for t in args:
if isinstance(t, expressions.MappingRuleExpression):
param_name = t.source
if isinstance(param_name, expressions.Constant):
if isinstance(param_name, expressions.KeywordConstant):
param_name = param_name.value
if not isinstance(param_name, six.string_types):
else:
raise exceptions.MappingTranslationException()
kw_args[param_name] = t.destination
else:

316
yaql/language/specs.py
View File

@ -13,7 +13,6 @@
# under the License.
import inspect
import types
import six
@ -43,21 +42,23 @@ class ParameterDefinition(object):
class FunctionDefinition(object):
def __init__(self, name, parameters, payload, doc='',
is_function=True, is_method=False,
returns_context=False, no_kwargs=False):
def __init__(self, name, payload, parameters=None, doc='', meta=None,
is_function=True, is_method=False, no_kwargs=False):
self.is_method = is_method
self.is_function = is_function
self.name = name
self.parameters = parameters
self.parameters = {} if not parameters else parameters
self.payload = payload
self.doc = doc
self.returns_context = returns_context
self.no_kwargs = no_kwargs
self.meta = meta or {}
def __call__(self, sender, engine, context):
return lambda *args, **kwargs: \
self.get_delegate(sender, engine, args, kwargs)(context)[0]
def __call__(self, engine, context, sender=utils.NO_VALUE):
def func(*args, **kwargs):
if sender is not utils.NO_VALUE:
args = (sender,) + args
return self.get_delegate(sender, engine, context, args, kwargs)()
return func
def clone(self):
parameters = dict(
@ -65,12 +66,135 @@ class FunctionDefinition(object):
for key, p in six.iteritems(self.parameters))
res = FunctionDefinition(
self.name, parameters, self.payload,
self.doc, self.is_function, self.is_method,
self.returns_context, self.no_kwargs)
self.name, self.payload, parameters, self.doc,
self.meta, self.is_function, self.is_method, self.no_kwargs)
return res
def map_args(self, args, kwargs):
def strip_hidden_parameters(self):
fd = self.clone()
keys_to_remove = set()
for k, v in fd.parameters.iteritems():
if not isinstance(v.value_type, yaqltypes.HiddenParameterType):
continue
keys_to_remove.add(k)
if v.position is not None:
for v2 in fd.parameters.itervalues():
if v2.position is not None and v2.position > v.position:
v2.position -= 1
for key in keys_to_remove:
del fd.parameters[key]
return fd
def set_parameter(self, name, value_type=None, nullable=None,
alias=None, overwrite=False):
if isinstance(name, ParameterDefinition):
if name.name in self.parameters and not overwrite:
raise exceptions.DuplicateParameterDecoratorException(
function_name=self.name or self.payload.__name__,
param_name=name.name)
self.parameters[name.name] = name
return name
if six.PY2:
spec = inspect.getargspec(self.payload)
if isinstance(name, int):
if 0 <= name < len(spec.args):
name = spec.args[name]
elif name == inspect.getargspec(self.payload) \
and spec.varargs is not None:
name = spec.varargs
else:
raise IndexError('argument position is out of range')
arg_name = name
if name == spec.keywords:
position = None
arg_name = '**'
elif name == spec.varargs:
position = len(spec.args)
arg_name = '*'
elif name not in spec.args:
raise exceptions.NoParameterFoundException(
function_name=self.name or self.payload.__name__,
param_name=name)
else:
position = spec.args.index(name)
default = NO_DEFAULT
if spec.defaults is not None and name in spec.args:
index = spec.args.index(name) - len(spec.args)
if index >= -len(spec.defaults):
default = spec.defaults[index]
else:
spec = inspect.getfullargspec(self.payload)
if isinstance(name, int):
if 0 <= name < len(spec.args):
name = spec.args[name]
elif name == inspect.getargspec(self.payload) \
and spec.varargs is not None:
name = spec.varargs
else:
raise IndexError('argument position is out of range')
arg_name = name
if name == spec.varkw:
position = None
arg_name = '**'
elif name == spec.varargs:
position = len(spec.args)
arg_name = '*'
elif name in spec.kwonlyargs:
position = None
elif name not in spec.args:
raise exceptions.NoParameterFoundException(
function_name=self.name or self.payload.__name__,
param_name=name)
else:
position = spec.args.index(name)
default = NO_DEFAULT
if spec.defaults is not None and name in spec.args:
index = spec.args.index(name) - len(spec.args)
if index >= -len(spec.defaults):
default = spec.defaults[index]
elif spec.kwonlydefaults is not None:
default = spec.kwonlydefaults.get(name, NO_DEFAULT)
if arg_name in self.parameters and not overwrite:
raise exceptions.DuplicateParameterDecoratorException(
function_name=self.name or self.payload.__name__,
param_name=name)
yaql_type = value_type
p_nullable = nullable
if value_type is None:
if p_nullable is None:
p_nullable = True
base_type = object \
if default in (None, NO_DEFAULT, utils.NO_VALUE) \
else type(default)
yaql_type = yaqltypes.PythonType(base_type, p_nullable)
elif not isinstance(value_type, yaqltypes.SmartType):
if p_nullable is None:
p_nullable = default is None
yaql_type = yaqltypes.PythonType(value_type, p_nullable)
pd = ParameterDefinition(
name, yaql_type, position, alias, default
)
self.parameters[arg_name] = pd
return pd
def insert_parameter(self, name, value_type=None, nullable=None,
alias=None, overwrite=False):
pd = self.set_parameter(name, value_type, nullable, alias, overwrite)
for p in self.parameters.itervalues():
if p is pd:
continue
if p.position is not None and p.position >= pd.position:
p.position += 1
def map_args(self, args, kwargs, context):
kwargs = dict(kwargs)
positional_args = len(args) * [
self.parameters.get('*', utils.NO_VALUE)]
@ -126,27 +250,29 @@ class FunctionDefinition(object):
value = args[i]
if value is utils.NO_VALUE:
value = positional_args[i].default
if not positional_args[i].value_type.check(value):
if not positional_args[i].value_type.check(value, context):
return None
for kwd in six.iterkeys(kwargs):
if not keyword_args[kwd].value_type.check(kwargs[kwd]):
if not keyword_args[kwd].value_type.check(kwargs[kwd], context):
return None
return tuple(positional_args), keyword_args
def get_delegate(self, sender, engine, args, kwargs):
def get_delegate(self, sender, engine, context, args, kwargs):
def checked(val, param):
if not param.value_type.check(val):
if not param.value_type.check(val, context):
raise exceptions.ArgumentException(param.name)
def convert_arg_func(context):
def convert_arg_func(context2):
try:
return param.value_type.convert(
val, sender, context, self, engine)
val, sender, context2, self, engine)
except exceptions.ArgumentValueException:
raise exceptions.ArgumentException(param.name)
return convert_arg_func
kwargs = kwargs.copy()
kwargs = dict(kwargs)
positional = 0
for arg_name, p in six.iteritems(self.parameters):
if p.position is not None and arg_name != '*':
@ -207,7 +333,7 @@ class FunctionDefinition(object):
else:
raise exceptions.ArgumentException('**')
def func(context):
def func():
new_context = context.create_child_context()
result = self.payload(
*tuple(map(lambda t: t(new_context),
@ -215,14 +341,7 @@ class FunctionDefinition(object):
**dict(map(lambda t: (t[0], t[1](new_context)),
six.iteritems(keyword_args)))
)
if self.returns_context:
if isinstance(result, types.GeneratorType):
result_context = next(result)
return result, result_context
result_value, result_context = result
return result_value, result_context
else:
return result, new_context
return result
return func
@ -241,40 +360,53 @@ class FunctionDefinition(object):
def _get_function_definition(func):
if not hasattr(func, '__yaql_function__'):
fd = FunctionDefinition(None, {}, func, func.__doc__)
fd = FunctionDefinition(None, func, {}, func.__doc__)
func.__yaql_function__ = fd
return func.__yaql_function__
def _infer_parameter_type(name):
if name == 'context' or name == '__context':
return yaqltypes.Context()
elif name == 'engine' or name == '__engine':
return yaqltypes.Engine()
def get_function_definition(func, name=None, function=None, method=None,
convention=None):
convention=None, parameter_type_func=None):
if parameter_type_func is None:
parameter_type_func = _infer_parameter_type
fd = _get_function_definition(func).clone()
if six.PY2:
spec = inspect.getargspec(func)
for arg in spec.args:
if arg not in fd.parameters:
parameter(arg, function_definition=fd)(func)
fd.set_parameter(arg, parameter_type_func(arg))
if spec.varargs and '*' not in fd.parameters:
parameter(spec.varargs, function_definition=fd)(func)
fd.set_parameter(spec.varargs, parameter_type_func(spec.varargs))
if spec.keywords and '**' not in fd.parameters:
parameter(spec.keywords, function_definition=fd)(func)
fd.set_parameter(spec.keywords, parameter_type_func(spec.keywords))
else:
spec = inspect.getfullargspec(func)
for arg in spec.args + spec.kwonlyargs:
if arg not in fd.parameters:
parameter(arg, function_definition=fd)(func)
fd.set_parameter(arg, parameter_type_func(arg))
if spec.varargs and '*' not in fd.parameters:
parameter(spec.varargs, function_definition=fd)(func)
fd.set_parameter(spec.varargs, parameter_type_func(spec.varargs))
if spec.varkw and '**' not in fd.parameters:
parameter(spec.varkw, function_definition=fd)(func)
fd.set_parameter(spec.varkw, parameter_type_func(spec.varkw))
if name is not None:
fd.name = name
elif fd.name is None:
if convention is not None:
fd.name = convention.convert_function_name(fd.payload.__name__)
fd.name = convention.convert_function_name(
fd.payload.__name__.rstrip('_'))
else:
fd.name = fd.payload.__name__
fd.name = fd.payload.__name__.rstrip('_')
elif convention is not None:
fd.name = convention.convert_function_name(fd.name.rstrip('_'))
if function is not None:
fd.is_function = function
if method is not None:
@ -282,109 +414,31 @@ def get_function_definition(func, name=None, function=None, method=None,
if convention:
for p in six.itervalues(fd.parameters):
if p.alias is None:
p.alias = convention.convert_parameter_name(p.name)
p.alias = convention.convert_parameter_name(p.name.rstrip('_'))
return fd
def _parameter(name, value_type=None, nullable=None, alias=None,
function_definition=None):
def _parameter(name, value_type=None, nullable=None, alias=None):
def wrapper(func):
fd = function_definition or _get_function_definition(func)
if six.PY2:
spec = inspect.getargspec(func)
arg_name = name
if name == spec.keywords:
position = None
arg_name = '**'
elif name == spec.varargs:
position = len(spec.args)
arg_name = '*'
elif name not in spec.args:
raise exceptions.NoParameterFoundException(
function_name=fd.name or func.__name__,
param_name=name)
else:
position = spec.args.index(name)
default = NO_DEFAULT
if spec.defaults is not None and name in spec.args:
index = spec.args.index(name) - len(spec.args)
if index >= -len(spec.defaults):
default = spec.defaults[index]
else:
spec = inspect.getfullargspec(func)
arg_name = name
if name == spec.varkw:
position = None
arg_name = '**'
elif name == spec.varargs:
position = len(spec.args)
arg_name = '*'
elif name in spec.kwonlyargs:
position = None
elif name not in spec.args:
raise exceptions.NoParameterFoundException(
function_name=fd.name or func.__name__,
param_name=name)
else:
position = spec.args.index(name)
default = NO_DEFAULT
if spec.defaults is not None and name in spec.args:
index = spec.args.index(name) - len(spec.args)
if index >= -len(spec.defaults):
default = spec.defaults[index]
elif spec.kwonlydefaults is not None:
default = spec.kwonlydefaults.get(name, NO_DEFAULT)
if arg_name in fd.parameters:
raise exceptions.DuplicateParameterDecoratorException(
function_name=fd.name or func.__name__,
param_name=name)
yaql_type = value_type
p_nullable = nullable
if value_type is None:
if p_nullable is None:
p_nullable = True
if name == 'context':
yaql_type = yaqltypes.Context()
elif name == 'engine':
yaql_type = yaqltypes.Engine()
else:
base_type = object \
if default in (None, NO_DEFAULT, utils.NO_VALUE) \
else type(default)
yaql_type = yaqltypes.PythonType(base_type, p_nullable)
elif not isinstance(value_type, yaqltypes.SmartType):
if p_nullable is None:
p_nullable = default is None
yaql_type = yaqltypes.PythonType(value_type, p_nullable)
fd.parameters[arg_name] = ParameterDefinition(
name, yaql_type, position, alias, default
)
fd = _get_function_definition(func)
fd.set_parameter(name, value_type, nullable, alias)
return func
return wrapper
def parameter(name, value_type=None, nullable=None, alias=None,
function_definition=None):
def parameter(name, value_type=None, nullable=None, alias=None):
if value_type is not None and isinstance(
value_type, yaqltypes.HiddenParameterType):
raise ValueError('Use inject() for hidden parameters')
return _parameter(name, value_type, nullable=nullable, alias=alias,
function_definition=function_definition)
return _parameter(name, value_type, nullable=nullable, alias=alias)
def inject(name, value_type=None, nullable=None, alias=None,
function_definition=None):
def inject(name, value_type=None, nullable=None, alias=None):
if value_type is not None and not isinstance(
value_type, yaqltypes.HiddenParameterType):
raise ValueError('Use parameter() for normal function parameters')
return _parameter(name, value_type, nullable=nullable, alias=alias,
function_definition=function_definition)
return _parameter(name, value_type, nullable=nullable, alias=alias)
def name(function_name):
@ -409,13 +463,15 @@ def extension_method(func):
return func
def returns_context(func):
fd = _get_function_definition(func)
fd.returns_context = True
return func
def no_kwargs(func):
fd = _get_function_definition(func)
fd.no_kwargs = True
return func
def meta(name, value):
def wrapper(func):
fd = _get_function_definition(func)
fd.meta[name] = value
return func
return wrapper

16
yaql/language/utils.py
View File

@ -202,3 +202,19 @@ def limit_memory_usage(engine, *args):
total += t[0] * sys.getsizeof(t[1], 0)
if total > quota:
raise exceptions.MemoryQuotaExceededException()
def to_extension_method(name, context):
layers = context.collect_functions(
name,
lambda t, ctx: not t.is_function or not t.is_method,
use_convention=True)
if len(layers) > 1:
raise ValueError(
'Multi layer functions are not supported by this helper method')
if len(layers) > 0:
for spec in layers[0]:
spec = spec.clone()
spec.is_function = True
spec.is_method = True
yield spec

136
yaql/language/yaqltypes.py
View File

@ -23,7 +23,7 @@ from yaql.language import utils
class HiddenParameterType(object):
# noinspection PyMethodMayBeStatic,PyUnusedLocal
def check(self, value):
def check(self, value, context):
return True
@ -35,13 +35,13 @@ class SmartType(object):
def __init__(self, nullable):
self.nullable = nullable
def check(self, value):
def check(self, value, context):
if value is None and not self.nullable:
return False
return True
def convert(self, value, sender, context, function_spec, engine):
if not self.check(value):
if not self.check(value, context):
raise exceptions.ArgumentValueException()
utils.limit_memory_usage(engine, (1, value))
@ -49,7 +49,35 @@ class SmartType(object):
return False
class PythonType(SmartType):
class GenericType(SmartType):
def __init__(self, nullable, checker=None, converter=None):
super(GenericType, self).__init__(nullable)
self.checker = checker
self.converter = converter
def check(self, value, context):
if isinstance(value, expressions.Constant):
value = value.value
if not super(GenericType, self).check(value, context):
return False
if value is None or isinstance(value, expressions.Expression):
return True
if not self.checker:
return True
return self.checker(value, context)
def convert(self, value, sender, context, function_spec, engine):
if isinstance(value, expressions.Constant):
value = value.value
super(GenericType, self).convert(
value, sender, context, function_spec, engine)
if value is None or not self.converter:
return value
return self.converter(value, sender, context, function_spec, engine)
class PythonType(GenericType):
def __init__(self, python_type, nullable=True, validators=None):
self.python_type = python_type
if not validators:
@ -57,45 +85,39 @@ class PythonType(SmartType):
if not isinstance(validators, (list, tuple)):
validators = [validators]
self.validators = validators
super(PythonType, self).__init__(nullable)
def check(self, value):
if isinstance(value, expressions.Constant):
value = value.value
return super(PythonType, self).check(value) and (
value is None or isinstance(value, expressions.Expression) or (
isinstance(value, self.python_type) and all(
map(lambda t: t(value), self.validators))))
def convert(self, value, sender, context, function_spec, engine):
super(PythonType, self).convert(value, sender, context,
function_spec, engine)
if isinstance(value, expressions.Constant):
value = value.value
super(PythonType, self).convert(value, sender, context,
function_spec, engine)
return value
super(PythonType, self).__init__(
nullable,
lambda value, context: isinstance(
value, self.python_type) and all(
map(lambda t: t(value), self.validators)))
def is_specialization_of(self, other):
return (
isinstance(other, PythonType)
and issubclass(self.python_type, other.python_type)
and not issubclass(other.python_type, self.python_type)
)
if not isinstance(other, PythonType):
return False
try:
len(self.python_type)
len(other.python_type)
except Exception:
return (
issubclass(self.python_type, other.python_type)
and not issubclass(other.python_type, self.python_type)
)
else:
return False
class MappingRule(LazyParameterType, SmartType):
def __init__(self):
super(MappingRule, self).__init__(False)
def check(self, value):
def check(self, value, context):
return isinstance(value, expressions.MappingRuleExpression)
def convert(self, value, sender, context, function_spec, engine):
super(MappingRule, self).convert(value, sender, context,
function_spec, engine)
wrap = lambda func: lambda: func(sender, context, engine)[0]
wrap = lambda func: lambda: func(sender, context, engine)
return utils.MappingRule(wrap(value.source), wrap(value.destination))
@ -145,17 +167,16 @@ class Number(PythonType):
class Lambda(LazyParameterType, SmartType):
def __init__(self, with_context=False, method=False, return_context=False):
def __init__(self, with_context=False, method=False):
super(Lambda, self).__init__(True)
self.return_context = return_context
self.with_context = with_context
self.method = method
def check(self, value):
def check(self, value, context):
if self.method and isinstance(
value, expressions.Expression) and not value.uses_sender:
return False
return super(Lambda, self).check(value)
return super(Lambda, self).check(value, context)
@staticmethod
def _publish_params(context, args, kwargs):
@ -168,17 +189,17 @@ class Lambda(LazyParameterType, SmartType):
self._publish_params(context, args, kwargs)
if isinstance(value, expressions.Expression):
result = value(sender, context, engine)
elif six.callable(value):
result = value, context
else:
result = value, context
return result[0] if not self.return_context else result
return result
def convert(self, value, sender, context, function_spec, engine):
super(Lambda, self).convert(value, sender, context,
function_spec, engine)
if value is None:
return None
elif six.callable(value) and hasattr(value, '__unwrapped__'):
value = value.__unwrapped__
def func(*args, **kwargs):
if self.method and self.with_context:
@ -198,13 +219,12 @@ class Lambda(LazyParameterType, SmartType):
return self._call(value, new_sender, new_context,
engine, args, kwargs)
func.__unwrapped__ = value
return func
class Super(HiddenParameterType, SmartType):
def __init__(self, with_context=False, method=None, return_context=False,
with_name=False):
self.return_context = return_context
def __init__(self, with_context=False, method=None, with_name=False):
self.with_context = with_context
self.method = method
self.with_name = with_name
@ -221,6 +241,9 @@ class Super(HiddenParameterType, SmartType):
spec.name)
def convert(self, value, sender, context, function_spec, engine):
if six.callable(value) and hasattr(value, '__unwrapped__'):
value = value.__unwrapped__
def func(*args, **kwargs):
function_context = self._find_function_context(
function_spec, context)
@ -248,8 +271,8 @@ class Super(HiddenParameterType, SmartType):
new_context = context.create_child_context()
return parent_function_context(
new_name, engine, new_sender, new_context,
self.return_context)(*args, **kwargs)
new_name, engine, new_sender, new_context)(*args, **kwargs)
func.__unwrapped__ = value
return func
@ -262,15 +285,16 @@ class Context(HiddenParameterType, SmartType):
class Delegate(HiddenParameterType, SmartType):
def __init__(self, name=None, with_context=False, method=False,
return_context=False):
def __init__(self, name=None, with_context=False, method=False):
super(Delegate, self).__init__(False)
self.name = name
self.return_context = return_context
self.with_context = with_context
self.method = method
def convert(self, value, sender, context, function_spec, engine):
if six.callable(value) and hasattr(value, '__unwrapped__'):
value = value.__unwrapped__
def func(*args, **kwargs):
name = self.name
if not name:
@ -288,8 +312,8 @@ class Delegate(HiddenParameterType, SmartType):
new_context = context.create_child_context()
return new_context(
name, engine, new_sender, return_context=self.return_context,
use_convention=True)(*args, **kwargs)
name, engine, new_sender, use_convention=True)(*args, **kwargs)
func.__unwrapped__ = value
return func
@ -322,8 +346,8 @@ class Constant(SmartType):
self.expand = expand
super(Constant, self).__init__(nullable)
def check(self, value):
return super(Constant, self).check(value.value) and (
def check(self, value, context):
return super(Constant, self).check(value.value, context) and (
value is None or isinstance(value, expressions.Constant))
def convert(self, value, sender, context, function_spec, engine):
@ -336,7 +360,7 @@ class YaqlExpression(LazyParameterType, SmartType):
def __init__(self):
super(YaqlExpression, self).__init__(False)
def check(self, value):
def check(self, value, context):
return isinstance(value, expressions.Expression)
def convert(self, value, sender, context, function_spec, engine):
@ -349,8 +373,8 @@ class StringConstant(Constant):
def __init__(self, nullable=False):
super(StringConstant, self).__init__(nullable)
def check(self, value):
return super(StringConstant, self).check(value) and (
def check(self, value, context):
return super(StringConstant, self).check(value, context) and (
value is None or isinstance(value.value, six.string_types))
@ -358,7 +382,7 @@ class Keyword(Constant):
def __init__(self, expand=True):
super(Keyword, self).__init__(False, expand)
def check(self, value):
def check(self, value, context):
return isinstance(value, expressions.KeywordConstant)
@ -366,8 +390,8 @@ class BooleanConstant(Constant):
def __init__(self, nullable=False, expand=True):
super(BooleanConstant, self).__init__(nullable, expand)
def check(self, value):
return super(BooleanConstant, self).check(value) and (
def check(self, value, context):
return super(BooleanConstant, self).check(value, context) and (
value is None or type(value.value) is bool)
@ -375,8 +399,8 @@ class NumericConstant(Constant):
def __init__(self, nullable=False, expand=True):
super(NumericConstant, self).__init__(nullable, expand)
def check(self, value):
return super(NumericConstant, self).check(value) and (
def check(self, value, context):
return super(NumericConstant, self).check(value, context) and (
value is None or isinstance(
value.value, six.integer_types + (float,)) and
type(value.value) is not bool)

5
yaql/legacy.py
View File

@ -18,9 +18,10 @@ from yaql.standard_library import legacy as std_legacy
class YaqlFactory(factory.YaqlFactory):
def __init__(self):
def __init__(self, allow_delegates=False):
# noinspection PyTypeChecker
super(YaqlFactory, self).__init__(keyword_operator=None)
super(YaqlFactory, self).__init__(
keyword_operator=None, allow_delegates=allow_delegates)
self.insert_operator(
'or', True, '=>',
factory.OperatorType.BINARY_LEFT_ASSOCIATIVE, True)

12
yaql/standard_library/collections.py
View File

@ -33,6 +33,17 @@ def list_(delegate, *args):
return delegate(rec(args))
@specs.method
@specs.parameter('collection', yaqltypes.Iterable())
def flatten(collection):
for t in collection:
if utils.is_iterable(t):
for t2 in flatten(t):
yield t2
else:
yield t
@specs.method
@specs.parameter('collection', yaqltypes.Iterable())
def to_list(collection):
@ -499,6 +510,7 @@ def register(context, no_sets=False):
context.register_function(list_)
context.register_function(build_list)
context.register_function(to_list)
context.register_function(flatten)
context.register_function(list_indexer)
context.register_function(dict_)
context.register_function(dict_, name='#map')

31
yaql/standard_library/legacy.py
View File

@ -16,6 +16,7 @@ import itertools
import six
from yaql.language import contexts
from yaql.language import expressions
from yaql.language import specs
from yaql.language import utils
@ -26,9 +27,9 @@ from yaql.language import yaqltypes
@specs.name('#operator_=>')
def build_tuple(left, right, context, engine):
if isinstance(left, expressions.BinaryOperator) and left.operator == '=>':
return left(utils.NO_VALUE, context, engine)[0] + (right,)
return left(utils.NO_VALUE, context, engine) + (right,)
else:
return left(utils.NO_VALUE, context, engine)[0], right
return left(utils.NO_VALUE, context, engine), right
@specs.parameter('tuples', tuple)
@ -82,6 +83,13 @@ def switch(*conditions):
return None
@specs.parameter('sender', contexts.ContextBase)
@specs.parameter('expr', yaqltypes.Lambda(with_context=True, method=True))
@specs.name('#operator_.')
def op_dot_context(sender, expr):
return expr(sender['$0'], sender)
@specs.parameter('mappings', yaqltypes.Lambda())
@specs.method
def as_(context, sender, *mappings):
@ -92,17 +100,8 @@ def as_(context, sender, *mappings):
if len(tt) != 2:
raise ValueError('as() tuples must be of size 2')
context[tt[1]] = tt[0]
return sender
def _to_extension_method(name, context):
for spec in context.parent.get_functions(
name, lambda t: not t.is_function or not t.is_method,
use_convention=True):
spec = spec.clone()
spec.is_function = True
spec.is_method = True
context.register_function(spec)
context['$0'] = sender
return context
def register(context, tuples):
@ -117,7 +116,9 @@ def register(context, tuples):
context.register_function(range_)
context.register_function(switch, exclusive=True)
context.register_function(as_)
context.register_function(op_dot_context)
for t in ('get', 'list', 'bool', 'int', 'float', 'select', 'where',
'join', 'sum', 'take_while'):
_to_extension_method(t, context)
'join', 'sum', 'take_while', 'len'):
for spec in utils.to_extension_method(t, context):
context.register_function(spec)

4
yaql/standard_library/math.py
View File

@ -118,10 +118,14 @@ def neq(left, right):
def int_(value):
if value is None:
return 0
return int(value)
def float_(value):
if value is None:
return 0.0
return float(value)

8
yaql/standard_library/regex.py
View File

@ -42,6 +42,13 @@ def matches(regexp, string):
return regexp.search(string) is not None
@specs.parameter('string', yaqltypes.String())
@specs.parameter('regexp', yaqltypes.String())
@specs.method
def matches_(string, regexp):
return re.search(regexp, string) is not None
@specs.parameter('regexp', REGEX_TYPE)
@specs.parameter('string', yaqltypes.String())
@specs.name('#operator_=~')
@ -189,6 +196,7 @@ def escape_regex(string):
def register(context):
context.register_function(regex)
context.register_function(matches)
context.register_function(matches_)
context.register_function(matches_operator_string)
context.register_function(matches_operator_regex)
context.register_function(not_matches_operator_string)

20
yaql/standard_library/strings.py
View File

@ -103,9 +103,18 @@ def right_split(string, separator=None, max_splits=-1):
@specs.parameter('sequence', yaqltypes.Iterable())
@specs.parameter('separator', yaqltypes.String())
@specs.inject('str_delegate', yaqltypes.Delegate('str'))
@specs.method
def join(sequence, separator):
return separator.join(sequence)
def join(sequence, separator, str_delegate):
return separator.join(six.moves.map(str_delegate, sequence))
@specs.parameter('sequence', yaqltypes.Iterable())
@specs.parameter('separator', yaqltypes.String())
@specs.inject('str_delegate', yaqltypes.Delegate('str'))
@specs.method
def join_(separator, sequence, str_delegate):
return join(sequence, separator, str_delegate)
@specs.parameter('value', nullable=True)
@ -184,10 +193,10 @@ def replace_with_dict(string, str_func, replacements, count=-1):
return string
@specs.parameter('__format_string__', yaqltypes.String())
@specs.parameter('__format_string', yaqltypes.String())
@specs.extension_method
def format_(__format_string__, *args, **kwargs):
return __format_string__.format(*args, **kwargs)
def format_(__format_string, *args, **kwargs):
return __format_string.format(*args, **kwargs)
@specs.parameter('left', yaqltypes.String())
@ -342,6 +351,7 @@ def register(context):
context.register_function(split)
context.register_function(right_split)
context.register_function(join)
context.register_function(join_)
context.register_function(str_)
context.register_function(concat)
context.register_function(concat, name='#operator_+')

49
yaql/standard_library/system.py
View File

@ -16,6 +16,7 @@ import itertools
import six
from yaql.language import contexts
from yaql.language import specs
from yaql.language import utils
from yaql.language import yaqltypes
@ -27,26 +28,19 @@ def get_context_data(name, context):
return context[name]
@specs.parameter('sender', yaqltypes.Lambda(with_context=True,
return_context=True))
@specs.parameter('expr', yaqltypes.Lambda(with_context=True, method=True,
return_context=True))
@specs.parameter('expr', yaqltypes.Lambda(method=True))
@specs.name('#operator_.')
@specs.returns_context
def op_dot(context, sender, expr):
return expr(*sender(context))
def op_dot(sender, expr):
return expr(sender)
@specs.parameter('sender',
yaqltypes.Lambda(with_context=True, return_context=True))
@specs.parameter('expr', yaqltypes.YaqlExpression())
@specs.inject('operator', yaqltypes.Delegate('#operator_.', with_context=True))
@specs.inject('operator', yaqltypes.Delegate('#operator_.'))
@specs.name('#operator_?.')
def elvis_operator(context, operator, sender, expr):
sender, context = sender(context)
def elvis_operator(operator, sender, expr):
if sender is None:
return None
return operator(context, sender, expr)
return operator(sender, expr)
@specs.parameter('sequence', yaqltypes.Iterable())
@ -63,12 +57,13 @@ def unpack(sequence, context, *args):
else:
for i, t in enumerate(sequence, 1):
context[str(i)] = t
return lst
return context
def with_(context, *args):
for i, t in enumerate(args, 1):
context[str(i)] = t
return context
@specs.inject('__context__', yaqltypes.Context())
@ -78,6 +73,7 @@ def let(__context__, *args, **kwargs):
for key, value in six.iteritems(kwargs):
__context__[key] = value
return __context__
@specs.parameter('name', yaqltypes.String())
@ -88,20 +84,20 @@ def def_(name, func, context):
return func(*args, **kwargs)
context.register_function(wrapper)
return context
@specs.parameter('left', yaqltypes.Lambda(return_context=True))
@specs.parameter('left', contexts.ContextBase)
@specs.parameter('right', yaqltypes.Lambda(with_context=True))
@specs.name('#operator_->')
def send_context(left, right):
context = left()[1]
return right(context)
return right(left)
@specs.method
@specs.parameter('condition', yaqltypes.Lambda())
@specs.parameter('message', yaqltypes.String())
def assert_(engine, obj, condition, message=u'Assertion failed'):
def assert__(engine, obj, condition, message=u'Assertion failed'):
if utils.is_iterator(obj):
obj = utils.memorize(obj, engine)
if not condition(obj):
@ -109,7 +105,17 @@ def assert_(engine, obj, condition, message=u'Assertion failed'):
return obj
def register(context):
@specs.name('#call')
def call(callable_, *args, **kwargs):
return callable_(*args, **kwargs)
@specs.parameter('func', yaqltypes.Lambda())
def lambda_(func):
return func
def register(context, delegates=False):
context.register_function(get_context_data)
context.register_function(op_dot)
context.register_function(unpack)
@ -118,4 +124,7 @@ def register(context):
context.register_function(let)
context.register_function(def_)
context.register_function(elvis_operator)
context.register_function(assert_)
context.register_function(assert__)
if delegates:
context.register_function(call)
context.register_function(lambda_)

4
yaql/tests/__init__.py
View File

@ -38,7 +38,7 @@ class TestCase(testtools.TestCase):
def create_engine(self):
func = TestCase._default_engine
if func is None:
engine_factory = factory.YaqlFactory()
engine_factory = factory.YaqlFactory(allow_delegates=True)
TestCase._default_engine = func = engine_factory.create(
options=self.engine_options)
return func
@ -54,7 +54,7 @@ class TestCase(testtools.TestCase):
@property
def context(self):
if self._context is None:
self._context = yaql.create_context()
self._context = yaql.create_context(delegates=True)
return self._context
@property

7
yaql/tests/test_collections.py
View File

@ -96,8 +96,9 @@ class TestCollections(yaql.tests.TestCase):
self.eval('$.toDict($, $*$)', data=[1, 2, 3]))
def test_keyword_dict_access(self):
data = {'a': 12, 'b c': 44}
self.assertEqual(12, self.eval('$.a', data=data))
data = {'A': 12, 'b c': 44, '__d': 99, '_e': 999}
self.assertEqual(12, self.eval('$.A', data=data))
self.assertEqual(999, self.eval('$._e', data=data))
self.assertRaises(exceptions.NoMatchingFunctionException,
self.eval, "$.'b c'", data=data)
@ -105,6 +106,8 @@ class TestCollections(yaql.tests.TestCase):
self.eval, "$.123", data=data)
self.assertRaises(KeyError,
self.eval, "$.b", data=data)
self.assertRaises(exceptions.YaqlLexicalException,
self.eval, "$.__d", data=data)
def test_keyword_collection_access(self):
data = [{'a': 2}, {'a': 4}]

9
yaql/tests/test_engine.py
View File

@ -19,7 +19,6 @@ import six
import yaql
from yaql.language import exceptions
from yaql.language import specs
from yaql.language import utils
from yaql.language import yaqltypes
from yaql import tests
@ -31,11 +30,11 @@ class TestEngine(tests.TestCase):
sys.stderr = six.StringIO()
try:
debug_opts = dict(self.engine_options)
debug_opts["yaql.debug"] = True
debug_opts['yaql.debug'] = True
yaql.factory.YaqlFactory().create(options=debug_opts)
sys.stderr.seek(0)
err_out = sys.stderr.read()
self.assertEqual("Generating LALR tables\n", err_out)
self.assertEqual('Generating LALR tables\n', err_out)
finally:
# put stderr back
sys.stderr = copy
@ -123,12 +122,12 @@ class TestEngine(tests.TestCase):
self.assertEqual(
(1, 2, (5, 7), {'kw1': 'x', 'kw2': None}),
fd(utils.NO_VALUE, self.engine, self.context)(
fd(self.engine, self.context)(
1, 2, 5, 7, kw1='x', kw2=None))
self.assertEqual(
(1, 5, (), {}),
fd(utils.NO_VALUE, self.engine, self.context)(1, b=5))
fd(self.engine, self.context)(1, b=5))
def test_eval(self):
self.assertEqual(

28
yaql/tests/test_legacy.py
View File

@ -50,29 +50,29 @@ class TestLegacy(yaql.tests.TestCase):
def test_int(self):
self.assertEqual(5, self.eval("'5'.int()"))
self.assertEqual(5, self.eval("5.2.int()"))
self.assertEqual(5, self.eval("int('5')"))
self.assertEqual(5, self.eval("int(5.2)"))
self.assertEqual(5, self.eval('5.2.int()'))
self.assertEqual(0, self.eval('null.int()'))
def test_float(self):
self.assertAlmostEqual(5.1, self.eval("'5.1'.float()"))
self.assertAlmostEqual(5.0, self.eval("5.float()"))
self.assertAlmostEqual(5.0, self.eval('5.float()'))
self.assertAlmostEqual(5.1, self.eval("float('5.1')"))
self.assertAlmostEqual(5.0, self.eval("float(5)"))
self.assertEqual(0, self.eval('null.float()'))
def test_bool(self):
self.assertFalse(self.eval("null.bool()"))
self.assertFalse(self.eval('null.bool()'))
self.assertFalse(self.eval("''.bool()"))
self.assertFalse(self.eval("0.bool()"))
self.assertFalse(self.eval("false.bool()"))
self.assertFalse(self.eval("[].bool()"))
self.assertFalse(self.eval("{}.bool()"))
self.assertFalse(self.eval('0.bool()'))
self.assertFalse(self.eval('false.bool()'))
self.assertFalse(self.eval('[].bool()'))
self.assertFalse(self.eval('{}.bool()'))
self.assertTrue(self.eval("' '.bool()"))
self.assertTrue(self.eval("x.bool()"))
self.assertTrue(self.eval("1.bool()"))
self.assertTrue(self.eval("true.bool()"))
self.assertTrue(self.eval("[1].bool()"))
self.assertTrue(self.eval("{a=>b}.bool()"))
self.assertTrue(self.eval('x.bool()'))
self.assertTrue(self.eval('1.bool()'))
self.assertTrue(self.eval('true.bool()'))
self.assertTrue(self.eval('[1].bool()'))
self.assertTrue(self.eval('{a=>b}.bool()'))
def test_filter(self):
self.assertEqual(2, self.eval("list(1,2,3)[1]"))

10
yaql/tests/test_math.py
View File

@ -123,12 +123,14 @@ class TestMath(yaql.tests.TestCase):
self.assertIsInstance(res, bool)
self.assertTrue(res)
self.assertFalse(self.eval('3 < 3'))
self.assertTrue(self.eval('2.5 < 3'))
def test_gte(self):
res = self.eval('5 >= 3')
self.assertIsInstance(res, bool)
self.assertTrue(res)
self.assertTrue(self.eval('3 >= 3'))
self.assertTrue(self.eval('3.5 > 3'))
self.assertFalse(self.eval('2 >= 3'))
def test_lte(self):
@ -140,10 +142,12 @@ class TestMath(yaql.tests.TestCase):
def test_eq(self):
self.assertTrue(self.eval('5 = 5'))
self.assertTrue(self.eval('1.0 = 1'))
self.assertFalse(self.eval('5 = 6'))
def test_neq(self):
self.assertFalse(self.eval('5 != 5'))
self.assertFalse(self.eval('0 != 0.0'))
self.assertTrue(self.eval('5 != 6'))
def test_zero_division(self):
@ -153,8 +157,14 @@ class TestMath(yaql.tests.TestCase):
self.assertTrue(self.eval('with(random()) -> $ >= 0 and $ < 1'))
self.assertTrue(self.eval('with(random(2, 5)) -> $ >= 2 and $ <= 5'))
def test_int(self):
self.assertEqual(5, self.eval("int('5')"))
self.assertEqual(5, self.eval('int(5.2)'))
self.assertEqual(0, self.eval('int(null)'))
def test_float(self):
self.assertAlmostEqual(-1.23, self.eval("float('-1.23')"))
self.assertEqual(0.0, self.eval('float(null)'))
def test_bitwise_or(self):
self.assertEqual(3, self.eval('bitwiseOr(1, 3)'))

4
yaql/tests/test_regex.py
View File

@ -20,6 +20,10 @@ class TestRegex(yaql.tests.TestCase):
self.assertTrue(self.eval("regex('a.b').matches(axb)"))
self.assertFalse(self.eval("regex('a.b').matches(abx)"))
def test_matches_string_method(self):
self.assertTrue(self.eval("axb.matches('a.b')"))
self.assertFalse(self.eval("abx.matches('a.b')"))
def test_matches_operator_regex(self):
self.assertTrue(self.eval("axb =~ regex('a.b')"))
self.assertFalse(self.eval("abx =~ regex('a.b')"))

3
yaql/tests/test_strings.py
View File

@ -68,6 +68,9 @@ class TestStrings(yaql.tests.TestCase):
def test_join(self):
self.assertEqual('some-text', self.eval("[some, text].join('-')"))
def test_join_pythonic(self):
self.assertEqual('some-text', self.eval("'-'.join([some, text])"))
def test_is_empty(self):
self.assertTrue(self.eval("isEmpty('')"))
self.assertTrue(self.eval("isEmpty(null)"))

69
yaql/tests/test_system.py
View File

@ -66,3 +66,72 @@ class TestSystem(yaql.tests.TestCase):
self.assertEqual(
3,
self.eval('[2].select($ + 1).assert(len($) = 1).first()'))
def test_lambda_passing(self):
delegate = lambda x: x ** 2
self.assertEqual(
9,
self.eval('$(3)', data=delegate))
def test_function_passing(self):
def func(x, y, z):
return (x - y) * z
context = self.context
context['func'] = func
self.assertEqual(
8,
self.eval('$func(5, z => 2, y => 1)', data=func))
def test_lambda_expression(self):
delegate = lambda x: x ** 2
self.assertEqual(
9,
self.eval('$.x[0](3)', data={'x': [delegate]}))
self.assertEqual(
9,
self.eval('($.x[0])(3)', data={'x': [delegate]}))
def test_2nd_order_lambda(self):
delegate = lambda y: lambda x: x ** y
self.assertEqual(
16,
self.eval('$(2)(4)', data=delegate))
def test_2nd_order_lambda_expression(self):
delegate = lambda y: {'key': lambda x: x ** y}
self.assertEqual(
16,
self.eval('$(2)[key](4)', data=delegate))
def test_2nd_order_lambda_collection_expression(self):
delegate = lambda y: lambda x: y ** x
self.assertEqual(
[1, 8, 27],
self.eval(
'let(func => $) -> [1, 2, 3].select($func($)).select($(3))',
data=delegate))
def test_lambda_func(self):
self.assertEqual(
[2, 4, 6],
self.eval('let(func => lambda(2 * $)) -> $.select($func($))',
data=[1, 2, 3]))
def test_lambda_func_2nd_order(self):
self.assertEqual(
5,
self.eval('lambda(let(outer => $) -> lambda($outer - $))(7)(2)'))
def test_lambda_closure(self):
data = [1, 2, 3, 4, 5, 6]
self.assertEqual([3, 4, 5, 6], self.eval(
'$.where(lambda($ > 3)($+1))',
data=data))
# lambda can access value from "where"'s context
# so we can omit parameter
self.assertEqual([4, 5, 6], self.eval(
'$.where(lambda($ > 3)())',
data=data))