Small improvements to yaql

Contexts:
* context module was renamed to contexts
* convention property is now part of ContextBase class.
* Standard Context implementation automatically uses convention
  from parent context
* Context interface was enhanced to add capability to check key presence,
   get data with different default or not try to access parent context
* MultiContext to virtually merge several contexts without of making them related.
  All merged contexts may have parents of their own which are also merged.
  Source contexts are not modified. This is sort of context mix-in

Parser:
* keyword arguments (name => value) now require name to be keyword token.
   So f('abc' => value) will not work. This doesnt affect dict() and similar functions
   as they use different argument format.
* tokens that start with two underscores (__) are no more valid. This is done so that it
  would be possible to have auto-injected Python arguments like "__context" without
  affecting possible kwargs of the same function

Delegates:
* Added ability to call delegate (callable) passed as a context value.
  The syntax is $var(args) (or even $(args)).
* Delegate doesn't have to be a context value but also can be result of expression:
  func(args1)(args2), (f() op g())() and so on
* Delegates are disabled by default for security reasons. "allow_delegates" parameter
  was added to both YaqlFactory classes to enable them
* (expr)(args) will be translated to #call(expr, args). So for this to work #call function
  must be present in context. Standard context doesn't provide this function by default.
  Use delegates=True parameter of create_context method (including legacy mode version)
  to register #call() and lambda() functions
* additional lambda(expression) method that returns delegate to passed expression thus
  making it 2nd order lambda. This delegate may be stored in context using let() method
  (or alternatives) or be called as usual. lambda(expression)(args) is equal to expression(args)

Function specs:
* FunctionDefinition now has "meta" dictionary to store arbitrary extended metadata for the
   function (for example version etc.)
* use @specs.meta('key', 'value') decorator to assign meta value to the function. Several
  decorators may be applied to single function
* Context.__call__() / runner.call() now accept optional function_filter parameter that is a
  predicate (functionDefinition, context -> boolean) to additionally filter function candidates
  based on context values, function metadata and so on.
* returns_context decorator and functionality was removed because it was shawn to be useless
* "__context" and "__engine" parameters where not explicitly declared treated the
  same way as  "context" and "engine"
* added ability to control what type be assigned to parameters without explicit specification
  based on parameter name
* added ability to get function definition with stripped hidden parameters for function wrappers
* refactoring of ParameterDefinition creation code. Code from decorator was moved to a more
  generic set_parameter method of FunctionDefinition to support wider range of scenarios
  (for example to overwrite auto-generated parameter specs)
* FunctionDefinition.__call__ was changed:
  a) order of parameter was changed so the sender could have default value
  b) args now automatically prefixed with sender when provided.
       There is no need to provide it twice anymore
* a helper functions was added to utils to get yaql 0.2 style extension methods specs
  for functions that already registered in context as pure functions or methods

Smart types:
* "context" argument was added to each check() method so that it would be possible to do
  checks using context values. Non of standard smart-types use it, however custom types may do.
* return_context flag was removed from Lambda/Delegate/Super types for the same reasons as in FD
* GenericType helper class was added as a base class for custom non-lazy smart-types simplifying
  their development
* added ability to pass received Lambda (Delegate etc) to another method (or base method version)
  when the later expects it in another format (for example sender had Lambda() while the receiver
  had Lambda(with_context=True))

Standard library:
* "#operator_." method for obj.method() now expects first argument (sender expression) to be object
   (pre-evaluated) rather than Lambda. This doesn't brake anything but allows to override this function
   in child contexts for some more specific types of sender without getting resolution error because of
   several versions of the same function being differ by parameter laziness
* collection.flatten() method was added
* string.matches(regexpString) method was added to complement
   regexp.matches(string) that was before
* string.join(collection) method was added to complement collection.join(string)
   that was before. Also now both functions apply str() function to each element
   of collection
* now int(null) = 0 and float(null) = 0.0

Also bumps requirements to latest OpenStack global-requirements and removes version
number from setup.cfg to use git tag-driven versioning of pbr

Change-Id: I0dd06bf1fb70296157ebb0e9831d2b70d93ca137

requirements.txt

@@ -1,5 +1,5 @@
-pbr>=0.6,!=0.7,<1.0
+pbr>=0.11,<2.0
-Babel>=0.9.6
+Babel>=1.3
 
 ply<=3.6
-six>=1.7.0
+six>=1.9.0

setup.cfg

@@ -1,6 +1,5 @@
 [metadata]
 name = yaql
-version = 1.0.0b3
 summary = YAQL - Yet Another Query Language
 description-file =
     README.rst

test-requirements.txt

@@ -1,11 +1,11 @@
-hacking>=0.5.6,<0.8
+hacking>=0.10.0,<0.11
 
 coverage>=3.6
 discover
-fixtures>=0.3.14
+fixtures>=1.3.1
-python-subunit
+python-subunit>=0.0.18
-sphinx>=1.1.2
+sphinx>=1.1.2,!=1.2.0,!=1.3b1,<1.3
-oslosphinx
+oslosphinx>=2.5.0
-testrepository>=0.0.17
+testrepository>=0.0.18
-testscenarios>=0.4,<0.5
+testscenarios>=0.4
-testtools>=0.9.32
+testtools>=1.4.0

tox.ini

@@ -28,9 +28,10 @@ commands = python setup.py build_sphinx
 # H803 skipped on purpose per list discussion.
 # E123, E125 skipped as they are invalid PEP-8.
 # H404  multi line docstring should start with a summary
+# H405 multi line docstring summary not separated with an empty line
 ## TODO(ruhe) following checks should be fixed
 # E721 do not compare types, use 'isinstance()'
 show-source = True
-ignore = E123,E125,E721,H404,H803
+ignore = E123,E125,E721,H404,H405,H803
 builtins = _
 exclude=.venv,.git,.tox,dist,doc,*openstack/common*,*lib/python*,*egg,build

yaql/__init__.py

@@ -15,7 +15,7 @@
 import os.path
 import pkg_resources
 
-from yaql.language import context as yaqlcontext
+from yaql.language import contexts
 from yaql.language import conventions
 from yaql.language import factory
 from yaql.language import specs
@@ -36,10 +36,10 @@ _cached_engine = None
 _default_context = None
 
 
-def _setup_context(data, context, finalizer):
+def _setup_context(data, context, finalizer, convention):
     if context is None:
-        context = yaqlcontext.Context(
+        context = contexts.Context(
-            convention=conventions.CamelCaseConvention())
+            convention=convention or conventions.CamelCaseConvention())
 
     if finalizer is None:
         @specs.parameter('iterator', yaqltypes.Iterable())
@@ -67,11 +67,12 @@ def create_context(data=utils.NO_VALUE, context=None, system=True,
                    common=True, boolean=True, strings=True,
                    math=True, collections=True, queries=True,
                    regex=True, branching=True,
-                   no_sets=False, finalizer=None):
+                   no_sets=False, finalizer=None, delegates=False,
+                   convention=None):
 
-    context = _setup_context(data, context, finalizer)
+    context = _setup_context(data, context, finalizer, convention)
     if system:
-        std_system.register(context)
+        std_system.register(context, delegates)
     if common:
         std_common.register(context)
     if boolean:

yaql/language/contexts.py

@@ -21,8 +21,11 @@ from yaql.language import utils
 
 
 class ContextBase(object):
-    def __init__(self, parent_context):
+    def __init__(self, parent_context=None, convention=None):
         self._parent_context = parent_context
+        self._convention = convention
+        if convention is None and parent_context:
+            self._convention = parent_context.convention
 
     @property
     def parent(self):
@@ -31,8 +34,11 @@ class ContextBase(object):
     def register_function(self, spec, *args, **kwargs):
         pass
 
+    def get_data(self, name, default=None, ask_parent=True):
+        return default
+
     def __getitem__(self, name):
-        return None
+        return self.get_data(name)
 
     def __setitem__(self, name, value):
         pass
@@ -40,10 +46,15 @@ class ContextBase(object):
     def __delitem__(self, name):
         pass
 
+    def __contains__(self, item):
+        return self.get_data(item, utils.NO_VALUE) is not utils.NO_VALUE
+
     def __call__(self, name, engine, sender=utils.NO_VALUE,
-                 data_context=None, return_context=False,
+                 data_context=None, use_convention=False,
-                 use_convention=False):
+                 function_filter=None):
-        raise exceptions.NoFunctionRegisteredException(name)
+        return lambda *args, **kwargs: runner.call(
+            name, self, args, kwargs, engine, sender,
+            data_context, use_convention, function_filter)
 
     def get_functions(self, name, predicate=None, use_convention=False):
         return []
@@ -54,14 +65,17 @@ class ContextBase(object):
     def create_child_context(self):
         return type(self)(self)
 
+    @property
+    def convention(self):
+        return self._convention
+
 
 class Context(ContextBase):
     def __init__(self, parent_context=None, data=utils.NO_VALUE,
                  convention=None):
-        super(Context, self).__init__(parent_context)
+        super(Context, self).__init__(parent_context, convention)
         self._functions = {}
         self._data = {}
-        self._convention = convention
         if data is not utils.NO_VALUE:
             self['$'] = data
 
@@ -84,6 +98,7 @@ class Context(ContextBase):
         self._functions.setdefault(spec.name, list()).append((spec, exclusive))
 
     def get_functions(self, name, predicate=None, use_convention=False):
+        name = name.rstrip('_')
         if use_convention and self._convention is not None:
             name = self._convention.convert_function_name(name)
         if predicate is None:
@@ -93,6 +108,7 @@ class Context(ContextBase):
             self._functions.get(name, list()))))
 
     def collect_functions(self, name, predicate=None, use_convention=False):
+        name = name.rstrip('_')
         if use_convention and self._convention is not None:
             name = self._convention.convert_function_name(name)
         overloads = []
@@ -105,20 +121,13 @@ class Context(ContextBase):
                 for spec, exclusive in layer_overloads:
                     if exclusive:
                         p = None
-                    if predicate and not predicate(spec):
+                    if predicate and not predicate(spec, self):
                         continue
                     layer.append(spec)
                 if layer:
                     overloads.append(layer)
         return overloads
 
-    def __call__(self, name, engine, sender=utils.NO_VALUE,
-                 data_context=None, return_context=False,
-                 use_convention=False):
-        return lambda *args, **kwargs: runner.call(
-            name, self, args, kwargs, engine, sender,
-            data_context, return_context, use_convention)
-
     @staticmethod
     def _normalize_name(name):
         if not name.startswith('$'):
@@ -130,15 +139,80 @@ class Context(ContextBase):
     def __setitem__(self, name, value):
         self._data[self._normalize_name(name)] = value
 
-    def __getitem__(self, name):
+    def get_data(self, name, default=None, ask_parent=True):
         name = self._normalize_name(name)
         if name in self._data:
             return self._data[name]
-        if self.parent:
+        if self.parent and ask_parent:
-            return self.parent[name]
+            return self.parent.get_data(name, default, ask_parent)
+        return default
 
     def __delitem__(self, name):
         self._data.pop(self._normalize_name(name))
 
     def create_child_context(self):
         return Context(self, convention=self._convention)
+
+
+class MultiContext(ContextBase):
+    def __init__(self, context_list, convention=None):
+        self._context_list = context_list
+        if convention is None:
+            convention = context_list[0].convention
+        parents = six.moves.filter(
+            lambda t: t,
+            six.moves.map(lambda t: t.parent, context_list))
+        if not parents:
+            super(MultiContext, self).__init__(None, convention)
+        elif len(parents) == 1:
+            super(MultiContext, self).__init__(parents[0], convention)
+        else:
+            super(MultiContext, self).__init__(MultiContext(parents),
+                                               convention)
+
+    def register_function(self, spec, *args, **kwargs):
+        self._context_list[0].register_function(spec, *args, **kwargs)
+
+    def get_data(self, name, default=None, ask_parent=True):
+        for context in self._context_list:
+            result = context.get_data(name, utils.NO_VALUE, False)
+            if result is not utils.NO_VALUE:
+                return result
+        if ask_parent and self.parent:
+            return self.parent.get_data(name, default, ask_parent)
+        return default
+
+    def __setitem__(self, name, value):
+        self._context_list[0][name] = value
+
+    def get_functions(self, name, predicate=None, use_convention=False):
+        result = []
+        for context in self._context_list:
+            result.extend(context.get_functions(
+                name, predicate, use_convention))
+        return result
+
+    def collect_functions(self, name, predicate=None, use_convention=False):
+        functions = six.moves.map(
+            lambda ctx: ctx.collect_functions(name, predicate, use_convention),
+            self._context_list)
+        i = 0
+        result = []
+        while True:
+            level = []
+            has_level = False
+            for f in functions:
+                if len(f) > i:
+                    has_level = True
+                    level.extend(f[i])
+            if not has_level:
+                return result
+            i += 1
+            result.append(level)
+
+    def __delitem__(self, name):
+        for context in self._context_list:
+            del context[name]
+
+    def create_child_context(self):
+        return Context(self)

yaql/language/conventions.py

@@ -28,13 +28,13 @@ class PythonConvention(Convention):
         if not name or not name[0].isalpha():
             return name
 
-        return name.rstrip('_')
+        return name
 
     def convert_parameter_name(self, name):
         if not name or not name[0].isalpha():
             return name
 
-        return name.rstrip('_')
+        return name
 
 
 class CamelCaseConvention(Convention):
@@ -45,12 +45,12 @@ class CamelCaseConvention(Convention):
         if not name or not name[0].isalpha():
             return name
 
-        return self._to_camel_case(name.strip('_'))
+        return self._to_camel_case(name)
 
     def convert_parameter_name(self, name):
         if not name or not name[0].isalpha():
             return name
-        return self._to_camel_case(name.rstrip('_', ))
+        return self._to_camel_case(name)
 
     def _to_camel_case(self, name):
         return self.regex.sub(lambda m: m.group(1).upper(), name)

yaql/language/expressions.py

@@ -34,8 +34,7 @@ class Function(Expression):
         self.uses_sender = True
 
     def __call__(self, sender, context, engine):
-        return context(self.name, engine, sender, context,
+        return context(self.name, engine, sender, context)(*self.args)
-                       return_context=True)(*self.args)
 
     def __str__(self):
         return u'{0}({1})'.format(self.name, ', '.join(
@@ -105,7 +104,7 @@ class Constant(Expression):
         return six.text_type(self.value)
 
     def __call__(self, sender, context, engine):
-        return self.value, context
+        return self.value
 
 
 class KeywordConstant(Constant):
@@ -137,8 +136,8 @@ class MappingRuleExpression(Expression):
 
     def __call__(self, sender, context, engine):
         return utils.MappingRule(
-            self.source(sender, context, engine)[0],
+            self.source(sender, context, engine),
-            self.destination(sender, context, engine)[0]), context
+            self.destination(sender, context, engine))
 
 
 @six.python_2_unicode_compatible
@@ -158,12 +157,12 @@ class Statement(Function):
         except exceptions.WrappedException as e:
             six.reraise(type(e.wrapped), e.wrapped, sys.exc_info()[2])
 
-    def evaluate(self, data=utils.NO_VALUE, context=utils.NO_VALUE):
+    def evaluate(self, data=utils.NO_VALUE, context=None):
-        if context is utils.NO_VALUE:
+        if context is None or context is utils.NO_VALUE:
             context = yaql.create_context()
         if data is not utils.NO_VALUE:
             context['$'] = utils.convert_input_data(data)
-        return self(utils.NO_VALUE, context, self.engine)[0]
+        return self(utils.NO_VALUE, context, self.engine)
 
     def __str__(self):
         return str(self.expression)

yaql/language/factory.py

@@ -82,8 +82,9 @@ class YaqlEngine(object):
 
 
 class YaqlFactory(object):
-    def __init__(self, keyword_operator='=>'):
+    def __init__(self, keyword_operator='=>', allow_delegates=False):
         self._keyword_operator = keyword_operator
+        self._allow_delegates = allow_delegates
         self.operators = self._standard_operators()
         if keyword_operator:
             self.operators.insert(0, (keyword_operator,
@@ -93,6 +94,10 @@ class YaqlFactory(object):
     def keyword_operator(self):
         return self._keyword_operator
 
+    @property
+    def allow_delegates(self):
+        return self._allow_delegates
+
     # noinspection PyMethodMayBeStatic
     def _standard_operators(self):
         return [
@@ -218,7 +223,7 @@ class YaqlFactory(object):
 
     # noinspection PyMethodMayBeStatic
     def _create_parser(self, lexer_rules, operators):
-        return parser.Parser(lexer_rules, operators)
+        return parser.Parser(lexer_rules, operators, self)
 
     def create(self, options=None):
         names = self._name_generator()
@@ -227,7 +232,7 @@ class YaqlFactory(object):
         ply_lexer = lex.lex(object=lexer_rules, reflags=re.UNICODE)
         ply_parser = yacc.yacc(
             module=self._create_parser(lexer_rules, operators),
-            debug=False if not options else options.get("yaql.debug", False),
+            debug=False if not options else options.get('yaql.debug', False),
             tabmodule='m' + uuid.uuid4().hex, write_tables=False)
 
         return YaqlEngine(ply_lexer, ply_parser, options, self)

yaql/language/lexer.py

@@ -104,7 +104,7 @@ class Lexer(object):
 
     def t_KEYWORD_STRING(self, t):
         """
-        \\b[^\\W\\d]\\w*\\b
+        (?!__)\\b[^\\W\\d]\\w*\\b
         """
         if t.value in self._operators_table:
             t.type = self._operators_table[t.value][2]

yaql/language/parser.py

@@ -20,12 +20,12 @@ from yaql.language import utils
 
 
 class Parser(object):
-    def __init__(self, lexer, yaql_operators):
+    def __init__(self, lexer, yaql_operators, engine):
         self.tokens = lexer.tokens
         self._aliases = {}
-        self._generate_operator_funcs(yaql_operators)
+        self._generate_operator_funcs(yaql_operators, engine)
 
-    def _generate_operator_funcs(self, yaql_operators):
+    def _generate_operator_funcs(self, yaql_operators, engine):
         binary_doc = ''
         unary_doc = ''
         precedence_dict = {}
@@ -84,6 +84,18 @@ class Parser(object):
         precedence.reverse()
         self.precedence = tuple(precedence)
 
+        def p_value_call(this, p):
+            """
+            func : value '(' args ')'
+            """
+            arg = ()
+            if len(p) > 4:
+                arg = p[3]
+            p[0] = expressions.Function('#call', p[1], *arg)
+
+        if engine.allow_delegates:
+            self.p_value_call = six.create_bound_method(p_value_call, self)
+
     @staticmethod
     def p_value_to_const(p):
         """

yaql/language/runner.py

@@ -23,15 +23,18 @@ from yaql.language import yaqltypes
 
 
 def call(name, context, args, kwargs, engine, sender=utils.NO_VALUE,
-         data_context=None, return_context=False, use_convention=False):
+         data_context=None, use_convention=False, function_filter=None):
 
     if data_context is None:
         data_context = context
 
+    if function_filter is None:
+        function_filter = lambda fd, ctx: True
+
     if sender is utils.NO_VALUE:
-        predicate = lambda fd: fd.is_function
+        predicate = lambda fd, ctx: fd.is_function and function_filter(fd, ctx)
     else:
-        predicate = lambda fd: fd.is_method
+        predicate = lambda fd, ctx: fd.is_method and function_filter(fd, ctx)
 
     all_overloads = context.collect_functions(
         name, predicate, use_convention=use_convention)
@@ -46,8 +49,8 @@ def call(name, context, args, kwargs, engine, sender=utils.NO_VALUE,
                                    data_context, args, kwargs)
         try:
             result = delegate()
-            utils.limit_memory_usage(engine, (1, result[0]))
+            utils.limit_memory_usage(engine, (1, result))
-            return result if return_context else result[0]
+            return result
         except StopIteration as e:
             six.reraise(
                 exceptions.WrappedException,
@@ -82,7 +85,7 @@ def choose_overload(name, candidates, engine, sender, context, args, kwargs):
             elif no_kwargs != c.no_kwargs:
                 raise_ambiguous()
 
-            mapping = c.map_args(args, kwargs)
+            mapping = c.map_args(args, kwargs, context)
             if mapping is None:
                 continue
             pos, kwd = mapping
@@ -105,7 +108,7 @@ def choose_overload(name, candidates, engine, sender, context, args, kwargs):
         raise_not_found()
 
     arg_evaluator = lambda i, arg: (
-        arg(utils.NO_VALUE, context, engine)[0]
+        arg(utils.NO_VALUE, context, engine)
         if (i not in lazy_params and isinstance(arg, expressions.Expression)
             and not isinstance(arg, expressions.Constant))
         else arg
@@ -120,7 +123,7 @@ def choose_overload(name, candidates, engine, sender, context, args, kwargs):
     for level in candidates2:
         for c, mapping in level:
             try:
-                d = c.get_delegate(sender, engine, args, kwargs)
+                d = c.get_delegate(sender, engine, context, args, kwargs)
             except exceptions.ArgumentException:
                 pass
             else:
@@ -136,7 +139,7 @@ def choose_overload(name, candidates, engine, sender, context, args, kwargs):
 
     if delegate is None:
         raise_not_found()
-    return lambda: delegate(context)
+    return lambda: delegate()
 
 
 def _translate_args(without_kwargs, args, kwargs):
@@ -145,13 +148,13 @@ def _translate_args(without_kwargs, args, kwargs):
             raise exceptions.ArgumentException(six.next(iter(kwargs)))
         return args, {}
     pos_args = []
-    kw_args = dict(kwargs)
+    kw_args = {}
     for t in args:
         if isinstance(t, expressions.MappingRuleExpression):
             param_name = t.source
-            if isinstance(param_name, expressions.Constant):
+            if isinstance(param_name, expressions.KeywordConstant):
                 param_name = param_name.value
-            if not isinstance(param_name, six.string_types):
+            else:
                 raise exceptions.MappingTranslationException()
             kw_args[param_name] = t.destination
         else:

yaql/language/specs.py

@@ -13,7 +13,6 @@
 #    under the License.
 
 import inspect
-import types
 
 import six
 
@@ -43,21 +42,23 @@ class ParameterDefinition(object):
 
 
 class FunctionDefinition(object):
-    def __init__(self, name, parameters, payload, doc='',
+    def __init__(self, name, payload, parameters=None, doc='', meta=None,
-                 is_function=True, is_method=False,
+                 is_function=True, is_method=False, no_kwargs=False):
-                 returns_context=False, no_kwargs=False):
         self.is_method = is_method
         self.is_function = is_function
         self.name = name
-        self.parameters = parameters
+        self.parameters = {} if not parameters else parameters
         self.payload = payload
         self.doc = doc
-        self.returns_context = returns_context
         self.no_kwargs = no_kwargs
+        self.meta = meta or {}
 
-    def __call__(self, sender, engine, context):
+    def __call__(self, engine, context, sender=utils.NO_VALUE):
-        return lambda *args, **kwargs: \
+        def func(*args, **kwargs):
-            self.get_delegate(sender, engine, args, kwargs)(context)[0]
+            if sender is not utils.NO_VALUE:
+                args = (sender,) + args
+            return self.get_delegate(sender, engine, context, args, kwargs)()
+        return func
 
     def clone(self):
         parameters = dict(
@@ -65,12 +66,135 @@ class FunctionDefinition(object):
             for key, p in six.iteritems(self.parameters))
 
         res = FunctionDefinition(
-            self.name, parameters, self.payload,
+            self.name, self.payload, parameters, self.doc,
-            self.doc, self.is_function, self.is_method,
+            self.meta, self.is_function, self.is_method, self.no_kwargs)
-            self.returns_context, self.no_kwargs)
         return res
 
-    def map_args(self, args, kwargs):
+    def strip_hidden_parameters(self):
+        fd = self.clone()
+        keys_to_remove = set()
+
+        for k, v in fd.parameters.iteritems():
+            if not isinstance(v.value_type, yaqltypes.HiddenParameterType):
+                continue
+            keys_to_remove.add(k)
+            if v.position is not None:
+                for v2 in fd.parameters.itervalues():
+                    if v2.position is not None and v2.position > v.position:
+                        v2.position -= 1
+        for key in keys_to_remove:
+            del fd.parameters[key]
+        return fd
+
+    def set_parameter(self, name, value_type=None, nullable=None,
+                      alias=None, overwrite=False):
+        if isinstance(name, ParameterDefinition):
+            if name.name in self.parameters and not overwrite:
+                raise exceptions.DuplicateParameterDecoratorException(
+                    function_name=self.name or self.payload.__name__,
+                    param_name=name.name)
+            self.parameters[name.name] = name
+            return name
+
+        if six.PY2:
+            spec = inspect.getargspec(self.payload)
+            if isinstance(name, int):
+                if 0 <= name < len(spec.args):
+                    name = spec.args[name]
+                elif name == inspect.getargspec(self.payload) \
+                        and spec.varargs is not None:
+                    name = spec.varargs
+                else:
+                    raise IndexError('argument position is out of range')
+
+            arg_name = name
+            if name == spec.keywords:
+                position = None
+                arg_name = '**'
+            elif name == spec.varargs:
+                position = len(spec.args)
+                arg_name = '*'
+            elif name not in spec.args:
+                raise exceptions.NoParameterFoundException(
+                    function_name=self.name or self.payload.__name__,
+                    param_name=name)
+            else:
+                position = spec.args.index(name)
+            default = NO_DEFAULT
+            if spec.defaults is not None and name in spec.args:
+                index = spec.args.index(name) - len(spec.args)
+                if index >= -len(spec.defaults):
+                    default = spec.defaults[index]
+        else:
+            spec = inspect.getfullargspec(self.payload)
+            if isinstance(name, int):
+                if 0 <= name < len(spec.args):
+                    name = spec.args[name]
+                elif name == inspect.getargspec(self.payload) \
+                        and spec.varargs is not None:
+                    name = spec.varargs
+                else:
+                    raise IndexError('argument position is out of range')
+
+            arg_name = name
+            if name == spec.varkw:
+                position = None
+                arg_name = '**'
+            elif name == spec.varargs:
+                position = len(spec.args)
+                arg_name = '*'
+            elif name in spec.kwonlyargs:
+                position = None
+            elif name not in spec.args:
+                raise exceptions.NoParameterFoundException(
+                    function_name=self.name or self.payload.__name__,
+                    param_name=name)
+            else:
+                position = spec.args.index(name)
+
+            default = NO_DEFAULT
+            if spec.defaults is not None and name in spec.args:
+                index = spec.args.index(name) - len(spec.args)
+                if index >= -len(spec.defaults):
+                    default = spec.defaults[index]
+            elif spec.kwonlydefaults is not None:
+                default = spec.kwonlydefaults.get(name, NO_DEFAULT)
+
+        if arg_name in self.parameters and not overwrite:
+            raise exceptions.DuplicateParameterDecoratorException(
+                function_name=self.name or self.payload.__name__,
+                param_name=name)
+
+        yaql_type = value_type
+        p_nullable = nullable
+        if value_type is None:
+            if p_nullable is None:
+                p_nullable = True
+            base_type = object \
+                if default in (None, NO_DEFAULT, utils.NO_VALUE) \
+                else type(default)
+            yaql_type = yaqltypes.PythonType(base_type, p_nullable)
+        elif not isinstance(value_type, yaqltypes.SmartType):
+            if p_nullable is None:
+                p_nullable = default is None
+            yaql_type = yaqltypes.PythonType(value_type, p_nullable)
+
+        pd = ParameterDefinition(
+            name, yaql_type, position, alias, default
+        )
+        self.parameters[arg_name] = pd
+        return pd
+
+    def insert_parameter(self, name, value_type=None, nullable=None,
+                         alias=None, overwrite=False):
+        pd = self.set_parameter(name, value_type, nullable, alias, overwrite)
+        for p in self.parameters.itervalues():
+            if p is pd:
+                continue
+            if p.position is not None and p.position >= pd.position:
+                p.position += 1
+
+    def map_args(self, args, kwargs, context):
         kwargs = dict(kwargs)
         positional_args = len(args) * [
             self.parameters.get('*', utils.NO_VALUE)]
@@ -126,27 +250,29 @@ class FunctionDefinition(object):
             value = args[i]
             if value is utils.NO_VALUE:
                 value = positional_args[i].default
-            if not positional_args[i].value_type.check(value):
+            if not positional_args[i].value_type.check(value, context):
                 return None
         for kwd in six.iterkeys(kwargs):
-            if not keyword_args[kwd].value_type.check(kwargs[kwd]):
+            if not keyword_args[kwd].value_type.check(kwargs[kwd], context):
                 return None
 
         return tuple(positional_args), keyword_args
 
-    def get_delegate(self, sender, engine, args, kwargs):
+    def get_delegate(self, sender, engine, context, args, kwargs):
         def checked(val, param):
-            if not param.value_type.check(val):
+            if not param.value_type.check(val, context):
                 raise exceptions.ArgumentException(param.name)
 
-            def convert_arg_func(context):
+            def convert_arg_func(context2):
                 try:
                     return param.value_type.convert(
-                        val, sender, context, self, engine)
+                        val, sender, context2, self, engine)
                 except exceptions.ArgumentValueException:
                     raise exceptions.ArgumentException(param.name)
             return convert_arg_func
 
+        kwargs = kwargs.copy()
+        kwargs = dict(kwargs)
         positional = 0
         for arg_name, p in six.iteritems(self.parameters):
             if p.position is not None and arg_name != '*':
@@ -207,7 +333,7 @@ class FunctionDefinition(object):
             else:
                 raise exceptions.ArgumentException('**')
 
-        def func(context):
+        def func():
             new_context = context.create_child_context()
             result = self.payload(
                 *tuple(map(lambda t: t(new_context),
@@ -215,14 +341,7 @@ class FunctionDefinition(object):
                 **dict(map(lambda t: (t[0], t[1](new_context)),
                            six.iteritems(keyword_args)))
             )
-            if self.returns_context:
+            return result
-                if isinstance(result, types.GeneratorType):
-                    result_context = next(result)
-                    return result, result_context
-                result_value, result_context = result
-                return result_value, result_context
-            else:
-                return result, new_context
 
         return func
 
@@ -241,40 +360,53 @@ class FunctionDefinition(object):
 
 def _get_function_definition(func):
     if not hasattr(func, '__yaql_function__'):
-        fd = FunctionDefinition(None, {}, func, func.__doc__)
+        fd = FunctionDefinition(None, func, {}, func.__doc__)
         func.__yaql_function__ = fd
     return func.__yaql_function__
 
 
+def _infer_parameter_type(name):
+    if name == 'context' or name == '__context':
+        return yaqltypes.Context()
+    elif name == 'engine' or name == '__engine':
+        return yaqltypes.Engine()
+
+
 def get_function_definition(func, name=None, function=None, method=None,
-                            convention=None):
+                            convention=None, parameter_type_func=None):
+    if parameter_type_func is None:
+        parameter_type_func = _infer_parameter_type
     fd = _get_function_definition(func).clone()
     if six.PY2:
         spec = inspect.getargspec(func)
         for arg in spec.args:
             if arg not in fd.parameters:
-                parameter(arg, function_definition=fd)(func)
+                fd.set_parameter(arg, parameter_type_func(arg))
         if spec.varargs and '*' not in fd.parameters:
-            parameter(spec.varargs, function_definition=fd)(func)
+            fd.set_parameter(spec.varargs, parameter_type_func(spec.varargs))
         if spec.keywords and '**' not in fd.parameters:
-            parameter(spec.keywords, function_definition=fd)(func)
+            fd.set_parameter(spec.keywords, parameter_type_func(spec.keywords))
     else:
         spec = inspect.getfullargspec(func)
         for arg in spec.args + spec.kwonlyargs:
             if arg not in fd.parameters:
-                parameter(arg, function_definition=fd)(func)
+                fd.set_parameter(arg, parameter_type_func(arg))
         if spec.varargs and '*' not in fd.parameters:
-            parameter(spec.varargs, function_definition=fd)(func)
+            fd.set_parameter(spec.varargs, parameter_type_func(spec.varargs))
         if spec.varkw and '**' not in fd.parameters:
-            parameter(spec.varkw, function_definition=fd)(func)
+            fd.set_parameter(spec.varkw, parameter_type_func(spec.varkw))
 
     if name is not None:
         fd.name = name
     elif fd.name is None:
         if convention is not None:
-            fd.name = convention.convert_function_name(fd.payload.__name__)
+            fd.name = convention.convert_function_name(
+                fd.payload.__name__.rstrip('_'))
         else:
-            fd.name = fd.payload.__name__
+            fd.name = fd.payload.__name__.rstrip('_')
+    elif convention is not None:
+        fd.name = convention.convert_function_name(fd.name.rstrip('_'))
+
     if function is not None:
         fd.is_function = function
     if method is not None:
@@ -282,109 +414,31 @@ def get_function_definition(func, name=None, function=None, method=None,
     if convention:
         for p in six.itervalues(fd.parameters):
             if p.alias is None:
-                p.alias = convention.convert_parameter_name(p.name)
+                p.alias = convention.convert_parameter_name(p.name.rstrip('_'))
 
     return fd
 
 
-def _parameter(name, value_type=None, nullable=None, alias=None,
+def _parameter(name, value_type=None, nullable=None, alias=None):
-               function_definition=None):
     def wrapper(func):
-        fd = function_definition or _get_function_definition(func)
+        fd = _get_function_definition(func)
-        if six.PY2:
+        fd.set_parameter(name, value_type, nullable, alias)
-            spec = inspect.getargspec(func)
-            arg_name = name
-            if name == spec.keywords:
-                position = None
-                arg_name = '**'
-            elif name == spec.varargs:
-                position = len(spec.args)
-                arg_name = '*'
-            elif name not in spec.args:
-                raise exceptions.NoParameterFoundException(
-                    function_name=fd.name or func.__name__,
-                    param_name=name)
-            else:
-                position = spec.args.index(name)
-            default = NO_DEFAULT
-            if spec.defaults is not None and name in spec.args:
-                index = spec.args.index(name) - len(spec.args)
-                if index >= -len(spec.defaults):
-                    default = spec.defaults[index]
-        else:
-            spec = inspect.getfullargspec(func)
-            arg_name = name
-            if name == spec.varkw:
-                position = None
-                arg_name = '**'
-            elif name == spec.varargs:
-                position = len(spec.args)
-                arg_name = '*'
-            elif name in spec.kwonlyargs:
-                position = None
-            elif name not in spec.args:
-                raise exceptions.NoParameterFoundException(
-                    function_name=fd.name or func.__name__,
-                    param_name=name)
-            else:
-                position = spec.args.index(name)
-
-            default = NO_DEFAULT
-            if spec.defaults is not None and name in spec.args:
-                index = spec.args.index(name) - len(spec.args)
-                if index >= -len(spec.defaults):
-                    default = spec.defaults[index]
-            elif spec.kwonlydefaults is not None:
-                default = spec.kwonlydefaults.get(name, NO_DEFAULT)
-
-        if arg_name in fd.parameters:
-            raise exceptions.DuplicateParameterDecoratorException(
-                function_name=fd.name or func.__name__,
-                param_name=name)
-
-        yaql_type = value_type
-        p_nullable = nullable
-        if value_type is None:
-            if p_nullable is None:
-                p_nullable = True
-            if name == 'context':
-                yaql_type = yaqltypes.Context()
-            elif name == 'engine':
-                yaql_type = yaqltypes.Engine()
-            else:
-                base_type = object \
-                    if default in (None, NO_DEFAULT, utils.NO_VALUE) \
-                    else type(default)
-                yaql_type = yaqltypes.PythonType(base_type, p_nullable)
-        elif not isinstance(value_type, yaqltypes.SmartType):
-            if p_nullable is None:
-                p_nullable = default is None
-            yaql_type = yaqltypes.PythonType(value_type, p_nullable)
-
-        fd.parameters[arg_name] = ParameterDefinition(
-            name, yaql_type, position, alias, default
-        )
-
         return func
     return wrapper
 
 
-def parameter(name, value_type=None, nullable=None, alias=None,
+def parameter(name, value_type=None, nullable=None, alias=None):
-              function_definition=None):
     if value_type is not None and isinstance(
             value_type, yaqltypes.HiddenParameterType):
         raise ValueError('Use inject() for hidden parameters')
-    return _parameter(name, value_type, nullable=nullable, alias=alias,
+    return _parameter(name, value_type, nullable=nullable, alias=alias)
-                      function_definition=function_definition)
 
 
-def inject(name, value_type=None, nullable=None, alias=None,
+def inject(name, value_type=None, nullable=None, alias=None):
-           function_definition=None):
     if value_type is not None and not isinstance(
             value_type, yaqltypes.HiddenParameterType):
         raise ValueError('Use parameter() for normal function parameters')
-    return _parameter(name, value_type, nullable=nullable, alias=alias,
+    return _parameter(name, value_type, nullable=nullable, alias=alias)
-                      function_definition=function_definition)
 
 
 def name(function_name):
@@ -409,13 +463,15 @@ def extension_method(func):
     return func
 
 
-def returns_context(func):
-    fd = _get_function_definition(func)
-    fd.returns_context = True
-    return func
-
-
 def no_kwargs(func):
     fd = _get_function_definition(func)
     fd.no_kwargs = True
     return func
+
+
+def meta(name, value):
+    def wrapper(func):
+        fd = _get_function_definition(func)
+        fd.meta[name] = value
+        return func
+    return wrapper

yaql/language/utils.py

@@ -202,3 +202,19 @@ def limit_memory_usage(engine, *args):
         total += t[0] * sys.getsizeof(t[1], 0)
         if total > quota:
             raise exceptions.MemoryQuotaExceededException()
+
+
+def to_extension_method(name, context):
+    layers = context.collect_functions(
+        name,
+        lambda t, ctx: not t.is_function or not t.is_method,
+        use_convention=True)
+    if len(layers) > 1:
+        raise ValueError(
+            'Multi layer functions are not supported by this helper method')
+    if len(layers) > 0:
+        for spec in layers[0]:
+            spec = spec.clone()
+            spec.is_function = True
+            spec.is_method = True
+            yield spec

yaql/language/yaqltypes.py

@@ -23,7 +23,7 @@ from yaql.language import utils
 
 class HiddenParameterType(object):
     # noinspection PyMethodMayBeStatic,PyUnusedLocal
-    def check(self, value):
+    def check(self, value, context):
         return True
 
 
@@ -35,13 +35,13 @@ class SmartType(object):
     def __init__(self, nullable):
         self.nullable = nullable
 
-    def check(self, value):
+    def check(self, value, context):
         if value is None and not self.nullable:
             return False
         return True
 
     def convert(self, value, sender, context, function_spec, engine):
-        if not self.check(value):
+        if not self.check(value, context):
             raise exceptions.ArgumentValueException()
         utils.limit_memory_usage(engine, (1, value))
 
@@ -49,7 +49,35 @@ class SmartType(object):
         return False
 
 
-class PythonType(SmartType):
+class GenericType(SmartType):
+    def __init__(self, nullable, checker=None, converter=None):
+        super(GenericType, self).__init__(nullable)
+        self.checker = checker
+        self.converter = converter
+
+    def check(self, value, context):
+        if isinstance(value, expressions.Constant):
+            value = value.value
+
+        if not super(GenericType, self).check(value, context):
+            return False
+        if value is None or isinstance(value, expressions.Expression):
+            return True
+        if not self.checker:
+            return True
+        return self.checker(value, context)
+
+    def convert(self, value, sender, context, function_spec, engine):
+        if isinstance(value, expressions.Constant):
+            value = value.value
+        super(GenericType, self).convert(
+            value, sender, context, function_spec, engine)
+        if value is None or not self.converter:
+            return value
+        return self.converter(value, sender, context, function_spec, engine)
+
+
+class PythonType(GenericType):
     def __init__(self, python_type, nullable=True, validators=None):
         self.python_type = python_type
         if not validators:
@@ -57,45 +85,39 @@ class PythonType(SmartType):
         if not isinstance(validators, (list, tuple)):
             validators = [validators]
         self.validators = validators
-        super(PythonType, self).__init__(nullable)
 
-    def check(self, value):
+        super(PythonType, self).__init__(
-        if isinstance(value, expressions.Constant):
+            nullable,
-            value = value.value
+            lambda value, context: isinstance(
-
+                value, self.python_type) and all(
-        return super(PythonType, self).check(value) and (
+                map(lambda t: t(value), self.validators)))
-            value is None or isinstance(value, expressions.Expression) or (
-                isinstance(value, self.python_type) and all(
-                    map(lambda t: t(value), self.validators))))
-
-    def convert(self, value, sender, context, function_spec, engine):
-        super(PythonType, self).convert(value, sender, context,
-                                        function_spec, engine)
-        if isinstance(value, expressions.Constant):
-            value = value.value
-        super(PythonType, self).convert(value, sender, context,
-                                        function_spec, engine)
-        return value
 
     def is_specialization_of(self, other):
+        if not isinstance(other, PythonType):
+            return False
+        try:
+            len(self.python_type)
+            len(other.python_type)
+        except Exception:
             return (
-            isinstance(other, PythonType)
+                issubclass(self.python_type, other.python_type)
-            and issubclass(self.python_type, other.python_type)
                 and not issubclass(other.python_type, self.python_type)
             )
+        else:
+            return False
 
 
 class MappingRule(LazyParameterType, SmartType):
     def __init__(self):
         super(MappingRule, self).__init__(False)
 
-    def check(self, value):
+    def check(self, value, context):
         return isinstance(value, expressions.MappingRuleExpression)
 
     def convert(self, value, sender, context, function_spec, engine):
         super(MappingRule, self).convert(value, sender, context,
                                          function_spec, engine)
-        wrap = lambda func: lambda: func(sender, context, engine)[0]
+        wrap = lambda func: lambda: func(sender, context, engine)
 
         return utils.MappingRule(wrap(value.source), wrap(value.destination))
 
@@ -145,17 +167,16 @@ class Number(PythonType):
 
 
 class Lambda(LazyParameterType, SmartType):
-    def __init__(self, with_context=False, method=False, return_context=False):
+    def __init__(self, with_context=False, method=False):
         super(Lambda, self).__init__(True)
-        self.return_context = return_context
         self.with_context = with_context
         self.method = method
 
-    def check(self, value):
+    def check(self, value, context):
         if self.method and isinstance(
                 value, expressions.Expression) and not value.uses_sender:
             return False
-        return super(Lambda, self).check(value)
+        return super(Lambda, self).check(value, context)
 
     @staticmethod
     def _publish_params(context, args, kwargs):
@@ -168,17 +189,17 @@ class Lambda(LazyParameterType, SmartType):
         self._publish_params(context, args, kwargs)
         if isinstance(value, expressions.Expression):
             result = value(sender, context, engine)
-        elif six.callable(value):
-            result = value, context
         else:
             result = value, context
-        return result[0] if not self.return_context else result
+        return result
 
     def convert(self, value, sender, context, function_spec, engine):
         super(Lambda, self).convert(value, sender, context,
                                     function_spec, engine)
         if value is None:
             return None
+        elif six.callable(value) and hasattr(value, '__unwrapped__'):
+            value = value.__unwrapped__
 
         def func(*args, **kwargs):
             if self.method and self.with_context:
@@ -198,13 +219,12 @@ class Lambda(LazyParameterType, SmartType):
             return self._call(value, new_sender, new_context,
                               engine, args, kwargs)
 
+        func.__unwrapped__ = value
         return func
 
 
 class Super(HiddenParameterType, SmartType):
-    def __init__(self, with_context=False, method=None, return_context=False,
+    def __init__(self, with_context=False, method=None, with_name=False):
-                 with_name=False):
-        self.return_context = return_context
         self.with_context = with_context
         self.method = method
         self.with_name = with_name
@@ -221,6 +241,9 @@ class Super(HiddenParameterType, SmartType):
             spec.name)
 
     def convert(self, value, sender, context, function_spec, engine):
+        if six.callable(value) and hasattr(value, '__unwrapped__'):
+            value = value.__unwrapped__
+
         def func(*args, **kwargs):
             function_context = self._find_function_context(
                 function_spec, context)
@@ -248,8 +271,8 @@ class Super(HiddenParameterType, SmartType):
                 new_context = context.create_child_context()
 
             return parent_function_context(
-                new_name, engine, new_sender, new_context,
+                new_name, engine, new_sender, new_context)(*args, **kwargs)
-                self.return_context)(*args, **kwargs)
+        func.__unwrapped__ = value
         return func
 
 
@@ -262,15 +285,16 @@ class Context(HiddenParameterType, SmartType):
 
 
 class Delegate(HiddenParameterType, SmartType):
-    def __init__(self, name=None, with_context=False, method=False,
+    def __init__(self, name=None, with_context=False, method=False):
-                 return_context=False):
         super(Delegate, self).__init__(False)
         self.name = name
-        self.return_context = return_context
         self.with_context = with_context
         self.method = method
 
     def convert(self, value, sender, context, function_spec, engine):
+        if six.callable(value) and hasattr(value, '__unwrapped__'):
+            value = value.__unwrapped__
+
         def func(*args, **kwargs):
             name = self.name
             if not name:
@@ -288,8 +312,8 @@ class Delegate(HiddenParameterType, SmartType):
                 new_context = context.create_child_context()
 
             return new_context(
-                name, engine, new_sender, return_context=self.return_context,
+                name, engine, new_sender, use_convention=True)(*args, **kwargs)
-                use_convention=True)(*args, **kwargs)
+        func.__unwrapped__ = value
         return func
 
 
@@ -322,8 +346,8 @@ class Constant(SmartType):
         self.expand = expand
         super(Constant, self).__init__(nullable)
 
-    def check(self, value):
+    def check(self, value, context):
-        return super(Constant, self).check(value.value) and (
+        return super(Constant, self).check(value.value, context) and (
             value is None or isinstance(value, expressions.Constant))
 
     def convert(self, value, sender, context, function_spec, engine):
@@ -336,7 +360,7 @@ class YaqlExpression(LazyParameterType, SmartType):
     def __init__(self):
         super(YaqlExpression, self).__init__(False)
 
-    def check(self, value):
+    def check(self, value, context):
         return isinstance(value, expressions.Expression)
 
     def convert(self, value, sender, context, function_spec, engine):
@@ -349,8 +373,8 @@ class StringConstant(Constant):
     def __init__(self, nullable=False):
         super(StringConstant, self).__init__(nullable)
 
-    def check(self, value):
+    def check(self, value, context):
-        return super(StringConstant, self).check(value) and (
+        return super(StringConstant, self).check(value, context) and (
             value is None or isinstance(value.value, six.string_types))
 
 
@@ -358,7 +382,7 @@ class Keyword(Constant):
     def __init__(self, expand=True):
         super(Keyword, self).__init__(False, expand)
 
-    def check(self, value):
+    def check(self, value, context):
         return isinstance(value, expressions.KeywordConstant)
 
 
@@ -366,8 +390,8 @@ class BooleanConstant(Constant):
     def __init__(self, nullable=False, expand=True):
         super(BooleanConstant, self).__init__(nullable, expand)
 
-    def check(self, value):
+    def check(self, value, context):
-        return super(BooleanConstant, self).check(value) and (
+        return super(BooleanConstant, self).check(value, context) and (
             value is None or type(value.value) is bool)
 
 
@@ -375,8 +399,8 @@ class NumericConstant(Constant):
     def __init__(self, nullable=False, expand=True):
         super(NumericConstant, self).__init__(nullable, expand)
 
-    def check(self, value):
+    def check(self, value, context):
-        return super(NumericConstant, self).check(value) and (
+        return super(NumericConstant, self).check(value, context) and (
             value is None or isinstance(
                 value.value, six.integer_types + (float,)) and
             type(value.value) is not bool)

yaql/legacy.py

@@ -18,9 +18,10 @@ from yaql.standard_library import legacy as std_legacy
 
 
 class YaqlFactory(factory.YaqlFactory):
-    def __init__(self):
+    def __init__(self, allow_delegates=False):
         # noinspection PyTypeChecker
-        super(YaqlFactory, self).__init__(keyword_operator=None)
+        super(YaqlFactory, self).__init__(
+            keyword_operator=None, allow_delegates=allow_delegates)
         self.insert_operator(
             'or', True, '=>',
             factory.OperatorType.BINARY_LEFT_ASSOCIATIVE, True)

yaql/standard_library/collections.py

@@ -33,6 +33,17 @@ def list_(delegate, *args):
     return delegate(rec(args))
 
 
+@specs.method
+@specs.parameter('collection', yaqltypes.Iterable())
+def flatten(collection):
+    for t in collection:
+        if utils.is_iterable(t):
+            for t2 in flatten(t):
+                yield t2
+        else:
+            yield t
+
+
 @specs.method
 @specs.parameter('collection', yaqltypes.Iterable())
 def to_list(collection):
@@ -499,6 +510,7 @@ def register(context, no_sets=False):
     context.register_function(list_)
     context.register_function(build_list)
     context.register_function(to_list)
+    context.register_function(flatten)
     context.register_function(list_indexer)
     context.register_function(dict_)
     context.register_function(dict_, name='#map')

yaql/standard_library/legacy.py

@@ -16,6 +16,7 @@ import itertools
 
 import six
 
+from yaql.language import contexts
 from yaql.language import expressions
 from yaql.language import specs
 from yaql.language import utils
@@ -26,9 +27,9 @@ from yaql.language import yaqltypes
 @specs.name('#operator_=>')
 def build_tuple(left, right, context, engine):
     if isinstance(left, expressions.BinaryOperator) and left.operator == '=>':
-        return left(utils.NO_VALUE, context, engine)[0] + (right,)
+        return left(utils.NO_VALUE, context, engine) + (right,)
     else:
-        return left(utils.NO_VALUE, context, engine)[0], right
+        return left(utils.NO_VALUE, context, engine), right
 
 
 @specs.parameter('tuples', tuple)
@@ -82,6 +83,13 @@ def switch(*conditions):
     return None
 
 
+@specs.parameter('sender', contexts.ContextBase)
+@specs.parameter('expr', yaqltypes.Lambda(with_context=True, method=True))
+@specs.name('#operator_.')
+def op_dot_context(sender, expr):
+    return expr(sender['$0'], sender)
+
+
 @specs.parameter('mappings', yaqltypes.Lambda())
 @specs.method
 def as_(context, sender, *mappings):
@@ -92,17 +100,8 @@ def as_(context, sender, *mappings):
         if len(tt) != 2:
             raise ValueError('as() tuples must be of size 2')
         context[tt[1]] = tt[0]
-    return sender
+        context['$0'] = sender
-
+    return context
-
-def _to_extension_method(name, context):
-    for spec in context.parent.get_functions(
-            name, lambda t: not t.is_function or not t.is_method,
-            use_convention=True):
-        spec = spec.clone()
-        spec.is_function = True
-        spec.is_method = True
-        context.register_function(spec)
 
 
 def register(context, tuples):
@@ -117,7 +116,9 @@ def register(context, tuples):
     context.register_function(range_)
     context.register_function(switch, exclusive=True)
     context.register_function(as_)
+    context.register_function(op_dot_context)
 
     for t in ('get', 'list', 'bool', 'int', 'float', 'select', 'where',
-              'join', 'sum', 'take_while'):
+              'join', 'sum', 'take_while', 'len'):
-        _to_extension_method(t, context)
+        for spec in utils.to_extension_method(t, context):
+            context.register_function(spec)

yaql/standard_library/math.py

@@ -118,10 +118,14 @@ def neq(left, right):
 
 
 def int_(value):
+    if value is None:
+        return 0
     return int(value)
 
 
 def float_(value):
+    if value is None:
+        return 0.0
     return float(value)
 
 

yaql/standard_library/regex.py

@@ -42,6 +42,13 @@ def matches(regexp, string):
     return regexp.search(string) is not None
 
 
+@specs.parameter('string', yaqltypes.String())
+@specs.parameter('regexp', yaqltypes.String())
+@specs.method
+def matches_(string, regexp):
+    return re.search(regexp, string) is not None
+
+
 @specs.parameter('regexp', REGEX_TYPE)
 @specs.parameter('string', yaqltypes.String())
 @specs.name('#operator_=~')
@@ -189,6 +196,7 @@ def escape_regex(string):
 def register(context):
     context.register_function(regex)
     context.register_function(matches)
+    context.register_function(matches_)
     context.register_function(matches_operator_string)
     context.register_function(matches_operator_regex)
     context.register_function(not_matches_operator_string)

yaql/standard_library/strings.py

@@ -103,9 +103,18 @@ def right_split(string, separator=None, max_splits=-1):
 
 @specs.parameter('sequence', yaqltypes.Iterable())
 @specs.parameter('separator', yaqltypes.String())
+@specs.inject('str_delegate', yaqltypes.Delegate('str'))
 @specs.method
-def join(sequence, separator):
+def join(sequence, separator, str_delegate):
-    return separator.join(sequence)
+    return separator.join(six.moves.map(str_delegate, sequence))
+
+
+@specs.parameter('sequence', yaqltypes.Iterable())
+@specs.parameter('separator', yaqltypes.String())
+@specs.inject('str_delegate', yaqltypes.Delegate('str'))
+@specs.method
+def join_(separator, sequence, str_delegate):
+    return join(sequence, separator, str_delegate)
 
 
 @specs.parameter('value', nullable=True)
@@ -184,10 +193,10 @@ def replace_with_dict(string, str_func, replacements, count=-1):
     return string
 
 
-@specs.parameter('__format_string__', yaqltypes.String())
+@specs.parameter('__format_string', yaqltypes.String())
 @specs.extension_method
-def format_(__format_string__, *args, **kwargs):
+def format_(__format_string, *args, **kwargs):
-    return __format_string__.format(*args, **kwargs)
+    return __format_string.format(*args, **kwargs)
 
 
 @specs.parameter('left', yaqltypes.String())
@@ -342,6 +351,7 @@ def register(context):
     context.register_function(split)
     context.register_function(right_split)
     context.register_function(join)
+    context.register_function(join_)
     context.register_function(str_)
     context.register_function(concat)
     context.register_function(concat, name='#operator_+')

yaql/standard_library/system.py

@@ -16,6 +16,7 @@ import itertools
 
 import six
 
+from yaql.language import contexts
 from yaql.language import specs
 from yaql.language import utils
 from yaql.language import yaqltypes
@@ -27,26 +28,19 @@ def get_context_data(name, context):
     return context[name]
 
 
-@specs.parameter('sender', yaqltypes.Lambda(with_context=True,
+@specs.parameter('expr', yaqltypes.Lambda(method=True))
-                                            return_context=True))
-@specs.parameter('expr', yaqltypes.Lambda(with_context=True, method=True,
-                                          return_context=True))
 @specs.name('#operator_.')
-@specs.returns_context
+def op_dot(sender, expr):
-def op_dot(context, sender, expr):
+    return expr(sender)
-    return expr(*sender(context))
 
 
-@specs.parameter('sender',
-                 yaqltypes.Lambda(with_context=True, return_context=True))
 @specs.parameter('expr', yaqltypes.YaqlExpression())
-@specs.inject('operator', yaqltypes.Delegate('#operator_.', with_context=True))
+@specs.inject('operator', yaqltypes.Delegate('#operator_.'))
 @specs.name('#operator_?.')
-def elvis_operator(context, operator, sender, expr):
+def elvis_operator(operator, sender, expr):
-    sender, context = sender(context)
     if sender is None:
         return None
-    return operator(context, sender, expr)
+    return operator(sender, expr)
 
 
 @specs.parameter('sequence', yaqltypes.Iterable())
@@ -63,12 +57,13 @@ def unpack(sequence, context, *args):
     else:
         for i, t in enumerate(sequence, 1):
             context[str(i)] = t
-    return lst
+    return context
 
 
 def with_(context, *args):
     for i, t in enumerate(args, 1):
         context[str(i)] = t
+    return context
 
 
 @specs.inject('__context__', yaqltypes.Context())
@@ -78,6 +73,7 @@ def let(__context__, *args, **kwargs):
 
     for key, value in six.iteritems(kwargs):
         __context__[key] = value
+    return __context__
 
 
 @specs.parameter('name', yaqltypes.String())
@@ -88,20 +84,20 @@ def def_(name, func, context):
         return func(*args, **kwargs)
 
     context.register_function(wrapper)
+    return context
 
 
-@specs.parameter('left', yaqltypes.Lambda(return_context=True))
+@specs.parameter('left', contexts.ContextBase)
 @specs.parameter('right', yaqltypes.Lambda(with_context=True))
 @specs.name('#operator_->')
 def send_context(left, right):
-    context = left()[1]
+    return right(left)
-    return right(context)
 
 
 @specs.method
 @specs.parameter('condition', yaqltypes.Lambda())
 @specs.parameter('message', yaqltypes.String())
-def assert_(engine, obj, condition, message=u'Assertion failed'):
+def assert__(engine, obj, condition, message=u'Assertion failed'):
     if utils.is_iterator(obj):
         obj = utils.memorize(obj, engine)
     if not condition(obj):
@@ -109,7 +105,17 @@ def assert_(engine, obj, condition, message=u'Assertion failed'):
     return obj
 
 
-def register(context):
+@specs.name('#call')
+def call(callable_, *args, **kwargs):
+    return callable_(*args, **kwargs)
+
+
+@specs.parameter('func', yaqltypes.Lambda())
+def lambda_(func):
+    return func
+
+
+def register(context, delegates=False):
     context.register_function(get_context_data)
     context.register_function(op_dot)
     context.register_function(unpack)
@@ -118,4 +124,7 @@ def register(context):
     context.register_function(let)
     context.register_function(def_)
     context.register_function(elvis_operator)
-    context.register_function(assert_)
+    context.register_function(assert__)
+    if delegates:
+        context.register_function(call)
+        context.register_function(lambda_)

yaql/tests/__init__.py

@@ -38,7 +38,7 @@ class TestCase(testtools.TestCase):
     def create_engine(self):
         func = TestCase._default_engine
         if func is None:
-            engine_factory = factory.YaqlFactory()
+            engine_factory = factory.YaqlFactory(allow_delegates=True)
             TestCase._default_engine = func = engine_factory.create(
                 options=self.engine_options)
         return func
@@ -54,7 +54,7 @@ class TestCase(testtools.TestCase):
     @property
     def context(self):
         if self._context is None:
-            self._context = yaql.create_context()
+            self._context = yaql.create_context(delegates=True)
         return self._context
 
     @property

yaql/tests/test_collections.py

@@ -96,8 +96,9 @@ class TestCollections(yaql.tests.TestCase):
                          self.eval('$.toDict($, $*$)', data=[1, 2, 3]))
 
     def test_keyword_dict_access(self):
-        data = {'a': 12, 'b c': 44}
+        data = {'A': 12, 'b c': 44, '__d': 99, '_e': 999}
-        self.assertEqual(12, self.eval('$.a', data=data))
+        self.assertEqual(12, self.eval('$.A', data=data))
+        self.assertEqual(999, self.eval('$._e', data=data))
 
         self.assertRaises(exceptions.NoMatchingFunctionException,
                           self.eval, "$.'b c'", data=data)
@@ -105,6 +106,8 @@ class TestCollections(yaql.tests.TestCase):
                           self.eval, "$.123", data=data)
         self.assertRaises(KeyError,
                           self.eval, "$.b", data=data)
+        self.assertRaises(exceptions.YaqlLexicalException,
+                          self.eval, "$.__d", data=data)
 
     def test_keyword_collection_access(self):
         data = [{'a': 2}, {'a': 4}]

yaql/tests/test_engine.py

@@ -19,7 +19,6 @@ import six
 import yaql
 from yaql.language import exceptions
 from yaql.language import specs
-from yaql.language import utils
 from yaql.language import yaqltypes
 from yaql import tests
 
@@ -31,11 +30,11 @@ class TestEngine(tests.TestCase):
         sys.stderr = six.StringIO()
         try:
             debug_opts = dict(self.engine_options)
-            debug_opts["yaql.debug"] = True
+            debug_opts['yaql.debug'] = True
             yaql.factory.YaqlFactory().create(options=debug_opts)
             sys.stderr.seek(0)
             err_out = sys.stderr.read()
-            self.assertEqual("Generating LALR tables\n", err_out)
+            self.assertEqual('Generating LALR tables\n', err_out)
         finally:
             # put stderr back
             sys.stderr = copy
@@ -123,12 +122,12 @@ class TestEngine(tests.TestCase):
 
         self.assertEqual(
             (1, 2, (5, 7), {'kw1': 'x', 'kw2': None}),
-            fd(utils.NO_VALUE, self.engine, self.context)(
+            fd(self.engine, self.context)(
                 1, 2, 5, 7, kw1='x', kw2=None))
 
         self.assertEqual(
             (1, 5, (), {}),
-            fd(utils.NO_VALUE, self.engine, self.context)(1, b=5))
+            fd(self.engine, self.context)(1, b=5))
 
     def test_eval(self):
         self.assertEqual(

yaql/tests/test_legacy.py

@@ -50,29 +50,29 @@ class TestLegacy(yaql.tests.TestCase):
 
     def test_int(self):
         self.assertEqual(5, self.eval("'5'.int()"))
-        self.assertEqual(5, self.eval("5.2.int()"))
+        self.assertEqual(5, self.eval('5.2.int()'))
-        self.assertEqual(5, self.eval("int('5')"))
+        self.assertEqual(0, self.eval('null.int()'))
-        self.assertEqual(5, self.eval("int(5.2)"))
 
     def test_float(self):
         self.assertAlmostEqual(5.1, self.eval("'5.1'.float()"))
-        self.assertAlmostEqual(5.0, self.eval("5.float()"))
+        self.assertAlmostEqual(5.0, self.eval('5.float()'))
         self.assertAlmostEqual(5.1, self.eval("float('5.1')"))
         self.assertAlmostEqual(5.0, self.eval("float(5)"))
+        self.assertEqual(0, self.eval('null.float()'))
 
     def test_bool(self):
-        self.assertFalse(self.eval("null.bool()"))
+        self.assertFalse(self.eval('null.bool()'))
         self.assertFalse(self.eval("''.bool()"))
-        self.assertFalse(self.eval("0.bool()"))
+        self.assertFalse(self.eval('0.bool()'))
-        self.assertFalse(self.eval("false.bool()"))
+        self.assertFalse(self.eval('false.bool()'))
-        self.assertFalse(self.eval("[].bool()"))
+        self.assertFalse(self.eval('[].bool()'))
-        self.assertFalse(self.eval("{}.bool()"))
+        self.assertFalse(self.eval('{}.bool()'))
         self.assertTrue(self.eval("' '.bool()"))
-        self.assertTrue(self.eval("x.bool()"))
+        self.assertTrue(self.eval('x.bool()'))
-        self.assertTrue(self.eval("1.bool()"))
+        self.assertTrue(self.eval('1.bool()'))
-        self.assertTrue(self.eval("true.bool()"))
+        self.assertTrue(self.eval('true.bool()'))
-        self.assertTrue(self.eval("[1].bool()"))
+        self.assertTrue(self.eval('[1].bool()'))
-        self.assertTrue(self.eval("{a=>b}.bool()"))
+        self.assertTrue(self.eval('{a=>b}.bool()'))
 
     def test_filter(self):
         self.assertEqual(2, self.eval("list(1,2,3)[1]"))

yaql/tests/test_math.py

@@ -123,12 +123,14 @@ class TestMath(yaql.tests.TestCase):
         self.assertIsInstance(res, bool)
         self.assertTrue(res)
         self.assertFalse(self.eval('3 < 3'))
+        self.assertTrue(self.eval('2.5 < 3'))
 
     def test_gte(self):
         res = self.eval('5 >= 3')
         self.assertIsInstance(res, bool)
         self.assertTrue(res)
         self.assertTrue(self.eval('3 >= 3'))
+        self.assertTrue(self.eval('3.5 > 3'))
         self.assertFalse(self.eval('2 >= 3'))
 
     def test_lte(self):
@@ -140,10 +142,12 @@ class TestMath(yaql.tests.TestCase):
 
     def test_eq(self):
         self.assertTrue(self.eval('5 = 5'))
+        self.assertTrue(self.eval('1.0 = 1'))
         self.assertFalse(self.eval('5 = 6'))
 
     def test_neq(self):
         self.assertFalse(self.eval('5 != 5'))
+        self.assertFalse(self.eval('0 != 0.0'))
         self.assertTrue(self.eval('5 != 6'))
 
     def test_zero_division(self):
@@ -153,8 +157,14 @@ class TestMath(yaql.tests.TestCase):
         self.assertTrue(self.eval('with(random()) -> $ >= 0 and $ < 1'))
         self.assertTrue(self.eval('with(random(2, 5)) -> $ >= 2 and $ <= 5'))
 
+    def test_int(self):
+        self.assertEqual(5, self.eval("int('5')"))
+        self.assertEqual(5, self.eval('int(5.2)'))
+        self.assertEqual(0, self.eval('int(null)'))
+
     def test_float(self):
         self.assertAlmostEqual(-1.23, self.eval("float('-1.23')"))
+        self.assertEqual(0.0, self.eval('float(null)'))
 
     def test_bitwise_or(self):
         self.assertEqual(3, self.eval('bitwiseOr(1, 3)'))

yaql/tests/test_regex.py

@@ -20,6 +20,10 @@ class TestRegex(yaql.tests.TestCase):
         self.assertTrue(self.eval("regex('a.b').matches(axb)"))
         self.assertFalse(self.eval("regex('a.b').matches(abx)"))
 
+    def test_matches_string_method(self):
+        self.assertTrue(self.eval("axb.matches('a.b')"))
+        self.assertFalse(self.eval("abx.matches('a.b')"))
+
     def test_matches_operator_regex(self):
         self.assertTrue(self.eval("axb =~ regex('a.b')"))
         self.assertFalse(self.eval("abx =~ regex('a.b')"))

yaql/tests/test_strings.py

@@ -68,6 +68,9 @@ class TestStrings(yaql.tests.TestCase):
     def test_join(self):
         self.assertEqual('some-text', self.eval("[some, text].join('-')"))
 
+    def test_join_pythonic(self):
+        self.assertEqual('some-text', self.eval("'-'.join([some, text])"))
+
     def test_is_empty(self):
         self.assertTrue(self.eval("isEmpty('')"))
         self.assertTrue(self.eval("isEmpty(null)"))

yaql/tests/test_system.py

@@ -66,3 +66,72 @@ class TestSystem(yaql.tests.TestCase):
         self.assertEqual(
             3,
             self.eval('[2].select($ + 1).assert(len($) = 1).first()'))
+
+    def test_lambda_passing(self):
+        delegate = lambda x: x ** 2
+        self.assertEqual(
+            9,
+            self.eval('$(3)', data=delegate))
+
+    def test_function_passing(self):
+        def func(x, y, z):
+            return (x - y) * z
+
+        context = self.context
+        context['func'] = func
+        self.assertEqual(
+            8,
+            self.eval('$func(5, z => 2, y => 1)', data=func))
+
+    def test_lambda_expression(self):
+        delegate = lambda x: x ** 2
+        self.assertEqual(
+            9,
+            self.eval('$.x[0](3)', data={'x': [delegate]}))
+
+        self.assertEqual(
+            9,
+            self.eval('($.x[0])(3)', data={'x': [delegate]}))
+
+    def test_2nd_order_lambda(self):
+        delegate = lambda y: lambda x: x ** y
+        self.assertEqual(
+            16,
+            self.eval('$(2)(4)', data=delegate))
+
+    def test_2nd_order_lambda_expression(self):
+        delegate = lambda y: {'key': lambda x: x ** y}
+        self.assertEqual(
+            16,
+            self.eval('$(2)[key](4)', data=delegate))
+
+    def test_2nd_order_lambda_collection_expression(self):
+        delegate = lambda y: lambda x: y ** x
+        self.assertEqual(
+            [1, 8, 27],
+            self.eval(
+                'let(func => $) -> [1, 2, 3].select($func($)).select($(3))',
+                data=delegate))
+
+    def test_lambda_func(self):
+        self.assertEqual(
+            [2, 4, 6],
+            self.eval('let(func => lambda(2 * $)) -> $.select($func($))',
+                      data=[1, 2, 3]))
+
+    def test_lambda_func_2nd_order(self):
+        self.assertEqual(
+            5,
+            self.eval('lambda(let(outer => $) -> lambda($outer - $))(7)(2)'))
+
+    def test_lambda_closure(self):
+        data = [1, 2, 3, 4, 5, 6]
+        self.assertEqual([3, 4, 5, 6], self.eval(
+            '$.where(lambda($ > 3)($+1))',
+            data=data))
+
+        # lambda can access value from "where"'s context
+        # so we can omit parameter
+        self.assertEqual([4, 5, 6], self.eval(
+            '$.where(lambda($ > 3)())',
+            data=data))