Language reference
==================

YAQL is a single expression language and as such does not have any block
constructs, line formatting, end of statement marks or comments. The expression
can be of any length. All whitespace characters (including newline) that are
not enclosed in quote marks are stripped. Thus, the expressions may span
multiple lines.

Expressions consist of:

* Literals
* Keywords
* Variable access
* Function calls
* Binary and unary operators
* List expressions
* Dictionary expressions
* Index expressions
* Delegate expressions

Terminology
~~~~~~~~~~~

* `YAQL` - the name of the language - acronym for `Yet Another Query Language`

* `yaql` - Python implementation of the YAQL language (this package)

* `expression` - a YAQL query that takes context as an input and produces
  result value

* `context` - an object that (directly or indirectly) holds all the data
  available to expression and all the function implementations accessible to
  expression

* `host` - the application that hosts the yaql interpreter. The host uses yaql
  to evaluate expressions, provides initial data, and decides which functions
  are going to be available to the expression. The host has ultimate power
  to customize yaql - provide additional functions, operators, decide not to
  use standard library or use only parts of it, override function and operator
  behavior

* `variable` - any data item that is available through the context

* `function` - a Python callable that is exposed to the YAQL expression and
  can be called either explicitly or implicitly

* `delegate` - a Python callable that is available as a context variable (in
  expression data rather than registered in context)

* `operator` - a form of implicit function on one (unary operator) or two
  (binary operator) operands

* `alphanumeric` - consists of latin letters and digits (`A-Z`, `a-z`, `0-9`)


Literals
~~~~~~~~

Literals refer to fixed values in expressions. YAQL has the following literals:

* Integer literals: ``123``
* Floating point literals: ``1.23``, ``1.0``
* Boolean and null literals represented by `keywords` (see below)
* String literals enclosed in either single (') or double (") quotes:
  ``"abc"``, ``'def'``. The backslash (\) character is used to escape
  characters that otherwise have a special meaning, such as newline, backslash
  itself, or the quote character
* Verbatim strings enclosed in back quote characters, for example ```abc```,
  are used to suppress escape sequences. This is equivalent to ``r'strings'``
  in Python and is especially useful for regular expressions


Keywords
~~~~~~~~

Keyword is a sequence of characters that conforms to the following criteria:

* Consists of non-zero alphanumeric characters and an underscore (`_`)
* Doesn't start with a digit
* Doesn't start with two underscore characters (`__`)
* Is not enclosed in quote marks of any type

YAQL has only three predefined keywords: `true`, `false`, and `null` that have
the value of similar JSON keywords.

There are also four keyword operators: `and`, `or`, `not`, `in`. However, this
list is not fixed. The yaql host may decide to have additional keyword
operators or not to have any of the four aforementioned keywords.

All other keywords have the value of their string representation. Thus, except
for the predefined keywords and operators they can be considered as string
literals and can be used anywhere where string is expected. However the
opposite is not true. That is, keywords can be used as string literals but
string literals cannot be used where a token is expected.

Examples:

* ``John + Snow`` - the same as ``"John" + "Snow"``
* ``true + love`` - syntactically valid, but cannot be evaluated because
  there is no plus operator that accepts boolean and string (unless you define
  one)
* ``not true`` - evaluates to `false`, `not` is an operator
* ``"foo"()`` - invalid expression because the function name must be a token
* ``John Snow`` - invalid expression - two tokens with no operator between
  them


Variable access
~~~~~~~~~~~~~~~

Each YAQL expression is a function that takes inputs (arguments) and produces
the result value (usually by doing some computations on those inputs).
Expressions get the input through a `context` - an object that holds all the
data and a list of functions, available for expression.

Besides the argument values, expressions may populate additional data items
to the context. All these data are collectively known as a `variables` and
available to all parts of an expression (unless overwritten with another
value).

The syntax for accessing variable values is ``$variableName`` where
`variableName` is the name of the variable. Variable names may consist of
alphanumeric and underscore characters only. Unlike tokens, variable names
may start with digit, any number of underscores and even be an empty string.
By convention, the first (usually the single) function parameter is accessible
through ``$`` expression (i.e. empty string variable name) which is an alias
for ``$1``. The usual case is to pass the main expression data in a single
structure (document) and access it through the ``$`` variable.

If the variable with given name is not provided, it is assumed to be `null`.
There is no built-in syntax to check if a variable exists to distinguish cases
where it does not and when it is just set to null. However in the future such a
function might be added to yaql standard library.

When the yaql parser encounters the ``$variable`` expression, it automatically
translates it to the ``#get_context_data("$variable")`` function call.
By default, the `#get_context_data` function returns a variable value from the
current context. However the yaql host may decide to override it and provide
another behavior. For example, the host may try to look up the value in an
external data source (database) or throw an exception due to a missing
variable.


Function calls
~~~~~~~~~~~~~~

The power of YAQL comes from the fact that almost everything in YAQL is a
function call (explicit or implicit) and any function may be overridden
by the host. In YAQL there are two types of functions:

* `explicit function` - those that can be called from expressions

* `implicit (system) functions` - functions with predefined names that get
  called upon some operations. For example, ``2 + 3`` is translated to
  ``#operator_+(2, 3)``. In this case, `#operator_+` is the name of the
  implicit function. However, because ``#operator_+(2, 3)`` is not a valid YAQL
  expression (because of `#`), implicit functions cannot be called explicitly
  but still can be redefined by the host.

The syntax for explicit function is:

.. productionlist::
   call: funcName "(" [parameters] ")"
   funcName: token
   parameters: positionalParameters |
             : keywordParameters |
             : positionalParameters "," keywordParameters
   positionalParameters: parameter ("," parameter)*
   parameter: expression | empty-string
   keywordParameters: keywordParameter ("," keywordParameter)
   keywordParameter: parameterName "=>" expression
   parameterName: token

In simple words:

* The function name must be a token.
* Parameters may be positional, keyword or both. But keyword parameters
  may not come before positional.
* Positional parameters can be skipped if they have a default value, for
  example, ``foo(1,,3)``.
* Keyword arguments must have a token name that must match the parameter name
  in the function declaration. Therefore, you must know the function signature
  for the right name.

Examples:

* ``foo(2 + 3)``
* ``bar(hello, world)``
* ``baz(a,b, kwparam1 => c, kwparam2 => d)``

Functions have ultimate control over how they can be called. In particular:

* Each parameter may (and usually does) have an associated type check. That is,
  the function may specify that the expected parameter type and if it can be
  null.
* Usually, any parameters can be passed either by positional or keyword syntax.
  However, function declaration may force one particular way and make it
  positional-only or keyword-only.
* A function may have a variable number of positional (aka `*args`) and/or
  keyword (aka `**kwarg`) arguments.
* In most languages, function arguments are evaluated prior to function
  invocation. This is not always true in YAQL. In YAQL, a function may declare
  a lazy argument. In this case, it is not evaluated and the function
  implementation receives a passed value as a callable or even as an AST,
  depending on how the parameter was declared. Thus in YAQL there is no special
  syntax for lambdas. ``foo($ + 1)`` may mean either "call `foo` with value of
  ``$ + 1``" or "call `foo` with expression ``$ + 1`` as a parameter". In the
  latter case it corresponds to ``foo(lambda *args, **kwargs: args[0] + 1)`` in
  Python. Actual argument interpretation depends on the parameter declaration.
* Function may decide to disable keyword argument syntax altogether. For such
  functions, the ``name => expr`` expression will be interpreted as a
  positional parameter ``yaql.language.utils.MappingRule(name, expr)`` and
  the left side of `=>` can be any expression and not just a keyword. This
  allows for functions like ``switch($ > 0 => 1, $ < 0 => -1, $ = 0 => 0)``.

Additionally, there are three subtypes of explicit functions. Suppose that
there is a declared function ``foo(string, int)``. By default, the syntax to
call it will be ``foo(something, 123)``. But it can be declared as a `method`.
In this case, the syntax is going to be ``something.foo(123)``. Because of the
type checking, ``something.foo(123)`` will work since `something` is a
string, but not the ``123.foo(456)``. Thus `foo` becomes a method of a string
type.

A function may also be declared as being an extension method. If foo were to be
declared as an extension method it could be called both as a function
(``foo(string, int)``) and as a method (``something.foo(123)``).

YAQL makes use of a full function signature to determine which function
implementation needs to be executed. This allows several overloads of the same
function as long as they differ by parameter count or parameter type,
or anything else that allows unambiguous identification of the right overload
from the function call expression. For example, ``something.foo(123)`` may
be resolved to a completely different implementation of `foo` from that in
``foo(something, 123)`` if there are two functions with the name `foo` present
in the context, but one of them was declared as a function while the other as
a method. If several overloads are equally suitable for the call expression,
an `AmbiguousFunctionException` or `AmbiguousMethodException` exception gets
raised.


Operators
~~~~~~~~~

YAQL has both binary and unary operators, like most other languages do.
Parentheses and `=>` sequence are not considered as operators and handled
internally by the yaql parser. However, it is possible to configure yaql to use
sequence other than `=>` for that purpose.

The list of available operators is not fixed and can be modified by the host.
The following operators are available by default:

Binary operators:

+--------------------------+---------------------------------+
| Group                    | Operators                       |
+==========================+=================================+
| math operators           | `+`, `-`, `*`, `/`, `mod`       |
+--------------------------+---------------------------------+
| comparision operators    | `>`, `<`, `>=`, `<=`, `=`, `!=` |
+--------------------------+---------------------------------+
| logical operators        | `and`, `or`                     |
+--------------------------+---------------------------------+
| method/member access     | `.`, `?.`                       |
+--------------------------+---------------------------------+
| regex operators          | `=~`, `!~`                      |
+--------------------------+---------------------------------+
| membership operator      | `in`                            |
+--------------------------+---------------------------------+
| context passing operator | `->`                            |
+--------------------------+---------------------------------+


Unary operators:

+--------------------------+---------------------------------+
| Group                    | Operators                       |
+==========================+=================================+
| math operators           | `+`, `-`                        |
+--------------------------+---------------------------------+
| logical operators        | `not`                           |
+--------------------------+---------------------------------+


YAQL supports for both prefix and suffix unary operators. However, only the
prefix operators are provided by default.

In YAQL there are no built-in operators. The parser is given a list of all
possible operator names (symbols), their associativity, precedence, and type,
but it knows nothing about what operators are applicable for what operands.
Each time a parser recognizes the ``X OP Y`` construct and `OP` is a known
binary operator name, it translates the expression to ``#operator_OP(X, Y)``.
Thus. ``2 + 3`` becomes ``#operator_+(2, 3)`` where `#operator_+` is an
implicit function with several implementations including the one for number
addition and defined in standard library. The host may override it and even
completely disable it. For unary operators, ``OP X`` (or ``X OP`` for suffix
unary operators) becomes ``#unary_operator_OP(X)``.

Upon yaql parser initialization, an operator might be given an alias name.
In such cases, ``X OP Y`` is translated to ``*ALIAS(X, Y)`` and ``OP X`` to
``*ALIAS(X)``. This decouples the operator implementation from the operator
symbol. For example, the `=` operator has the `equal` alias. The host may
configure yaql to have the `==` operator instead of `=` keeping the same alias
so that operator implementation and all its consumers work equally well for the
new operator symbol. In default configuration only `=` and `!=` operators have
alias names.

For information on default operators, see the YAQL standard library reference.


List expressions
~~~~~~~~~~~~~~~~

List expressions have the following form:

.. productionlist::
   listExpression: "[" [expressions] "]"
   expressions: expression ("," expression)*

When a yaql parser encounters an expression of the form ``[A, B, C]``, it
translates it into ``#list(A, B, C)`` (for arbitrary number of arguments).

Default `#list` function implementation in standard library produces a list
(tuple) comprised of given elements. However, the host might decide to give it
a different implementation.


Map expressions
~~~~~~~~~~~~~~~

Map expressions have the following form:

.. productionlist::
   mapExpression: "{" [mappings] "}"
   mappings: mapping ("," mapping)*
   mapping: expression "=>" expression

When a yaql parser encounters an expression of the form ``{A => X, B => Y}``,
it translates it into ``#map(A => X, B => Y)``.

The default `#map` implementation disables the keyword arguments syntax and
thus receives a variable length list of mappings, which allows dictionary
keys to be expressions rather than a keyword. It returns a (frozen) dictionary
that itself can be used as a key in another map expression. For example,
``{{a => b} => {[2 + 2, 2 * 2] => 4}}`` is a valid YAQL expression though
yaql REPL utility will fail to display its output due to the fact that it is
not JSON-compatible.


Index expressions
~~~~~~~~~~~~~~~~~

Index expressions have the following form:

.. productionlist::
   indexExpression: expression listExpression


Examples:

* ``[1, 2, 3][0]``
* ``$arr[$index + 1]``
* ``{foo => 1, bar => 2}[foo]``

When a yaql parser encounters such an expression, it translates it into
``#indexer(expression, index)``.

The standard library provides a number of `#indexer` implementations for
different types.

The right side of the index expression is a list expression. Therefore, an
expression like ``$foo[1, x, null]`` is also a valid YAQL expression and will
be translated to ``#indexer($foo, 1, x, null)``. However, any attempt to
evaluate such expression will result in `NoMatchingFunctionException` exception
because there is no `#indexer` implementation that accepts such arguments
(unless the host defines one).


Delegate expressions
~~~~~~~~~~~~~~~~~~~~

Delegate expressions is an optional language feature that is disabled by
default. It makes possible to pass delegates (callables) as part of the context
data and invoke them from the expression. It has the same syntax as explicit
function calls with the only difference being that instead of function name
(keyword) there is a non-keyword expression that must produce the delegate.

Examples:

* ``$foo(1, arg => 2)`` - call delegate returned by ``$foo`` with parameters
  ``(1, arg => 2)``

* ``[$foo, $bar][0](x)`` - the same as ``$foo(x)``

* ``foo()()`` - can be written as ``(foo())()`` - ``foo()`` must return a
  delegate

Delegate expressions are translated into ``#call(callable, arguments)``.
Thus ``$foo(1, 2)`` becomes ``#call($foo, 1, 2)``.

The default implementation of ``#call`` invokes the result of the evaluation
of its first arguments with the given arguments.