Documentation structure update
Updated README to include some quickstart information and information on what YAQL is. Includes stubs for further expansion of YAQL documentation and proposes general structure of how YAQL documentation might look like. Targets blueprint: yaql-docs Change-Id: I65885bd132bb3efa5798aa0cb6f1774b2348a44e
This commit is contained in:
Kirill Zaitsev
parent
ab32af08be
commit
f134c743b0
155
README.rst
155
README.rst
@ -1,101 +1,68 @@
|
||||
YAQL - Yet Another Query Language
|
||||
=================================
|
||||
YAQL: Yet Another Query Language
|
||||
================================
|
||||
|
||||
At the beginning of millennium the growing trend towards data formats standardization and application integrability made
|
||||
XML extremely popular. XML became lingua franca of the data. Applications tended to process lots of XML files ranging
|
||||
from small config files to very large datasets. As these data often had a complex structure with many levels of
|
||||
nestedness it is quickly became obvious that there is a need for specially crafted domain specific languages to query
|
||||
these data sets. This is how XPath and later XQL were born.
|
||||
YAQL (Yet Another Query Language) is an embeddable and extensible query
|
||||
language, that allows performing complex queries against arbitrary objects. It
|
||||
has a vast and comprehensive standard library of frequently used querying
|
||||
functions and can be extend even further with user-specified functions. YAQL is
|
||||
written in python and is distributed via PyPI.
|
||||
|
||||
With later popularization of REST services and Web 2.0 JSON started to take XML’s place. JSON’s main advantage (besides
|
||||
being simpler than XML) is that is closely reassembles data structures found in most programming languages (arrays,
|
||||
dictionaries, scalars) making it very convenient for data serialization. As JSON lacked all the brilliant XML-related
|
||||
technologies like XSLT, XML Schema, XPath etc. various attempts to develop similar languages for JSON were made. One of
|
||||
those efforts was JSONPath library developed in 2007 by Stefan Gössner. Initial implementation was for PHP and
|
||||
JavaScript languages, but later on ports to other languages including Python were written.
|
||||
Quickstart
|
||||
----------
|
||||
Install the latest version of yaql:
|
||||
|
||||
JSONPath allows navigation and querying, well, JSONs.
|
||||
Suppose we have JSON as in following:
|
||||
.. code-block:: console
|
||||
|
||||
{
|
||||
"customers": [
|
||||
{
|
||||
"customer_id": 1,
|
||||
"name": "John",
|
||||
"orders": [{
|
||||
"order_id": 1,
|
||||
"item": "Guitar",
|
||||
"quantity": 1
|
||||
}]
|
||||
},{
|
||||
"customer_id": 2,
|
||||
"name": "Paul",
|
||||
"orders": [ {
|
||||
"order_id": 2,
|
||||
"item": "Banjo",
|
||||
"quantity": 2
|
||||
},{
|
||||
"order_id": 3,
|
||||
"item": "Piano",
|
||||
"quantity": 1
|
||||
}]
|
||||
}
|
||||
]
|
||||
}
|
||||
pip install yaql>=1.0.0
|
||||
..
|
||||
|
||||
Run yaql REPL:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
yaql
|
||||
..
|
||||
|
||||
Load a json file:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
yaql> @load my_file.json
|
||||
..
|
||||
|
||||
Check it loaded to current context, i.e. `$`:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
yaql> $
|
||||
|
||||
..
|
||||
|
||||
Run some queries:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
yaql> $.customers
|
||||
...
|
||||
yaql> $.customers.orders
|
||||
...
|
||||
yaql> $.customers.where($.age > 18)
|
||||
...
|
||||
yaql> $.customers.groupBy($.sex)
|
||||
...
|
||||
yaql> $.customers.where($.orders.len() >= 1 or name = "John")
|
||||
..
|
||||
|
||||
Project Resources
|
||||
-----------------
|
||||
|
||||
* `Official Documentation <http://yaql.readthedocs.org>`_
|
||||
|
||||
* Project status, bugs, and blueprints are tracked on
|
||||
`Launchpad <https://launchpad.net/yaql>`_
|
||||
|
||||
|
||||
then
|
||||
License
|
||||
-------
|
||||
|
||||
`jsonpath(data, "$.customers[0].name") -> [‘John’]`
|
||||
`jsonpath(data, "$.customers[*].orders[*].order_id") -> [1, 2, 3]`
|
||||
|
||||
But what if we need, for example to find order having ID = 2? Here is how it done in JSONPath:
|
||||
|
||||
`jsonpath(data, "$.customers[*].orders[?(@.order_id == 2)") -> [{'order_id': 2, 'item': 'Banjo', 'quantity': 2}]`
|
||||
|
||||
The construct `[?(expression)]` allows to filter items using any Python expression in our case. `@` character is
|
||||
replaced with current value and then the whole expression is evaluated. Evaluation of arbitrary Python expression
|
||||
requires using `eval()` function unless one wants to develop his own complete parser and interpreter of Python
|
||||
programming language. Needless to say that `eval()` is a great security breach. If JSONPath expressions are used to
|
||||
simplify program logic it would not be a big deal, but what if JSONPath is written by program users?
|
||||
|
||||
JSONPath expression is just a plain string. There is no such concept as parameter. That is if one want to find order
|
||||
having ID = some variable value he has to dynamically construct expression string using string formatting or
|
||||
concatenation. And again that is might be okay for internal usage but would became difficult for external usage and also
|
||||
open the doors for injection attacks (remember SQL injection?)
|
||||
|
||||
Another limitation of JSONPath is JSON itself. Technically speaking JSONPath operates not on the JSON itself (i.e. text
|
||||
representation) but on a JSON-like object model that is mixture of arrays, dictionaries and scalar values. But what is
|
||||
one want to query object model consisting of custom objects? What if some parts of this model are dynamically computed?
|
||||
Or the model is a graph rather than a tree?
|
||||
|
||||
It seems like JSONPath is good enough to use in Python code when you can `eval()` things and have many helper function
|
||||
to work with data besides JSONPath capabilities but is not enough for external use when you need to have sufficient
|
||||
power to query model without manual coding and have it still secure.
|
||||
This is why we designed YAQL. YAQL follows the JSONPath ideas and has very similar syntax but offers much more for data
|
||||
querying.
|
||||
|
||||
Expressions are quite similar to JSONPath. Here is how examples above can be translated to YAQL:
|
||||
|
||||
`$.customers[0].name -> $.customers[0].name (no change)`
|
||||
`$.customers[*].orders[*].order_id -> $.customers.orders.order_id`
|
||||
|
||||
the main addition to JSONPath is functions and operators. Consider the following YAQL expressions:
|
||||
|
||||
`$.customers.orders[$.quantity > 0].quantity.sum() -> 4`
|
||||
`$.customers.orders.select($.quantity * $.quantity).sum() -> 6`
|
||||
`$.customers.orders.order_id.orderDesc($) -> [3, 2, 1]`
|
||||
`$.customers.orders.order_id.orderDesc($).take(2) -> [3, 2]`
|
||||
`$.customers.orders.order_id.orderDesc($).first() -> 3`
|
||||
|
||||
Does it mean that YAQL has large built-in function and operator library?. Yes, YAQL library has a out of the box large
|
||||
set of commonly used functions. But they are not built-in. All the functions and operators (which are also function:
|
||||
`a + b = operator_+(a, b)` etc) are user-supplied. User is free to add other functions that could be used in expressions
|
||||
and to remove standard ones.
|
||||
|
||||
JSONPath library needs 2 arguments - input JSON data and an a expression. YAQL library requires third
|
||||
parameter - context.
|
||||
|
||||
Context is a repository of functions and variables that can be used in expressions. So all the functions above are just
|
||||
ordinary Python functions that are registered in Context object. But because they all need to be registered in Context
|
||||
user can always customize them, add his own model-specific ones and have full control over the expression evaluation.
|
||||
Apache License Version 2.0 http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
@ -4,16 +4,33 @@
|
||||
contain the root `toctree` directive.
|
||||
|
||||
Welcome to yaql's documentation!
|
||||
========================================================
|
||||
================================
|
||||
|
||||
Contents:
|
||||
Introduction
|
||||
~~~~~~~~~~~~
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
|
||||
readme
|
||||
installation
|
||||
what_is_yaql
|
||||
|
||||
Usage
|
||||
~~~~~
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
usage
|
||||
language_description
|
||||
standard_library
|
||||
|
||||
Contributing
|
||||
~~~~~~~~~~~~
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
contributing
|
||||
|
||||
Indices and tables
|
||||
|
||||
@ -1,12 +0,0 @@
|
||||
============
|
||||
Installation
|
||||
============
|
||||
|
||||
At the command line::
|
||||
|
||||
$ pip install yaql
|
||||
|
||||
Or, if you have virtualenvwrapper installed::
|
||||
|
||||
$ mkvirtualenv yaql
|
||||
$ pip install yaql
|
||||
19
doc/source/language_description.rst
Normal file
19
doc/source/language_description.rst
Normal file
@ -0,0 +1,19 @@
|
||||
Language description
|
||||
====================
|
||||
|
||||
This section is not ready yet
|
||||
|
||||
Syntax
|
||||
~~~~~~
|
||||
|
||||
Types
|
||||
~~~~~
|
||||
|
||||
Operators
|
||||
~~~~~~~~~
|
||||
|
||||
Context
|
||||
~~~~~~~
|
||||
|
||||
$ variable
|
||||
~~~~~~~~~~
|
||||
4
doc/source/standard_library.rst
Normal file
4
doc/source/standard_library.rst
Normal file
@ -0,0 +1,4 @@
|
||||
Standard Library
|
||||
================
|
||||
|
||||
This section is not ready yet.
|
||||
@ -1,7 +1,13 @@
|
||||
========
|
||||
Usage
|
||||
========
|
||||
=====
|
||||
|
||||
To use yaql in a project::
|
||||
This section is not ready yet.
|
||||
|
||||
import yaql
|
||||
REPL
|
||||
~~~~
|
||||
|
||||
Embedding YAQL
|
||||
~~~~~~~~~~~~~~
|
||||
|
||||
Extending YAQL
|
||||
~~~~~~~~~~~~~~
|
||||
|
||||
22
doc/source/what_is_yaql.rst
Normal file
22
doc/source/what_is_yaql.rst
Normal file
@ -0,0 +1,22 @@
|
||||
What is YAQL
|
||||
============
|
||||
|
||||
YAQL is a general purpose query language, that is designed to operate on
|
||||
objects of arbitrary complexity.
|
||||
YAQL has a large standard library of functions for filtering, grouping and
|
||||
aggregation of data. At the same time YAQL allows you to extend it by
|
||||
defining your own functions.
|
||||
|
||||
Why YAQL?
|
||||
=========
|
||||
|
||||
So why bother and create another solution for a task, that has been addressed
|
||||
by many before us? Obviously because we were not satisfied with flexibility
|
||||
and/or quality of any existing solution. Most notably we needed a tool for json
|
||||
data, that would support some complex data transformations.
|
||||
YAQL is a pure-python library and therefore is easily embeddable in any python
|
||||
application.
|
||||
YAQL is designed to be human-readable and has a SQL-like feel and look. It is
|
||||
inspired in part by LINQ for .NET.
|
||||
Since YAQL is extensible and embeddable it makes a perfect choice for becoming
|
||||
the basis for your DSLs.
|
||||
Loading…
Reference in New Issue
Block a user