diff --git a/README.rst b/README.rst index e583254..4a528ce 100644 --- a/README.rst +++ b/README.rst @@ -1,101 +1,68 @@ -YAQL - Yet Another Query Language -================================= +YAQL: Yet Another Query Language +================================ -At the beginning of millennium the growing trend towards data formats standardization and application integrability made -XML extremely popular. XML became lingua franca of the data. Applications tended to process lots of XML files ranging -from small config files to very large datasets. As these data often had a complex structure with many levels of -nestedness it is quickly became obvious that there is a need for specially crafted domain specific languages to query -these data sets. This is how XPath and later XQL were born. +YAQL (Yet Another Query Language) is an embeddable and extensible query +language, that allows performing complex queries against arbitrary objects. It +has a vast and comprehensive standard library of frequently used querying +functions and can be extend even further with user-specified functions. YAQL is +written in python and is distributed via PyPI. -With later popularization of REST services and Web 2.0 JSON started to take XML’s place. JSON’s main advantage (besides -being simpler than XML) is that is closely reassembles data structures found in most programming languages (arrays, -dictionaries, scalars) making it very convenient for data serialization. As JSON lacked all the brilliant XML-related -technologies like XSLT, XML Schema, XPath etc. various attempts to develop similar languages for JSON were made. One of -those efforts was JSONPath library developed in 2007 by Stefan Gössner. Initial implementation was for PHP and -JavaScript languages, but later on ports to other languages including Python were written. +Quickstart +---------- +Install the latest version of yaql: -JSONPath allows navigation and querying, well, JSONs. -Suppose we have JSON as in following: +.. code-block:: console - { - "customers": [ - { - "customer_id": 1, - "name": "John", - "orders": [{ - "order_id": 1, - "item": "Guitar", - "quantity": 1 - }] - },{ - "customer_id": 2, - "name": "Paul", - "orders": [ { - "order_id": 2, - "item": "Banjo", - "quantity": 2 - },{ - "order_id": 3, - "item": "Piano", - "quantity": 1 - }] - } - ] - } + pip install yaql>=1.0.0 +.. + +Run yaql REPL: + +.. code-block:: console + + yaql +.. + +Load a json file: + +.. code-block:: console + + yaql> @load my_file.json +.. + +Check it loaded to current context, i.e. `$`: + +.. code-block:: console + + yaql> $ + +.. + +Run some queries: + +.. code-block:: console + + yaql> $.customers + ... + yaql> $.customers.orders + ... + yaql> $.customers.where($.age > 18) + ... + yaql> $.customers.groupBy($.sex) + ... + yaql> $.customers.where($.orders.len() >= 1 or name = "John") +.. + +Project Resources +----------------- + +* `Official Documentation `_ + +* Project status, bugs, and blueprints are tracked on + `Launchpad `_ -then +License +------- -`jsonpath(data, "$.customers[0].name") -> [‘John’]` -`jsonpath(data, "$.customers[*].orders[*].order_id") -> [1, 2, 3]` - -But what if we need, for example to find order having ID = 2? Here is how it done in JSONPath: - -`jsonpath(data, "$.customers[*].orders[?(@.order_id == 2)") -> [{'order_id': 2, 'item': 'Banjo', 'quantity': 2}]` - -The construct `[?(expression)]` allows to filter items using any Python expression in our case. `@` character is -replaced with current value and then the whole expression is evaluated. Evaluation of arbitrary Python expression -requires using `eval()` function unless one wants to develop his own complete parser and interpreter of Python -programming language. Needless to say that `eval()` is a great security breach. If JSONPath expressions are used to -simplify program logic it would not be a big deal, but what if JSONPath is written by program users? - -JSONPath expression is just a plain string. There is no such concept as parameter. That is if one want to find order -having ID = some variable value he has to dynamically construct expression string using string formatting or -concatenation. And again that is might be okay for internal usage but would became difficult for external usage and also -open the doors for injection attacks (remember SQL injection?) - -Another limitation of JSONPath is JSON itself. Technically speaking JSONPath operates not on the JSON itself (i.e. text -representation) but on a JSON-like object model that is mixture of arrays, dictionaries and scalar values. But what is -one want to query object model consisting of custom objects? What if some parts of this model are dynamically computed? -Or the model is a graph rather than a tree? - -It seems like JSONPath is good enough to use in Python code when you can `eval()` things and have many helper function -to work with data besides JSONPath capabilities but is not enough for external use when you need to have sufficient -power to query model without manual coding and have it still secure. -This is why we designed YAQL. YAQL follows the JSONPath ideas and has very similar syntax but offers much more for data -querying. - -Expressions are quite similar to JSONPath. Here is how examples above can be translated to YAQL: - -`$.customers[0].name -> $.customers[0].name (no change)` -`$.customers[*].orders[*].order_id -> $.customers.orders.order_id` - -the main addition to JSONPath is functions and operators. Consider the following YAQL expressions: - -`$.customers.orders[$.quantity > 0].quantity.sum() -> 4` -`$.customers.orders.select($.quantity * $.quantity).sum() -> 6` -`$.customers.orders.order_id.orderDesc($) -> [3, 2, 1]` -`$.customers.orders.order_id.orderDesc($).take(2) -> [3, 2]` -`$.customers.orders.order_id.orderDesc($).first() -> 3` - -Does it mean that YAQL has large built-in function and operator library?. Yes, YAQL library has a out of the box large -set of commonly used functions. But they are not built-in. All the functions and operators (which are also function: -`a + b = operator_+(a, b)` etc) are user-supplied. User is free to add other functions that could be used in expressions -and to remove standard ones. - -JSONPath library needs 2 arguments - input JSON data and an a expression. YAQL library requires third -parameter - context. - -Context is a repository of functions and variables that can be used in expressions. So all the functions above are just -ordinary Python functions that are registered in Context object. But because they all need to be registered in Context -user can always customize them, add his own model-specific ones and have full control over the expression evaluation. +Apache License Version 2.0 http://www.apache.org/licenses/LICENSE-2.0 diff --git a/doc/source/index.rst b/doc/source/index.rst index 8212b33..f0aca26 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -4,16 +4,33 @@ contain the root `toctree` directive. Welcome to yaql's documentation! -======================================================== +================================ -Contents: +Introduction +~~~~~~~~~~~~ .. toctree:: :maxdepth: 2 readme - installation + what_is_yaql + +Usage +~~~~~ + +.. toctree:: + :maxdepth: 1 + usage + language_description + standard_library + +Contributing +~~~~~~~~~~~~ + +.. toctree:: + :maxdepth: 1 + contributing Indices and tables diff --git a/doc/source/installation.rst b/doc/source/installation.rst deleted file mode 100644 index 0f0b3b3..0000000 --- a/doc/source/installation.rst +++ /dev/null @@ -1,12 +0,0 @@ -============ -Installation -============ - -At the command line:: - - $ pip install yaql - -Or, if you have virtualenvwrapper installed:: - - $ mkvirtualenv yaql - $ pip install yaql \ No newline at end of file diff --git a/doc/source/language_description.rst b/doc/source/language_description.rst new file mode 100644 index 0000000..b347bca --- /dev/null +++ b/doc/source/language_description.rst @@ -0,0 +1,19 @@ +Language description +==================== + +This section is not ready yet + +Syntax +~~~~~~ + +Types +~~~~~ + +Operators +~~~~~~~~~ + +Context +~~~~~~~ + +$ variable +~~~~~~~~~~ diff --git a/doc/source/standard_library.rst b/doc/source/standard_library.rst new file mode 100644 index 0000000..4d10394 --- /dev/null +++ b/doc/source/standard_library.rst @@ -0,0 +1,4 @@ +Standard Library +================ + +This section is not ready yet. diff --git a/doc/source/usage.rst b/doc/source/usage.rst index 6f73a9e..75d5701 100644 --- a/doc/source/usage.rst +++ b/doc/source/usage.rst @@ -1,7 +1,13 @@ -======== Usage -======== +===== -To use yaql in a project:: +This section is not ready yet. - import yaql \ No newline at end of file +REPL +~~~~ + +Embedding YAQL +~~~~~~~~~~~~~~ + +Extending YAQL +~~~~~~~~~~~~~~ diff --git a/doc/source/what_is_yaql.rst b/doc/source/what_is_yaql.rst new file mode 100644 index 0000000..9798c5d --- /dev/null +++ b/doc/source/what_is_yaql.rst @@ -0,0 +1,22 @@ +What is YAQL +============ + +YAQL is a general purpose query language, that is designed to operate on +objects of arbitrary complexity. +YAQL has a large standard library of functions for filtering, grouping and +aggregation of data. At the same time YAQL allows you to extend it by +defining your own functions. + +Why YAQL? +========= + +So why bother and create another solution for a task, that has been addressed +by many before us? Obviously because we were not satisfied with flexibility +and/or quality of any existing solution. Most notably we needed a tool for json +data, that would support some complex data transformations. +YAQL is a pure-python library and therefore is easily embeddable in any python +application. +YAQL is designed to be human-readable and has a SQL-like feel and look. It is +inspired in part by LINQ for .NET. +Since YAQL is extensible and embeddable it makes a perfect choice for becoming +the basis for your DSLs.