diff --git a/README.md b/README.md index 016b88e..83afa1c 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,101 @@ -yaql +YAQL - Yet Another Query Language ==== -yet another query language +At the beginning of millennium the growing trend towards data formats standardization and application integrability made +XML extremely popular. XML became lingua franca of the data. Applications tended to process lots of XML files ranging +from small config files to very large datasets. As these data often had a complex structure with many levels of +nestedness it is quickly became obvious that there is a need for specially crafted domain specific languages to query +these data sets. This is how XPath and later XQL were born. + +With later popularization of REST services and Web 2.0 JSON started to take XML’s place. JSON’s main advantage (besides +being simpler than XML) is that is closely reassembles data structures found in most programming languages (arrays, +dictionaries, scalars) making it very convenient for data serialization. As JSON lacked all the brilliant XML-related +technologies like XSLT, XML Schema, XPath etc. various attempts to develop similar languages for JSON were made. One of +those efforts was JSONPath library developed in 2007 by Stefan Gössner. Initial implementation was for PHP and +JavaScript languages, but later on ports to other languages including Python were written. + +JSONPath allows navigation and querying, well, JSONs. +Suppose we have JSON as in following: + + { + "customers": [ + { + "customer_id": 1, + "name": "John", + "orders": [{ + "order_id": 1, + "item": "Guitar", + "quantity": 1 + }] + },{ + "customer_id": 2, + "name": "Paul", + "orders": [ { + "order_id": 2, + "item": "Banjo", + "quantity": 2 + },{ + "order_id": 3, + "item": "Piano", + "quantity": 1 + }] + } + ] + } + + +then + +`jsonpath(data, "$.customers[0].name") -> [‘John’]` +`jsonpath(data, "$.customers[*].orders[*].order_id") -> [1, 2, 3]` + +But what if we need, for example to find order having ID = 2? Here is how it done in JSONPath: + +`jsonpath(data, "$.customers[*].orders[?(@.order_id == 2)") -> [{'order_id': 2, 'item': 'Banjo', 'quantity': 2}]` + +The construct `[?(expression)]` allows to filter items using any Python expression in our case. `@` character is +replaced with current value and then the whole expression is evaluated. Evaluation of arbitrary Python expression +requires using `eval()` function unless one wants to develop his own complete parser and interpreter of Python +programming language. Needless to say that `eval()` is a great security breach. If JSONPath expressions are used to +simplify program logic it would not be a big deal, but what if JSONPath is written by program users? + +JSONPath expression is just a plain string. There is no such concept as parameter. That is if one want to find order +having ID = some variable value he has to dynamically construct expression string using string formatting or +concatenation. And again that is might be okay for internal usage but would became difficult for external usage and also +open the doors for injection attacks (remember SQL injection?) + +Another limitation of JSONPath is JSON itself. Technically speaking JSONPath operates not on the JSON itself (i.e. text +representation) but on a JSON-like object model that is mixture of arrays, dictionaries and scalar values. But what is +one want to query object model consisting of custom objects? What if some parts of this model are dynamically computed? +Or the model is a graph rather than a tree? + +It seems like JSONPath is good enough to use in Python code when you can `eval()` things and have many helper function +to work with data besides JSONPath capabilities but is not enough for external use when you need to have sufficient +power to query model without manual coding and have it still secure. +This is why we designed YAQL. YAQL follows the JSONPath ideas and has very similar syntax but offers much more for data +querying. + +Expressions are quite similar to JSONPath. Here is how examples above can be translated to YAQL: + +`$.customers[0].name -> $.customers[0].name (no change)` +`$.customers[*].orders[*].order_id -> $.customers.orders.order_id` + +the main addition to JSONPath is functions and operators. Consider the following YAQL expressions: + +`$.customers.orders[$.quantity > 0].quantity.sum() -> 4` +`$.customers.orders.select($.quantity * $.quantity).sum() -> 6` +`$.customers.orders.order_id.orderDesc($) -> [3, 2, 1]` +`$.customers.orders.order_id.orderDesc($).take(2) -> [3, 2]` +`$.customers.orders.order_id.orderDesc($).first() -> 3` + +Does it mean that YAQL has large built-in function and operator library?. Yes, YAQL library has a out of the box large +set of commonly used functions. But they are not built-in. All the functions and operators (which are also function: +`a + b = operator_+(a, b)` etc) are user-supplied. User is free to add other functions that could be used in expressions +and to remove standard ones. + +JSONPath library needs 2 arguments - input JSON data and an a expression. YAQL library requires third +parameter - context. + +Context is a repository of functions and variables that can be used in expressions. So all the functions above are just +ordinary Python functions that are registered in Context object. But because they all need to be registered in Context +user can always customize them, add his own model-specific ones and have full control over the expression evaluation. \ No newline at end of file