151 lines
4.6 KiB
Markdown
151 lines
4.6 KiB
Markdown
Please note that this API information is being superseded by a new client facing API (under development) This file will stay in place for now, but needs to be removed when the newer API is in place. Please see API.md for the newer information.
|
|
|
|
## REST API ##
|
|
|
|
The default deployment will build an environment where the Falcon API server listens on port 31901
|
|
|
|
The Airflow Web UI can be accessed on the localhost via port 32080
|
|
|
|
At the present moment, Shipyard expects authentication token to be passed, i.e. *10* for the shipyard
|
|
user and *admin* for the admin user. An authentication error like the one below will be thrown if the
|
|
token is not passed in or if an invalid token is used (note that keystone will be used to perform
|
|
authentication in the future).
|
|
|
|
```
|
|
$ http localhost:31901/api/experimental/dags/airflow_task_state/tasks/airflow_task_state
|
|
HTTP/1.1 401 Unauthorized
|
|
content-length: 97
|
|
content-type: application/json; charset=UTF-8
|
|
vary: Accept
|
|
x-shipyard-req: 1a35a1cf-29c4-4642-aba6-9c12e3fb7f04
|
|
|
|
{
|
|
"description": "This resource requires an authorized role.",
|
|
"title": "Authentication required"
|
|
}
|
|
|
|
```
|
|
|
|
Note: An optional X-Context-Marker can also be passed with the REST call
|
|
|
|
|
|
The same Airflow REST API [endpoints](https://airflow.incubator.apache.org/api.html) are used in
|
|
Shipyard to perform API calls to airflow
|
|
|
|
|
|
Note: The following examples make use of [HTTPie](https://httpie.org/) to perform RESTful API calls
|
|
|
|
### Example 1 ###
|
|
|
|
Retrieve information of a particular task
|
|
|
|
```
|
|
$ http localhost:31901/api/experimental/dags/airflow_task_state/tasks/airflow_task_state X-Auth-Token:admin
|
|
HTTP/1.1 200 OK
|
|
content-length: 1153
|
|
content-type: application/json; charset=UTF-8
|
|
x-shipyard-req: e3afc941-a31a-42ab-a1b5-8717777fe155
|
|
|
|
{
|
|
"adhoc": "False",
|
|
"airflow_command": "airflow task_state airflow_task_state airflow_task_state 2016-06-26T16:35:45.068576",
|
|
"airflow_dag_id": "airflow_task_state",
|
|
"airflow_execution_date": "2016-06-26T16:35:45.068576",
|
|
"airflow_task_id": "airflow_task_state",
|
|
"depends_on_past": "False",
|
|
"email": "['airflow@example.com']",
|
|
"email_on_failure": "False",
|
|
"email_on_retry": "False",
|
|
"end_date": "None",
|
|
"execution_timeout": "None",
|
|
"max_retry_delay": "None",
|
|
"on_failure_callback": "None",
|
|
"on_retry_callback": "None",
|
|
"on_success_callback": "None",
|
|
"owner": "airflow",
|
|
"params": "{}",
|
|
"pool": "None",
|
|
"priority_weight": "1",
|
|
"queue": "default",
|
|
"resources": "{'disk': {'_qty': 512, '_units_str': 'MB', '_name': 'Disk'}, 'gpus': {'_qty': 0, '_units_str': 'gpu(s)', '_name': 'GPU'}, 'ram': {'_qty': 512, '_units_str': 'MB', '_name': 'RAM'}, 'cpus': {'_qty': 1, '_units_str': 'core(s)', '_name': 'CPU'}}",
|
|
"retries": "1",
|
|
"retry_delay": "0:01:00",
|
|
"retry_exponential_backoff": "False",
|
|
"run_as_user": "None",
|
|
"sla": "None",
|
|
"start_date": "2017-06-25 00:00:00",
|
|
"task_id": "airflow_task_state",
|
|
"trigger_rule": "all_success",
|
|
"wait_for_downstream": "False"
|
|
}
|
|
|
|
```
|
|
|
|
|
|
Note that we will get an error response if we try and retrieve an invalid task:
|
|
|
|
```
|
|
$ http localhost:31901/api/experimental/dags/airflow_task_state/tasks/test123 X-Auth-Token:10
|
|
HTTP/1.1 400 Bad Request
|
|
content-length: 61
|
|
content-type: application/json; charset=UTF-8
|
|
x-shipyard-req: ec150a0c-30bf-40ba-a40e-ad4627d8770b
|
|
|
|
{
|
|
"error": "Task test123 not found in dag airflow_task_state"
|
|
}
|
|
|
|
```
|
|
|
|
|
|
Retrieval of invalid dag will end up with error as well:
|
|
|
|
```
|
|
$ http localhost:31901/api/experimental/dags/test123/tasks/airflow_task_state X-Auth-Token:10
|
|
HTTP/1.1 400 Bad Request
|
|
content-length: 34
|
|
content-type: application/json; charset=UTF-8
|
|
x-shipyard-req: d718820b-7f74-4893-affb-92e783ec4541
|
|
|
|
{
|
|
"error": "Dag test123 not found"
|
|
}
|
|
|
|
```
|
|
|
|
|
|
### Example 2 ###
|
|
|
|
Execute a Dag
|
|
|
|
```
|
|
$ http POST localhost:31901/api/experimental/dags/airflow_task_state/dag_runs X-Auth-Token:10 X-Context-Marker:airflow_task_state_testing
|
|
HTTP/1.1 200 OK
|
|
content-length: 0
|
|
content-type: application/json; charset=UTF-8
|
|
x-shipyard-req: a2f2a49c-82c8-4131-9764-fc15495b520c
|
|
|
|
```
|
|
|
|
In this example, the *airflow_task_state* dag will get executed and the information of the task instances
|
|
can be retrieved from the Airflow Web UI which is usually located at this [link](http://localhost:32080/admin/taskinstance/)
|
|
|
|
|
|
Note that we will get an error response if we try and execute an invalid dag:
|
|
|
|
```
|
|
$ http POST localhost:31901/api/experimental/dags/test123/dag_runs X-Auth-Token:10
|
|
HTTP/1.1 400 Bad Request
|
|
content-length: 67
|
|
content-type: application/json
|
|
x-shipyard-req: ad978d6c-eb5b-4392-a8cb-461dd42a9b00
|
|
|
|
{
|
|
"message": "Fail to Execute Dag",
|
|
"retry": false,
|
|
"type": "error"
|
|
}
|
|
|
|
```
|
|
|