Merge pull request #7 from eanylin/shipyard_doc

Shipyard doc
This commit is contained in:
Mark Burnett 2017-07-03 11:26:38 -05:00 committed by GitHub
commit eb548398f8
3 changed files with 360 additions and 0 deletions

5
docs/README.md Normal file
View File

@ -0,0 +1,5 @@
## Shipyard ##
Shipyard adopts the Falcon web framework and uses Apache Airflow as backend engine to programmatically
author, schedule and monitor workflows

148
docs/REST_API.md Normal file
View File

@ -0,0 +1,148 @@
## REST API ##
The default deployment will build an environment where the Falcon API server listens on port 31901
The Airflow Web UI can be accessed on the localhost via port 32080
At the present moment, Shipyard expects authentication token to be passed, i.e. *10* for the shipyard
user and *admin* for the admin user. An authentication error like the one below will be thrown if the
token is not passed in or if an invalid token is used (note that keystone will be used to perform
authentication in the future).
```
$ http localhost:31901/api/experimental/dags/airflow_task_state/tasks/airflow_task_state
HTTP/1.1 401 Unauthorized
content-length: 97
content-type: application/json; charset=UTF-8
vary: Accept
x-shipyard-req: 1a35a1cf-29c4-4642-aba6-9c12e3fb7f04
{
"description": "This resource requires an authorized role.",
"title": "Authentication required"
}
```
Note: An optional X-Context-Marker can also be passed with the REST call
The same Airflow REST API [endpoints](https://airflow.incubator.apache.org/api.html) are used in
Shipyard to perform API calls to airflow
Note: The following examples make use of [HTTPie](https://httpie.org/) to perform RESTful API calls
### Example 1 ###
Retrieve information of a particular task
```
$ http localhost:31901/api/experimental/dags/airflow_task_state/tasks/airflow_task_state X-Auth-Token:admin
HTTP/1.1 200 OK
content-length: 1153
content-type: application/json; charset=UTF-8
x-shipyard-req: e3afc941-a31a-42ab-a1b5-8717777fe155
{
"adhoc": "False",
"airflow_command": "airflow task_state airflow_task_state airflow_task_state 2016-06-26T16:35:45.068576",
"airflow_dag_id": "airflow_task_state",
"airflow_execution_date": "2016-06-26T16:35:45.068576",
"airflow_task_id": "airflow_task_state",
"depends_on_past": "False",
"email": "['airflow@example.com']",
"email_on_failure": "False",
"email_on_retry": "False",
"end_date": "None",
"execution_timeout": "None",
"max_retry_delay": "None",
"on_failure_callback": "None",
"on_retry_callback": "None",
"on_success_callback": "None",
"owner": "airflow",
"params": "{}",
"pool": "None",
"priority_weight": "1",
"queue": "default",
"resources": "{'disk': {'_qty': 512, '_units_str': 'MB', '_name': 'Disk'}, 'gpus': {'_qty': 0, '_units_str': 'gpu(s)', '_name': 'GPU'}, 'ram': {'_qty': 512, '_units_str': 'MB', '_name': 'RAM'}, 'cpus': {'_qty': 1, '_units_str': 'core(s)', '_name': 'CPU'}}",
"retries": "1",
"retry_delay": "0:01:00",
"retry_exponential_backoff": "False",
"run_as_user": "None",
"sla": "None",
"start_date": "2017-06-25 00:00:00",
"task_id": "airflow_task_state",
"trigger_rule": "all_success",
"wait_for_downstream": "False"
}
```
Note that we will get an error response if we try and retrieve an invalid task:
```
$ http localhost:31901/api/experimental/dags/airflow_task_state/tasks/test123 X-Auth-Token:10
HTTP/1.1 400 Bad Request
content-length: 61
content-type: application/json; charset=UTF-8
x-shipyard-req: ec150a0c-30bf-40ba-a40e-ad4627d8770b
{
"error": "Task test123 not found in dag airflow_task_state"
}
```
Retrieval of invalid dag will end up with error as well:
```
$ http localhost:31901/api/experimental/dags/test123/tasks/airflow_task_state X-Auth-Token:10
HTTP/1.1 400 Bad Request
content-length: 34
content-type: application/json; charset=UTF-8
x-shipyard-req: d718820b-7f74-4893-affb-92e783ec4541
{
"error": "Dag test123 not found"
}
```
### Example 2 ###
Execute a Dag
```
$ http POST localhost:31901/api/experimental/dags/airflow_task_state/dag_runs X-Auth-Token:10 X-Context-Marker:airflow_task_state_testing
HTTP/1.1 200 OK
content-length: 0
content-type: application/json; charset=UTF-8
x-shipyard-req: a2f2a49c-82c8-4131-9764-fc15495b520c
```
In this example, the *airflow_task_state* dag will get executed and the information of the task instances
can be retrieved from the Airflow Web UI which is usually located at this [link](http://localhost:32080/admin/taskinstance/)
Note that we will get an error response if we try and execute an invalid dag:
```
$ http POST localhost:31901/api/experimental/dags/test123/dag_runs X-Auth-Token:10
HTTP/1.1 400 Bad Request
content-length: 67
content-type: application/json
x-shipyard-req: ad978d6c-eb5b-4392-a8cb-461dd42a9b00
{
"message": "Fail to Execute Dag",
"retry": false,
"type": "error"
}
```

207
docs/deployment_guide.md Normal file
View File

@ -0,0 +1,207 @@
## Deployment Guide ##
*Note that Shipyard is still under active development and this guide will evolve along the way*
The current deployment makes use of OpenStack-Helm to set up the underlaying Kubernetes
infrastructure and helm charts to deploy the containerized applications.
1) Follow the steps in the OpenStack-Helm All-In-One [guide](http://openstack-helm.readthedocs.io/en/latest/install/all-in-one.html)
to set up the environment on an Ubuntu 16.04 Virtual Machine. Follow the steps from the start of the
wiki till the 'Deploy' section to get the base system up.
*Note that we do not need to install the OpenStack Helm Charts for Shipyard to function*
The environment should resemble the following after the executing the required steps in the guide:
```
# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default nfs-provisioner-3807301966-b7sh4 1/1 Running 0 22h
kube-system calico-etcd-p4fct 1/1 Running 0 22h
kube-system calico-node-35pbq 2/2 Running 0 22h
kube-system calico-policy-controller-1777954159-wvp9h 1/1 Running 0 22h
kube-system etcd-labinstance 1/1 Running 0 22h
kube-system kube-apiserver-labinstance 1/1 Running 0 22h
kube-system kube-controller-manager-labinstance 1/1 Running 0 22h
kube-system kube-dns-692378583-h9cq2 3/3 Running 0 22h
kube-system kube-proxy-259th 1/1 Running 0 22h
kube-system kube-scheduler-labinstance 1/1 Running 0 22h
kube-system tiller-deploy-1332173772-fplsk 1/1 Running 0 22h
```
2) Deploy Airflow Helm Chart
Note: The airflow chart requires a postgresql instance and rabbitmq to be running
Create airflow namespace:
```
kubectl create namespace airflow
```
Postgresql Helm Chart Installation:
Clone the [OpenStack-Helm-Addons](https://github.com/att-comdev/openstack-helm-addons.git) repository to
get the postgresql helm chart. Bring up the postgresql container using the postgresql helm chart:
```
helm install --name=airflow-postgresql postgresql/ --namespace=airflow
```
Note: Postgresql may take a short time to reach the 'Running' state. Verify that postgresql is running:
```
# kubectl get pods -n airflow
NAME READY STATUS RESTARTS AGE
postgresql-0 1/1 Running 0 1m
```
Rabbitmq Helm Chart Installation:
Go to the openstack-helm directory that was created in Step 1
Update the values.yaml of the rabbitmq charts to reflect the appropriate username and password for the
environment, e.g. *airflow / airflow*
Execute the following commands:
```
helm install --name=airflow-etcd-rabbitmq etcd/ --namespace=airflow
helm install --name=airflow-rabbitmq rabbitmq/ --namespace=airflow
```
Note: We need to make sure that the etcd chart is executed before the rabbitmq chart due to dependencies
```
# kubectl get pods -n airflow
NAME READY STATUS RESTARTS AGE
etcd-2810752095-054xb 1/1 Running 0 2m
postgresql-0 1/1 Running 0 3m
rabbitmq-646028817-0bwgp 1/1 Running 0 1m
rabbitmq-646028817-3hb1z 1/1 Running 0 1m
rabbitmq-646028817-sq6cw 1/1 Running 0 1m
```
Airflow Helm Chart Installation:
Clone the [AIC-Helm](https://github.com/att-comdev/aic-helm.git) repository to get the Airflow Helm Charts.
The charts can be found under the airflow directory.
```
helm install --name=airflow airflow/ --namespace=airflow
```
Verify that the airflow helm charts were successful deployed:
```
# kubectl get pods -n airflow
NAME READY STATUS RESTARTS AGE
etcd-2810752095-054xb 1/1 Running 0 1h
flower-57424757-xqzls 1/1 Running 0 1m
postgresql-0 1/1 Running 0 1h
rabbitmq-646028817-0bwgp 1/1 Running 0 1h
rabbitmq-646028817-3hb1z 1/1 Running 0 1h
rabbitmq-646028817-sq6cw 1/1 Running 0 1h
scheduler-1793121224-z57t9 1/1 Running 0 1m
web-1556478053-t06t9 1/1 Running 0 1m
worker-3775326852-0747g 1/1 Running 0 1m
```
To check that all resources are working as intended:
```
$ kubectl get all --namespace=airflow
NAME READY STATUS RESTARTS AGE
po/etcd-2810752095-054xb 1/1 Running 0 1h
po/flower-57424757-xqzls 1/1 Running 0 1m
po/postgresql-0 1/1 Running 0 1h
po/rabbitmq-646028817-0bwgp 1/1 Running 0 1h
po/rabbitmq-646028817-3hb1z 1/1 Running 0 1h
po/rabbitmq-646028817-sq6cw 1/1 Running 0 1h
po/scheduler-1793121224-z57t9 1/1 Running 0 1m
po/web-1556478053-t06t9 1/1 Running 0 1m
po/worker-3775326852-0747g 1/1 Running 0 1m
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
svc/etcd 10.102.166.90 <none> 2379/TCP 1h
svc/flower 10.105.87.167 <nodes> 5555:32081/TCP 1m
svc/postgresql 10.96.110.4 <none> 5432/TCP 1h
svc/rabbitmq 10.100.94.226 <none> 5672/TCP 1h
svc/web 10.103.245.128 <nodes> 8080:32080/TCP 1m
NAME DESIRED CURRENT AGE
statefulsets/postgresql 1 1 1h
NAME DESIRED SUCCESSFUL AGE
jobs/airflow-db-init-postgresql 1 1 1m
jobs/airflow-db-sync 1 1 1m
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
deploy/etcd 1 1 1 1 1h
deploy/flower 1 1 1 1 1m
deploy/rabbitmq 3 3 3 3 1h
deploy/scheduler 1 1 1 1 1m
deploy/web 1 1 1 1 1m
deploy/worker 1 1 1 1 1m
NAME DESIRED CURRENT READY AGE
rs/etcd-2810752095 1 1 1 1h
rs/flower-57424757 1 1 1 1m
rs/rabbitmq-646028817 3 3 3 1h
rs/scheduler-1793121224 1 1 1 1m
rs/web-1556478053 1 1 1 1m
rs/worker-3775326852 1 1 1 1m
```
3) Deploy Shipyard Helm Chart
The Shipyard helm chart is still under review at the moment and can be retrieved by cloning the *shipyard_chart*
branch with the following command:
```
$ git clone https://github.com/eanylin/aic-helm.git --branch shipyard_chart
```
Shipyard Helm Chart Installation:
```
$ helm install --name=airflow-shipyard shipyard/ --namespace=airflow
```
Check that all the helm charts have been properly deployed and that all services are up and running:
```
# kubectl get pods -n airflow
NAME READY STATUS RESTARTS AGE
etcd-2810752095-7xbj2 1/1 Running 0 14m
flower-3408049844-ntpt3 1/1 Running 0 6m
postgresql-0 1/1 Running 0 17m
rabbitmq-4130740871-716vm 1/1 Running 0 13m
rabbitmq-4130740871-dlxjk 1/1 Running 0 13m
rabbitmq-4130740871-rj0pd 1/1 Running 0 13m
scheduler-2853377807-jghld 1/1 Running 0 6m
shipyard-1823745968-t794w 1/1 Running 0 2m
web-3069981571-9tjt9 1/1 Running 0 6m
worker-2534280303-fcb49 1/1 Running 0 6m
# kubectl get service -n airflow
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
etcd 10.97.149.59 <none> 2379/TCP 13m
flower 10.108.197.84 <nodes> 5555:32081/TCP 5m
postgresql 10.108.17.95 <none> 5432/TCP 17m
rabbitmq 10.98.1.115 <none> 5672/TCP 12m
shipyard-api 10.108.189.157 <nodes> 9000:31901/TCP 1m
web 10.97.88.69 <nodes> 8080:32080/TCP 5m
```