commit
eb548398f8
|
@ -0,0 +1,5 @@
|
|||
## Shipyard ##
|
||||
|
||||
Shipyard adopts the Falcon web framework and uses Apache Airflow as backend engine to programmatically
|
||||
author, schedule and monitor workflows
|
||||
|
|
@ -0,0 +1,148 @@
|
|||
## REST API ##
|
||||
|
||||
The default deployment will build an environment where the Falcon API server listens on port 31901
|
||||
|
||||
The Airflow Web UI can be accessed on the localhost via port 32080
|
||||
|
||||
At the present moment, Shipyard expects authentication token to be passed, i.e. *10* for the shipyard
|
||||
user and *admin* for the admin user. An authentication error like the one below will be thrown if the
|
||||
token is not passed in or if an invalid token is used (note that keystone will be used to perform
|
||||
authentication in the future).
|
||||
|
||||
```
|
||||
$ http localhost:31901/api/experimental/dags/airflow_task_state/tasks/airflow_task_state
|
||||
HTTP/1.1 401 Unauthorized
|
||||
content-length: 97
|
||||
content-type: application/json; charset=UTF-8
|
||||
vary: Accept
|
||||
x-shipyard-req: 1a35a1cf-29c4-4642-aba6-9c12e3fb7f04
|
||||
|
||||
{
|
||||
"description": "This resource requires an authorized role.",
|
||||
"title": "Authentication required"
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
Note: An optional X-Context-Marker can also be passed with the REST call
|
||||
|
||||
|
||||
The same Airflow REST API [endpoints](https://airflow.incubator.apache.org/api.html) are used in
|
||||
Shipyard to perform API calls to airflow
|
||||
|
||||
|
||||
Note: The following examples make use of [HTTPie](https://httpie.org/) to perform RESTful API calls
|
||||
|
||||
### Example 1 ###
|
||||
|
||||
Retrieve information of a particular task
|
||||
|
||||
```
|
||||
$ http localhost:31901/api/experimental/dags/airflow_task_state/tasks/airflow_task_state X-Auth-Token:admin
|
||||
HTTP/1.1 200 OK
|
||||
content-length: 1153
|
||||
content-type: application/json; charset=UTF-8
|
||||
x-shipyard-req: e3afc941-a31a-42ab-a1b5-8717777fe155
|
||||
|
||||
{
|
||||
"adhoc": "False",
|
||||
"airflow_command": "airflow task_state airflow_task_state airflow_task_state 2016-06-26T16:35:45.068576",
|
||||
"airflow_dag_id": "airflow_task_state",
|
||||
"airflow_execution_date": "2016-06-26T16:35:45.068576",
|
||||
"airflow_task_id": "airflow_task_state",
|
||||
"depends_on_past": "False",
|
||||
"email": "['airflow@example.com']",
|
||||
"email_on_failure": "False",
|
||||
"email_on_retry": "False",
|
||||
"end_date": "None",
|
||||
"execution_timeout": "None",
|
||||
"max_retry_delay": "None",
|
||||
"on_failure_callback": "None",
|
||||
"on_retry_callback": "None",
|
||||
"on_success_callback": "None",
|
||||
"owner": "airflow",
|
||||
"params": "{}",
|
||||
"pool": "None",
|
||||
"priority_weight": "1",
|
||||
"queue": "default",
|
||||
"resources": "{'disk': {'_qty': 512, '_units_str': 'MB', '_name': 'Disk'}, 'gpus': {'_qty': 0, '_units_str': 'gpu(s)', '_name': 'GPU'}, 'ram': {'_qty': 512, '_units_str': 'MB', '_name': 'RAM'}, 'cpus': {'_qty': 1, '_units_str': 'core(s)', '_name': 'CPU'}}",
|
||||
"retries": "1",
|
||||
"retry_delay": "0:01:00",
|
||||
"retry_exponential_backoff": "False",
|
||||
"run_as_user": "None",
|
||||
"sla": "None",
|
||||
"start_date": "2017-06-25 00:00:00",
|
||||
"task_id": "airflow_task_state",
|
||||
"trigger_rule": "all_success",
|
||||
"wait_for_downstream": "False"
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
|
||||
Note that we will get an error response if we try and retrieve an invalid task:
|
||||
|
||||
```
|
||||
$ http localhost:31901/api/experimental/dags/airflow_task_state/tasks/test123 X-Auth-Token:10
|
||||
HTTP/1.1 400 Bad Request
|
||||
content-length: 61
|
||||
content-type: application/json; charset=UTF-8
|
||||
x-shipyard-req: ec150a0c-30bf-40ba-a40e-ad4627d8770b
|
||||
|
||||
{
|
||||
"error": "Task test123 not found in dag airflow_task_state"
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
|
||||
Retrieval of invalid dag will end up with error as well:
|
||||
|
||||
```
|
||||
$ http localhost:31901/api/experimental/dags/test123/tasks/airflow_task_state X-Auth-Token:10
|
||||
HTTP/1.1 400 Bad Request
|
||||
content-length: 34
|
||||
content-type: application/json; charset=UTF-8
|
||||
x-shipyard-req: d718820b-7f74-4893-affb-92e783ec4541
|
||||
|
||||
{
|
||||
"error": "Dag test123 not found"
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
|
||||
### Example 2 ###
|
||||
|
||||
Execute a Dag
|
||||
|
||||
```
|
||||
$ http POST localhost:31901/api/experimental/dags/airflow_task_state/dag_runs X-Auth-Token:10 X-Context-Marker:airflow_task_state_testing
|
||||
HTTP/1.1 200 OK
|
||||
content-length: 0
|
||||
content-type: application/json; charset=UTF-8
|
||||
x-shipyard-req: a2f2a49c-82c8-4131-9764-fc15495b520c
|
||||
|
||||
```
|
||||
|
||||
In this example, the *airflow_task_state* dag will get executed and the information of the task instances
|
||||
can be retrieved from the Airflow Web UI which is usually located at this [link](http://localhost:32080/admin/taskinstance/)
|
||||
|
||||
|
||||
Note that we will get an error response if we try and execute an invalid dag:
|
||||
|
||||
```
|
||||
$ http POST localhost:31901/api/experimental/dags/test123/dag_runs X-Auth-Token:10
|
||||
HTTP/1.1 400 Bad Request
|
||||
content-length: 67
|
||||
content-type: application/json
|
||||
x-shipyard-req: ad978d6c-eb5b-4392-a8cb-461dd42a9b00
|
||||
|
||||
{
|
||||
"message": "Fail to Execute Dag",
|
||||
"retry": false,
|
||||
"type": "error"
|
||||
}
|
||||
|
||||
```
|
||||
|
|
@ -0,0 +1,207 @@
|
|||
## Deployment Guide ##
|
||||
|
||||
*Note that Shipyard is still under active development and this guide will evolve along the way*
|
||||
|
||||
The current deployment makes use of OpenStack-Helm to set up the underlaying Kubernetes
|
||||
infrastructure and helm charts to deploy the containerized applications.
|
||||
|
||||
|
||||
1) Follow the steps in the OpenStack-Helm All-In-One [guide](http://openstack-helm.readthedocs.io/en/latest/install/all-in-one.html)
|
||||
to set up the environment on an Ubuntu 16.04 Virtual Machine. Follow the steps from the start of the
|
||||
wiki till the 'Deploy' section to get the base system up.
|
||||
|
||||
*Note that we do not need to install the OpenStack Helm Charts for Shipyard to function*
|
||||
|
||||
The environment should resemble the following after the executing the required steps in the guide:
|
||||
|
||||
```
|
||||
# kubectl get pods --all-namespaces
|
||||
NAMESPACE NAME READY STATUS RESTARTS AGE
|
||||
default nfs-provisioner-3807301966-b7sh4 1/1 Running 0 22h
|
||||
kube-system calico-etcd-p4fct 1/1 Running 0 22h
|
||||
kube-system calico-node-35pbq 2/2 Running 0 22h
|
||||
kube-system calico-policy-controller-1777954159-wvp9h 1/1 Running 0 22h
|
||||
kube-system etcd-labinstance 1/1 Running 0 22h
|
||||
kube-system kube-apiserver-labinstance 1/1 Running 0 22h
|
||||
kube-system kube-controller-manager-labinstance 1/1 Running 0 22h
|
||||
kube-system kube-dns-692378583-h9cq2 3/3 Running 0 22h
|
||||
kube-system kube-proxy-259th 1/1 Running 0 22h
|
||||
kube-system kube-scheduler-labinstance 1/1 Running 0 22h
|
||||
kube-system tiller-deploy-1332173772-fplsk 1/1 Running 0 22h
|
||||
```
|
||||
|
||||
2) Deploy Airflow Helm Chart
|
||||
|
||||
Note: The airflow chart requires a postgresql instance and rabbitmq to be running
|
||||
|
||||
|
||||
Create airflow namespace:
|
||||
|
||||
```
|
||||
kubectl create namespace airflow
|
||||
```
|
||||
|
||||
|
||||
Postgresql Helm Chart Installation:
|
||||
|
||||
Clone the [OpenStack-Helm-Addons](https://github.com/att-comdev/openstack-helm-addons.git) repository to
|
||||
get the postgresql helm chart. Bring up the postgresql container using the postgresql helm chart:
|
||||
|
||||
```
|
||||
helm install --name=airflow-postgresql postgresql/ --namespace=airflow
|
||||
```
|
||||
|
||||
Note: Postgresql may take a short time to reach the 'Running' state. Verify that postgresql is running:
|
||||
|
||||
```
|
||||
# kubectl get pods -n airflow
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
postgresql-0 1/1 Running 0 1m
|
||||
```
|
||||
|
||||
|
||||
Rabbitmq Helm Chart Installation:
|
||||
|
||||
Go to the openstack-helm directory that was created in Step 1
|
||||
|
||||
Update the values.yaml of the rabbitmq charts to reflect the appropriate username and password for the
|
||||
environment, e.g. *airflow / airflow*
|
||||
|
||||
Execute the following commands:
|
||||
|
||||
```
|
||||
helm install --name=airflow-etcd-rabbitmq etcd/ --namespace=airflow
|
||||
helm install --name=airflow-rabbitmq rabbitmq/ --namespace=airflow
|
||||
```
|
||||
|
||||
Note: We need to make sure that the etcd chart is executed before the rabbitmq chart due to dependencies
|
||||
|
||||
```
|
||||
# kubectl get pods -n airflow
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
etcd-2810752095-054xb 1/1 Running 0 2m
|
||||
postgresql-0 1/1 Running 0 3m
|
||||
rabbitmq-646028817-0bwgp 1/1 Running 0 1m
|
||||
rabbitmq-646028817-3hb1z 1/1 Running 0 1m
|
||||
rabbitmq-646028817-sq6cw 1/1 Running 0 1m
|
||||
```
|
||||
|
||||
|
||||
Airflow Helm Chart Installation:
|
||||
|
||||
Clone the [AIC-Helm](https://github.com/att-comdev/aic-helm.git) repository to get the Airflow Helm Charts.
|
||||
The charts can be found under the airflow directory.
|
||||
|
||||
```
|
||||
helm install --name=airflow airflow/ --namespace=airflow
|
||||
```
|
||||
|
||||
Verify that the airflow helm charts were successful deployed:
|
||||
|
||||
```
|
||||
# kubectl get pods -n airflow
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
etcd-2810752095-054xb 1/1 Running 0 1h
|
||||
flower-57424757-xqzls 1/1 Running 0 1m
|
||||
postgresql-0 1/1 Running 0 1h
|
||||
rabbitmq-646028817-0bwgp 1/1 Running 0 1h
|
||||
rabbitmq-646028817-3hb1z 1/1 Running 0 1h
|
||||
rabbitmq-646028817-sq6cw 1/1 Running 0 1h
|
||||
scheduler-1793121224-z57t9 1/1 Running 0 1m
|
||||
web-1556478053-t06t9 1/1 Running 0 1m
|
||||
worker-3775326852-0747g 1/1 Running 0 1m
|
||||
|
||||
```
|
||||
|
||||
|
||||
To check that all resources are working as intended:
|
||||
|
||||
```
|
||||
$ kubectl get all --namespace=airflow
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
po/etcd-2810752095-054xb 1/1 Running 0 1h
|
||||
po/flower-57424757-xqzls 1/1 Running 0 1m
|
||||
po/postgresql-0 1/1 Running 0 1h
|
||||
po/rabbitmq-646028817-0bwgp 1/1 Running 0 1h
|
||||
po/rabbitmq-646028817-3hb1z 1/1 Running 0 1h
|
||||
po/rabbitmq-646028817-sq6cw 1/1 Running 0 1h
|
||||
po/scheduler-1793121224-z57t9 1/1 Running 0 1m
|
||||
po/web-1556478053-t06t9 1/1 Running 0 1m
|
||||
po/worker-3775326852-0747g 1/1 Running 0 1m
|
||||
|
||||
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
|
||||
svc/etcd 10.102.166.90 <none> 2379/TCP 1h
|
||||
svc/flower 10.105.87.167 <nodes> 5555:32081/TCP 1m
|
||||
svc/postgresql 10.96.110.4 <none> 5432/TCP 1h
|
||||
svc/rabbitmq 10.100.94.226 <none> 5672/TCP 1h
|
||||
svc/web 10.103.245.128 <nodes> 8080:32080/TCP 1m
|
||||
|
||||
NAME DESIRED CURRENT AGE
|
||||
statefulsets/postgresql 1 1 1h
|
||||
|
||||
NAME DESIRED SUCCESSFUL AGE
|
||||
jobs/airflow-db-init-postgresql 1 1 1m
|
||||
jobs/airflow-db-sync 1 1 1m
|
||||
|
||||
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
|
||||
deploy/etcd 1 1 1 1 1h
|
||||
deploy/flower 1 1 1 1 1m
|
||||
deploy/rabbitmq 3 3 3 3 1h
|
||||
deploy/scheduler 1 1 1 1 1m
|
||||
deploy/web 1 1 1 1 1m
|
||||
deploy/worker 1 1 1 1 1m
|
||||
|
||||
NAME DESIRED CURRENT READY AGE
|
||||
rs/etcd-2810752095 1 1 1 1h
|
||||
rs/flower-57424757 1 1 1 1m
|
||||
rs/rabbitmq-646028817 3 3 3 1h
|
||||
rs/scheduler-1793121224 1 1 1 1m
|
||||
rs/web-1556478053 1 1 1 1m
|
||||
rs/worker-3775326852 1 1 1 1m
|
||||
|
||||
```
|
||||
|
||||
|
||||
3) Deploy Shipyard Helm Chart
|
||||
|
||||
The Shipyard helm chart is still under review at the moment and can be retrieved by cloning the *shipyard_chart*
|
||||
branch with the following command:
|
||||
|
||||
```
|
||||
$ git clone https://github.com/eanylin/aic-helm.git --branch shipyard_chart
|
||||
```
|
||||
|
||||
Shipyard Helm Chart Installation:
|
||||
|
||||
```
|
||||
$ helm install --name=airflow-shipyard shipyard/ --namespace=airflow
|
||||
```
|
||||
|
||||
Check that all the helm charts have been properly deployed and that all services are up and running:
|
||||
|
||||
```
|
||||
# kubectl get pods -n airflow
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
etcd-2810752095-7xbj2 1/1 Running 0 14m
|
||||
flower-3408049844-ntpt3 1/1 Running 0 6m
|
||||
postgresql-0 1/1 Running 0 17m
|
||||
rabbitmq-4130740871-716vm 1/1 Running 0 13m
|
||||
rabbitmq-4130740871-dlxjk 1/1 Running 0 13m
|
||||
rabbitmq-4130740871-rj0pd 1/1 Running 0 13m
|
||||
scheduler-2853377807-jghld 1/1 Running 0 6m
|
||||
shipyard-1823745968-t794w 1/1 Running 0 2m
|
||||
web-3069981571-9tjt9 1/1 Running 0 6m
|
||||
worker-2534280303-fcb49 1/1 Running 0 6m
|
||||
|
||||
|
||||
# kubectl get service -n airflow
|
||||
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
|
||||
etcd 10.97.149.59 <none> 2379/TCP 13m
|
||||
flower 10.108.197.84 <nodes> 5555:32081/TCP 5m
|
||||
postgresql 10.108.17.95 <none> 5432/TCP 17m
|
||||
rabbitmq 10.98.1.115 <none> 5672/TCP 12m
|
||||
shipyard-api 10.108.189.157 <nodes> 9000:31901/TCP 1m
|
||||
web 10.97.88.69 <nodes> 8080:32080/TCP 5m
|
||||
|
||||
```
|
||||
|
Loading…
Reference in New Issue