airship
/
shipyard
Code Issues Proposed changes

Update Deployment Guide 

- Change Namespace to shipyard instead of airflow
- Add steps to add custom dags and operators to mount point on host machine
- Update steps to retrieve shipyard chart

Change-Id: Ia54201a7d1241b0aa6da092bb60705a15b5d9c71
This commit is contained in:
Anthony Lin 2017-08-10 01:05:53 +00:00
parent 2d7f7dc57c
commit 065bf9259a
1 changed files with 62 additions and 48 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

110
docs/deployment_guide.md
View File

@ -2,11 +2,11 @@
*Note that Shipyard is still under active development and this guide will evolve along the way*
The current deployment makes use of OpenStack-Helm to set up the underlaying Kubernetes
The current deployment makes use of OpenStack-Helm to set up the underlaying Kubernetes
infrastructure and helm charts to deploy the containerized applications.
1) Follow the steps in the OpenStack-Helm All-In-One [guide](http://openstack-helm.readthedocs.io/en/latest/install/all-in-one.html)
1) Follow the steps in the OpenStack-Helm All-In-One [guide](http://openstack-helm.readthedocs.io/en/latest/install/all-in-one.html)
to set up the environment on an Ubuntu 16.04 Virtual Machine. Follow the steps from the start of the
wiki till the 'Deploy' section to get the base system up.
@ -35,49 +35,49 @@ infrastructure and helm charts to deploy the containerized applications.
Note: The airflow chart requires a postgresql instance and rabbitmq to be running
Create airflow namespace:
Create shipyard namespace:
```
kubectl create namespace airflow
```
kubectl create namespace shipyard
```
Postgresql Helm Chart Installation:
Clone the [OpenStack-Helm-Addons](https://github.com/att-comdev/openstack-helm-addons.git) repository to
get the postgresql helm chart. Bring up the postgresql container using the postgresql helm chart:
```
helm install --name=airflow-postgresql postgresql/ --namespace=airflow
helm install --name=shipyard-postgresql postgresql/ --namespace=shipyard
```
Note: Postgresql may take a short time to reach the 'Running' state. Verify that postgresql is running:
```
# kubectl get pods -n airflow
# kubectl get pods -n shipyard
NAME READY STATUS RESTARTS AGE
postgresql-0 1/1 Running 0 1m
```
Rabbitmq Helm Chart Installation:
Go to the openstack-helm directory that was created in Step 1
Update the values.yaml of the rabbitmq charts to reflect the appropriate username and password for the
environment, e.g. *airflow / airflow*
Execute the following commands:
```
helm install --name=airflow-etcd-rabbitmq etcd/ --namespace=airflow
helm install --name=airflow-rabbitmq rabbitmq/ --namespace=airflow
helm install --name=shipyard-etcd-rabbitmq etcd/ --namespace=shipyard
helm install --name=shipyard-rabbitmq rabbitmq/ --namespace=shipyard
```
Note: We need to make sure that the etcd chart is executed before the rabbitmq chart due to dependencies
```
# kubectl get pods -n airflow
# kubectl get pods -n shipyard
NAME READY STATUS RESTARTS AGE
etcd-2810752095-054xb 1/1 Running 0 2m
postgresql-0 1/1 Running 0 3m
@ -85,21 +85,34 @@ infrastructure and helm charts to deploy the containerized applications.
rabbitmq-646028817-3hb1z 1/1 Running 0 1m
rabbitmq-646028817-sq6cw 1/1 Running 0 1m
```
Airflow Helm Chart Installation:
Clone the [AIC-Helm](https://github.com/att-comdev/aic-helm.git) repository to get the Airflow Helm Charts.
Create the following directories on the target host machine:
```
$ mkdir -p /home/ubuntu/workbench/dags
$ mkdir -p /home/ubuntu/workbench/plugins
$ mkdir -p /home/ubuntu/workbench/logs
```
Copy the [rest_api_plugin](https://github.com/att-comdev/shipyard/blob/master/shipyard_airflow/plugins/rest_api_plugin.py)
into the newly created plugins directory, i.e. `/home/ubuntu/workbench/plugins` so that it can be loaded by Airflow during
startup. Note that other custom operators can also be added to the directory as required.
Note: Custom dags should be added into the newly created dags directory, i.e. `/home/ubuntu/workbench/dags`
Next, proceed to clone the [AIC-Helm](https://github.com/att-comdev/aic-helm.git) repository to get the Airflow Helm Charts.
The charts can be found under the airflow directory.
```
helm install --name=airflow airflow/ --namespace=airflow
helm install --name=airflow airflow/ --namespace=shipyard
```
Verify that the airflow helm charts were successful deployed:
```
# kubectl get pods -n airflow
# kubectl get pods -n shipyard
NAME READY STATUS RESTARTS AGE
etcd-2810752095-054xb 1/1 Running 0 1h
flower-57424757-xqzls 1/1 Running 0 1m
@ -110,14 +123,14 @@ infrastructure and helm charts to deploy the containerized applications.
scheduler-1793121224-z57t9 1/1 Running 0 1m
web-1556478053-t06t9 1/1 Running 0 1m
worker-3775326852-0747g 1/1 Running 0 1m
```
To check that all resources are working as intended:
```
$ kubectl get all --namespace=airflow
$ kubectl get all --namespace=shipyard
NAME READY STATUS RESTARTS AGE
po/etcd-2810752095-054xb 1/1 Running 0 1h
po/flower-57424757-xqzls 1/1 Running 0 1m
@ -128,21 +141,21 @@ infrastructure and helm charts to deploy the containerized applications.
po/scheduler-1793121224-z57t9 1/1 Running 0 1m
po/web-1556478053-t06t9 1/1 Running 0 1m
po/worker-3775326852-0747g 1/1 Running 0 1m
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
svc/etcd 10.102.166.90 <none> 2379/TCP 1h
svc/flower 10.105.87.167 <nodes> 5555:32081/TCP 1m
svc/postgresql 10.96.110.4 <none> 5432/TCP 1h
svc/rabbitmq 10.100.94.226 <none> 5672/TCP 1h
svc/web 10.103.245.128 <nodes> 8080:32080/TCP 1m
NAME DESIRED CURRENT AGE
statefulsets/postgresql 1 1 1h
NAME DESIRED SUCCESSFUL AGE
jobs/airflow-db-init-postgresql 1 1 1m
jobs/airflow-db-sync 1 1 1m
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
deploy/etcd 1 1 1 1 1h
deploy/flower 1 1 1 1 1m
@ -150,7 +163,7 @@ infrastructure and helm charts to deploy the containerized applications.
deploy/scheduler 1 1 1 1 1m
deploy/web 1 1 1 1 1m
deploy/worker 1 1 1 1 1m
NAME DESIRED CURRENT READY AGE
rs/etcd-2810752095 1 1 1 1h
rs/flower-57424757 1 1 1 1m
@ -158,29 +171,30 @@ infrastructure and helm charts to deploy the containerized applications.
rs/scheduler-1793121224 1 1 1 1m
rs/web-1556478053 1 1 1 1m
rs/worker-3775326852 1 1 1 1m
```
3) Deploy Shipyard Helm Chart
The Shipyard helm chart is still under review at the moment and can be retrieved by cloning the *shipyard_chart*
branch with the following command:
The Shipyard helm chart is still under review at the moment and can be retrieved by performing the
the following steps:
```
$ git clone https://github.com/eanylin/aic-helm.git --branch shipyard_chart
$ git clone http://review.gerrithub.io/att-comdev/aic-helm
$ git fetch https://review.gerrithub.io/att-comdev/aic-helm refs/changes/91/373591/1 && git checkout FETCH_HEAD
```
Shipyard Helm Chart Installation:
```
$ helm install --name=airflow-shipyard shipyard/ --namespace=airflow
$ helm install --name=shipyard shipyard/ --namespace=shipyard
```
Check that all the helm charts have been properly deployed and that all services are up and running:
```
# kubectl get pods -n airflow
# kubectl get pods -n shipyard
NAME READY STATUS RESTARTS AGE
etcd-2810752095-7xbj2 1/1 Running 0 14m
flower-3408049844-ntpt3 1/1 Running 0 6m
@ -194,7 +208,7 @@ infrastructure and helm charts to deploy the containerized applications.
worker-2534280303-fcb49 1/1 Running 0 6m
# kubectl get service -n airflow
# kubectl get service -n shipyard
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
etcd 10.97.149.59 <none> 2379/TCP 13m
flower 10.108.197.84 <nodes> 5555:32081/TCP 5m