openstack/qinling
Code Issues Proposed changes

Add some documentation for Qinling

Browse Source 
- Add README.rst
- Add Vagrantfile for Qinling trial.
- Rename python2.7 dir to python2
This commit is contained in:
Lingxian Kong 2017-05-12 17:00:44 +12:00
parent df32a9f412
commit 6c71504654
8 changed files with 341 additions and 39 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

233
README.rst
View File

@ -1,19 +1,236 @@
=======
qinling
Qinling
=======
Function as a Service (Documentation needs to be added, please stay tuned!)
.. note::
Please fill here a long description which must be at least 3 lines wrapped on
80 cols, so that distribution package maintainers can use it in their packages.
Note that this is a hard requirement.
Qinling refers to Qinling Mountains in southern Shaanxi Province in China.
The mountains provide a natural boundary between North and South China and
support a huge variety of plant and wildlife, some of which is found
nowhere else on Earth.
Qinling is Function as a Service for OpenStack. This project aims to provide a
platform to support serverless functions (like AWS Lambda). Qinling supports
different container orchestration platforms (Kubernetes/Swarm, etc.) and
different function package storage backends (local/Swift/S3) by nature using
plugin mechanism.
* Free software: Apache license
* Documentation: http://docs.openstack.org/developer/qinling
* Source: http://git.openstack.org/cgit/openstack/qinling
* Features: https://blueprints.launchpad.net/qinling
* Bugs: http://bugs.launchpad.net/qinling
Features
--------
Quick Start
~~~~~~~~~~~
* TODO
Installation
------------
A fast and simple way to try Qinling is to create a Vagrant VM including all
related components and dependencies of Qinling service. For your convenience,
Qinling team already provide a Vagrantfile in ``tools/vagrant`` folder.
Qinling is a FaaS implemented on top of container orchestration system such as
Kubernetes, Swarm, etc. Particularly, Kubernetes is a reference backend
considering its popularity. So, you need to setup Kubernetes first before
installing Qinling. The easiest way to setup Kubernetes is to use `Minikube
<https://kubernetes.io/docs/getting-started-guides/minikube/>`_, it runs a
single-node Kubernetes cluster inside a VM alongside Qinling vagrant VM, so
they can communicate with each other without any network configuration.
.. note::
In order to manage resources on Kubernetes, it is recommended to install
`kubectl <https://kubernetes.io/docs/tasks/tools/install-kubectl/>`_
command line tool.
After Kubernetes installation, perform the following commands on your local
host.
#. Setup HTTP proxy to access the Kubernetes API:
.. code-block:: console
$ kubectl proxy --accept-hosts='.*' --address='0.0.0.0'
Starting to serve on [::]:8001
.. end
#. Clone Qinling repo and go to ``vagrant`` directory:
.. code-block:: console
$ git clone https://github.com/LingxianKong/qinling.git
$ cd qinling/tools/vagrant
.. end
#. Modify Qinling sample config file according to your own environment. Suppose
your IP address of your local host is ``192.168.200.50``, default Kubernetes
API HTTP proxy port is ``8001``, and Qinling vagrant VM IP address is
``192.168.33.18`` (the default value in ``Vagrantfile``):
.. code-block:: console
$ sed -i 's/KUBERNETES_API_HOST/192.168.200.50/' qinling.conf.sample
$ sed -i 's/KUBERNETES_API_PORT/8001/' qinling.conf.sample
$ sed -i 's/QINLING_API_ADDRESS/192.168.33.18/' qinling.conf.sample
.. end
#. Now, start Qinling vagrant VM:
.. code-block:: console
$ vagrant up
.. end
Getting started with Qinling
----------------------------
**Currently, RESTful API is the only way to interact with Qinling,
python-qinlingclient is still under development.**
``httpie`` is a convenient tool to send HTTP request, make sure you installed
``httpie`` via ``pip install httpie`` before playing with Qinling.
Perform following commands on your local host, the process will create
runtime/function/execution in Qinling.
#. First, build a docker image that is used to create runtime in Qinling and
upload to docker hub. Only ``Python 2`` runtime is supported for now, but it
is very easy to add another program language support. Run the commands in
``qinling`` repo directory, replace ``DOCKER_USER`` with your docker hub
username:
.. code-block:: console
$ cd runtimes/python2
$ docker build -t DOCKER_USER/python-runtime .
$ docker push DOCKER_USER/python-runtime
.. end
#. Create runtime. A runtime in Qinling is running environment for a specific
language, this resource is supposed to be created/deleted/updated by cloud
operator. After creation, check the runtime status is ``available``:
.. code-block:: console
$ http POST http://192.168.33.18:7070/v1/runtimes name=python2.7 \
image=DOCKER_USER/python-runtime
HTTP/1.1 201 Created
Connection: keep-alive
Content-Length: 194
Content-Type: application/json
Date: Fri, 12 May 2017 04:37:08 GMT
{
"created_at": "2017-05-12 04:37:08.129860",
"id": "c1d78623-56bf-4487-9a72-1299b2c55e65",
"image": "DOCKER_USER/python-runtime",
"name": "python2.7",
"project_id": "default",
"status": "creating"
}
$ http GET http://192.168.33.18:7070/v1/runtimes/c1d78623-56bf-4487-9a72-1299b2c55e65
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 246
Content-Type: application/json
Date: Fri, 12 May 2017 04:37:50 GMT
{
"created_at": "2017-05-12 04:37:08",
"description": null,
"id": "c1d78623-56bf-4487-9a72-1299b2c55e65",
"image": "DOCKER_USER/python-runtime",
"name": "python2.7",
"project_id": "default",
"status": "available",
"updated_at": "2017-05-12 04:37:08"
}
.. end
#. Create a customized function package:
.. code-block:: console
$ mkdir ~/qinling_test
$ cat <<EOF > ~/qinling_test/main.py
import requests
def main():
r = requests.get('https://api.github.com/events')
return len(r.json())
if __name__ == '__main__':
main()
EOF
$ pip install requests -t ~/qinling_test
$ zip ~/qinling_test/qinling_test.zip ~/qinling_test/*
.. end
#. Create function, ``runtime_id`` comes from the output of above command:
.. code-block:: console
$ http -f POST http://192.168.33.18:7070/v1/functions name=github_test \
runtime_id=c1d78623-56bf-4487-9a72-1299b2c55e65 \
code='{"package": "true"}' \
package@~/qinling_test/qinling_test.zip
HTTP/1.1 201 Created
Connection: keep-alive
Content-Length: 234
Content-Type: application/json
Date: Fri, 12 May 2017 04:49:59 GMT
{
"code": {
"package": "true"
},
"created_at": "2017-05-12 04:49:59.659345",
"description": null,
"entry": "main",
"id": "352e4c02-3c6b-4860-9b85-f72344b1f986",
"name": "github_test",
"runtime_id": "c1d78623-56bf-4487-9a72-1299b2c55e65"
}
.. end
#. Invoke the function by specifying ``function_id``:
.. code-block:: console
$ http POST http://192.168.33.18:7070/v1/executions \
function_id=352e4c02-3c6b-4860-9b85-f72344b1f986
HTTP/1.1 201 Created
Connection: keep-alive
Content-Length: 255
Content-Type: application/json
Date: Thu, 11 May 2017 23:46:12 GMT
{
"created_at": "2017-05-12 04:51:10",
"function_id": "352e4c02-3c6b-4860-9b85-f72344b1f986",
"id": "80cd55be-d369-49b8-8bd5-e0bfc1d20d25",
"input": null,
"output": "{\"result\": 30}",
"status": "success",
"sync": true,
"updated_at": "2017-05-12 04:51:23"
}
.. end
If you invoke the same function again, you will find it is much faster
thanks to Qinling cache mechanism.

31
runtimes/python2.7/README.md
View File

@ -1,31 +0,0 @@
# Qinling: Python Environment
This is the Python environment for Qinling.
It's a Docker image containing a Python 2.7 runtime, along with a
dynamic loader. A few common dependencies are included in the
requirements.txt file.
## Customizing this image
To add package dependencies, edit requirements.txt to add what you
need, and rebuild this image (instructions below).
You also may want to customize what's available to the function in its
request context. You can do this by editing server.py (see the
comment in that file about customizing request context).
## Rebuilding and pushing the image
You'll need access to a Docker registry to push the image: you can
sign up for Docker hub at hub.docker.com, or use registries from
gcr.io, quay.io, etc. Let's assume you're using a docker hub account
called USER. Build and push the image to the the registry:
```
docker build -t USER/python-env . && docker push USER/python-env
```
## Using the image in Qinling
TBD

0
runtimes/python2.7/Dockerfile → runtimes/python2/Dockerfile
View File

22
runtimes/python2/README.md Normal file
View File

@ -0,0 +1,22 @@
# Qinling: Python Environment
This is the Python environment for Qinling.
It's a Docker image containing a Python 2.7 runtime, along with a
dynamic loader. A few common dependencies are included in the
requirements.txt file. End users need to provide their own dependencies
in their function packages through Qinling API or CLI.
## Rebuilding and pushing the image
You'll need access to a Docker registry to push the image, by default it's
docker hub. After modification, build a new image and upload to docker hub:
docker build -t USER/python-runtime. && docker push USER/python-runtime
## Using the image in Qinling
After the image is ready in docker hub, create a runtime in Qinling:
http POST http://127.0.0.1:7070/v1/runtimes name=python2.7 image=USER/python-runtime

0
runtimes/python2.7/requirements.txt → runtimes/python2/requirements.txt
View File

0
runtimes/python2.7/server.py → runtimes/python2/server.py
View File

71
tools/vagrant/Vagrantfile vendored Normal file
View File

@ -0,0 +1,71 @@
# -*- mode: ruby -*-
# vi: set ft=ruby :
VAGRANTFILE_API_VERSION = "2"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
config.vm.box = "ubuntu/xenial64"
config.vm.hostname = "qinling"
config.vm.network "private_network", ip: "192.168.33.18"
config.vm.network "forwarded_port", guest: 7070, host: 7070
config.vm.provider "virtualbox" do |vb|
vb.customize ["modifyvm", :id, "--memory", "1024"]
vb.customize ["modifyvm", :id, "--cpus", "1"]
vb.gui = false
end
config.vm.provision "shell", privileged: false, inline: <<-SHELL
#!/usr/bin/env bash
sudo apt-get update
sudo apt-get -y upgrade
sudo apt-get -y install python-dev python-setuptools libffi-dev \
libxslt1-dev libxml2-dev libyaml-dev libssl-dev rabbitmq-server git
# Install mysql and initialize database.
echo mysql-server-5.5 mysql-server/root_password password password | sudo debconf-set-selections
echo mysql-server-5.5 mysql-server/root_password_again password password | sudo debconf-set-selections
echo mysql-server-5.5 mysql-server/start_on_boot boolean true | sudo debconf-set-selections
sudo apt-get -y install mysql-server python-mysqldb
sudo sed -i 's/127.0.0.1/0.0.0.0/g' /etc/mysql/my.cnf
sudo sed -i '44 i skip-name-resolve' /etc/mysql/my.cnf
sudo service mysql restart
HOSTNAME="127.0.0.1"
PORT="3306"
USERNAME="root"
PASSWORD="password"
DBNAME="qinling"
create_db_sql="create database IF NOT EXISTS ${DBNAME}"
mysql -h${HOSTNAME} -P${PORT} -u${USERNAME} -p${PASSWORD} -e "${create_db_sql}"
# Change rabbitmq credential.
sudo rabbitmqctl change_password guest password
# Install pip.
curl -O https://bootstrap.pypa.io/get-pip.py && sudo python get-pip.py
sudo pip install httpie
# Install Qinling.
git clone https://github.com/LingxianKong/qinling.git
cd qinling
sudo pip install -e .
# Initialize Qinling configuration.
sudo mkdir -p /vagrant/etc/qinling
sudo mkdir -p /vagrant/log
sudo mkdir -p /opt/qinling/funtion
sudo chown ubuntu:ubuntu /opt/qinling/funtion
cp /vagrant/qinling.conf.sample /vagrant/etc/qinling/qinling.conf
# Qinling db migration.
qinling-db-manage --config-file /vagrant/etc/qinling/qinling.conf upgrade head
# Start Qinling service.
python qinling/cmd/launch.py --server api,engine --config-file /vagrant/etc/qinling/qinling.conf &
SHELL
end

23
tools/vagrant/qinling.conf.sample Normal file
View File

@ -0,0 +1,23 @@
[DEFAULT]
debug=True
verbose=False
log_file=/vagrant/log/qinling.log
logging_default_format_string=%(asctime)s.%(msecs)03d %(process)d %(levelname)s %(instance)s%(message)s (%(name)s) [-]
logging_context_format_string=%(asctime)s.%(msecs)03d %(process)d %(levelname)s %(instance)s%(message)s (%(name)s) [%(request_id)s %(user_identity)s]
logging_user_identity_format=%(user)s %(tenant)s
[api]
api_workers=1
[database]
connection=mysql://root:password@localhost:3306/qinling
[oslo_messaging_rabbit]
rabbit_password=password
[pecan]
auth_enable = false
[kubernetes]
kube_host = KUBERNETES_API_HOST:KUBERNETES_API_PORT
qinling_service_address = QINLING_API_ADDRESS