diff --git a/README.rst b/README.rst index ef6e355a..d7e41977 100644 --- a/README.rst +++ b/README.rst @@ -1,19 +1,236 @@ ======= -qinling +Qinling ======= -Function as a Service (Documentation needs to be added, please stay tuned!) +.. note:: -Please fill here a long description which must be at least 3 lines wrapped on -80 cols, so that distribution package maintainers can use it in their packages. -Note that this is a hard requirement. + Qinling refers to Qinling Mountains in southern Shaanxi Province in China. + The mountains provide a natural boundary between North and South China and + support a huge variety of plant and wildlife, some of which is found + nowhere else on Earth. + +Qinling is Function as a Service for OpenStack. This project aims to provide a +platform to support serverless functions (like AWS Lambda). Qinling supports +different container orchestration platforms (Kubernetes/Swarm, etc.) and +different function package storage backends (local/Swift/S3) by nature using +plugin mechanism. * Free software: Apache license * Documentation: http://docs.openstack.org/developer/qinling * Source: http://git.openstack.org/cgit/openstack/qinling +* Features: https://blueprints.launchpad.net/qinling * Bugs: http://bugs.launchpad.net/qinling -Features --------- +Quick Start +~~~~~~~~~~~ -* TODO +Installation +------------ + +A fast and simple way to try Qinling is to create a Vagrant VM including all +related components and dependencies of Qinling service. For your convenience, +Qinling team already provide a Vagrantfile in ``tools/vagrant`` folder. + +Qinling is a FaaS implemented on top of container orchestration system such as +Kubernetes, Swarm, etc. Particularly, Kubernetes is a reference backend +considering its popularity. So, you need to setup Kubernetes first before +installing Qinling. The easiest way to setup Kubernetes is to use `Minikube +`_, it runs a +single-node Kubernetes cluster inside a VM alongside Qinling vagrant VM, so +they can communicate with each other without any network configuration. + +.. note:: + + In order to manage resources on Kubernetes, it is recommended to install + `kubectl `_ + command line tool. + +After Kubernetes installation, perform the following commands on your local +host. + +#. Setup HTTP proxy to access the Kubernetes API: + + .. code-block:: console + + $ kubectl proxy --accept-hosts='.*' --address='0.0.0.0' + + Starting to serve on [::]:8001 + + .. end + +#. Clone Qinling repo and go to ``vagrant`` directory: + + .. code-block:: console + + $ git clone https://github.com/LingxianKong/qinling.git + $ cd qinling/tools/vagrant + + .. end + +#. Modify Qinling sample config file according to your own environment. Suppose + your IP address of your local host is ``192.168.200.50``, default Kubernetes + API HTTP proxy port is ``8001``, and Qinling vagrant VM IP address is + ``192.168.33.18`` (the default value in ``Vagrantfile``): + + .. code-block:: console + + $ sed -i 's/KUBERNETES_API_HOST/192.168.200.50/' qinling.conf.sample + $ sed -i 's/KUBERNETES_API_PORT/8001/' qinling.conf.sample + $ sed -i 's/QINLING_API_ADDRESS/192.168.33.18/' qinling.conf.sample + + .. end + +#. Now, start Qinling vagrant VM: + + .. code-block:: console + + $ vagrant up + + .. end + +Getting started with Qinling +---------------------------- + +**Currently, RESTful API is the only way to interact with Qinling, +python-qinlingclient is still under development.** + +``httpie`` is a convenient tool to send HTTP request, make sure you installed +``httpie`` via ``pip install httpie`` before playing with Qinling. + +Perform following commands on your local host, the process will create +runtime/function/execution in Qinling. + +#. First, build a docker image that is used to create runtime in Qinling and + upload to docker hub. Only ``Python 2`` runtime is supported for now, but it + is very easy to add another program language support. Run the commands in + ``qinling`` repo directory, replace ``DOCKER_USER`` with your docker hub + username: + + .. code-block:: console + + $ cd runtimes/python2 + $ docker build -t DOCKER_USER/python-runtime . + $ docker push DOCKER_USER/python-runtime + + .. end + +#. Create runtime. A runtime in Qinling is running environment for a specific + language, this resource is supposed to be created/deleted/updated by cloud + operator. After creation, check the runtime status is ``available``: + + .. code-block:: console + + $ http POST http://192.168.33.18:7070/v1/runtimes name=python2.7 \ + image=DOCKER_USER/python-runtime + + HTTP/1.1 201 Created + Connection: keep-alive + Content-Length: 194 + Content-Type: application/json + Date: Fri, 12 May 2017 04:37:08 GMT + + { + "created_at": "2017-05-12 04:37:08.129860", + "id": "c1d78623-56bf-4487-9a72-1299b2c55e65", + "image": "DOCKER_USER/python-runtime", + "name": "python2.7", + "project_id": "default", + "status": "creating" + } + + $ http GET http://192.168.33.18:7070/v1/runtimes/c1d78623-56bf-4487-9a72-1299b2c55e65 + + HTTP/1.1 200 OK + Connection: keep-alive + Content-Length: 246 + Content-Type: application/json + Date: Fri, 12 May 2017 04:37:50 GMT + + { + "created_at": "2017-05-12 04:37:08", + "description": null, + "id": "c1d78623-56bf-4487-9a72-1299b2c55e65", + "image": "DOCKER_USER/python-runtime", + "name": "python2.7", + "project_id": "default", + "status": "available", + "updated_at": "2017-05-12 04:37:08" + } + + .. end + +#. Create a customized function package: + + .. code-block:: console + + $ mkdir ~/qinling_test + $ cat < ~/qinling_test/main.py + import requests + def main(): + r = requests.get('https://api.github.com/events') + return len(r.json()) + if __name__ == '__main__': + main() + EOF + $ pip install requests -t ~/qinling_test + $ zip ~/qinling_test/qinling_test.zip ~/qinling_test/* + + .. end + +#. Create function, ``runtime_id`` comes from the output of above command: + + .. code-block:: console + + $ http -f POST http://192.168.33.18:7070/v1/functions name=github_test \ + runtime_id=c1d78623-56bf-4487-9a72-1299b2c55e65 \ + code='{"package": "true"}' \ + package@~/qinling_test/qinling_test.zip + + HTTP/1.1 201 Created + Connection: keep-alive + Content-Length: 234 + Content-Type: application/json + Date: Fri, 12 May 2017 04:49:59 GMT + + { + "code": { + "package": "true" + }, + "created_at": "2017-05-12 04:49:59.659345", + "description": null, + "entry": "main", + "id": "352e4c02-3c6b-4860-9b85-f72344b1f986", + "name": "github_test", + "runtime_id": "c1d78623-56bf-4487-9a72-1299b2c55e65" + } + + .. end + +#. Invoke the function by specifying ``function_id``: + + .. code-block:: console + + $ http POST http://192.168.33.18:7070/v1/executions \ + function_id=352e4c02-3c6b-4860-9b85-f72344b1f986 + + HTTP/1.1 201 Created + Connection: keep-alive + Content-Length: 255 + Content-Type: application/json + Date: Thu, 11 May 2017 23:46:12 GMT + + { + "created_at": "2017-05-12 04:51:10", + "function_id": "352e4c02-3c6b-4860-9b85-f72344b1f986", + "id": "80cd55be-d369-49b8-8bd5-e0bfc1d20d25", + "input": null, + "output": "{\"result\": 30}", + "status": "success", + "sync": true, + "updated_at": "2017-05-12 04:51:23" + } + + .. end + + If you invoke the same function again, you will find it is much faster + thanks to Qinling cache mechanism. diff --git a/runtimes/python2.7/README.md b/runtimes/python2.7/README.md deleted file mode 100644 index 3529849e..00000000 --- a/runtimes/python2.7/README.md +++ /dev/null @@ -1,31 +0,0 @@ -# Qinling: Python Environment - -This is the Python environment for Qinling. - -It's a Docker image containing a Python 2.7 runtime, along with a -dynamic loader. A few common dependencies are included in the -requirements.txt file. - -## Customizing this image - -To add package dependencies, edit requirements.txt to add what you -need, and rebuild this image (instructions below). - -You also may want to customize what's available to the function in its -request context. You can do this by editing server.py (see the -comment in that file about customizing request context). - -## Rebuilding and pushing the image - -You'll need access to a Docker registry to push the image: you can -sign up for Docker hub at hub.docker.com, or use registries from -gcr.io, quay.io, etc. Let's assume you're using a docker hub account -called USER. Build and push the image to the the registry: - -``` - docker build -t USER/python-env . && docker push USER/python-env -``` - -## Using the image in Qinling - -TBD diff --git a/runtimes/python2.7/Dockerfile b/runtimes/python2/Dockerfile similarity index 100% rename from runtimes/python2.7/Dockerfile rename to runtimes/python2/Dockerfile diff --git a/runtimes/python2/README.md b/runtimes/python2/README.md new file mode 100644 index 00000000..277a6efa --- /dev/null +++ b/runtimes/python2/README.md @@ -0,0 +1,22 @@ +# Qinling: Python Environment + +This is the Python environment for Qinling. + +It's a Docker image containing a Python 2.7 runtime, along with a +dynamic loader. A few common dependencies are included in the +requirements.txt file. End users need to provide their own dependencies +in their function packages through Qinling API or CLI. + +## Rebuilding and pushing the image + +You'll need access to a Docker registry to push the image, by default it's +docker hub. After modification, build a new image and upload to docker hub: + + docker build -t USER/python-runtime. && docker push USER/python-runtime + + +## Using the image in Qinling + +After the image is ready in docker hub, create a runtime in Qinling: + + http POST http://127.0.0.1:7070/v1/runtimes name=python2.7 image=USER/python-runtime diff --git a/runtimes/python2.7/requirements.txt b/runtimes/python2/requirements.txt similarity index 100% rename from runtimes/python2.7/requirements.txt rename to runtimes/python2/requirements.txt diff --git a/runtimes/python2.7/server.py b/runtimes/python2/server.py similarity index 100% rename from runtimes/python2.7/server.py rename to runtimes/python2/server.py diff --git a/tools/vagrant/Vagrantfile b/tools/vagrant/Vagrantfile new file mode 100644 index 00000000..e0cf5a41 --- /dev/null +++ b/tools/vagrant/Vagrantfile @@ -0,0 +1,71 @@ +# -*- mode: ruby -*- +# vi: set ft=ruby : + +VAGRANTFILE_API_VERSION = "2" + +Vagrant.configure(VAGRANTFILE_API_VERSION) do |config| + config.vm.box = "ubuntu/xenial64" + config.vm.hostname = "qinling" + + config.vm.network "private_network", ip: "192.168.33.18" + config.vm.network "forwarded_port", guest: 7070, host: 7070 + + config.vm.provider "virtualbox" do |vb| + vb.customize ["modifyvm", :id, "--memory", "1024"] + vb.customize ["modifyvm", :id, "--cpus", "1"] + vb.gui = false + end + + config.vm.provision "shell", privileged: false, inline: <<-SHELL + + #!/usr/bin/env bash + sudo apt-get update + sudo apt-get -y upgrade + sudo apt-get -y install python-dev python-setuptools libffi-dev \ + libxslt1-dev libxml2-dev libyaml-dev libssl-dev rabbitmq-server git + + # Install mysql and initialize database. + echo mysql-server-5.5 mysql-server/root_password password password | sudo debconf-set-selections + echo mysql-server-5.5 mysql-server/root_password_again password password | sudo debconf-set-selections + echo mysql-server-5.5 mysql-server/start_on_boot boolean true | sudo debconf-set-selections + + sudo apt-get -y install mysql-server python-mysqldb + sudo sed -i 's/127.0.0.1/0.0.0.0/g' /etc/mysql/my.cnf + sudo sed -i '44 i skip-name-resolve' /etc/mysql/my.cnf + sudo service mysql restart + + HOSTNAME="127.0.0.1" + PORT="3306" + USERNAME="root" + PASSWORD="password" + DBNAME="qinling" + create_db_sql="create database IF NOT EXISTS ${DBNAME}" + mysql -h${HOSTNAME} -P${PORT} -u${USERNAME} -p${PASSWORD} -e "${create_db_sql}" + + # Change rabbitmq credential. + sudo rabbitmqctl change_password guest password + + # Install pip. + curl -O https://bootstrap.pypa.io/get-pip.py && sudo python get-pip.py + sudo pip install httpie + + # Install Qinling. + git clone https://github.com/LingxianKong/qinling.git + cd qinling + sudo pip install -e . + + # Initialize Qinling configuration. + sudo mkdir -p /vagrant/etc/qinling + sudo mkdir -p /vagrant/log + sudo mkdir -p /opt/qinling/funtion + sudo chown ubuntu:ubuntu /opt/qinling/funtion + cp /vagrant/qinling.conf.sample /vagrant/etc/qinling/qinling.conf + + # Qinling db migration. + qinling-db-manage --config-file /vagrant/etc/qinling/qinling.conf upgrade head + + # Start Qinling service. + python qinling/cmd/launch.py --server api,engine --config-file /vagrant/etc/qinling/qinling.conf & + + SHELL +end \ No newline at end of file diff --git a/tools/vagrant/qinling.conf.sample b/tools/vagrant/qinling.conf.sample new file mode 100644 index 00000000..3c86254e --- /dev/null +++ b/tools/vagrant/qinling.conf.sample @@ -0,0 +1,23 @@ +[DEFAULT] +debug=True +verbose=False +log_file=/vagrant/log/qinling.log +logging_default_format_string=%(asctime)s.%(msecs)03d %(process)d %(levelname)s %(instance)s%(message)s (%(name)s) [-] +logging_context_format_string=%(asctime)s.%(msecs)03d %(process)d %(levelname)s %(instance)s%(message)s (%(name)s) [%(request_id)s %(user_identity)s] +logging_user_identity_format=%(user)s %(tenant)s + +[api] +api_workers=1 + +[database] +connection=mysql://root:password@localhost:3306/qinling + +[oslo_messaging_rabbit] +rabbit_password=password + +[pecan] +auth_enable = false + +[kubernetes] +kube_host = KUBERNETES_API_HOST:KUBERNETES_API_PORT +qinling_service_address = QINLING_API_ADDRESS \ No newline at end of file