openstack/heat
Code Issues Proposed changes

Create a guide for scaling out Heat API's

Browse Source 
This documentation describes a guide for scaling out the ReST and
CFN API's based on a devstack installation. HAProxy is used as a load
balancer, redirecting calls from CLI to the several API's instances.

Change-Id: I6efe988ce970f2a3aa08f39050f85dc1621e6774
Closes-bug: #1072958
This commit is contained in:
Samuel de Medeiros Queiroz 2013-10-29 10:49:21 -03:00
parent 95c5739533
commit 82ba0dd68f
2 changed files with 315 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
doc/source/index.rst
View File

@ -36,6 +36,7 @@ Getting Started
getting_started/index
templates/index
template_guide/index
scaleout_apis
glossary
Man Pages

314
doc/source/scaleout_apis.rst Normal file
View File

@ -0,0 +1,314 @@
..
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
================
Scaling Out APIs
================
Working in an architecture where a large number of incoming requests need to be
handled, the API services can be overloaded. In those scenarios, in order to
increase the system performance, it can be helpful to run multiple APIs and use
a load balancing mechanism.
This guide details how to scale out the ReST and CFN APIs, also known as the
*heat-api* and *heat-api-cfn* services, respectively.
Assumptions
===========
This guide, using a devstack installation of Openstack, assumes that:
1. You have configured devstack from `Single Machine Installation Guide
<http://devstack.org/guides/single-machine.html>`_;
2. You have set up Heat on devstack, as defined at `Heat and Devstack
<http://docs.openstack.org/developer/heat/getting_started/
on_devstack.html>`_;
3. You have installed `HAProxy <http://haproxy.1wt.eu>`_ on the devstack
server.
Architecture
============
This section shows the basic Heat architecture, the load balancing mechanism
used and the target scaling out architecture.
Basic Architecture
------------------
The Heat architecture is as defined at `Heat Architecture
<http://docs.openstack.org/developer/heat/architecture.html>`_ and shown in the
diagram below, where we have a CLI that sends HTTP requests to the ReST and CFN
APIs, which in turn make calls using AMQP to the Heat engine.
::
|- [ REST API ] -|
[ CLI ] -- < HTTP > -- -- < AMQP > -- [ ENGINE ]
|- [ CFN API ] -|
Load Balancing
--------------
As there is a need to use a load balancer mechanism between the multiple APIs
and the CLI, a proxy has to be deployed.
Because the Heat CLI and APIs communicate by exchanging HTTP requests and
responses, a `HAProxy <http://haproxy.1wt.eu>`_ HTTP load balancer server will
be deployed between them.
This way, the proxy will take the CLIs requests to the APIs and act on their
behalf. Once the proxy receives a response, it will be redirected to the caller
CLI.
Target Architecture
-------------------
A scaling out Heat architecture is represented in the diagram below:
::
|- [ REST-API ] -|
|- ... -|
|- [ REST-API ] -|
[ CLI ] -- < HTTP > -- [ PROXY ] -- -- < AMQP > -- [ ENGINE ]
|- [ API-CFN ] -|
|- ... -|
|- [ API-CFN ] -|
Thus, a request sent from the CLI looks like:
1. CLI contacts the proxy;
2. The HAProxy server, acting as a load balancer, redirects the call to an
API instance;
3. The API server sends messages to RabbitMQ, and the engine server picks up
messages from the RabbitMQ queue.
Deploying Multiple APIs
=======================
In order to run a Heat component separately, you have to execute one of the
python scripts located at the *bin* directory of your Heat repository.
These scripts take as argument a configuration file. When using devstack, the
configuration file is located at */etc/heat/heat.conf*. For instance, to start
new ReST and CFN API services, you must run:
::
python bin/heat-api --config-file=/etc/heat/heat.conf
python bin/heat-api-cfn --config-file=/etc/heat/heat.conf
Running multiple APIs require that you create as many copies of this file as the
number of instances you want to deploy.
Each API service must have a unique address to listen. This address have to be
defined in the configuration file. For ReST and CFN APIs, modify the
*[heat_api]* and *[heat_api_cfn]* blocks, respectively.
::
[heat_api]
bind_port = {API_PORT}
bind_host = {API_HOST}
...
[heat_api_cfn]
bind_port = {API_CFN_PORT}
bind_host = {API_CFN_HOST}
In addition, if you want to run some API services in different machines than
the devstack server, you have to update the loopback addresses found at the
*sql_connection* and *rabbit_host* properties to the devstack server's IP,
which must be reachable from the remote machine.
Deploying the Proxy
===================
In order to simplify the deployment of the HAProxy server, we will replace
the ReST and CFN APIs deployed when installing devstack by the HAProxy server.
This way, there is no need to update, on the CLI, the addresses where it should
look for the APIs. In this case, when it makes a call to any API, it will find
the proxy, acting on their behalf.
Note that the addresses that the HAProxy will be listening to are the pairs
*API_HOST:API-PORT* and *API_CFN_HOST:API_CFN_PORT*, found at the *[heat_api]*
and *[heat_api_cfn]* blocks on the devstack server's configuration file. In
addition, the original *heat-api* and *heat-api-cfn* processes running in these
ports have to be killed, because these addresses must be free to be used by the
proxy.
To deploy the HAProxy server on the devstack server, run
*haproxy -f apis-proxy.conf*, where this configuration file looks like:
::
global
daemon
maxconn 4000
defaults
log global
maxconn 8000
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout check 10s
listen rest_api_proxy
# The values required below are the original ones that were in
# /etc/heat/heat.conf on the devstack server.
bind {API_HOST}:{API_PORT}
balance source
option tcpka
option httpchk
# The values required below are the different addresses supplied when
# running the ReST API instances.
server SERVER_1 {HOST_1}:{PORT_1}
...
server SERVER_N {HOST_N}:{PORT_N}
listen cfn_api_proxy
# The values required below are the original ones that were in
# /etc/heat/heat.conf on the devstack server.
bind {API_CFN_HOST}:{API_CFN_PORT}
balance source
option tcpka
option httpchk
# The values required below are the different addresses supplied when
# running the CFN API instances.
server SERVER_1 {HOST_1}:{PORT_1}
...
server SERVER_N {HOST_N}:{PORT_N}
Sample
======
This section aims to clarify some aspects of the scaling out solution, as well
as to show more details of the configuration by describing a concrete sample.
Architecture
------------
This subsection shows a basic OpenStack architecture and the target one that
will be meet.
Basic Architecture
^^^^^^^^^^^^^^^^^^
For this sample, consider that:
1. We have an architecture composed by 3 machines configured in a LAN, with
the addresses A: 10.0.0.1; B: 10.0.0.2; and C: 10.0.0.3;
2. The OpenStack devstack installation, including the Heat module, has been
done in the machine A, as shown in the Assumptions section.
Target Architecture
^^^^^^^^^^^^^^^^^^^
At this moment, everything is running in a single devstack server. The next
subsections show how to deploy a scaling out Heat architecture by:
1. Running one ReST and one CFN API on the machines B and C;
2. Setting up the HAProxy server on the machine A.
Running API Services
--------------------
For each machine, B and C, you must do the following steps:
1. Clone the `Heat repository https://github.com/openstack/heat`_;
2. Create a local copy of the configuration file */etc/heat/heat.conf* from
the machine A;
3. Make required changes on the configuration file;
4. Enter the Heat local repository and run:
::
python bin/heat-api --config-file=/etc/heat/heat.conf
python bin/heat-api-cfn --config-file=/etc/heat/heat.conf
Changes On Configuration
^^^^^^^^^^^^^^^^^^^^^^^^
The original file from A looks like:
::
[DEFAULT]
...
sql_connection = mysql://root:admin@127.0.0.1/heat?charset=utf8
rabbit_host = localhost
...
[heat_api]
bind_port = 8004
bind_host = 10.0.0.1
...
[heat_api_cfn]
bind_port = 8000
bind_host = 10.0.0.1
After the changes for B, it looks like:
::
[DEFAULT]
...
sql_connection = mysql://root:admin@10.0.0.1/heat?charset=utf8
rabbit_host = 10.0.0.1
...
[heat_api]
bind_port = 8004
bind_host = 10.0.0.2
...
[heat_api_cfn]
bind_port = 8000
bind_host = 10.0.0.2
Setting Up HAProxy
------------------
On the machine A, kill the *heat-api* and *heat-api-cfn* processes by running
*pkill heat-api* and *pkill heat-api-cfn*. After, run
*haproxy -f apis-proxy.conf* with the following configuration:
::
global
daemon
maxconn 4000
defaults
log global
maxconn 8000
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout check 10s
listen rest_api_proxy
bind 10.0.0.1:8004
balance source
option tcpka
option httpchk
server rest-server-1 10.0.0.2:8004
server rest-server-2 10.0.0.3:8004
listen cfn_api_proxy
bind 10.0.0.1:8000
balance source
option tcpka
option httpchk
server cfn-server-1 10.0.0.2:8000
server cfn-server-2 10.0.0.3:8000