heat/doc/source/scale_deployment.rst

Scaling Out APIs

Working in an architecture where a large number of incoming requests need to be handled, the API services can be overloaded. In those scenarios, in order to increase the system performance, it can be helpful to run multiple APIs and use a load balancing mechanism.

This guide details how to scale out the ReST and CFN APIs, also known as the heat-api and heat-api-cfn services, respectively.

Assumptions

This guide, using a devstack installation of Openstack, assumes that:

1. You have configured devstack from Single Machine Installation Guide;
2. You have set up Heat on devstack, as defined at Heat and Devstack <http://docs.openstack.org/developer/heat/getting_started/ on_devstack.html>;
3. You have installed HAProxy on the devstack server.

Architecture

This section shows the basic Heat architecture, the load balancing mechanism used and the target scaling out architecture.

Basic Architecture

The Heat architecture is as defined at Heat Architecture and shown in the diagram below, where we have a CLI that sends HTTP requests to the ReST and CFN APIs, which in turn make calls using AMQP to the Heat engine. :

|- [ REST API ] -|
[ CLI ] -- < HTTP > --                  -- < AMQP > -- [ ENGINE ]
|- [ CFN API ]  -|

Load Balancing

As there is a need to use a load balancer mechanism between the multiple APIs and the CLI, a proxy has to be deployed.

Because the Heat CLI and APIs communicate by exchanging HTTP requests and responses, a HAProxy HTTP load balancer server will be deployed between them.

This way, the proxy will take the CLIs requests to the APIs and act on their behalf. Once the proxy receives a response, it will be redirected to the caller CLI.

Target Architecture

A scaling out Heat architecture is represented in the diagram below: :

|- [ REST-API ] -|
|-     ...      -|
|- [ REST-API ] -|
[ CLI ] -- < HTTP > -- [ PROXY ] --                  -- < AMQP > -- [ ENGINE ]
|- [ API-CFN ]  -|
|-     ...      -|
|- [ API-CFN ]  -|

Thus, a request sent from the CLI looks like:

1. CLI contacts the proxy;
2. The HAProxy server, acting as a load balancer, redirects the call to an API instance;
3. The API server sends messages to RabbitMQ, and the engine server picks up messages from the RabbitMQ queue.

Deploying Multiple APIs

In order to run a Heat component separately, you have to execute one of the python scripts located at the bin directory of your Heat repository.

These scripts take as argument a configuration file. When using devstack, the configuration file is located at /etc/heat/heat.conf. For instance, to start new ReST and CFN API services, you must run: :

python bin/heat-api --config-file=/etc/heat/heat.conf
python bin/heat-api-cfn --config-file=/etc/heat/heat.conf

Running multiple APIs require that you create as many copies of this file as the number of instances you want to deploy.

Each API service must have a unique address to listen. This address have to be defined in the configuration file. For ReST and CFN APIs, modify the [heat_api] and [heat_api_cfn] blocks, respectively. :

[heat_api]
bind_port = {API_PORT}
bind_host = {API_HOST}

...

[heat_api_cfn]
bind_port = {API_CFN_PORT}
bind_host = {API_CFN_HOST}

In addition, if you want to run some API services in different machines than the devstack server, you have to update the loopback addresses found at the sql_connection and rabbit_host properties to the devstack server's IP, which must be reachable from the remote machine.

Deploying the Proxy

In order to simplify the deployment of the HAProxy server, we will replace the ReST and CFN APIs deployed when installing devstack by the HAProxy server. This way, there is no need to update, on the CLI, the addresses where it should look for the APIs. In this case, when it makes a call to any API, it will find the proxy, acting on their behalf.

Note that the addresses that the HAProxy will be listening to are the pairs API_HOST:API-PORT and API_CFN_HOST:API_CFN_PORT, found at the [heat_api] and [heat_api_cfn] blocks on the devstack server's configuration file. In addition, the original heat-api and heat-api-cfn processes running in these ports have to be killed, because these addresses must be free to be used by the proxy.

To deploy the HAProxy server on the devstack server, run haproxy -f apis-proxy.conf, where this configuration file looks like: :

global
    daemon
    maxconn 4000

defaults
    log  global
    maxconn  8000
    option  redispatch
    retries  3
    timeout  http-request 10s
    timeout  queue 1m
    timeout  connect 10s
    timeout  client 1m
    timeout  server 1m
    timeout  check 10s

listen rest_api_proxy
    # The values required below are the original ones that were in
    # /etc/heat/heat.conf on the devstack server.
    bind {API_HOST}:{API_PORT}
    balance  source
    option  tcpka
    option  httpchk
    # The values required below are the different addresses supplied when
    # running the ReST API instances.
    server SERVER_1 {HOST_1}:{PORT_1}
    ...
    server SERVER_N {HOST_N}:{PORT_N}

listen cfn_api_proxy
    # The values required below are the original ones that were in
    # /etc/heat/heat.conf on the devstack server.
    bind {API_CFN_HOST}:{API_CFN_PORT}
    balance  source
    option  tcpka
    option  httpchk
    # The values required below are the different addresses supplied when
    # running the CFN API instances.
    server SERVER_1 {HOST_1}:{PORT_1}
    ...
    server SERVER_N {HOST_N}:{PORT_N}

Sample

This section aims to clarify some aspects of the scaling out solution, as well as to show more details of the configuration by describing a concrete sample.

Architecture

This subsection shows a basic OpenStack architecture and the target one that will be meet.

Basic Architecture

For this sample, consider that:

1. We have an architecture composed by 3 machines configured in a LAN, with the addresses A: 10.0.0.1; B: 10.0.0.2; and C: 10.0.0.3;
2. The OpenStack devstack installation, including the Heat module, has been done in the machine A, as shown in the Assumptions section.

Target Architecture

At this moment, everything is running in a single devstack server. The next subsections show how to deploy a scaling out Heat architecture by:

1. Running one ReST and one CFN API on the machines B and C;
2. Setting up the HAProxy server on the machine A.

Running API Services

For each machine, B and C, you must do the following steps:

1. Clone the Heat repository https://github.com/openstack/heat;
2. Create a local copy of the configuration file /etc/heat/heat.conf from the machine A;
3. Make required changes on the configuration file;
4. Enter the Heat local repository and run:

python bin/heat-api --config-file=/etc/heat/heat.conf
python bin/heat-api-cfn --config-file=/etc/heat/heat.conf

Changes On Configuration

The original file from A looks like: :

[DEFAULT]
...
sql_connection = mysql://root:admin@127.0.0.1/heat?charset=utf8
rabbit_host = localhost
...
[heat_api]
bind_port = 8004
bind_host = 10.0.0.1
...
[heat_api_cfn]
bind_port = 8000
bind_host = 10.0.0.1

After the changes for B, it looks like: :

[DEFAULT]
...
sql_connection = mysql://root:admin@10.0.0.1/heat?charset=utf8
rabbit_host = 10.0.0.1
...
[heat_api]
bind_port = 8004
bind_host = 10.0.0.2
...
[heat_api_cfn]
bind_port = 8000
bind_host = 10.0.0.2

Setting Up HAProxy

On the machine A, kill the heat-api and heat-api-cfn processes by running pkill heat-api and pkill heat-api-cfn. After, run haproxy -f apis-proxy.conf with the following configuration: :

global
   daemon
   maxconn 4000

defaults
   log  global
   maxconn  8000
   option  redispatch
   retries  3
   timeout  http-request 10s
   timeout  queue 1m
   timeout  connect 10s
   timeout  client 1m
   timeout  server 1m
   timeout  check 10s

listen rest_api_proxy
   bind 10.0.0.1:8004
   balance  source
   option  tcpka
   option  httpchk
   server rest-server-1 10.0.0.2:8004
   server rest-server-2 10.0.0.3:8004

listen cfn_api_proxy
   bind 10.0.0.1:8000
   balance  source
   option  tcpka
   option  httpchk
   server cfn-server-1 10.0.0.2:8000
   server cfn-server-2 10.0.0.3:8000