1f11e61068
from commit 925eb734a9d3cb46bb89a89ec1a78281d2d7afe9 it says qpid is removed so nova should not guide user to use qpid. Change-Id: Idf745ec5ccc08a3bab55c3e4394e4a4da900f56d
329 lines
13 KiB
ReStructuredText
329 lines
13 KiB
ReStructuredText
..
|
|
Copyright (c) 2010 Citrix Systems, Inc.
|
|
All Rights Reserved.
|
|
|
|
Licensed under the Apache License, Version 2.0 (the "License"); you may
|
|
not use this file except in compliance with the License. You may obtain
|
|
a copy of the License at
|
|
|
|
http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
Unless required by applicable law or agreed to in writing, software
|
|
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
|
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
|
License for the specific language governing permissions and limitations
|
|
under the License.
|
|
|
|
AMQP and Nova
|
|
=============
|
|
|
|
AMQP is the messaging technology chosen by the OpenStack cloud. The AMQP
|
|
broker, default to Rabbitmq, sits between any two Nova components and
|
|
allows them to communicate in a loosely coupled fashion. More precisely, Nova
|
|
components (the compute fabric of OpenStack) use Remote Procedure Calls (RPC
|
|
hereinafter) to communicate to one another; however such a paradigm is built
|
|
atop the publish/subscribe paradigm so that the following benefits can be
|
|
achieved:
|
|
|
|
* Decoupling between client and servant (such as the client does not need to
|
|
know where the servant's reference is).
|
|
|
|
* Full a-synchronism between client and servant (such as the client does not
|
|
need the servant to run at the same time of the remote call).
|
|
|
|
* Random balancing of remote calls (such as if more servants are up and
|
|
running, one-way calls are transparently dispatched to the first available
|
|
servant).
|
|
|
|
Nova uses direct, fanout, and topic-based exchanges. The architecture looks
|
|
like the one depicted in the figure below:
|
|
|
|
.. image:: /_static/images/rpc-arch.png
|
|
:width: 60%
|
|
|
|
Nova implements RPC (both request+response, and one-way, respectively nicknamed
|
|
``rpc.call`` and ``rpc.cast``) over AMQP by providing an adapter class which
|
|
take cares of marshaling and unmarshaling of messages into function calls. Each
|
|
Nova service (for example Compute, Scheduler, etc.) create two queues at the
|
|
initialization time, one which accepts messages with routing keys
|
|
``NODE-TYPE.NODE-ID`` (for example ``compute.hostname``) and another, which
|
|
accepts messages with routing keys as generic ``NODE-TYPE`` (for example
|
|
``compute``). The former is used specifically when Nova-API needs to redirect
|
|
commands to a specific node like ``openstack server delete $instance``. In this
|
|
case, only the compute node whose host's hypervisor is running the virtual
|
|
machine can kill the instance. The API acts as a consumer when RPC calls are
|
|
request/response, otherwise it acts as a publisher only.
|
|
|
|
Nova RPC Mappings
|
|
-----------------
|
|
|
|
The figure below shows the internals of a message broker node (referred to as a
|
|
RabbitMQ node in the diagrams) when a single instance is deployed and shared in
|
|
an OpenStack cloud. Every Nova component connects to the message broker and,
|
|
depending on its personality (for example a compute node or a network node),
|
|
may use the queue either as an Invoker (such as API or Scheduler) or a Worker
|
|
(such as Compute or Network). Invokers and Workers do not actually exist in the
|
|
Nova object model, but we are going to use them as an abstraction for sake of
|
|
clarity. An Invoker is a component that sends messages in the queuing system
|
|
via two operations: 1) ``rpc.call`` and ii) ``rpc.cast``; a Worker is a
|
|
component that receives messages from the queuing system and reply accordingly
|
|
to ``rpc.call`` operations.
|
|
|
|
Figure 2 shows the following internal elements:
|
|
|
|
Topic Publisher
|
|
A Topic Publisher comes to life when an ``rpc.call`` or an ``rpc.cast``
|
|
operation is executed; this object is instantiated and used to push a message
|
|
to the queuing system. Every publisher connects always to the same
|
|
topic-based exchange; its life-cycle is limited to the message delivery.
|
|
|
|
Direct Consumer
|
|
A Direct Consumer comes to life if (and only if) an ``rpc.call`` operation is
|
|
executed; this object is instantiated and used to receive a response message
|
|
from the queuing system. Every consumer connects to a unique direct-based
|
|
exchange via a unique exclusive queue; its life-cycle is limited to the
|
|
message delivery; the exchange and queue identifiers are determined by a UUID
|
|
generator, and are marshaled in the message sent by the Topic Publisher (only
|
|
``rpc.call`` operations).
|
|
|
|
Topic Consumer
|
|
A Topic Consumer comes to life as soon as a Worker is instantiated and exists
|
|
throughout its life-cycle; this object is used to receive messages from the
|
|
queue and it invokes the appropriate action as defined by the Worker role. A
|
|
Topic Consumer connects to the same topic-based exchange either via a shared
|
|
queue or via a unique exclusive queue. Every Worker has two topic consumers,
|
|
one that is addressed only during ``rpc.cast`` operations (and it connects to
|
|
a shared queue whose exchange key is ``topic``) and the other that is
|
|
addressed only during ``rpc.call`` operations (and it connects to a unique
|
|
queue whose exchange key is ``topic.host``).
|
|
|
|
Direct Publisher
|
|
A Direct Publisher comes to life only during ``rpc.call`` operations and it
|
|
is instantiated to return the message required by the request/response
|
|
operation. The object connects to a direct-based exchange whose identity is
|
|
dictated by the incoming message.
|
|
|
|
Topic Exchange
|
|
The Exchange is a routing table that exists in the context of a virtual host
|
|
(the multi-tenancy mechanism provided by RabbitMQ etc); its type (such as
|
|
topic vs. direct) determines the routing policy; a message broker node will
|
|
have only one topic-based exchange for every topic in Nova.
|
|
|
|
Direct Exchange
|
|
This is a routing table that is created during ``rpc.call`` operations; there
|
|
are many instances of this kind of exchange throughout the life-cycle of a
|
|
message broker node, one for each ``rpc.call`` invoked.
|
|
|
|
Queue Element
|
|
A Queue is a message bucket. Messages are kept in the queue until a Consumer
|
|
(either Topic or Direct Consumer) connects to the queue and fetch it. Queues
|
|
can be shared or can be exclusive. Queues whose routing key is ``topic`` are
|
|
shared amongst Workers of the same personality.
|
|
|
|
.. image:: /_static/images/rpc-rabt.png
|
|
:width: 60%
|
|
|
|
RPC Calls
|
|
---------
|
|
|
|
The diagram below shows the message flow during an ``rpc.call`` operation:
|
|
|
|
1. A Topic Publisher is instantiated to send the message request to the queuing
|
|
system; immediately before the publishing operation, a Direct Consumer is
|
|
instantiated to wait for the response message.
|
|
|
|
2. Once the message is dispatched by the exchange, it is fetched by the Topic
|
|
Consumer dictated by the routing key (such as 'topic.host') and passed to
|
|
the Worker in charge of the task.
|
|
|
|
3. Once the task is completed, a Direct Publisher is allocated to send the
|
|
response message to the queuing system.
|
|
|
|
4. Once the message is dispatched by the exchange, it is fetched by the Direct
|
|
Consumer dictated by the routing key (such as ``msg_id``) and passed to the
|
|
Invoker.
|
|
|
|
.. image:: /_static/images/rpc-flow-1.png
|
|
:width: 60%
|
|
|
|
RPC Casts
|
|
---------
|
|
|
|
The diagram below shows the message flow during an ``rpc.cast`` operation:
|
|
|
|
1. A Topic Publisher is instantiated to send the message request to the queuing
|
|
system.
|
|
|
|
2. Once the message is dispatched by the exchange, it is fetched by the Topic
|
|
Consumer dictated by the routing key (such as 'topic') and passed to the
|
|
Worker in charge of the task.
|
|
|
|
.. image:: /_static/images/rpc-flow-2.png
|
|
:width: 60%
|
|
|
|
AMQP Broker Load
|
|
----------------
|
|
|
|
At any given time the load of a message broker node running RabbitMQ etc
|
|
is function of the following parameters:
|
|
|
|
Throughput of API calls
|
|
The number of API calls (more precisely ``rpc.call`` ops) being served by the
|
|
OpenStack cloud dictates the number of direct-based exchanges, related queues
|
|
and direct consumers connected to them.
|
|
|
|
Number of Workers
|
|
There is one queue shared amongst workers with the same personality; however
|
|
there are as many exclusive queues as the number of workers; the number of
|
|
workers dictates also the number of routing keys within the topic-based
|
|
exchange, which is shared amongst all workers.
|
|
|
|
The figure below shows the status of a RabbitMQ node after Nova components'
|
|
bootstrap in a test environment. Exchanges and queues being created by Nova
|
|
components are:
|
|
|
|
* Exchanges
|
|
|
|
1. nova (topic exchange)
|
|
|
|
* Queues
|
|
|
|
1. ``compute.phantom`` (``phantom`` is hostname)
|
|
2. ``compute``
|
|
3. ``network.phantom`` (``phantom`` is hostname)
|
|
4. ``network``
|
|
5. ``scheduler.phantom`` (``phantom`` is hostname)
|
|
6. ``scheduler``
|
|
|
|
.. image:: /_static/images/rpc-state.png
|
|
:width: 60%
|
|
|
|
RabbitMQ Gotchas
|
|
----------------
|
|
|
|
Nova uses Kombu to connect to the RabbitMQ environment. Kombu is a Python
|
|
library that in turn uses AMQPLib, a library that implements the standard AMQP
|
|
0.8 at the time of writing. When using Kombu, Invokers and Workers need the
|
|
following parameters in order to instantiate a Connection object that connects
|
|
to the RabbitMQ server (please note that most of the following material can be
|
|
also found in the Kombu documentation; it has been summarized and revised here
|
|
for sake of clarity):
|
|
|
|
``hostname``
|
|
The hostname to the AMQP server.
|
|
|
|
``userid``
|
|
A valid username used to authenticate to the server.
|
|
|
|
``password``
|
|
The password used to authenticate to the server.
|
|
|
|
``virtual_host``
|
|
The name of the virtual host to work with. This virtual host must exist on
|
|
the server, and the user must have access to it. Default is "/".
|
|
|
|
``port``
|
|
The port of the AMQP server. Default is ``5672`` (amqp).
|
|
|
|
The following parameters are default:
|
|
|
|
``insist``
|
|
Insist on connecting to a server. In a configuration with multiple
|
|
load-sharing servers, the Insist option tells the server that the client is
|
|
insisting on a connection to the specified server. Default is False.
|
|
|
|
``connect_timeout``
|
|
The timeout in seconds before the client gives up connecting to the server.
|
|
The default is no timeout.
|
|
|
|
``ssl``
|
|
Use SSL to connect to the server. The default is False.
|
|
|
|
More precisely Consumers need the following parameters:
|
|
|
|
``connection``
|
|
The above mentioned Connection object.
|
|
|
|
``queue``
|
|
Name of the queue.
|
|
|
|
``exchange``
|
|
Name of the exchange the queue binds to.
|
|
|
|
``routing_key``
|
|
The interpretation of the routing key depends on the value of the
|
|
``exchange_type`` attribute.
|
|
|
|
Direct exchange
|
|
If the routing key property of the message and the ``routing_key`` attribute of
|
|
the queue are identical, then the message is forwarded to the queue.
|
|
|
|
Fanout exchange
|
|
Messages are forwarded to the queues bound the exchange, even if the
|
|
binding does not have a key.
|
|
|
|
Topic exchange
|
|
If the routing key property of the message matches the routing key of the
|
|
key according to a primitive pattern matching scheme, then the message is
|
|
forwarded to the queue. The message routing key then consists of words
|
|
separated by dots (``.``, like domain names), and two special characters
|
|
are available; star (``*``) and hash (``#``). The star matches any word,
|
|
and the hash matches zero or more words. For example ``.stock.#`` matches
|
|
the routing keys ``usd.stock`` and ``eur.stock.db`` but not
|
|
``stock.nasdaq``.
|
|
|
|
``durable``
|
|
This flag determines the durability of both exchanges and queues; durable
|
|
exchanges and queues remain active when a RabbitMQ server restarts.
|
|
Non-durable exchanges/queues (transient exchanges/queues) are purged when a
|
|
server restarts. It is worth noting that AMQP specifies that durable queues
|
|
cannot bind to transient exchanges. Default is True.
|
|
|
|
``auto_delete``
|
|
If set, the exchange is deleted when all queues have finished using it.
|
|
Default is False.
|
|
|
|
``exclusive``
|
|
Exclusive queues (such as non-shared) may only be consumed from by the
|
|
current connection. When exclusive is on, this also implies ``auto_delete``.
|
|
Default is False.
|
|
|
|
``exchange_type``
|
|
AMQP defines several default exchange types (routing algorithms) that covers
|
|
most of the common messaging use cases.
|
|
|
|
``auto_ack``
|
|
Acknowledgment is handled automatically once messages are received. By
|
|
default ``auto_ack`` is set to False, and the receiver is required to manually
|
|
handle acknowledgment.
|
|
|
|
``no_ack``
|
|
It disable acknowledgment on the server-side. This is different from
|
|
``auto_ack`` in that acknowledgment is turned off altogether. This
|
|
functionality increases performance but at the cost of reliability. Messages
|
|
can get lost if a client dies before it can deliver them to the application.
|
|
|
|
``auto_declare``
|
|
If this is True and the exchange name is set, the exchange will be
|
|
automatically declared at instantiation. Auto declare is on by default.
|
|
|
|
Publishers specify most the parameters of Consumers (such as they do not
|
|
specify a queue name), but they can also specify the following:
|
|
|
|
``delivery_mode``
|
|
The default delivery mode used for messages. The value is an integer. The
|
|
following delivery modes are supported by RabbitMQ:
|
|
|
|
``1`` (transient)
|
|
The message is transient. Which means it is stored in memory only, and is
|
|
lost if the server dies or restarts.
|
|
|
|
``2`` (persistent)
|
|
The message is persistent. Which means the message is stored both
|
|
in-memory, and on disk, and therefore preserved if the server dies or
|
|
restarts.
|
|
|
|
The default value is ``2`` (persistent). During a send operation, Publishers
|
|
can override the delivery mode of messages so that, for example, transient
|
|
messages can be sent over a durable queue.
|