438 lines
19 KiB
XML
438 lines
19 KiB
XML
<?xml version="1.0" encoding="utf-8"?>
|
||
<chapter xmlns="http://docbook.org/ns/docbook"
|
||
xmlns:xi="http://www.w3.org/2001/XInclude"
|
||
xmlns:xlink="http://www.w3.org/1999/xlink" version="5.0"
|
||
xml:id="module001-ch008-queues-messaging">
|
||
<title>OpenStack Messaging and Queues</title>
|
||
<figure>
|
||
<title>Messaging in OpenStack</title>
|
||
<mediaobject>
|
||
<imageobject>
|
||
<imagedata fileref="figures/image04.png" contentwidth="7in"/>
|
||
</imageobject>
|
||
</mediaobject>
|
||
</figure>
|
||
<para>AMQP is the messaging technology chosen by the OpenStack
|
||
cloud. The AMQP broker, either RabbitMQ or Qpid, sits between any
|
||
two Nova components and allows them to communicate in a loosely
|
||
coupled fashion. More precisely, Nova components (the compute
|
||
fabric of OpenStack) use Remote Procedure Calls (RPC hereinafter)
|
||
to communicate to one another; however such a paradigm is built
|
||
atop the publish/subscribe paradigm so that the following benefits
|
||
can be achieved:</para>
|
||
<itemizedlist>
|
||
<listitem>
|
||
<para>Decoupling between client and servant (such as the client
|
||
does not need to know where the servant reference
|
||
is).</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para>Full a-synchronism between client and servant (such as the
|
||
client does not need the servant to run at the same time of
|
||
the remote call).</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para>Random balancing of remote calls (such as if more servants
|
||
are up and running, one-way calls are transparently dispatched
|
||
to the first available servant).</para>
|
||
</listitem>
|
||
</itemizedlist>
|
||
<para>Nova uses direct, fanout, and topic-based exchanges. The
|
||
architecture looks like the one depicted in the figure
|
||
below:</para>
|
||
<figure>
|
||
<title>AMQP</title>
|
||
<mediaobject>
|
||
<imageobject>
|
||
<imagedata fileref="figures/image24.png" contentwidth="7in"/>
|
||
</imageobject>
|
||
</mediaobject>
|
||
</figure>
|
||
<para>Nova implements RPC (both request+response, and one-way,
|
||
respectively nicknamed ‘rpc.call’ and ‘rpc.cast’) over AMQP by
|
||
providing an adapter class which take cares of marshaling and
|
||
un-marshaling of messages into function calls. Each Nova service,
|
||
such as Compute, Scheduler, and so on, creates two queues at the
|
||
initialization time, one which accepts messages with routing keys
|
||
‘NODE-TYPE.NODE-ID’, for example, compute.hostname, and another,
|
||
which accepts messages with routing keys as generic ‘NODE-TYPE’, for example compute. The former is used specifically when
|
||
Nova-API needs to redirect commands to a specific node like
|
||
‘euca-terminate instance’. In this case, only the compute node
|
||
whose host’s hypervisor is running the virtual machine can kill
|
||
the instance. The API acts as a consumer when RPC calls are
|
||
request/response, otherwise is acts as publisher only.</para>
|
||
<para><guilabel>Nova RPC Mappings</guilabel></para>
|
||
<para>The figure below shows the internals of a message broker node
|
||
(referred to as a RabbitMQ node in the diagrams) when a single
|
||
instance is deployed and shared in an OpenStack cloud. Every component
|
||
within Nova connects to the message broker and, depending on its
|
||
personality, such as a compute node or a network node, may
|
||
use the queue either as an Invoker (such as API or Scheduler) or a
|
||
Worker (such as Compute or Network). Invokers and Workers do not
|
||
actually exist in the Nova object model, but in this example they are used
|
||
as an abstraction for the sake of clarity. An Invoker is a
|
||
component that sends messages in the queuing system using <command>rpc.call
|
||
</command> and <command>rpc.cast</command>. A worker is a component
|
||
that receives messages from the queuing system and replies
|
||
accordingly to rcp.call operations.</para>
|
||
<para>Figure 2 shows the following internal elements:</para>
|
||
<itemizedlist>
|
||
<listitem>
|
||
<para><emphasis role="bold">Topic Publisher:</emphasis> A Topic
|
||
Publisher comes to life when an rpc.call or an rpc.cast
|
||
operation is executed; this object is instantiated and used to
|
||
push a message to the queuing system. Every publisher connects
|
||
always to the same topic-based exchange; its life-cycle is
|
||
limited to the message delivery.</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para><emphasis role="bold">Direct Consumer:</emphasis> A Direct
|
||
Consumer comes to life if (an only if) a rpc.call operation is
|
||
executed; this object is instantiated and used to receive a
|
||
response message from the queuing system; Every consumer
|
||
connects to a unique direct-based exchange via a unique
|
||
exclusive queue; its life-cycle is limited to the message
|
||
delivery; the exchange and queue identifiers are determined by
|
||
a UUID generator, and are marshaled in the message sent by the
|
||
Topic Publisher (only rpc.call operations).</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para><emphasis role="bold">Topic Consumer:</emphasis> A Topic
|
||
Consumer comes to life as soon as a Worker is instantiated and
|
||
exists throughout its life-cycle; this object is used to
|
||
receive messages from the queue and it invokes the appropriate
|
||
action as defined by the Worker role. A Topic Consumer
|
||
connects to the same topic-based exchange either via a shared
|
||
queue or via a unique exclusive queue. Every Worker has two
|
||
topic consumers, one that is addressed only during rpc.cast
|
||
operations (and it connects to a shared queue whose exchange
|
||
key is ‘topic’) and the other that is addressed only during
|
||
rpc.call operations (and it connects to a unique queue whose
|
||
exchange key is ‘topic.host’).</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para><emphasis role="bold">Direct Publisher:</emphasis> A Direct
|
||
Publisher comes to life only during rpc.call operations and it
|
||
is instantiated to return the message required by the
|
||
request/response operation. The object connects to a
|
||
direct-based exchange whose identity is dictated by the
|
||
incoming message.</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para><emphasis role="bold">Topic Exchange:</emphasis> The
|
||
Exchange is a routing table that exists in the context of a
|
||
virtual host (the multi-tenancy mechanism provided by Qpid or
|
||
RabbitMQ); its type (such as topic vs. direct) determines the
|
||
routing policy; a message broker node will have only one
|
||
topic-based exchange for every topic in Nova.</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para><emphasis role="bold">Direct Exchange:</emphasis> This is a
|
||
routing table that is created during rpc.call operations;
|
||
there are many instances of this kind of exchange throughout
|
||
the life-cycle of a message broker node, one for each rpc.call
|
||
invoked.</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para><emphasis role="bold">Queue Element:</emphasis> A Queue is
|
||
a message bucket. Messages are kept in the queue until a
|
||
Consumer (either Topic or Direct Consumer) connects to the
|
||
queue and fetch it. Queues can be shared or can be exclusive.
|
||
Queues whose routing key is ‘topic’ are shared amongst Workers
|
||
of the same personality.</para>
|
||
</listitem>
|
||
</itemizedlist>
|
||
<figure>
|
||
<title>RabbitMQ</title>
|
||
<mediaobject>
|
||
<imageobject>
|
||
<imagedata fileref="figures/image20.png" contentwidth="7in"/>
|
||
</imageobject>
|
||
</mediaobject>
|
||
</figure>
|
||
<para><guilabel>RPC Calls</guilabel></para>
|
||
<para>The diagram below shows the message flow during an rp.call
|
||
operation:</para>
|
||
<orderedlist>
|
||
<listitem>
|
||
<para>A Topic Publisher is instantiated to send the message
|
||
request to the queuing system; immediately before the
|
||
publishing operation. A Direct Consumer is instantiated to
|
||
wait for the response message.</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para>Once the message is dispatched by the exchange, it is
|
||
fetched by the Topic Consumer dictated by the routing key
|
||
(such as ‘topic.host’) and passed to the Worker in charge of
|
||
the task.</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para>Once the task is completed, a Direct Publisher is
|
||
allocated to send the response message to the queuing
|
||
system.</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para>Once the message is dispatched by the exchange, it is
|
||
fetched by the Direct Consumer dictated by the routing key
|
||
(such as ‘msg_id’) and passed to the Invoker.</para>
|
||
</listitem>
|
||
</orderedlist>
|
||
<figure>
|
||
<title>RabbitMQ</title>
|
||
<mediaobject>
|
||
<imageobject>
|
||
<imagedata fileref="figures/image28.png" contentwidth="7in"/>
|
||
</imageobject>
|
||
</mediaobject>
|
||
</figure>
|
||
<para><guilabel>RPC Casts</guilabel></para>
|
||
<para>The diagram below the message flow during an rp.cast
|
||
operation:</para>
|
||
<orderedlist>
|
||
<listitem>
|
||
<para>A Topic Publisher is instantiated to send the message
|
||
request to the queuing system.</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para>Once the message is dispatched by the exchange, it is
|
||
fetched by the Topic Consumer dictated by the routing key
|
||
(such as ‘topic’) and passed to the Worker in charge of the
|
||
task.</para>
|
||
</listitem>
|
||
</orderedlist>
|
||
<figure>
|
||
<title>RabbitMQ</title>
|
||
<mediaobject>
|
||
<imageobject>
|
||
<imagedata fileref="figures/image20.png" contentwidth="7in"/>
|
||
</imageobject>
|
||
</mediaobject>
|
||
</figure>
|
||
<para><guilabel>AMQP Broker Load</guilabel></para>
|
||
<para>At any given time the load of a message broker node running
|
||
either Qpid or RabbitMQ is a function of the following
|
||
parameters:</para>
|
||
<itemizedlist>
|
||
<listitem>
|
||
<para>Throughput of API calls: the number of API calls (more
|
||
precisely rpc.call ops) being served by the OpenStack cloud
|
||
dictates the number of direct-based exchanges, related queues
|
||
and direct consumers connected to them.</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para>Number of Workers: there is one queue shared amongst
|
||
workers with the same personality; however there are as many
|
||
exclusive queues as the number of workers; the number of
|
||
workers dictates also the number of routing keys within the
|
||
topic-based exchange, which is shared amongst all
|
||
workers.</para>
|
||
</listitem>
|
||
</itemizedlist>
|
||
<para>The figure below shows the status of a RabbitMQ node after
|
||
Nova components’ bootstrap in a test environment. Exchanges and
|
||
queues being created by Nova components are:</para>
|
||
<itemizedlist>
|
||
<listitem>
|
||
<para>Exchanges</para>
|
||
</listitem>
|
||
</itemizedlist>
|
||
<orderedlist>
|
||
<listitem>
|
||
<para>nova (topic exchange)</para>
|
||
</listitem>
|
||
</orderedlist>
|
||
<itemizedlist>
|
||
<listitem>
|
||
<para>Queues</para>
|
||
</listitem>
|
||
</itemizedlist>
|
||
<orderedlist>
|
||
<listitem>
|
||
<para>compute.phantom (phantom is the hostname)</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para>compute</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para>network.phantom (phantom is the hostname)</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para>network</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para>scheduler.phantom (phantom is the hostname)</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para>scheduler</para>
|
||
</listitem>
|
||
</orderedlist>
|
||
<para><guilabel>RabbitMQ Gotchas</guilabel></para>
|
||
<para>Nova uses Kombu to connect to the RabbitMQ environment. Kombu
|
||
is a Python library that in turn uses AMQPLib, a library that
|
||
implements the standard AMQP 0.8 at the time of writing. When
|
||
using Kombu, Invokers and Workers need the following parameters in
|
||
order to instantiate a Connection object that connects to the
|
||
RabbitMQ server (please note that most of the following material
|
||
can be also found in the Kombu documentation; it has been
|
||
summarized and revised here for the sake of clarity):</para>
|
||
<itemizedlist>
|
||
<listitem>
|
||
<para><emphasis role="bold">Hostname:</emphasis> The hostname to
|
||
the AMQP server.</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para><emphasis role="bold">Userid:</emphasis> A valid username
|
||
used to authenticate to the server.</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para><emphasis role="bold">Password:</emphasis> The password
|
||
used to authenticate to the server.</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para><emphasis role="bold">Virtual_host:</emphasis> The name of
|
||
the virtual host to work with. This virtual host must exist on
|
||
the server, and the user must have access to it. Default is
|
||
“/”.</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para><emphasis role="bold">Port:</emphasis> The port of the
|
||
AMQP server. Default is 5672 (amqp).</para>
|
||
</listitem>
|
||
</itemizedlist>
|
||
<para>The following parameters are default:</para>
|
||
<itemizedlist>
|
||
<listitem>
|
||
<para><emphasis role="bold">Insist:</emphasis> Insist on
|
||
connecting to a server. In a configuration with multiple
|
||
load-sharing servers, the Insist option tells the server that
|
||
the client is insisting on a connection to the specified
|
||
server. Default is False.</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para><emphasis role="bold">Connect_timeout:</emphasis> The
|
||
timeout in seconds before the client gives up connecting to
|
||
the server. The default is no timeout.</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para><emphasis role="bold">SSL:</emphasis> Use SSL to connect
|
||
to the server. The default is False.</para>
|
||
</listitem>
|
||
</itemizedlist>
|
||
<para>More precisely consumers need the following parameters:</para>
|
||
<itemizedlist>
|
||
<listitem>
|
||
<para><emphasis role="bold">Connection:</emphasis> The above
|
||
mentioned Connection object.</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para><emphasis role="bold">Queue:</emphasis> Name of the
|
||
queue.</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para><emphasis role="bold">Exchange:</emphasis> Name of the
|
||
exchange the queue binds to.</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para><emphasis role="bold">Routing_key:</emphasis> The
|
||
interpretation of the routing key depends on the value of the
|
||
exchange_type attribute.</para>
|
||
</listitem>
|
||
</itemizedlist>
|
||
<itemizedlist>
|
||
<listitem>
|
||
<para><emphasis role="bold">Direct exchange:</emphasis> If the
|
||
routing key property of the message and the routing_key
|
||
attribute of the queue are identical, then the message is
|
||
forwarded to the queue.</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para><emphasis role="bold">Fanout exchange:</emphasis> Messages
|
||
are forwarded to the queues bound the exchange, even if the
|
||
binding does not have a key.</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para><emphasis role="bold">Topic exchange:</emphasis> If the
|
||
routing key property of the message matches the routing key of
|
||
the key according to a primitive pattern matching scheme, then
|
||
the message is forwarded to the queue. The message routing key
|
||
then consists of words separated by dots (”.”, like domain
|
||
names), and two special characters are available; star (“”)
|
||
and hash (“#”). The star matches any word, and the hash
|
||
matches zero or more words. For example ”.stock.#” matches the
|
||
routing keys “usd.stock” and “eur.stock.db” but not
|
||
“stock.nasdaq”.</para>
|
||
</listitem>
|
||
</itemizedlist>
|
||
<itemizedlist>
|
||
<listitem>
|
||
<para><emphasis role="bold">Durable:</emphasis> This flag
|
||
determines the durability of both exchanges and queues;
|
||
durable exchanges and queues remain active when a RabbitMQ
|
||
server restarts. Non-durable exchanges/queues (transient
|
||
exchanges/queues) are purged when a server restarts. It is
|
||
worth noting that AMQP specifies that durable queues cannot
|
||
bind to transient exchanges. Default is True.</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para><emphasis role="bold">Auto_delete:</emphasis> If set, the
|
||
exchange is deleted when all queues have finished using it.
|
||
Default is False.</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para><emphasis role="bold">Exclusive:</emphasis> Exclusive
|
||
queues (such as non-shared) may only be consumed from by the
|
||
current connection. When exclusive is on, this also implies
|
||
auto_delete. Default is False.</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para><emphasis role="bold">Exchange_type:</emphasis> AMQP
|
||
defines several default exchange types (routing algorithms)
|
||
that covers most of the common messaging use cases.</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para><emphasis role="bold">Auto_ack:</emphasis> Acknowledgement
|
||
is handled automatically once messages are received. By
|
||
default auto_ack is set to False, and the receiver is required
|
||
to manually handle acknowledgment.</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para><emphasis role="bold">No_ack:</emphasis> It disables
|
||
acknowledgement on the server-side. This is different from
|
||
auto_ack in that acknowledgement is turned off altogether.
|
||
This functionality increases performance but at the cost of
|
||
reliability. Messages can get lost if a client dies before it
|
||
can deliver them to the application.</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para><emphasis role="bold">Auto_declare:</emphasis> If this is
|
||
True and the exchange name is set, the exchange will be
|
||
automatically declared at instantiation. Auto declare is on by
|
||
default. Publishers specify most the parameters of consumers
|
||
(they do not specify a queue name), but they can also
|
||
specify the following:</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para><emphasis role="bold">Delivery_mode:</emphasis> The default
|
||
delivery mode used for messages. The value is an integer. The
|
||
following delivery modes are supported by RabbitMQ:</para>
|
||
</listitem>
|
||
</itemizedlist>
|
||
<itemizedlist>
|
||
<listitem>
|
||
<para><emphasis role="bold">1 or “transient”:</emphasis> The
|
||
message is transient. Which means it is stored in memory only,
|
||
and is lost if the server dies or restarts.</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para><emphasis role="bold">2 or “persistent”:</emphasis> The
|
||
message is persistent. Which means the message is stored both
|
||
in-memory, and on disk, and therefore preserved if the server
|
||
dies or restarts.</para>
|
||
</listitem>
|
||
</itemizedlist>
|
||
<para>The default value is 2 (persistent). During a send operation,
|
||
Publishers can override the delivery mode of messages so that, for
|
||
example, transient messages can be sent over a durable
|
||
queue.</para>
|
||
</chapter>
|