openstack/cinder
Code Issues Proposed changes
3c6c2ee656
cinder/doc/source/contributor/rpc.rst
Sean McGinnis d5b539be36
Doc8: Stop skipping D001: Line too long 
This cleans up the cases where we had D001 violations so we can stop
skipping that check in doc8 runs.

Change-Id: Ie52f6ecac1a645fcbcc643b9ca63e033b622d830
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
2019-02-19 16:51:56 -06:00

281 lines
14 KiB
ReStructuredText
Raw Blame History

..
Copyright (c) 2010 Citrix Systems, Inc.
All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
AMQP and Cinder
===============
AMQP is the messaging technology chosen by the OpenStack cloud. The AMQP
broker, either RabbitMQ or Qpid, sits between any two Cinder components and
allows them to communicate in a loosely coupled fashion. More precisely, Cinder
components (the compute fabric of OpenStack) use Remote Procedure Calls (RPC
hereinafter) to communicate to one another; however such a paradigm is built
atop the publish/subscribe paradigm so that the following benefits can be
achieved:
* Decoupling between client and servant (such as the client does not need
to know where the servant's reference is).
* Full a-synchronism between client and servant (such as the client does
not need the servant to run at the same time of the remote call).
* Random balancing of remote calls (such as if more servants are up and
running, one-way calls are transparently dispatched to the first
available servant).
Cinder uses direct, fanout, and topic-based exchanges. The architecture looks
like the one depicted in the figure below:
.. image:: /images/rpc/arch.png
:width: 60%
..
Cinder implements RPC (both request+response, and one-way, respectively
nicknamed 'rpc.call' and 'rpc.cast') over AMQP by providing an adapter class
which take cares of marshaling and unmarshaling of messages into function
calls. Each Cinder service (for example Scheduler, Volume, etc.) create two
queues at the initialization time, one which accepts messages with routing keys
'NODE-TYPE.NODE-ID' (for example cinder-volume.hostname) and another, which
accepts messages with routing keys as generic 'NODE-TYPE' (for example
cinder-volume). The API acts as a consumer when RPC calls are request/response,
otherwise is acts as publisher only.
Cinder RPC Mappings
-------------------
The figure below shows the internals of a message broker node (referred to as a
RabbitMQ node in the diagrams) when a single instance is deployed and shared in
an OpenStack cloud. Every Cinder component connects to the message broker and,
depending on its personality, may use the queue either as an Invoker (such as
API or Scheduler) or a Worker (such as Volume). Invokers and Workers do not
actually exist in the Cinder object model, but we are going to use them as an
abstraction for sake of clarity. An Invoker is a component that sends messages
in the queuing system via two operations: 1) rpc.call and ii) rpc.cast; a
Worker is a component that receives messages from the queuing system and reply
accordingly to rpc.call operations.
Figure 2 shows the following internal elements:
* Topic Publisher: a Topic Publisher comes to life when an rpc.call or an
rpc.cast operation is executed; this object is instantiated and used to
push a message to the queuing system. Every publisher connects always to
the same topic-based exchange; its life-cycle is limited to the message
delivery.
* Direct Consumer: a Direct Consumer comes to life if (an only if) a
rpc.call operation is executed; this object is instantiated and used to
receive a response message from the queuing system; Every consumer
connects to a unique direct-based exchange via a unique exclusive queue;
its life-cycle is limited to the message delivery; the exchange and queue
identifiers are determined by a UUID generator, and are marshaled in the
message sent by the Topic Publisher (only rpc.call operations).
* Topic Consumer: a Topic Consumer comes to life as soon as a Worker is
instantiated and exists throughout its life-cycle; this object is used to
receive messages from the queue and it invokes the appropriate action as
defined by the Worker role. A Topic Consumer connects to the same
topic-based exchange either via a shared queue or via a unique exclusive
queue. Every Worker has two topic consumers, one that is addressed only
during rpc.cast operations (and it connects to a shared queue whose
exchange key is 'topic') and the other that is addressed only during
rpc.call operations (and it connects to a unique queue whose exchange key
is 'topic.host').
* Direct Publisher: a Direct Publisher comes to life only during rpc.call
operations and it is instantiated to return the message required by the
request/response operation. The object connects to a direct-based
exchange whose identity is dictated by the incoming message.
* Topic Exchange: The Exchange is a routing table that exists in the
context of a virtual host (the multi-tenancy mechanism provided by Qpid
or RabbitMQ); its type (such as topic vs. direct) determines the routing
policy; a message broker node will have only one topic-based exchange for
every topic in Cinder.
* Direct Exchange: this is a routing table that is created during rpc.call
operations; there are many instances of this kind of exchange throughout
the life-cycle of a message broker node, one for each rpc.call invoked.
* Queue Element: A Queue is a message bucket. Messages are kept in the
queue until a Consumer (either Topic or Direct Consumer) connects to the
queue and fetch it. Queues can be shared or can be exclusive. Queues
whose routing key is 'topic' are shared amongst Workers of the same
personality.
.. image:: /images/rpc/rabt.png
:width: 60%
..
RPC Calls
---------
The diagram below shows the message flow during an rpc.call operation:
1. a Topic Publisher is instantiated to send the message request to the
queuing system; immediately before the publishing operation, a Direct
Consumer is instantiated to wait for the response message.
2. once the message is dispatched by the exchange, it is fetched by the
Topic Consumer dictated by the routing key (such as 'topic.host') and
passed to the Worker in charge of the task.
3. once the task is completed, a Direct Publisher is allocated to send the
response message to the queuing system.
4. once the message is dispatched by the exchange, it is fetched by the
Direct Consumer dictated by the routing key (such as 'msg_id') and
passed to the Invoker.
.. image:: /images/rpc/flow1.png
:width: 60%
..
RPC Casts
---------
The diagram below the message flow during an rpc.cast operation:
1. A Topic Publisher is instantiated to send the message request to the
queuing system.
2. Once the message is dispatched by the exchange, it is fetched by the
Topic Consumer dictated by the routing key (such as 'topic') and passed
to the Worker in charge of the task.
.. image:: /images/rpc/flow2.png
:width: 60%
..
AMQP Broker Load
----------------
At any given time the load of a message broker node running either Qpid or
RabbitMQ is function of the following parameters:
* Throughput of API calls: the number of API calls (more precisely
rpc.call ops) being served by the OpenStack cloud dictates the number of
direct-based exchanges, related queues and direct consumers connected to
them.
* Number of Workers: there is one queue shared amongst workers with the
same personality; however there are as many exclusive queues as the
number of workers; the number of workers dictates also the number of
routing keys within the topic-based exchange, which is shared amongst all
workers.
The figure below shows the status of a RabbitMQ node after Cinder components'
bootstrap in a test environment (phantom is hostname). Exchanges and queues
being created by Cinder components are:
* Exchanges
1. cinder-scheduler_fanout (fanout exchange)
2. cinder-volume.phantom@lvm_fanout (fanout exchange)
3. cinder-volume_fanout (fanout exchange)
4. openstack (topic exchange)
* Queues
1. cinder-scheduler
2. cinder-scheduler.phantom
3. cinder-scheduler_fanout_572c35c0fbf94560b4c49572d5868ea5
4. cinder-volume
5. cinder-volume.phantom@lvm
6. cinder-volume.phantom@lvm.phantom
7. cinder-volume.phantom@lvm_fanout_cb3387f7a7684b1c9ee5f2f88325b7d5
8. cinder-volume_fanout_9017a1a7f4b44867983dcddfb56531a2
.. image:: /images/rpc/state.png
:width: 60%
..
RabbitMQ Gotchas
----------------
Cinder uses Kombu to connect to the RabbitMQ environment. Kombu is a Python
library that in turn uses AMQPLib, a library that implements the standard
AMQP 0.8 at the time of writing. When using Kombu, Invokers and Workers need
the following parameters in order to instantiate a Connection object that
connects to the RabbitMQ server (please note that most of the following
material can be also found in the Kombu documentation; it has been summarized
and revised here for sake of clarity):
* Hostname: The hostname to the AMQP server.
* Userid: A valid username used to authenticate to the server.
* Password: The password used to authenticate to the server.
* Virtual_host: The name of the virtual host to work with. This virtual
host must exist on the server, and the user must have access to it.
Default is "/".
* Port: The port of the AMQP server. Default is 5672 (amqp).
The following parameters are default:
* Insist: insist on connecting to a server. In a configuration with
multiple load-sharing servers, the Insist option tells the server that
the client is insisting on a connection to the specified server. Default
is False.
* Connect_timeout: the timeout in seconds before the client gives up
connecting to the server. The default is no timeout.
* SSL: use SSL to connect to the server. The default is False.
More precisely Consumers need the following parameters:
* Connection: the above mentioned Connection object.
* Queue: name of the queue.
* Exchange: name of the exchange the queue binds to.
* Routing_key: the interpretation of the routing key depends on the value
of the exchange_type attribute.
* Direct exchange: if the routing key property of the message and the
routing_key attribute of the queue are identical, then the message is
forwarded to the queue.
* Fanout exchange: messages are forwarded to the queues bound the
exchange, even if the binding does not have a key.
* Topic exchange: if the routing key property of the message matches the
routing key of the key according to a primitive pattern matching
scheme, then the message is forwarded to the queue. The message routing
key then consists of words separated by dots (".", like domain names),
and two special characters are available; star ("*") and hash ("#").
The star matches any word, and the hash matches zero or more words. For
example ".stock.#" matches the routing keys "usd.stock" and
"eur.stock.db" but not "stock.nasdaq".
* Durable: this flag determines the durability of both exchanges and
queues; durable exchanges and queues remain active when a RabbitMQ server
restarts. Non-durable exchanges/queues (transient exchanges/queues) are
purged when a server restarts. It is worth noting that AMQP specifies
that durable queues cannot bind to transient exchanges. Default is True.
* Auto_delete: if set, the exchange is deleted when all queues have
finished using it. Default is False.
* Exclusive: exclusive queues (such as non-shared) may only be consumed
from by the current connection. When exclusive is on, this also implies
auto_delete. Default is False.
* Exchange_type: AMQP defines several default exchange types (routing
algorithms) that covers most of the common messaging use cases.
* Auto_ack: acknowledgement is handled automatically once messages are
received. By default auto_ack is set to False, and the receiver is
required to manually handle acknowledgment.
* No_ack: it disable acknowledgement on the server-side. This is different
from auto_ack in that acknowledgement is turned off altogether. This
functionality increases performance but at the cost of reliability.
Messages can get lost if a client dies before it can deliver them to the
application.
* Auto_declare: if this is True and the exchange name is set, the exchange
will be automatically declared at instantiation. Auto declare is on by
default.
Publishers specify most the parameters of Consumers (such as they do not
specify a queue name), but they can also specify the following:
* Delivery_mode: the default delivery mode used for messages. The value is
an integer. The following delivery modes are supported by RabbitMQ:
* 1 or "transient": the message is transient. Which means it is
stored in memory only, and is lost if the server dies or restarts.
* 2 or "persistent": the message is persistent. Which means the
message is stored both in-memory, and on disk, and therefore
preserved if the server dies or restarts.
The default value is 2 (persistent). During a send operation, Publishers can
override the delivery mode of messages so that, for example, transient messages
can be sent over a durable queue.
View Git Blame Copy Permalink