Merge "doc: Cleanup admin docs"
This commit is contained in:
Zuul
Gerrit Code Review
commit
46cdd88358
@ -1,15 +1,14 @@
|
||||
-----------------------------------------
|
||||
=========================================
|
||||
AMQP 1.0 Protocol Driver Deployment Guide
|
||||
-----------------------------------------
|
||||
=========================================
|
||||
|
||||
.. currentmodule:: oslo_messaging
|
||||
|
||||
============
|
||||
Introduction
|
||||
============
|
||||
------------
|
||||
|
||||
The AMQP 1.0 Protocol Driver is a messaging transport backend
|
||||
supported in oslo.messaging. The driver maps the base oslo.messaging
|
||||
supported in oslo.messaging. The driver maps the base *oslo.messaging*
|
||||
capabilities for RPC and Notification message exchange onto version
|
||||
1.0 of the Advanced Message Queuing Protocol (AMQP 1.0, ISO/IEC
|
||||
19464). The driver is intended to support any messaging intermediary
|
||||
@ -17,20 +16,19 @@ capabilities for RPC and Notification message exchange onto version
|
||||
protocol.
|
||||
|
||||
More detail regarding the AMQP 1.0 Protocol is available from the
|
||||
`AMQP specification`_.
|
||||
|
||||
.. _AMQP specification: http://docs.oasis-open.org/amqp/core/v1.0/os/amqp-core-overview-v1.0-os.html
|
||||
`AMQP specification`__.
|
||||
|
||||
More detail regarding the driver's implementation is available from
|
||||
the `oslo specification`_.
|
||||
the `oslo specification`__.
|
||||
|
||||
__ http://docs.oasis-open.org/amqp/core/v1.0/os/amqp-core-overview-v1.0-os.html
|
||||
__ https://opendev.org/openstack/oslo-specs/src/branch/master/specs/juno/amqp10-driver-implementation.rst
|
||||
|
||||
.. _oslo specification: https://opendev.org/openstack/oslo-specs/src/branch/master/specs/juno/amqp10-driver-implementation.rst
|
||||
|
||||
========
|
||||
Abstract
|
||||
========
|
||||
--------
|
||||
|
||||
The AMQP 1.0 driver is one of a family of oslo.messaging backend
|
||||
The AMQP 1.0 driver is one of a family of *oslo.messaging* backend
|
||||
drivers. It currently supports two types of message intermediaries.
|
||||
The first type is an AMQP 1.0 messaging broker and the second type is
|
||||
an AMQP 1.0 message router. The driver should support additional
|
||||
@ -49,7 +47,7 @@ configuration parameters in order to do so.
|
||||
+--------------+-----------+------------+------------+-----------+
|
||||
|
||||
Direct Messaging
|
||||
----------------
|
||||
~~~~~~~~~~~~~~~~
|
||||
|
||||
The RPC messaging pattern is a synchronous exchange between
|
||||
client and server that is temporally bracketed. The direct messaging
|
||||
@ -61,7 +59,7 @@ instances of a message router to working with a large scale routed
|
||||
mesh interconnect topology.
|
||||
|
||||
Store and Forward
|
||||
-----------------
|
||||
~~~~~~~~~~~~~~~~~
|
||||
|
||||
The Notification messaging pattern is an asynchronous exchange from
|
||||
a notifier to a listener (e.g. consumer). The listener need not be
|
||||
@ -81,7 +79,7 @@ consumer. The message router does not provide durability or
|
||||
store-and-forward capabilities for notification messages.
|
||||
|
||||
Hybrid Messaging Backends
|
||||
-------------------------
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Oslo.messaging provides a mechanism to configure separate backends for
|
||||
RPC and Notification communications. This is supported through the
|
||||
@ -94,7 +92,7 @@ This document provides deployment and configuration information for use
|
||||
of this driver in hybrid messaging configurations.
|
||||
|
||||
Addressing
|
||||
----------
|
||||
~~~~~~~~~~
|
||||
|
||||
A new address syntax was added to the driver to support efficient
|
||||
direct message routing. This new syntax will also work with a broker
|
||||
@ -108,7 +106,7 @@ override this dynamic behavior and force the use of either the legacy
|
||||
or routable address syntax.
|
||||
|
||||
Message Acknowledgement
|
||||
-----------------------
|
||||
~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
A primary functional difference between a router and a
|
||||
broker intermediary type is when message acknowledgement occurs.
|
||||
@ -197,43 +195,43 @@ each notifier client can have at most one message outstanding
|
||||
clients are less than 100 the batch limit will never be met. This will
|
||||
effectively pause all notifying clients until the batch timeout expires.
|
||||
|
||||
=============
|
||||
|
||||
Prerequisites
|
||||
=============
|
||||
-------------
|
||||
|
||||
Protocol Engine
|
||||
---------------
|
||||
~~~~~~~~~~~~~~~
|
||||
|
||||
This driver uses the Apache QPID `Proton`_ AMQP 1.0 protocol engine.
|
||||
This driver uses the Apache QPID `Proton`__ AMQP 1.0 protocol engine.
|
||||
This engine consists of a platform specific library and a python
|
||||
binding. The driver does not directly interface with the engine API,
|
||||
as the API is a very low-level interface to the AMQP protocol.
|
||||
Instead, the driver uses the pure python `Pyngus`_ client API, which
|
||||
Instead, the driver uses the pure python `Pyngus`__ client API, which
|
||||
is layered on top of the protocol engine.
|
||||
|
||||
In order to run the driver the Proton Python bindings, Proton
|
||||
library, Proton header files, and Pyngus must be installed.
|
||||
|
||||
.. _Proton: http://qpid.apache.org/proton/index.html
|
||||
.. _Pyngus: https://github.com/kgiusti/pyngus
|
||||
__ http://qpid.apache.org/proton/index.html
|
||||
__ https://github.com/kgiusti/pyngus
|
||||
|
||||
Source packages for the `Pyngus client API`_ are available via PyPI.
|
||||
Source packages for the `Pyngus client API`__ are available via PyPI.
|
||||
|
||||
.. _Pyngus client API: https://pypi.org/project/pyngus
|
||||
__ https://pypi.org/project/pyngus
|
||||
|
||||
Pyngus depends on the Proton Python bindings. Source packages for the
|
||||
`Proton Python bindings`_ are also available via PyPI.
|
||||
`Proton Python bindings`__ are also available via PyPI.
|
||||
|
||||
.. _Proton Python bindings: https://pypi.org/project/python-qpid-proton
|
||||
__ https://pypi.org/project/python-qpid-proton
|
||||
|
||||
Since the AMQP 1.0 driver is an optional extension to Oslo.Messaging
|
||||
these packages are not installed by default. Use the 'amqp1' extras
|
||||
tag when installing Oslo.Messaging in order to pull in these extra
|
||||
packages:
|
||||
|
||||
::
|
||||
.. code-block:: shell
|
||||
|
||||
python -m pip install oslo.messaging[amqp1]
|
||||
$ python -m pip install oslo.messaging[amqp1]
|
||||
|
||||
The Proton package includes a C extension that links to the Proton
|
||||
library. The C extension is built locally when the Proton source
|
||||
@ -242,13 +240,9 @@ locally, there are a number of tools and libraries that need to be
|
||||
present on the system:
|
||||
|
||||
* The tools and library necessary for Python C development
|
||||
* The `SWIG`_ wrapper generator
|
||||
* The `OpenSSL`_ development libraries and headers
|
||||
* The `Cyrus SASL`_ development libraries and headers
|
||||
|
||||
.. _SWIG: http://www.swig.org/index.php
|
||||
.. _OpenSSL: https://www.openssl.org
|
||||
.. _Cyrus SASL: https://cyrusimap.org
|
||||
* The `SWIG`__ wrapper generator
|
||||
* The `OpenSSL`__ development libraries and headers
|
||||
* The `Cyrus SASL`__ development libraries and headers
|
||||
|
||||
**Note well**: Currently the Proton PyPI package only supports building
|
||||
the C extension on Linux systems.
|
||||
@ -257,74 +251,77 @@ Pre-built packages for both Pyngus and Proton engine are available for
|
||||
various Linux distributions (see `packages`_ below). It is recommended
|
||||
to use the pre-built packages if they are available for your platform.
|
||||
|
||||
__ http://www.swig.org/index.php
|
||||
__ https://www.openssl.org
|
||||
__ https://cyrusimap.org
|
||||
|
||||
Router Intermediary
|
||||
-------------------
|
||||
~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
This driver supports a *router* intermediary that supports version 1.0
|
||||
of the AMQP protocol. The direct messaging capabilities provided by
|
||||
this intermediary type are recommended for oslo.messaging RPC.
|
||||
|
||||
The driver has been tested with `qpid-dispatch-router`_ router in a
|
||||
The driver has been tested with `qpid-dispatch-router`__ router in a
|
||||
`devstack`_ environment. The version of qpid-dispatch-router
|
||||
**must** be at least 0.7.0. The qpid-dispatch-router also uses the
|
||||
Proton engine for its AMQP 1.0 support, so the Proton library must be
|
||||
installed on the system hosting the qpid-dispatch-router daemon.
|
||||
|
||||
.. _qpid-dispatch-router: http://qpid.apache.org/components/dispatch-router/
|
||||
|
||||
Pre-built packages for the router are available. See `packages`_ below.
|
||||
|
||||
__ http://qpid.apache.org/components/dispatch-router/
|
||||
|
||||
Broker Intermediary
|
||||
-------------------
|
||||
~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
This driver supports a *broker* intermediary that supports version 1.0
|
||||
of the AMQP protocol. The store and forward capabilities provided by
|
||||
this intermediary type are recommended for oslo.messaging Notifications.
|
||||
this intermediary type are recommended for *oslo.messaging* Notifications.
|
||||
|
||||
The driver has been tested with the `qpidd`_ broker in a `devstack`_
|
||||
The driver has been tested with the `qpidd`__ broker in a `devstack`_
|
||||
environment. The version of qpidd **must** be at least
|
||||
0.34. qpidd also uses the Proton engine for its AMQP 1.0 support, so
|
||||
the Proton library must be installed on the system hosting the qpidd
|
||||
daemon.
|
||||
|
||||
.. _qpidd: http://qpid.apache.org/components/cpp-broker/index.html
|
||||
|
||||
Pre-built packages for the broker are available. See `packages`_ below.
|
||||
|
||||
See the `oslo specification`_ for additional information regarding testing
|
||||
See the `oslo specification`__ for additional information regarding testing
|
||||
done on the driver.
|
||||
|
||||
__ http://qpid.apache.org/components/cpp-broker/index.html
|
||||
__ https://opendev.org/openstack/oslo-specs/src/branch/master/specs/juno/amqp10-driver-implementation.rst
|
||||
|
||||
|
||||
=============
|
||||
Configuration
|
||||
=============
|
||||
-------------
|
||||
|
||||
.. _transport urls:
|
||||
|
||||
Transport URL Enable
|
||||
--------------------
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
In oslo.messaging, the transport_url parameters define the OpenStack service
|
||||
backends for RPC and Notify. The url is of the form:
|
||||
backends for RPC and Notify. The url is of the form::
|
||||
|
||||
transport://user:pass@host1:port[,hostN:portN]/virtual_host
|
||||
|
||||
Where the transport value specifies the rpc or notification backend as
|
||||
one of **amqp**, rabbit, kafka, etc.
|
||||
|
||||
To specify and enable the AMQP 1.0 driver for RPC, in the section
|
||||
[DEFAULT] of the service configuration file, specify the
|
||||
'transport_url' parameter:
|
||||
To specify and enable the AMQP 1.0 driver for RPC, in the ``[DEFAULT]``
|
||||
section of the service configuration file, specify the
|
||||
``transport_url`` parameter:
|
||||
|
||||
::
|
||||
.. code-block:: ini
|
||||
|
||||
[DEFAULT]
|
||||
transport_url = amqp://username:password@routerhostname:5672
|
||||
|
||||
To specify and enable the AMQP 1.0 driver for Notify, in the section
|
||||
[NOTIFICATIONS] of the service configuration file, specify the
|
||||
'transport_url' parameter:
|
||||
To specify and enable the AMQP 1.0 driver for Notify, in the
|
||||
``[NOTIFICATIONS]`` section of the service configuration file, specify the
|
||||
``transport_url`` parameter:
|
||||
|
||||
::
|
||||
|
||||
@ -336,125 +333,63 @@ Note, that if a 'transport_url' parameter is not specified in the
|
||||
for both RPC and Notify backends.
|
||||
|
||||
Driver Options
|
||||
--------------
|
||||
~~~~~~~~~~~~~~
|
||||
|
||||
It is recommended that the default configuration options provided by
|
||||
the AMQP 1.0 driver be used. The configuration options can be modified
|
||||
in the oslo_messaging_amqp section of the service configuration file.
|
||||
in the :oslo.config:group:`oslo_messaging_amqp` section of the service
|
||||
configuration file.
|
||||
|
||||
Connection Options
|
||||
^^^^^^^^^^^^^^^^^^
|
||||
|
||||
In section [oslo_messaging_amqp]:
|
||||
|
||||
#. idle_timeout: Timeout in seconds for inactive connections.
|
||||
Default is disabled.
|
||||
|
||||
#. connection_retry_interval: Seconds to pause before attempting to
|
||||
re-connect.
|
||||
|
||||
#. connection_retry_backoff: Connection retry interval increment after
|
||||
unsuccessful failover attempt.
|
||||
|
||||
#. connection_retry_interval_max: The maximum duration for a
|
||||
connection retry interval.
|
||||
- :oslo.config:option:`oslo_messaging_amqp.idle_timeout`
|
||||
- :oslo.config:option:`oslo_messaging_amqp.connection_retry_interval`
|
||||
- :oslo.config:option:`oslo_messaging_amqp.connection_retry_backoff`
|
||||
- :oslo.config:option:`oslo_messaging_amqp.connection_retry_interval_max`
|
||||
|
||||
Message Send Options
|
||||
^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
In section [oslo_messaging_amqp]:
|
||||
|
||||
#. pre_settled: Send message types as pre-settled. Pre-settled messages
|
||||
will not receive acknowledgement from the peer.
|
||||
|
||||
#. link_retry_delay: Time to pause between re-connecting to an AMQP 1.0 link.
|
||||
|
||||
#. default_reply_timeout: The deadline for an rpc reply message delivery.
|
||||
|
||||
#. default_send_timeout: The deadline for an rpc cast or call message delivery.
|
||||
|
||||
#. default_notify_timeout: The deadline for a sent notification
|
||||
message delivery.
|
||||
- :oslo.config:option:`oslo_messaging_amqp.pre_settled`
|
||||
- :oslo.config:option:`oslo_messaging_amqp.link_retry_delay`
|
||||
- :oslo.config:option:`oslo_messaging_amqp.default_reply_timeout`
|
||||
- :oslo.config:option:`oslo_messaging_amqp.default_send_timeout`
|
||||
- :oslo.config:option:`oslo_messaging_amqp.default_notify_timeout`
|
||||
|
||||
.. _address mode:
|
||||
|
||||
Addressing Options
|
||||
^^^^^^^^^^^^^^^^^^
|
||||
|
||||
In section [oslo_messaging_amqp]:
|
||||
|
||||
#. addressing_mode: Indicates addressing mode used by the driver.
|
||||
|
||||
#. server_request_prefix: Legacy address prefix used when sending to a
|
||||
specific server.
|
||||
|
||||
#. broadcast_prefix: Legacy broadcast prefix used when broadcasting to
|
||||
all servers.
|
||||
|
||||
#. group_request_prefix: Legacy address prefix when sending to any
|
||||
server in a group.
|
||||
|
||||
#. rpc_address_prefix: Routable address prefix for all generated RPC addresses.
|
||||
|
||||
#. notify_address_prefix: Routable address prefix for all generated
|
||||
Notification addresses.
|
||||
|
||||
#. multicast_address: Appended to address prefix when sending a fanout address.
|
||||
|
||||
#. unicast_address: Appended to address prefix when sending to a
|
||||
particular RPC/Notification server.
|
||||
|
||||
#. anycast_address: Appended to address prefix when sending to a group
|
||||
of consumers.
|
||||
|
||||
#. default_notification_exchange: Exchange name used in notification
|
||||
addresses if not supplied by the application.
|
||||
|
||||
#. default_rpc_exchange: Exchange name used in RPC addresses if not
|
||||
supplied by the application.
|
||||
- :oslo.config:option:`oslo_messaging_amqp.addressing_mode`
|
||||
- :oslo.config:option:`oslo_messaging_amqp.server_request_prefix`
|
||||
- :oslo.config:option:`oslo_messaging_amqp.broadcast_prefix`
|
||||
- :oslo.config:option:`oslo_messaging_amqp.group_request_prefix`
|
||||
- :oslo.config:option:`oslo_messaging_amqp.rpc_address_prefix`
|
||||
- :oslo.config:option:`oslo_messaging_amqp.notify_address_prefix`
|
||||
- :oslo.config:option:`oslo_messaging_amqp.multicast_address`
|
||||
- :oslo.config:option:`oslo_messaging_amqp.unicast_address`
|
||||
- :oslo.config:option:`oslo_messaging_amqp.anycast_address`
|
||||
- :oslo.config:option:`oslo_messaging_amqp.default_notification_exchange`
|
||||
- :oslo.config:option:`oslo_messaging_amqp.default_rpc_exchange`
|
||||
|
||||
SSL Options
|
||||
^^^^^^^^^^^
|
||||
|
||||
In section [oslo_messaging_amqp]:
|
||||
|
||||
#. ssl: Attempt to connect via SSL. If no other ssl-related parameters
|
||||
are given, use the system's CA-bundle to verify the server's certificate.
|
||||
|
||||
#. ssl_ca_file: A file containing the trusted Certificate Authority's
|
||||
digital certificate (in PEM format). This certificate is used to
|
||||
authenticate the messaging backend.
|
||||
|
||||
#. ssl_cert_file: A file containing a digital certificate (in PEM
|
||||
format) that is used to identify the driver with the messaging bus
|
||||
(i.e. client authentication).
|
||||
|
||||
#. ssl_key_file:A file containing the private key used to sign the
|
||||
ssl_cert_file certificate (PEM format, optional if private key is
|
||||
stored in the certificate itself).
|
||||
|
||||
#. ssl_key_password: The password used to decrypt the private key
|
||||
(not required if private key is not encrypted).
|
||||
- :oslo.config:option:`oslo_messaging_amqp.ssl`
|
||||
- :oslo.config:option:`oslo_messaging_amqp.ssl_ca_file`
|
||||
- :oslo.config:option:`oslo_messaging_amqp.ssl_cert_file`
|
||||
- :oslo.config:option:`oslo_messaging_amqp.ssl_key_file`
|
||||
- :oslo.config:option:`oslo_messaging_amqp.ssl_key_password`
|
||||
|
||||
SASL Options
|
||||
^^^^^^^^^^^^
|
||||
|
||||
In section [oslo_messaging_amqp]:
|
||||
|
||||
#. sasl_mechanisms: Space separated list of acceptable SASL mechanisms.
|
||||
|
||||
#. sasl_config_dir: Path to the *directory* that contains the SASL
|
||||
configuration.
|
||||
|
||||
#. sasl_config_name: The name of SASL configuration file (without
|
||||
.conf suffix) in sasl_config_dir
|
||||
|
||||
#. sasl_default_realm: SASL realm to use if no realm present in username.
|
||||
|
||||
#. username: SASL user identifier for authentication with the message
|
||||
bus. Can be overridden by URL.
|
||||
|
||||
#. password: Password for username
|
||||
- :oslo.config:option:`oslo_messaging_amqp.sasl_mechanisms`
|
||||
- :oslo.config:option:`oslo_messaging_amqp.sasl_config_dir`
|
||||
- :oslo.config:option:`oslo_messaging_amqp.sasl_config_name`
|
||||
- :oslo.config:option:`oslo_messaging_amqp.sasl_default_realm`
|
||||
|
||||
AMQP Generic Options (**Note Well**)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
@ -464,23 +399,23 @@ options used by pre-1.0 drivers such as *amqp_durable_queues* or
|
||||
*amqp_auto_delete*.
|
||||
|
||||
qpid-dispatch-router
|
||||
--------------------
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
First, verify that the Proton library has been installed and is
|
||||
imported by the qpid-dispatch-router intermediary. This can be checked
|
||||
imported by the ``qpid-dispatch-router`` intermediary. This can be checked
|
||||
by running:
|
||||
|
||||
::
|
||||
|
||||
$ qdrouterd --help
|
||||
|
||||
and looking for references to qpid-dispatch include and config path
|
||||
options in the help text. If no qpid-dispatch information is listed,
|
||||
and looking for references to ``qpid-dispatch`` include and config path
|
||||
options in the help text. If no ``qpid-dispatch`` information is listed,
|
||||
verify that the Proton libraries are installed and that the version of
|
||||
the qdrouterd is greater than or equal to 0.6.0.
|
||||
``qdrouterd`` is greater than or equal to 0.6.0.
|
||||
|
||||
Second, configure the address patterns used by the driver. This is
|
||||
done by adding the following to /etc/qpid-dispatch/qdrouterd.conf.
|
||||
done by adding the following to ``/etc/qpid-dispatch/qdrouterd.conf``.
|
||||
|
||||
If the legacy syntax for the addressing mode is required, include the
|
||||
following:
|
||||
@ -539,15 +474,15 @@ For the routable syntax addressing mode, include the following:
|
||||
|
||||
**Note well**: For any customization of the `address mode`_ and syntax used,
|
||||
it is required that the address entity configurations in the
|
||||
/etc/qpid-dispatch/qdrouterd.conf be updated.
|
||||
`1`/etc/qpid-dispatch/qdrouterd.conf`` be updated.
|
||||
|
||||
qpidd
|
||||
-----
|
||||
~~~~~
|
||||
|
||||
First, verify that the Proton library has been installed and is
|
||||
imported by the qpidd broker. This can checked by running:
|
||||
|
||||
::
|
||||
.. code-block:: shell
|
||||
|
||||
$ qpidd --help
|
||||
|
||||
@ -557,9 +492,9 @@ that the version of qpidd is greater than or equal to 0.34.
|
||||
|
||||
Second, configure the default address patterns used by the
|
||||
driver for a broker-based backend. This is done by adding the
|
||||
following to /etc/qpid/qpidd.conf:
|
||||
following to ``/etc/qpid/qpidd.conf``:
|
||||
|
||||
::
|
||||
.. code-block:: ini
|
||||
|
||||
queue-patterns=exclusive
|
||||
queue-patterns=unicast
|
||||
@ -568,22 +503,22 @@ following to /etc/qpid/qpidd.conf:
|
||||
These patterns, *exclusive*, *unicast*, and *broadcast* are the
|
||||
legacy addressing values used by the driver. These can be overridden via the
|
||||
driver configuration options if desired (see above). If manually overridden,
|
||||
update the qpidd.conf values to match.
|
||||
update the ``qpidd.conf`` values to match.
|
||||
|
||||
|
||||
.. _devstack:
|
||||
|
||||
================
|
||||
DevStack Support
|
||||
================
|
||||
----------------
|
||||
|
||||
The plugin for the AMQP 1.0 oslo.messaging driver is supported by
|
||||
DevStack. The plugin supports the deployment of several different
|
||||
message bus configurations.
|
||||
|
||||
In local.conf [localrc] section, the `devstack-plugin-amqp1`_ plugin
|
||||
repository must be enabled. For example:
|
||||
In the ``[localrc]`` section of ``local.conf``, the `devstack-plugin-amqp1`__
|
||||
plugin repository must be enabled. For example:
|
||||
|
||||
::
|
||||
.. code-block:: ini
|
||||
|
||||
[[local|localrc]]
|
||||
enable_plugin amqp1 https://opendev.org/openstack/devstack-plugin-amqp1
|
||||
@ -591,7 +526,7 @@ repository must be enabled. For example:
|
||||
Set the username and password variables if needed for the
|
||||
configuration:
|
||||
|
||||
::
|
||||
.. code-block:: shell
|
||||
|
||||
AMQP1_USERNAME=queueuser
|
||||
AMQP1_PASSWORD=queuepassword
|
||||
@ -627,60 +562,50 @@ for the RPC and Notification messaging backends:
|
||||
| | | |
|
||||
+---------------+------------------+------------------+
|
||||
|
||||
__ https://github.com/openstack/devstack-plugin-amqp1.git
|
||||
|
||||
.. _devstack-plugin-amqp1: https://github.com/openstack/devstack-plugin-amqp1.git
|
||||
|
||||
.. _packages:
|
||||
|
||||
======================
|
||||
Platforms and Packages
|
||||
======================
|
||||
----------------------
|
||||
|
||||
PyPi
|
||||
----
|
||||
~~~~
|
||||
|
||||
Packages for `Pyngus pypi`_ and the `Proton pypi`_ engine are available on Pypi.
|
||||
|
||||
.. _Pyngus pypi: https://pypi.org/project/pyngus
|
||||
.. _Proton pypi: https://pypi.org/project/python-qpid-proton
|
||||
Packages for `Pyngus`__ and the `Proton`__ engine are available on PyPI.
|
||||
|
||||
__ https://pypi.org/project/pyngus
|
||||
__ https://pypi.org/project/python-qpid-proton
|
||||
|
||||
RHEL and Fedora
|
||||
---------------
|
||||
~~~~~~~~~~~~~~~
|
||||
|
||||
Packages exist in EPEL for RHEL/Centos 7, and Fedora 26+.
|
||||
Unfortunately, RHEL/Centos 6 base packages include a very old version
|
||||
of qpidd that does not support AMQP 1.0. EPEL's policy does not allow
|
||||
a newer version of qpidd for RHEL/Centos 6.
|
||||
Packages exist in EPEL for RHEL/Centos 7 and 8, and Fedora 26+.
|
||||
|
||||
The following packages must be installed on the system running the
|
||||
intermediary daemon:
|
||||
``qdrouterd`` daemon:
|
||||
|
||||
+--------------+--------------------------+
|
||||
| Intermediary | Package |
|
||||
+--------------+--------------------------+
|
||||
| qdrouterd | qpid-dispatch-router |
|
||||
| | python-qpid-proton |
|
||||
+--------------+--------------------------+
|
||||
| qpidd | qpid-cpp-server |
|
||||
| | qpid-proton-c |
|
||||
+--------------+--------------------------+
|
||||
- ``qpid-dispatch-router``
|
||||
- ``python-qpid-proton``
|
||||
|
||||
The following packages must be installed on the system running the
|
||||
``qpidd`` daemon:
|
||||
|
||||
qpidd daemon:
|
||||
|
||||
- qpid-cpp-server (version 0.26+)
|
||||
- qpid-proton-c
|
||||
- ``qpid-cpp-server`` (version 0.26+)
|
||||
- ``qpid-proton-c``
|
||||
|
||||
The following packages must be installed on the systems running the
|
||||
services that use the new driver:
|
||||
|
||||
- Proton libraries: qpid-proton-c-devel
|
||||
- Proton python bindings: python-qpid-proton
|
||||
- pyngus (via Pypi)
|
||||
- Proton libraries: ``qpid-proton-c-devel``
|
||||
- Proton python bindings: ``python-qpid-proton``
|
||||
- ``pyngus`` (via PyPI)
|
||||
|
||||
Debian and Ubuntu
|
||||
-----------------
|
||||
~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. todo:: Is this still true?
|
||||
|
||||
Packages for the Proton library, headers, and Python bindings are
|
||||
available in the Debian/Testing repository. Proton packages are not
|
||||
@ -688,29 +613,27 @@ yet available in the Ubuntu repository. The version of qpidd on both
|
||||
platforms is too old and does not support AMQP 1.0.
|
||||
|
||||
Until the proper package version arrive the latest packages can be
|
||||
pulled from the `Apache Qpid PPA`_ on Launchpad:
|
||||
pulled from the `Apache Qpid PPA`__ on Launchpad:
|
||||
|
||||
::
|
||||
.. code-block:: shell
|
||||
|
||||
sudo add-apt-repository ppa:qpid/released
|
||||
|
||||
.. _Apache Qpid PPA: https://launchpad.net/~qpid/+archive/ubuntu/released
|
||||
$ sudo add-apt-repository ppa:qpid/released
|
||||
|
||||
The following packages must be installed on the system running the
|
||||
qdrouterd daemon:
|
||||
``qdrouterd`` daemon:
|
||||
|
||||
- qdrouterd (version 0.8.0+)
|
||||
- ``qdrouterd`` (version 0.8.0+)
|
||||
|
||||
The following packages must be installed on the system running the
|
||||
qpidd daemon:
|
||||
``qpidd`` daemon:
|
||||
|
||||
- qpidd (version 0.34+)
|
||||
- ``qpidd`` (version 0.34+)
|
||||
|
||||
The following packages must be installed on the systems running the
|
||||
services that use the new driver:
|
||||
|
||||
- Proton libraries: libqpid-proton2-dev
|
||||
- Proton python bindings: python-qpid-proton
|
||||
- pyngus (via Pypi)
|
||||
- Proton libraries: ``libqpid-proton2-dev``
|
||||
- Proton python bindings: ``python-qpid-proton``
|
||||
- ``pyngus`` (via Pypi)
|
||||
|
||||
.. LocalWords: Acknowledgement acknowledgement
|
||||
__ https://launchpad.net/~qpid/+archive/ubuntu/released
|
||||
|
||||
@ -1,61 +1,59 @@
|
||||
-----------------------------
|
||||
=============================
|
||||
Kafka Driver Deployment Guide
|
||||
-----------------------------
|
||||
=============================
|
||||
|
||||
============
|
||||
Introduction
|
||||
============
|
||||
------------
|
||||
|
||||
The Kafka Driver is an experimental messaging transport backend
|
||||
in *oslo.messaging*. The driver maps the base oslo.messaging
|
||||
capabilities for notification message exchange onto V2.0 of the
|
||||
in *oslo.messaging*. The driver maps the base *oslo.messaging*
|
||||
capabilities for notification message exchange onto v2.0 of the
|
||||
Apache Kafka distributed streaming platform. More detail regarding
|
||||
the Apache Kafka server is available from the `Apache Kafka website`__.
|
||||
|
||||
__ https://kafka.apache.org/
|
||||
|
||||
More detail regarding the driver's implementation is available from
|
||||
the `adding kafka driver specification`_ and the `update kafka driver
|
||||
specification`_ .
|
||||
the `adding kafka driver specification`__ and the `update kafka driver
|
||||
specification`__.
|
||||
|
||||
__ https://kafka.apache.org/
|
||||
__ https://opendev.org/openstack/oslo-specs/src/branch/master/specs/liberty/adding-kafka-support.rst
|
||||
__ https://opendev.org/openstack/oslo-specs/src/branch/master/specs/queens/update-kafka-support.rst
|
||||
|
||||
.. _adding kafka driver specification: https://opendev.org/openstack/oslo-specs/src/branch/master/specs/liberty/adding-kafka-support.rst
|
||||
.. _update kafka driver specification: https://opendev.org/openstack/oslo-specs/src/branch/master/specs/queens/update-kafka-support.rst
|
||||
|
||||
========
|
||||
Overview
|
||||
========
|
||||
--------
|
||||
|
||||
The Kafka driver **only** supports use for sending and receiving
|
||||
*oslo.messaging* notifications. Specifically, the Kafka driver
|
||||
does not support *oslo.messaging* RPC transfers. Communications between
|
||||
the driver and Kafka server backend uses a `binary protocol over TCP`_
|
||||
the driver and Kafka server backend uses a `binary protocol over TCP`__
|
||||
that defines all APIs as request response message pairs. The Kafka
|
||||
driver integrates the confluent kafka python client for full
|
||||
driver integrates the ``confluent-kafka`` Python client for full
|
||||
protocol support and utilizes the Producer API to publish notification
|
||||
messages and the Consumer API for notification listener subscriptions.
|
||||
The driver is able to work with a single instance of a Kafka server or
|
||||
a clustered Kafka server deployment.
|
||||
|
||||
.. _binary protocol over TCP: https://kafka.apache.org/protocol.html
|
||||
__ https://kafka.apache.org/protocol.html
|
||||
|
||||
Hybrid Messaging Deployment
|
||||
---------------------------
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
*Oslo.messaging* provides a mechanism to configure separate messaging
|
||||
backends for RPC and notification communications. This is supported
|
||||
through the definition of separate RPC and notification
|
||||
`transport urls`_ in the service configuration. When the Kafka driver
|
||||
`transport urls`__ in the service configuration. When the Kafka driver
|
||||
is deployed for *oslo.messaging* notifications, a separate driver and
|
||||
messaging backend must be deployed for RPC communications. For these
|
||||
hybrid messaging configurations, either the `rabbit`_ or `amqp`_
|
||||
hybrid messaging configurations, either the `rabbit`__ or `amqp`__
|
||||
drivers can be deployed for *oslo.messaging* RPC.
|
||||
|
||||
.. _transport urls: https://docs.openstack.org/oslo.messaging/latest/reference/transport.html
|
||||
.. _rabbit: https://docs.openstack.org/oslo.messaging/latest/admin/drivers.html#rabbit
|
||||
.. _amqp: https://docs.openstack.org/oslo.messaging/latest/admin/AMQP1.0.html
|
||||
__ https://docs.openstack.org/oslo.messaging/latest/reference/transport.html
|
||||
__ https://docs.openstack.org/oslo.messaging/latest/admin/drivers.html#rabbit
|
||||
__ https://docs.openstack.org/oslo.messaging/latest/admin/AMQP1.0.html
|
||||
|
||||
Topics and vhost Support
|
||||
------------------------
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The Kafka topic is the feed name to which records are
|
||||
published. Topics in Kafka are multi-subscriber such that a topic can
|
||||
@ -70,7 +68,7 @@ per virtual host but **note** there is otherwise no access
|
||||
control or isolation provided by the Kafka server.
|
||||
|
||||
Listener Pools
|
||||
--------------
|
||||
~~~~~~~~~~~~~~
|
||||
|
||||
The Kafka driver provides support for listener pools. This capability
|
||||
is realized by mapping the listener pool name to a Kafka server
|
||||
@ -82,9 +80,8 @@ the Kafka driver and all listeners will be assigned to that
|
||||
group and the messages will effectively be load balanced across the
|
||||
competing listener instances.
|
||||
|
||||
|
||||
Synchronous Commit
|
||||
------------------
|
||||
~~~~~~~~~~~~~~~~~~
|
||||
|
||||
A primary functional difference between a Kafka server and a
|
||||
classic broker queue is that the offset or position of the
|
||||
@ -97,44 +94,43 @@ messages prior to the notification listener dispatch. Due to this, the
|
||||
driver does not support the re-queue operation and the driver can not
|
||||
replay messages from a Kafka partition.
|
||||
|
||||
=============
|
||||
Prerequisites
|
||||
=============
|
||||
|
||||
In order to run the driver the confluent-kafka python client must be
|
||||
installed. The Kafka driver integrates a `Python client based on librdkafka`_
|
||||
Prerequisites
|
||||
-------------
|
||||
|
||||
In order to run the driver the ``confluent-kafka`` Python client must be
|
||||
installed. The Kafka driver integrates a Python client based on `librdkafka`__
|
||||
for full protocol support and utilizes the Producer API to publish
|
||||
notification messages and the Consumer API for notification listener
|
||||
subscriptions.
|
||||
|
||||
.. _Python client based on librdkafka: https://github.com/confluentinc/confluent-kafka-python
|
||||
__ https://github.com/confluentinc/confluent-kafka-python
|
||||
|
||||
Source packages for the `confluent-kafka library`_ are available via PyPI.
|
||||
Source packages for the ``confluent-kafka`` library are available via `PyPI`__.
|
||||
Since the Kafka driver is an optional extension to *oslo.messaging*
|
||||
these packages are not installed by default. Use the 'kafka' extras
|
||||
these packages are not installed by default. Use the ``kafka`` extras
|
||||
tag when installing *oslo.messaging* in order to pull in these extra
|
||||
packages:
|
||||
|
||||
::
|
||||
.. code-block:: shell
|
||||
|
||||
python -m pip install oslo.messaging[kafka]
|
||||
$ python -m pip install oslo.messaging[kafka]
|
||||
|
||||
__ https://pypi.org/project/confluent-kafka/
|
||||
|
||||
|
||||
.. _confluent-kafka library: https://pypi.org/project/confluent-kafka/
|
||||
|
||||
=============
|
||||
Configuration
|
||||
=============
|
||||
-------------
|
||||
|
||||
Transport URL Enable
|
||||
--------------------
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
In *oslo.messaging*, the transport_url parameters define the OpenStack service
|
||||
backends for RPC and Notify. The url is of the form::
|
||||
In *oslo.messaging*, the ``transport_url`` parameters define the OpenStack
|
||||
service backends for RPC and Notify. The URL is of the form::
|
||||
|
||||
transport://user:pass@host1:port[,hostN:portN]/virtual_host
|
||||
|
||||
Where the transport value specifies the rpc or notification backend as
|
||||
Where the transport value specifies the RPC or notification backend as
|
||||
one of ``amqp``, ``rabbit``, ``kafka``, etc. To specify and enable the
|
||||
Kafka driver for notifications, in the section
|
||||
``[oslo_messaging_notifications]`` of the service configuration file,
|
||||
@ -144,90 +140,68 @@ specify the ``transport_url`` parameter::
|
||||
transport_url = kafka://username:password@kafkahostname:9092
|
||||
|
||||
Note, that if a ``transport_url`` parameter is not specified in the
|
||||
``[oslo_messaging_notifications]`` section, the ``[DEFAULT]``
|
||||
transport_url will be used for both RPC and Notify backends.
|
||||
``[oslo_messaging_notifications]`` section, the value of ``[DEFAULT]
|
||||
transport_url`` will be used for both RPC and notification backends.
|
||||
|
||||
Driver Options
|
||||
--------------
|
||||
~~~~~~~~~~~~~~
|
||||
|
||||
It is recommended that the default configuration options provided by
|
||||
the Kafka driver be used. The configuration options can be modified
|
||||
in the ``oslo_messaging_kafka`` section of the service configuration file.
|
||||
in the :oslo.config:group:`oslo_messaging_kafka` section of the service
|
||||
configuration file.
|
||||
|
||||
Notification Listener Options
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
kafka_max_fetch_bytes
|
||||
Initial maximum number of bytes per topic+partition to request
|
||||
when fetching messages from the broker.
|
||||
|
||||
kafka_consumer_timeout
|
||||
Consumer polling interval timeout.
|
||||
|
||||
consumer_group
|
||||
The default client group identifier.
|
||||
|
||||
enable_auto_commit
|
||||
Indicator to perform asynchronous consumer message commits.
|
||||
|
||||
max_poll_records
|
||||
The maximum number of messages to return per consume/poll operation.
|
||||
- :oslo.config:option:`oslo_messaging_kafka.kafka_max_fetch_bytes`
|
||||
- :oslo.config:option:`oslo_messaging_kafka.kafka_consumer_timeout`
|
||||
- :oslo.config:option:`oslo_messaging_kafka.consumer_group`
|
||||
- :oslo.config:option:`oslo_messaging_kafka.enable_auto_commit`
|
||||
- :oslo.config:option:`oslo_messaging_kafka.max_poll_records`
|
||||
|
||||
Notifier Options
|
||||
^^^^^^^^^^^^^^^^
|
||||
|
||||
producer_batch_timeout
|
||||
Delay (ms) to wait for messages in the producer queue to
|
||||
accumulate before constructing message sets to transmit to broker
|
||||
|
||||
producer_batch_size
|
||||
The maximum number of messages batched into one message set
|
||||
- :oslo.config:option:`oslo_messaging_kafka.producer_batch_timeout`
|
||||
- :oslo.config:option:`oslo_messaging_kafka.producer_batch_size`
|
||||
|
||||
Security Options
|
||||
^^^^^^^^^^^^^^^^
|
||||
|
||||
security_protocol
|
||||
The protocol used to communicate with the Kafka brokers.
|
||||
- :oslo.config:option:`oslo_messaging_kafka.security_protocol`
|
||||
- :oslo.config:option:`oslo_messaging_kafka.sasl_mechanism`
|
||||
- :oslo.config:option:`oslo_messaging_kafka.ssl_cafile`
|
||||
|
||||
sasl_mechanisms
|
||||
SASL mechanism to use for authentication. Current driver support
|
||||
is for PLAIN only.
|
||||
|
||||
ssl_cafile
|
||||
A file containing the trusted Certificate Authority's digital
|
||||
certificate (in PEM format). This certificate is used to
|
||||
authenticate the messaging backend.
|
||||
|
||||
================
|
||||
DevStack Support
|
||||
================
|
||||
----------------
|
||||
|
||||
The plugin for the Kafka *oslo.messaging* driver is supported by
|
||||
DevStack. As the Kafka driver can only be deployed for notifications,
|
||||
the plugin supports the deployment of several message bus configurations.
|
||||
In local.conf ``[localrc]`` section, the `devstack-plugin-kafka`_
|
||||
In the ``[localrc]`` section of ``local.conf``, the `devstack-plugin-kafka`__
|
||||
plugin repository must be enabled. For example:
|
||||
|
||||
::
|
||||
.. code-block:: ini
|
||||
|
||||
[[local|localrc]]
|
||||
enable_plugin kafka https://opendev.org/openstack/devstack-plugin-kafka
|
||||
|
||||
|
||||
Set the Kafka and Scala version and location variables if needed for
|
||||
the configuration
|
||||
|
||||
::
|
||||
.. code-block:: shell
|
||||
|
||||
KAFKA_VERSION=2.0.0
|
||||
KAFKA_BASEURL=http://archive.apache.org/dist/kafka
|
||||
SCALA_VERSION=2.12
|
||||
SCALA_BASEURL=http://www.scala-lang.org/riles/archive
|
||||
|
||||
The **RPC_** and **NOTIFY_** variables will define the message bus
|
||||
The ``RPC_`` and ``NOTIFY_`` variables will define the message bus
|
||||
configuration that will be used. The hybrid configurations will allow
|
||||
for the *rabbit* and *amqp* drivers to be used for the RPC transports
|
||||
while the *kafka* driver will be used for the Notify transport. The
|
||||
while the *kafka* driver will be used for the notification transport. The
|
||||
setting of the service variables will select which messaging
|
||||
intermediary is enabled for the configuration:
|
||||
|
||||
@ -241,5 +215,4 @@ intermediary is enabled for the configuration:
|
||||
| Config 1 | amqp | 5672 | kafka | 9092 |
|
||||
+------------+-----------+--------+-----------+--------+
|
||||
|
||||
|
||||
.. _devstack-plugin-kafka: https://github.com/openstack/devstack-plugin-kafka.git
|
||||
__ https://github.com/openstack/devstack-plugin-kafka.git
|
||||
|
||||
@ -23,6 +23,7 @@ copyright = '2018, Oslo Contributors'
|
||||
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
|
||||
extensions = [
|
||||
'sphinx.ext.autodoc',
|
||||
'sphinx.ext.todo',
|
||||
'openstackdocstheme',
|
||||
'stevedore.sphinxext',
|
||||
'oslo_config.sphinxext',
|
||||
|
||||
Loading…
Reference in New Issue
Block a user