Adds oslo-http driver spec
This is a proposal to create a new driver implementation which communicates RPC client and server directly over HTTP without messaging queue. Change-Id: I563e6455deca9a95f814541a7f2a6a6353b1a858
This commit is contained in:
Mitsuhiro Tanino
parent
5d99032ae8
commit
b440f3dccd
@ -7,6 +7,15 @@
|
||||
Oslo Design Specifications
|
||||
============================
|
||||
|
||||
Bobcat
|
||||
========
|
||||
|
||||
.. toctree::
|
||||
:glob:
|
||||
:maxdepth: 1
|
||||
|
||||
specs/bobcat/*
|
||||
|
||||
Zed
|
||||
========
|
||||
|
||||
|
||||
BIN
images/http-driver/conductor-health-check.png
Normal file
BIN
images/http-driver/conductor-health-check.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 362 KiB |
BIN
images/http-driver/conductor-service-list.png
Normal file
BIN
images/http-driver/conductor-service-list.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 789 KiB |
BIN
images/http-driver/consul-federation.png
Normal file
BIN
images/http-driver/consul-federation.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 341 KiB |
BIN
images/http-driver/consul-service-list.png
Normal file
BIN
images/http-driver/consul-service-list.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 267 KiB |
439
specs/bobcat/http-driver.rst
Normal file
439
specs/bobcat/http-driver.rst
Normal file
@ -0,0 +1,439 @@
|
||||
=============
|
||||
http driver
|
||||
=============
|
||||
|
||||
This is a proposal to create a new driver implementation which communicates
|
||||
RPC client and server directly over HTTP without messaging queue.
|
||||
|
||||
Problem description
|
||||
===================
|
||||
|
||||
In oslo.messaging, RabbitMQ driver is widely used and validated that
|
||||
the driver works very stably on most of OpenStack environment.
|
||||
However, in PUB/SUB model using message queue, message queue itself
|
||||
might be a single point of failure and also message queue is bit
|
||||
difficult to support large scale of client and server communications.
|
||||
|
||||
For example, if there are more than 10,000 hypervisors which host
|
||||
nova-compute and neutron-agent, each RabbitMQ node must always support
|
||||
large number (ex. more than 40,000) of network connections from nova
|
||||
and neutron clients. In this situation, if RabbitMQ cluster restart
|
||||
happens, large number of RPC clients started to connect RabbitMQ nodes
|
||||
simultaneously, this makes RabbitMQ nodes unstable.
|
||||
|
||||
Also due to large number of hypervisors, RabbitMQ must support huge
|
||||
number of message queues to allow to communicate neutron-server and
|
||||
neutron-agent, for example. All queue data must be synchronized between
|
||||
all RabbitMQ nodes, so having large number of queue also makes RabbitMQ
|
||||
cluster unstable.
|
||||
|
||||
Note, this is not a proposal to deny RabbitMQ (PUB/SUB model) backend.
|
||||
The proposal should be aware both PUB/SUB model and REQ/RESP model have
|
||||
different advantages described in the following section.
|
||||
|
||||
PUB/SUB advantage examples
|
||||
--------------------------
|
||||
|
||||
- Simple architecture in a small cluster
|
||||
- Easy ACL configuration in a secure cluster
|
||||
|
||||
REQ/RESP advantage examples
|
||||
---------------------------
|
||||
|
||||
- Fault tolerant against control plane (consul) down
|
||||
- Easy to scale out the cluster
|
||||
|
||||
As mentioned in above, the PUB/SUB model has simple and easy to use
|
||||
most of the cases, but for large scale OpenStack environment, REQ/RESP
|
||||
model for all RPC communications could be more proper and reliable.
|
||||
|
||||
This specification proposes to implement OpenAPI based REQ/RESP model
|
||||
driver called ``http driver`` which enable to communicates between RPC
|
||||
client to RPC server directly without a component which is single point
|
||||
of failure for OpenStack internal communication.
|
||||
|
||||
Proposed change
|
||||
===============
|
||||
|
||||
This is overview diagram of HTTP driver for RPC Call/Cast/Cast Fanout requests.
|
||||
There are 4 main components to realize the driver.
|
||||
|
||||
* RPC client
|
||||
* RPC server
|
||||
* Endpoint datastore (ex. Consul)
|
||||
* Broadcaster
|
||||
|
||||
.. code-block:: none
|
||||
|
||||
+--------------+
|
||||
+------>| Consul |<------------------+
|
||||
| +--------------+ |
|
||||
| +-------------+
|
||||
| +--->| RPC server |
|
||||
| | +-------------+
|
||||
+-------------+ Call/Cast RPC | +-------------+
|
||||
| RPC client +---------------------------> | RPC server |
|
||||
+-------------+ | +-------------+
|
||||
| | +-------------+
|
||||
| Cast Fanout +--->| RPC server |
|
||||
| | +-------------+
|
||||
| +--------------+ |
|
||||
+------>| Broadcaster |---------+
|
||||
+--------------+
|
||||
Send Cast to all RPC servers
|
||||
|
||||
|
||||
.. note:: Scope of the proposal
|
||||
|
||||
- The scope of this proposal covers Call/Cast/Cast Fanout RPC cases
|
||||
- Notification is out of scope in this proposal
|
||||
|
||||
Main points to be improved by the architecture
|
||||
----------------------------------------------
|
||||
* Separate control plane from data plane
|
||||
* Allow direct API call without single point of failure
|
||||
* Reduce RPC communication between multiple availability zones
|
||||
|
||||
Responsibility of each components like followings.
|
||||
|
||||
RPC Client
|
||||
^^^^^^^^^^
|
||||
Fetch the RPC server list from the consul cluster for
|
||||
an RPC target service.
|
||||
Pick up appropriate RPC server for the RPC, then send a RPC request
|
||||
over HTTP protocol.
|
||||
|
||||
RPC Server
|
||||
^^^^^^^^^^
|
||||
Listen and handle an in-coming rpc request over HTTP protocol
|
||||
|
||||
When the RPC server starts, the server registers its RPC server
|
||||
information to Consul cluster as Consul's "service".
|
||||
Also the PRC Server works as ``Push Style``, not poll style.
|
||||
|
||||
For RPC client/server communication over HTTP with RESTful API, OpenAPI
|
||||
allows us to define formatted RESTful API definition easily. Once API
|
||||
definition is generated, it can be launched as http server using
|
||||
Web framework and WSGI server. For example, they are candidate for Web
|
||||
framework and WSGI server to enable HTTP client and server.
|
||||
|
||||
* Web famework: `connexion <https://connexion.readthedocs.io/en/latest/>`_
|
||||
* WSGI server: eventlet
|
||||
|
||||
Endpoint datastore
|
||||
^^^^^^^^^^^^^^^^^^
|
||||
Store the RPC server information.
|
||||
|
||||
The server information includes following items.
|
||||
oslo.messaging layer information(exchange, topic and server)
|
||||
RPC server information (IP address, port and etc).
|
||||
|
||||
In this spec, `Consul <https://developer.hashicorp.com/consul/api-docs>`_
|
||||
is one example imeplementation for the endpoint datastore.
|
||||
|
||||
For other use-case, the service discovery feature code with backend
|
||||
selectable implementation should be added for users who would like
|
||||
to avoid introducing new dependency for datastore.
|
||||
|
||||
HTTP Broadcaster
|
||||
^^^^^^^^^^^^^^^^
|
||||
An amplifier for an HTTP request
|
||||
|
||||
HTTP protocol doesn't have broadcast mechanism though the oslo
|
||||
RPC has Cast Fanout RPC.
|
||||
The Broadcaster sends an in-coming RPC to all target RPC servers.
|
||||
|
||||
Sample RPC communication flow for Nova
|
||||
--------------------------------------
|
||||
.. code-block:: none
|
||||
|
||||
+--------------+
|
||||
+------>| Consul |
|
||||
| +--------------+
|
||||
|
|
||||
| * Call RPC
|
||||
+----------------+ (1) select_destinations +----------------+
|
||||
| nova-conductor +---------------------------> | nova-scheduler |
|
||||
| +------+ +----------------+
|
||||
+----------------+ | * Cast RPC
|
||||
| (2) build_and_run_instance
|
||||
| +----------------+
|
||||
+--------------------->| nova-compute |
|
||||
+----------------+
|
||||
|
||||
In this figure, nova-conductor on left side is RPC client and each
|
||||
nova-scheduler and nova-compute on right side are RPC servers.
|
||||
|
||||
select_destinations() Call RPC
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
During instance creation, in order to choose target nova-compute
|
||||
node to launch new instance, nova-conductor make a Call RPC request
|
||||
select_destinations() to nova-scheduler.
|
||||
In this case, Call RPC communication works with following steps.
|
||||
|
||||
#. A nova-conductor fetches nova-scheduler's hostn and port
|
||||
port information from Consul, and pick one of Call RPC
|
||||
target randomly
|
||||
#. A nova-conductor makes a select_destinations() Call RPC and
|
||||
wait a response.
|
||||
|
||||
build_and_run_instance() Cast RPC
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
During instance creation, once target nova-compute is choosen by
|
||||
scheduler, nova-conductor sends instance creation request to
|
||||
target nova-compute using build_and_run_instance() Cast RPC.
|
||||
In this case, Cast RPC communication works with following steps.
|
||||
|
||||
#. A nova-conductor fetches a target nova-compute's host and
|
||||
port information from Consul
|
||||
#. A nova-conductor makes a build_and_run_instance() Cast RPC and
|
||||
return immediately without waiting actual instance creation
|
||||
|
||||
|
||||
Sample RPC server information registered in Consul
|
||||
--------------------------------------------------
|
||||
Nova's RPC server information
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
This figure shows Nova's RPC server information registered into `nova` AZ.
|
||||
Nova has nova.scheduler, nova-conductor, nova-compute and nova-consoleauth
|
||||
as RPC server.
|
||||
|
||||
.. image:: ../../../../images/http-driver/consul-service-list.png
|
||||
:width: 850px
|
||||
:align: left
|
||||
|
||||
nova-condutor RPC server list
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
This figure shows nova-conductor RPC server list on `nova` AZ.
|
||||
Each nova-conductor has <service name>:<port number> and also
|
||||
<IP address>:<port number> identifier. RPC client access to
|
||||
RPC server via the <IP address>:<port number>.
|
||||
|
||||
.. image:: ../../../../images/http-driver/conductor-service-list.png
|
||||
:width: 850px
|
||||
:align: left
|
||||
|
||||
Health check status of one of nova-condutor RPC server
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
Consul has health check mechanism for registered HTTP endpoint.
|
||||
The figure shows one of nova-conductor RPC server respond 200 OK
|
||||
to Consul's periodical health check. If the endpoint is healthy,
|
||||
the endpoint marked as green check.
|
||||
|
||||
.. image:: ../../../../images/http-driver/conductor-health-check.png
|
||||
:width: 850px
|
||||
:align: left
|
||||
|
||||
Multi-AZ support
|
||||
^^^^^^^^^^^^^^^^
|
||||
Consul has a feature to federate multiple Consul cluster into one to
|
||||
support multiple datacenter case. In the figure, there are multiple
|
||||
nova AZ such as `nova`, `az-1`, `az-2` and `az-3`, then each nova AZ
|
||||
has one Consul cluster. These 4 Consule clusters are manged as one
|
||||
cluster using Consul federation.
|
||||
|
||||
.. image:: ../../../../images/http-driver/consul-federation.png
|
||||
:width: 850px
|
||||
:align: left
|
||||
|
||||
Alternatives
|
||||
============
|
||||
|
||||
Datastore for host/topic management
|
||||
-----------------------------------
|
||||
|
||||
In above architecture, Consul is chosen because it supports healthcheck
|
||||
mechanism and multi DC federation so that these features enable user to
|
||||
monitor RPC server status easily, and scale out easily to support large
|
||||
scale OpenStack.
|
||||
|
||||
But alternatively, managing service endpoint using Keystone service
|
||||
catalog, DNS or registering data into Redis cluster similar to ZeroMQ
|
||||
are another choices.
|
||||
|
||||
Once the endpoint datastore supports selectable mechanism, these
|
||||
alternative datasotre plugin can be developped based on use-cases.
|
||||
|
||||
Cast Fanout support design
|
||||
--------------------------
|
||||
|
||||
In above diagram, broadcaster process is splitted out as one another
|
||||
service from RPC client. The Cast Fanout could be implemented into
|
||||
RPC client, but it takes much CPU and memory if number of targets are
|
||||
quite huge. Instead, by splitting out the broadcaster, the user easily
|
||||
scale out number of broadcaster processes based on number targets on
|
||||
OpenStack cluster.
|
||||
|
||||
API framework
|
||||
-------------
|
||||
|
||||
For API schema definition, any approaches are available.
|
||||
For example, there are REST, OpenAPI, json-rpc, gRPC or any http
|
||||
protocol implementation are available if it allow to integrated with
|
||||
eventlet which is used by oslo.service.
|
||||
In above diagram, OpenAPI is chosen in order to realize HTTP protocol
|
||||
client-server model.
|
||||
|
||||
|
||||
Advanced feature
|
||||
================
|
||||
|
||||
Multi Availability zones support
|
||||
--------------------------------
|
||||
* Consul provides Federation mechanism between multiple Consul clusters.
|
||||
* By using the Federation, Consul cluster can be splitted per OpenStack
|
||||
Availability zones.
|
||||
* Splitting Consul to multiple cluster reduces workload of Consul and makes
|
||||
the cluster stable.
|
||||
* Also in basic, RPC client and server communication happen in one Availability
|
||||
zone. This reduces network communication between multiple AZ.
|
||||
|
||||
.. code-block:: none
|
||||
|
||||
Availability Zone-A Availability Zone-B
|
||||
+----------------+ +----------------+
|
||||
| | RPC Request | |
|
||||
| RPC server | +------->+ RPC server |
|
||||
| | | | | | |
|
||||
| | | | | | |
|
||||
| RPC client +--------+ | RPC client |
|
||||
| | | | | |
|
||||
| | | Federation | | |
|
||||
| Consul Cluster +<--------------->+ Consul Cluster |
|
||||
| | | |
|
||||
+----------------+ +----------------+
|
||||
|
||||
|
||||
|
||||
Impact on Existing APIs
|
||||
-----------------------
|
||||
|
||||
None.
|
||||
|
||||
Security impact
|
||||
---------------
|
||||
|
||||
* OpenAPI supports SSL based communication between client and server.
|
||||
This enables secure RPC communication same as other messaging drivers.
|
||||
|
||||
Performance Impact
|
||||
------------------
|
||||
|
||||
Performance should become better because RPC client communicates to
|
||||
RPC server directly over HTTP.
|
||||
|
||||
Configuration Impact
|
||||
--------------------
|
||||
|
||||
New config options to configure http server are needed.
|
||||
Here is sample config parameters which is defined in oslo.messaging
|
||||
library user configuration file like nova.conf, neutron.conf.
|
||||
|
||||
.. code-block:: ini
|
||||
|
||||
[oslo_messaging_http]
|
||||
port_range=20000:20100
|
||||
|
||||
[consul]
|
||||
port = 8500
|
||||
host = <Consul hostname>
|
||||
token = <Consul token>
|
||||
check_timeout = 60s
|
||||
check_deregister = 30m
|
||||
|
||||
[http_driver]
|
||||
http_token = <Token to access OpenAPI server>
|
||||
http_api_config_file = <OpenAPI definition>
|
||||
enable_ssl = True
|
||||
ssl_certfile = <Path for SSL Cert file>
|
||||
ssl_keyfile = <Path for SSL Key file>
|
||||
listen_timeout = 60
|
||||
|
||||
|
||||
Detailed design for port_range
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
For port_range, following spec could reduce complexity of this option.
|
||||
|
||||
#. If the port_range is not defined, unused ephemeral port is chosen randomly.
|
||||
For most of use cases, this satisfies the requirement.
|
||||
#. If the port_range is defined, RPC server pick port from the specified range
|
||||
during service launch.
|
||||
|
||||
Here is the reason why multiple ports are required for RPC server.
|
||||
For example, Nova has "conductor.workers" config parameter and Neutron-server
|
||||
has "rpc_workers" config parameter. By specifying more than "1" for these configs,
|
||||
multiple RPC workers are launched on single host. Therefore, from RPC client-server
|
||||
communication perspective, each RPC worker must have individual destination port
|
||||
number to launch it on a host.
|
||||
|
||||
And purpose of port_range is to support secure cloud environment which strictly
|
||||
manages network communication using dest/source port at firewall, it would be better
|
||||
that default behavior is random choice from unused ephemeral port.
|
||||
|
||||
Developer Impact
|
||||
----------------
|
||||
|
||||
If other OpenStack component use http rpc_backend, ``oslo_messaging_http``,
|
||||
``consul`` and ``http_driver`` configuration section must be defined based
|
||||
on Consul and OpenAPI configuration.
|
||||
|
||||
Testing Impact
|
||||
--------------
|
||||
|
||||
Basic unit test and integration test (using DevStack) will be implemented.
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Primary assignee:
|
||||
Mitsuhiro Tanino (mitsuhiro-tanino)
|
||||
Other contributors:
|
||||
Xiang Wang
|
||||
|
||||
Yushiro Furukawa (y-furukawa-2)
|
||||
|
||||
Masahito Muroi (masahito-muroi)
|
||||
|
||||
Milestones
|
||||
----------
|
||||
|
||||
None
|
||||
|
||||
Work Items
|
||||
----------
|
||||
|
||||
- Propose driver and test codes
|
||||
- Release note and docs
|
||||
|
||||
Documentation Impact
|
||||
====================
|
||||
|
||||
The http driver documentation will be added to the library.
|
||||
This documentation will follow the style of documentation provided by the
|
||||
other drivers and should include the following topics:
|
||||
|
||||
* Overview of http driver architecture
|
||||
* pre-requisites
|
||||
* driver options overview
|
||||
* Consul configuration
|
||||
* HTTP broadcaster configuration
|
||||
|
||||
|
||||
Dependencies
|
||||
============
|
||||
|
||||
None
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
None
|
||||
|
||||
.. note::
|
||||
|
||||
This work is licensed under a Creative Commons Attribution 3.0
|
||||
Unported License.
|
||||
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||
Loading…
Reference in New Issue
Block a user