diff --git a/specs/juno/amqp10-driver-implementation.rst b/specs/juno/amqp10-driver-implementation.rst new file mode 100644 index 0000000..eeed61e --- /dev/null +++ b/specs/juno/amqp10-driver-implementation.rst @@ -0,0 +1,398 @@ +==================================================== +Provide a AMQP 1.0 implementation for oslo.messaging +==================================================== + +Include the URL of your launchpad blueprint: + +https://blueprints.launchpad.net/oslo.messaging/+spec/amqp10-driver-implementation + +We already have rabbit and qpid drivers for earlier (and different!) +versions of AMQP, the proposal would be to add an additional driver +for a _protocol_ not a particular broker. + +Problem description +=================== + +The purpose of allowing for different drivers is to give users +choice. However every driver supported increases the maintenance +burden. + +By targeting a clear mapping on to a protocol, rather than a specific +implementation, we would simplify the task in the future for anyone +wishing to move to any other system that spoke AMQP 1.0[1]. That would no +longer require a new driver, merely different configuration and +deployment. That would then allow openstack to more easily take +advantage of any emerging innovations in this space. + +As an example, this would allow the use of the Dispatch Router from +the Apache Qpid project to be used. This has been designed as a +distributed router rather than a single-process, store-and-forward +broker and offers simple scalability and redundancy. + +Proposed change +=============== + +A new driver will be added to oslo.messaging (a patch for this has +been available for review at https://review.openstack.org/75815). + +The mapping outlined below considers the inter-mediated case +(i.e. where the driver connects to a message broker or network of +brokers). AMQP 1.0 would allow communication (in part or as a whole) +to be conducted directly between peers, without the use of +intermediaries. Development of this aspect however is considered a future +extension (see Appendix A for some more detailed discussion). + +The aim is to support as many of the existing 1.0 enabled +intermediaries as we can without overly distorting the use of the AMQP +1.0 protocol. + +In AMQP 1.0, addressable entities within a broker (or similar) are +called 'nodes' (e.g. queues or topics). Nodes are message 'sinks' or +'sources', to which messages are then typically sent by establishing a +sending or receiving 'link' to or from the given node. AMQP 1.0 does +not provide any standard configuration mechanism for configuring such +nodes on the fly (with the exception of dynamic reply queues). However +by allowing deployment specific prefixes to the addresses specified in +the source and target for receiving and sending links respectively, it +should be possible to get the required functionality working with a +variety of implementations and offer considerable flexibility to +deployers. + +In RPC, there are three patterns for invocations, and a different +address format will be used for each. + +(A) For invocations on a specific server the address will be formed by +concatenating the topic name and the server name and will be prefixed +by the exchange if specified and/or by a deployment specific +'server_request_prefix' string. The default value for this prefix is +'exclusive'. + +(B) For invocations on one of a group of servers the address will be +the topic name and will be prefixed by the exchange if specified +and/or by a deployment specific 'group_request_prefix' string. The +default value for this prefix is 'unicast'. + +(C) For invocations on all of a group of servers the address will be +the topic name, prefixed by the exchange if specified and also +prefixed with a configurable 'broadcast_prefix' string. The default +value of this prefix is '/broadcast/'. + +Each pattern of communication is therefore identifiable through a +customisable pattern of address. This allows either the broker (or +equivalent) to be configured to match what the driver is using, or it +allows the driver to be configured to take account of built-in (and +non-configurable) conventions for a given broker. + +So for example, Qpid Dispatch Router could be configured to recognise +three patterns based on the default prefixes. Any message for an +address starting with 'unicast' (or 'exclusive') would be routed to +only one subscriber for that address. Any message for an address +starting with 'broadcast' would be sent to all subscribers for that +address. [Some more detail on configuration is provided in Appendix B +for Qpid Dispatch Router and Appendix C for qpidd.] + +Some other broker might not be configurable in this way, but might +have built in conventions around address patterns. E.g. all address of +the form '/queues/foo' would be treated as queues and messages sent to +them would be allocated to only one consumer. Addresses of the form +'/topics/bar' (or '/exchanges/bar') would be considered pub-sub topics +and the message would be distributed to all subscribers. + +For each send request, the appropriate address will be deduced from +the specified target. If the server value on the target is specified, +it implies an invocation on a specific server and the address format +described above in (A) is used. If the server is not specified then +the fanout flag is considered. If that is specified it implies an +invocation on the entire group of servers identified by the topic and +the address format described in (C) is used. If neither the server +nor the fanout flag is specified it implies an invocation on one of +the group of servers identified by the topic and the address format +described in (B) is used. + +A cache of sender links per target will be maintained to avoid the +overhead of recreating them on each request. (The alternative of using +a single sending link and setting the 'to' field is also a +possibility, but not all intermediaries might support that so it would +be an unnecessary limitation on choice). If there is no sender link +yet for the specified address one will be created and added to the +cache. The cache will be cleared on reconnect. + +A single connection and session will be used for all the sender +links. If a timeout is specified for the request, this will be set as +a ttl on the request. The request will also have the 'to' field set to +the value used in the target for the link. This accommodates any +intermediary that may expect or prefer that means of addressing. + +There will be a single receiver link per driver instance for replies, +this will have the dynamic flag set on the source. Each request will +have the reply-to address set to the address to which this listener +is subscribed. Each request will have a message-id set. This will be +echoed back in the correlation-id of any response message allowing it +to be correlated back to the waiting send request. + +A server will subscribe for messages by creating three receiving +links: one to subscribe to requests specific to the server, one to +subscribe to requests for one of the servers group and one to +subscribe to broadcasts for all the servers in the group (i.e. one for +each form of address described in A, B and C above). The first two +receiver links will have the distribution mode set to move, the third +will have the distribution mode set to copy. (Note: the distribution +mode is defined by the AMQP 1.0 specification to determine if/how +message are distributed between multiple subscribers). + +Requests and responses for RPC will be sent pre-settled, meaning that +they are not explicitly acknowledged and therefore may be lost on +failover. Ideally notifications would be acknowledged both when +published and when received by a listener. (Note: at present the +rabbit driver acknowledges notifications when delivered, but published +notifications are not aknowledged. For the current qpid driver +notifications are not acknowledged either on publication or delivery). + +Alternatives +------------ + +The rabbit driver is the only one that appears to be adequately +maintained at present. The qpid driver suffers from an impedance +mismatch between the API offered by qpid.messaging and that used for +the rabbit driver. The attempt to use a common architecture for these +two quite different APIs has led to the qpid driver being hard to +understand and inefficient in its mapping to AMQP 0-10. + +One alternative to adding a new driver is to only support the rabbit +driver. That reduces the maintenance burden and gives clarity to +users. It does however restrict choice. + +Another alternative would be to choose a different protocol as the +basis of the new driver. MQTT is focused on a pub-sub pattern and +doesn't incorporate competing consumers as is required for the current +olso.messaging semantics. STOMP doesn't define interoperable +mechanisms for request-response. Since two of the existing drivers use +earlier versions of AMQP, AMQP supports all the patterns needed and +AMQP is an open standard (now standardised under ISO) it seems a +fairly obvious candidate. + +Though this driver is not being suggested as a replacement for existing +drivers (merely an alternative), it does offer a path to greater +consolidation as it could also accommodate the non-intermediated style +of communication embraced by ZeroMQ. Perhaps even more powerful would +be a hybrid approach, where again the use of a common, standard +protocol would be advantageous. + +Impact on Existing APIs +----------------------- + +None + +Security impact +--------------- + +The security implications are identical to that of the other +drivers. SSL will be supported as an option to secure the messaging +traffic where desired. + +Performance Impact +------------------ + +There would be no impact whatsoever unless the driver is selected for +use. The overall performance will depend on which server components +are selected for use with the driver (unlike the existing drivers, +selection of this driver doesn't restrict the choice to a single +broker implementation). + +Initial experience with the code[2] indicate that this driver in +conjunction with either qpidd or Qpid Dispatch Router compares very +favourably with existing drivers. + + +Configuration Impact +-------------------- + +The driver would be selected via the existing transport_url +option. The 'amqp' scheme in the url is used to select the AMQP +protocol. + +There would be further configuration options to tailor the behaviour +of the driver to different deployments if desired, such as the various +prefix options described above. The defaults have been selected to +make broker configuration simple. In general configuration of the +messaging infrastructure is preferred to configuration of the driver +as it allows for more transparent, centralised changes. + +Developer Impact +---------------- + +Any future changes to the driver API would need to be reflected in +another driver if this one were added. + +Implementation +============== + +Assignee(s) +----------- + +Primary assignee: + kgiusti@redhat.com + +Other contributors: + gsim@redhat.com + fpercoco@redhat.com + +Milestones +---------- + +Target Milestone for completion: + Juno-1 + +Work Items +---------- + +Code review; the driver has been available for review via Gerritt for +some time. There is quite a lot of code in this that is in fact very +generic, and would be applicable beyond the use in olso.messaging. It +has been suggested (requested?) that this should be moved out of the +driver into its own library. There is in fact already a library that +contains this generic code, pyngus, and one task would be to update +the driver patch to rely on that as a dependency. + +Automated testing should be set up. I have submitted a patch for +functional tests of the oslo.messaging API (which I used to test the +driver during development). In addition any relevant tests from +Tempest could be run using this driver and an appropriately configured +backend. + +I have tested against both qpidd and Qpid dispatch router. Based on my +experience with other applications, I believe the driver would work +against the current ActiveMQ release (as well as ApolloMQ). For +RabbitMQ, the lack of support for the 'dynamic' flag on link source is +the main issue at present (which could be worked around if +desired). In theory it should also work against Microsofts ServiceBus +and IITs SwiftMQ as they support the aspects of the protocol +used. Those might require much more manual configuration at present, I +haven't used them so am unable to say for sure. + +Allowing for reliable delivery of notifications by having publication +and delivery to consumers acknowledged. + +Incubation +========== + +Though this does not add a new (public) module, it would be prudent to +allow some time for the new driver to mature. Having some sort of +'beta' phase would be good, where the driver could be selected for +testing by interested parties and feedback provided. + +Adoption +-------- + +The driver should be usable by any service using oslo.messaging + +Library +------- + +oslo.messaging + +Anticipated API Stabilization +----------------------------- + +There is no API impact. Stabilization of the new driver option itself +will take some time. + +Documentation Impact +==================== + +The availability of an alternative option would at some point need to +be documented. + +Dependencies +============ + +This driver depends on the qpid-proton python library for AMQP 1.0 +support. Pyngus may be added as dependency, see Work Items above. + +Appendix A: Some discussion of direct communication +=================================================== + +Supporting direct communication, where messages do not go through an +application layer intermediary user process, requires that +communicants accept incoming connections and can determine the correct +hostname and port to connect to. + +Though this can be done by convention, e.g. using the server name as +the hostname and agreeing well-known ports, that can be restrictive +and cumbersome. A better approach is to have a registry of servers and +the host and ports they listen on. + +This registry is much like the matchmaker used for 0MQ and the choices +applicable there would also be applicable to this driver. + +However since there is already support for communication through n +intermediary, that can be used to dynamically distribute the data for +the registry to all communicants. This keeps configuration simple and +allows the system to adapt to changes. + +Different schemes would be possible. One example would be to add a +configuration option that caused servers to start listenting on a +particular port. They would then advertise this fact by attaching a +property to any reply sent back for a request that identifies them +directly. The RPC clients could then cache these alternate addresses +and use them for any subsequent requests to the same server. In this +approach the communicants use the messaging intermediaries to locate +each other, but having done so they then 'offload' further +communication to a direct connection to reduce the load on the +intermediaries. + +Another approach would be to have a special 'matchmaker' topic that +clients would subscribe to and servers would announce themselves +over. This would allow direct comunication even from the first request +to a given server. + +These and indeed other schemes could be easily accomplished in a backward +compatible manner. + +Appendix B: Configuring Qpid Dispatch Router +============================================ + +Using the default prefixes, the following address configuration, if +added to the configuration file for Dispatch Router, would setup the +required semantics for openstack:: + + fixed-address { + prefix: /unicast + fanout: single + bias: closest + } + + fixed-address { + prefix: /exclusive + fanout: single + bias: closest + } + + fixed-address { + prefix: /broadcast + fanout: multiple + } + + +Appendix C: Configuring Qpidd +============================= + +Using the default prefixes, passing the following options to qpidd +(0.28 or later) would setup the required semantics for openstack:: + + --queue-patterns exclusive --queue-patterns unicast --topic-patterns broadcast + + +References +========== + +[1] http://docs.oasis-open.org/amqp/core/v1.0/amqp-core-complete-v1.0.pdf + +[2] http://people.apache.org/~gsim/oslo.messaging_scalability.pdf + +.. note:: + + This work is licensed under a Creative Commons Attribution 3.0 + Unported License. + http://creativecommons.org/licenses/by/3.0/legalcode +