Message Queuing API Reference API v1 Message Queuing 2014-02-20 Copyright details are filled in by the build system. This document is intended for software developers interested in developing applications using the Message Queuing Application Programming Interface (API). 2014-02-20 Initial document for OpenStack incubation. Message Queuing is an open source, scalable, and highly available message and notifications service. Message Queuing supports a variety of messaging patterns. Users of this service can create and manage a producer-consumer or a publisher-subscriber model () from one simple API. Unlimited queues and messages give Message Queuing users the flexibility they need to create powerful web applications in the cloud. Message Queuing consists of a few basic components: queues, messages, claims, and statistics. In the producer-consumer model, users create queues in which producers, or servers, can post messages. Workers, or consumers, can then claim those messages and delete them after they complete the actions associated with the messages. A single claim can contain multiple messages, and administrators can query claims for status. In the publisher-subscriber model, messages are posted to a queue as in the producer-consumer model, but messages are never claimed. Instead, subscribers, or watchers, send GET requests to pull all messages that have been posted since their last request. In this model, a message remains in the queue, unclaimed, until the message's time to live (TTL) has expired. In both of these models, administrators can get queue statistics that display the most recent and oldest messages, the number of unclaimed messages, and more.

This document is intended for software developers who are interested in developing applications that use the Message Queuing API. It assumes that you have the following skills: RESTful web services HTTP/1.1 conventions JSON data serialization formats

This version of the document replaces and obsoletes all earlier versions. The most recent changes are described in the following table.

To use the Message Queueing API effectively, you should understand the key concepts described in this section.

Following is an overview of how Message Queuing works. For definitions of Message Queuing terms, see the . You create a queue to which producers or publishers post messages. Workers (consumers or subscribers) claim or get a message from the queue, complete the work in that message, and delete the message. If a worker will be off-line before it completes the work in a message, the worker can retire the claim's time to live (TTL), putting the message back into the queue for another worker to claim. Subscribers monitor the claims from these queues to track activity and help troubleshoot errors. For the majority of use cases, Message Queuing is not responsible for the ordering of messages. However, if there is only a single producer, Message Queuing ensures that messages are handled in a First In, First Out (FIFO) order.

The Message Queuing API supports a variety of messaging patterns including the following: producer-consumer publish-subscribe

The producer-consumer pattern has the following characteristics: A producer is programmed to send messages to a queue. Multiple workers (or consumers) are programmed to monitor a queue. Only one worker can claim a message so that no other worker can claim the message and duplicate the work. The worker must delete the message when work is done. TTL restores a message to an unclaimed state if the worker never finishes. This pattern is ideal for dispatching jobs to multiple processors.

Characteristics of the Publish-Subscribe pattern are: The publisher sends a message to a queue. All workers (or subscribers) listen to the messages in the queue. Workers do not claim a message. Multiple workers can take action on a message. Workers can send a marker or cursor to skip messages already seen. TTL eventually deletes messages. This pattern is ideal for notification of events to multiple workers at once.

Queuing systems are used to coordinate tasks within an application. Here are some examples: Backup: A backup application might use a queuing system to connect the actions that users do in the a control panel to the customer's backup agent on a server. When a customer wants to start a backup, they simply choose "start backup" on a panel. Doing so causes the producer to put a "startBackup" message into the queue. Every few minutes, the agent on the customers server (the worker) checks the queue to see if it has any new messages to act on. The agent claims the "startBackup" message and kicks off the backup on the customer's server. Storage: Gathering statistics for a large, distributed storage system can be a long process. The storage system can use a queuing system to ensure that jobs complete, even if one initially fails. Since messages are not deleted until after the worker has completed the job, the storage system can make sure that no job goes undone. If the worker fails to complete the job, the message stays in the queue to be completed by another server. In this case, a worker claims a message to perform a statistics job, but the claim's TTL expired and the message is put back into the queue when the job took too long to complete (meaning that it most likely failed). By giving the claim a TTL, applications can protect themselves from workers going off-line while processing a message. After a claim's TTL expires, the message is put back into the queue for another worker to claim. Email: The team for an email application is constantly migrating customer email from old versions to newer ones, so they develop a tool to let customers do it themselves. The migrations take a long time, so they cannot be done with single API calls, or by a single server. When a user starts a migration job from their portal, the migration tool sends messages to the queue with details of how to run the migration. A set of migration engines, the consumers in this case, periodically check the queues for new migration tasks, claim the messages, perform the migration, and update a database with the migration details. This process allows a set of servers to work together to accomplish large migrations in a timely manner. Following are some generic use cases for Message Queuing: Distribute tasks among multiple workers (transactional job queues) Forward events to data collectors (transactional event queues) Publish events to any number of subscribers (publish-subscribe) Send commands to one or more agents (point-to-point or publish-subscribe) Request an action or get information from a Remote Procedure Call (RPC) agent

The Message Queuing API is implemented using a RESTful web service interface. Like other OpenStack projects, Message Queuing shares a common token-based authentication system that allows seamless access between products and services. All requests to authenticate against and operate the service are performed using SSL over HTTP (HTTPS) on TCP port 443.

You can use cURL to try the authentication process in two steps: get a token, and send the token to a service. Get an authentication token by providing your user name and either your API key or your password. Here are examples of both approaches: You can request a token by providing your user name and your password. $ curl -X POST https://localhost:5000/v2.0/tokens -d '{"auth":{"passwordCredentials":{"username": "joecool", "password":"coolword"}, "tenantId":"5"}}' -H 'Content-type: application/json' Successful authentication returns a token which you can use as evidence that your identity has already been authenticated. To use the token, pass it to other services as an X-Auth-Token header. Authentication also returns a service catalog, listing the endpoints you can use for Cloud services. Use the authentication token to send a GET to a service you would like to use. Authentication tokens are typically valid for 24 hours. Applications should be designed to re-authenticate after receiving a 401 (Unauthorized) response from a service endpoint. If you programmatically parse an authentication response, be aware that service names are stable for the life of the particular service and can be used as keys. You should also be aware that a user's service catalog can include multiple uniquely-named services that perform similar functions.

The Message Queuing API supports JSON data serialization formats. You specify the request format by using the Content-Type header. The request format is required for calls that have a request body. You can specify the response format in requests either by using the Accept header or by adding a .json extension to the request URI. JSON is also used for the response format.

Response Format

Format	Accept Header	Query Extension	Default
JSON	application/json	.json	Yes

Accounts are configured with default thresholds, or limits, that manage capacity and prevent abuse of the system. The following table provides details about the absolute limits for the Message Queuing API. These limits are configurable.

Absolute Limits

Type	Description	Limit	Default
Limits on Messages and Queues per Request	Number of queue records per page of results when listing queues	1 - 1000	20
	Number of messages per page of results when listing messages	1 - 25	20
	Number of messages that can be posted in a single request	1 - 25	20
	Number of messages that can be claimed at one time	1 - 25	20
	Number of messages that can be deleted in a single bulk delete request	1 - 25	20
	Number of messages that can be requested in a bulk get by ID	1 - 25	20
Time Limits	Valid range for a message TTL	60 - 1209600 seconds (1 minute to 14 days)	1209600 seconds
	Valid range for a claim TTL	60 - 43200 seconds (1 minute to 12 hours)	43200 seconds
	Valid range for claim grace period	60 - 43200 seconds (1 minute to 12 hours)	43200 seconds
Data Limits	Queue metadata JSON	Cannot exceed 262144 bytes, including whitespace (256 KB)	65536 bytes
	Message body JSON	Cannot exceed 262144 bytes, including whitespace (256 KB)	262144 bytes

For the 256 KB data limit for the message body JSON, if multiple messages are included in the request, this limit also applies to the sum of the message bodies added together. For example, you may have one message body of 256 KB, or up to 10 messages for which the sum of the message bodies is 256 KB. A 400 error is triggered if a single message body or the sum of multiple message bodies exceed 256 KB in a single request.

Each request to the Message Queuing API must include certain standard and extended HTTP headers. These headers provide host, agent, authentication, and other pertinent information to the server. The following table provides the common headers used by the API.

Common Headers

Header	Description
Host	Host name of the API
Date	Current date and time
Accept	Media type desired; initially, only application/json is supported. Note: The Accept header is required.
Accept-Encoding	Specifies that the agent accepts gzip-encoded response bodies
Content-Type	application/json
Content-Length	For POST or PUT requests, the length in bytes of the message document being submitted
X-Auth-Token	Auth token
X-Project-Id	An ID for a project to which the value of X-Auth-Token grants access. Queues are created under this project.
Client-ID	A UUID for each client instance. The UUID must be submitted in its canonical form (for example, 3381af92-2b9e-11e3-b191-71861300734c). The client generates the Client-ID once. Client-ID persists between restarts of the client so the client should reuse that same Client-ID. Note: All message-related operations require the use of Client-ID in the headers to ensure that messages are not echoed back to the client that posted them, unless the client explicitly requests this.

For the display and consumption of date and time values, the Message Queuing service uses a date format that complies with ISO 8601. yyyy-MM-dd'T'HH:mm:ss.SSSZ For example, May 19, 2013 at 8:07:08 A.M., GMT-5 would have the following format: 2013-05-19T08:07:08-05:00 The following table describes the date and time format codes.

Explanation of Date and Time Format Codes

Code	Description
yyyy	Four-digit year
MM	Two-digit month
dd	Two-digit day of at the month
T	Separator for date and time
HH	Two-digit hour of the day (00-23)
mm	Two-digit minutes of the hour
ss	Two-digit seconds of the minute
SSS	Three-digit milliseconds of the second
Z	RFC 822 time zone

If any Message Queuing request results in an error, the queuing service returns an appropriate 4xx or 5xx HTTP status code, and the following information in the body: Title Description Internal code Link to more information An example of an error message follows. HTTP/1.1 400 Bad Request Content-Type: application/json { "title": "Unsupported limit", "description": "The given limit cannot be negative, and cannot be greater than 50.", "code": 1092, "link": { "rel": "help", "href": "http://docs.example.com/messages#limit", "text": "API documentation for the limit parameter" } } Detailed error information for each API operation is included in the following sections. Additionally, brief error information is included in each operation description in .

This section includes response codes and explanations for each of the Message Queuing API operations. The operations are described in .

The following table describes the response codes for the Create Queue operation.

Descriptions of Create Queue Response Codes

HTTP Response Code	Description
201	Success (created)
204	Success (already exists)
400	Queue has a long name (greater than 64 bytes)
400	Request header has missing fields
401	Request header has invalid auth token
429	Too many requests

The following table describes the response codes for the List Queues operation.

Descriptions of List Queues Response Codes

HTTP Response Code	Description
200	Success
200	URI has invalid parameter (something other than limit, marker, or metadata)
204	Request with non-existing value for marker
400	Request with negative value for limit
400	Request with zero value for limit
400	Request with non-boolean value for metadata
400	Header has missing fields
401	Request header has invalid auth token
404	Request a non-existing queue
406	Header has Accept != "application/json"
429	Too many requests

The following table describes the response codes for the Delete Queue operation.

Descriptions of Delete Queue Response Codes

HTTP Response Code	Description
200	URI has invalid parameters
204	Success
204	Delete a non-existing queue
400	Header has missing fields
401	Request header has invalid auth token
406	Header has Accept != "application/json"
429	Too many requests

The following table describes the response codes for the Check Queue Existence operation.

Descriptions of Check Queue Existence Response Codes

HTTP Response Code	Description
204	Check a non-existing queue
404	Request a non-existing queue

The following table describes the response codes for the Set Queue Metadata operation.

Descriptions of Set Queue Metadata Response Codes

HTTP Response Code	Description
204	Success
400	Request body is empty
400	Request body is greater than 64 KB
400	Request body has malformed JSON
400	Request with non-JSON body
406	Request header has Accept != "application/json"
429	Too many requests
	Request with UTF-16 char JSON body

The following table describes the response codes for the Get Queue Metadata operation.

Descriptions of Get Queue Metadata Response Codes

HTTP Response Code	Description
200	Success
200	No metadata exists for the queue
200	URI has invalid parameters
400	Header has missing fields
401	Request header has invalid auth token
404	Request metadata for a non-existing queue
406	Header has Accept != "application/json"
429	Too many requests

The following table describes the response codes for the Get Queue Stats operation.

Descriptions of Get Queue Stats Response Codes

HTTP Response Code	Description
200	Success
200	URI has invalid parameters
400	Header has missing fields
401	Request header has invalid auth token
404	Request stats for a non-existing queue
406	Header has Accept != "application/json"
429	Too many requests

The following table describes the response codes for the Post Message operation.

Descriptions of Post Message Response Codes

HTTP Response Code	Description
200	URI has invalid parameters
201	Success
201	Post multiple messages in a single request, few of them succeed (with "partial": true)
400	Post greater than 100 messages with a single request
400	Post with request body greater than 4 KB
400	Post with invalid JSON request body
400	Post with non-JSON request body
400	Post with non-JSON message
400	Post with TTL less than 60
400	Post with TTL greater than 1209600
400	Post with non-integer value of TTL
400	Post with negative value of TTL
400	Post without TTL in request body
400	Post without "body" in request body
400	Post with no request body
400	Post with messages encapsulated in multiple arrays
400	Header has missing fields
401	Header has invalid auth token
404	Post message to a non-existing queue
406	Header has Accept != "application/json"
429	Too many requests
503	Post multiple messages in a single request and all of them fail

The following table describes the response codes for the Get Messages operation.

Descriptions of Get Messages Response Codes

HTTP Response Code	Description
200	Success (query matched 1 or more messages)
200	Request message with invalid parameter (something other than marker, limit, or echo)
204	Success (query has no match)
204	Request message with non-existing marker
400	Request message with limit greater than 100
400	Post with request body greater than 4 KB
400	Request message with limit less than or equal to 0
400	Request message with non-boolean value for echo
401	Header has invalid auth token
404	Request message from a non-existing queue
406	Header has Accept != "application/json"
429	Too many requests

The following table describes the response codes for the Get a Set of Messages by ID operation.

Descriptions of Get a Set of Messages by ID Response Codes

HTTP Response Code	Description
200	Success
200	URI has invalid parameters
400	Request message with limit greater than 100
401	Post with request body greater than 4 KB
404	Request message with limit less than or equal to 0
404	Request message with non-boolean value for echo
406	Header has invalid auth token
429	Too many requests

The following table describes the response codes for the Delete a Set of Messages by ID operation.

Descriptions of Delete a Set of Messages by ID Response Codes

HTTP Response Code	Description
204	Success
204	Delete message from a non-existing queue
204	Delete a non-existing message
400	Header has missing fields
401	Header has invalid auth token
403	Delete with expired claim ID
403	Delete with non-existing claim ID
403	Delete claimed message without providing a claim ID
406	Header has Accept != "application/json"
429	Too many requests

The following table describes the response codes for the Get a Specific Message operation.

Descriptions of Get a Specific Message Response Codes

HTTP Response Code	Description
200	Success(query matched the message)
200	URI has invalid parameters
400	Header has missing fields
401	Header has invalid auth token
404	Request message from a non-existing queue
404	Request a non-existing message
404	Request an expired message
406	Header has Accept != "application/json"
429	Too many requests

The following table describes the response codes for the Delete Message operation.

Descriptions of Delete Message Response Codes

HTTP Response Code	Description
200	URI has invalid parameters
204	Success
204	Delete message from a non-existing queue
204	Delete a non-existing message
400	Header has missing fields
401	Header has invalid auth token
403	Delete with expired claim ID
403	Delete with non-existing claim ID
403	Delete claimed message without providing a claim ID
406	Header has Accept != "application/json"
429	Too many requests

The following table describes the response codes for the Claim Messages operation.

Descriptions of Claim Messages Response Codes

HTTP Response Code	Description
201	Success
204	Success (empty queue with no messages to claim)
204	Claim message from a non-existing queue
400	Claim message with invalid value for limit (non-integer, or less than or equal to 0)
400	Claim message with limit greater than 100)
400	Claim message with no request body
400	Claim message with no TTL field
400	Claim message with invalid TTL value (non-integer, or less than or equal to 0)
400	Claim message with invalid JSON
400	Claim message with non-JSON request body
400	Claim message with missing header fields
401	Claim message with invalid auth token
404	Item not found
406	Header has Accept != "application/json"
429	Too many requests

The following table describes the response codes for the Query Claim operation.

Descriptions of Query Claim Response Codes

HTTP Response Code	Description
200	Success
200	Query claim with invalid URI parameters
400	Query claim with missing header fields
401	Query claim with invalid auth token
404	Query expired claim
404	Query claim from a non-existing queue
404	Query claim with non-existing claim ID
406	Header has Accept != "application/json"
429	Too many requests

The following table describes the response codes for the Update Claim operation.

Descriptions of Update Claim Response Codes

HTTP Response Code	Description
200	Update claim with invalid URI parameters
204	Success
400	Update claim with no request body
400	Update claim with invalid JSON in request body
400	Update claim with non-JSON request body
400	Update claim with missing header fields
401	Update claim with invalid auth token
404	Update expired claim
404	Update claim on a non-existing queue
404	Update a non-existing claim
406	Update claim with content-type != application/json-patch
429	Too many requests

The following table describes the response codes for the Release Claim operation.

Descriptions of Release Claim Response Codes

HTTP Response Code	Description
200	Release claim with invalid URI parameters
204	Success
204	Release expired claim
204	Release a non-existing claim
400	Release claim with missing header fields
401	Release claim with invalid auth token
404	Release claim from a non-existing queue
406	Header has Accept != "application/json"
429	Too many requests

This chapter describes each of the Message Queuing operations, which are summarized in the following table.

The section describes how to get the home document for Message Queuing. The home document schema is currently not ratified in OpenStack and may change.

This section describes queue operations that are supported by the Message Queuing API.

This section describes message operations that are supported by the Message Queuing API. All message-related operations require that Client-ID be included in the headers. This is to ensure that messages are not echoed back to the client that posted them unless the client explicitly requests this. For more information, see Client-ID in .

This section describes claim operations that are supported by the Message Queuing API.

Claim The process of a worker checking out a message to perform a task. Claiming a message prevents other workers from attempting to process the same messages. Claim TTL Defines how long a message will be in claimed state. A message can be claimed by one worker at a time. Consumer A server that claims messages from the queue. Message A task, a notification, or any meaningful data that a producer or publisher sends to the queue. A message exists until it is deleted by a recipient or automatically by the system based on a TTL (time-to-live) value. Message TTL Defines how long a message will be accessible. Producer A server or application that sends messages to the queue. Producer - Consumer A pattern where each worker application that reads the queue has to claim the message in order to prevent duplicate processing. Later, when work is done, the worker is responsible for deleting the message. If message is not deleted in a predefined time, it can be claimed by other workers. Publisher A server or application that posts messages to the queue with the intent to distribute information or updates to multiple subscribers. Publisher - Subscriber A pattern where all worker applications have access to all messages in the queue. Workers cannot delete or update messages. Queue The entity that holds messages. Ideally, a queue is created per work type. For example, if you want to compress files, you would create a queue dedicated to this job. Any application that reads from this queue would only compress files. Subscriber An observer that watches messages like an RSS feed but does not claim any messages. TTL Time-to-live value. Worker A client that claims messages from the queue and performs actions based on those messages.