bfd29252f5
This patch modifies the way message markers are generated and used, such that Marconi can guarantee FIFO for a single message producer posting to a single queue (FIFO for multiple producers is not guaranteed.) At the same time, these changes guarantee that observer clients will recieve messages once-and-only once by removing race conditions inherent in timestamp-based markers. Along the way, some technical debt was also addressed, particulary regarding style, as well as some minor optimizations. A new service, marconi-mongo-gc, was also added, and is required when using the MongoDB storage driver for Marconi. This service was necessary due to the special requirement that at least the most recent message always remain in each queue, which is an a-priori assumption upon which the FIFO/pagination algorithm is based, and arises from the particular constraints imposed by MongoDB's semantics. Note: While implementing this blueprint, many calculated tradeoffs were made in an attempt to balance performance, risk, readability, and maintainability. The goal was to create a resonable baseline implementation that can be iterated upon pending comprehensive system and performance testing. Due to the many subtleties of solving the FIFO/once-and-only-once problem for the MongoDB driver, future contributors should excercise extreme caution when modifying the algorithm introduced in this patch. Changes include: * Align text in comments * Add counter to queue, messages * Markers are now converted over to using monotonic counter * Handle DuplicateKeyError * Return resources in body as a response to a message POST * Added mongo driver claims tests * Return 503 when no messages were enqueued due to marker conflict * Added backoff sleep between retries * Added marconi-mongo-gc service. This is a new required servi Implements: blueprint message-pagination Change-Id: Ifa0bb9e1bc393545adc4c804d14c6eb2df01848c
96 lines
2.9 KiB
Python
96 lines
2.9 KiB
Python
# Copyright (c) 2013 Red Hat, Inc.
|
|
#
|
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
# you may not use this file except in compliance with the License.
|
|
# You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing, software
|
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
|
# implied.
|
|
# See the License for the specific language governing permissions and
|
|
# limitations under the License.
|
|
|
|
|
|
class DoesNotExist(Exception):
|
|
pass
|
|
|
|
|
|
class NotPermitted(Exception):
|
|
pass
|
|
|
|
|
|
class Conflict(Exception):
|
|
"""Resource could not be created due to a conflict
|
|
with an existing resource.
|
|
"""
|
|
|
|
|
|
class MalformedID(ValueError):
|
|
"""ID was malformed."""
|
|
|
|
|
|
class MalformedMarker(ValueError):
|
|
"""Pagination marker was malformed."""
|
|
|
|
|
|
class MessageConflict(Conflict):
|
|
|
|
def __init__(self, queue, project, message_ids):
|
|
"""Initializes the error with contextual information.
|
|
|
|
:param queue: name of the queue to which the message was posted
|
|
:param project: name of the project to which the queue belongs
|
|
:param message_ids: list of IDs for messages successfully
|
|
posted. Note that these must be in the same order as the
|
|
list of messages originally submitted to be enqueued.
|
|
"""
|
|
msg = (_("Message could not be enqueued due to a conflict "
|
|
"with another message that is already in "
|
|
"queue %(queue)s for project %(project)s") %
|
|
dict(queue=queue, project=project))
|
|
|
|
super(MessageConflict, self).__init__(msg)
|
|
|
|
self._succeeded_ids = message_ids
|
|
|
|
@property
|
|
def succeeded_ids(self):
|
|
return self._succeeded_ids
|
|
|
|
|
|
class QueueDoesNotExist(DoesNotExist):
|
|
|
|
def __init__(self, name, project):
|
|
msg = (_("Queue %(name)s does not exist for project %(project)s") %
|
|
dict(name=name, project=project))
|
|
super(QueueDoesNotExist, self).__init__(msg)
|
|
|
|
|
|
class MessageDoesNotExist(DoesNotExist):
|
|
|
|
def __init__(self, mid, queue, project):
|
|
msg = (_("Message %(mid)s does not exist in "
|
|
"queue %(queue)s for project %(project)s") %
|
|
dict(mid=mid, queue=queue, project=project))
|
|
super(MessageDoesNotExist, self).__init__(msg)
|
|
|
|
|
|
class ClaimDoesNotExist(DoesNotExist):
|
|
|
|
def __init__(self, cid, queue, project):
|
|
msg = (_("Claim %(cid)s does not exist in "
|
|
"queue %(queue)s for project %(project)s") %
|
|
dict(cid=cid, queue=queue, project=project))
|
|
super(ClaimDoesNotExist, self).__init__(msg)
|
|
|
|
|
|
class ClaimNotPermitted(NotPermitted):
|
|
|
|
def __init__(self, mid, cid):
|
|
msg = (_("Message %(mid)s is not claimed by %(cid)s") %
|
|
dict(cid=cid, mid=mid))
|
|
super(ClaimNotPermitted, self).__init__(msg)
|