diff --git a/docs/examples.rst b/docs/examples.rst index 136e5af..332ab99 100644 --- a/docs/examples.rst +++ b/docs/examples.rst @@ -1,2 +1,242 @@ -Code Examples -============= +Examples +======== + +At this time the **wrapt** module does not provide any bundled decorators, +only the one decorator for creating other decorators. This document +provides various examples of decorators often described elsewhere, to +exhibit what can be done with decorators using the **wrapt** module, for +the purpose of comparison. + +Synchronization +--------------- + +Synchronization decorators are a simplified way of adding thread locking to +functions, methods, instances of classes or a class type. They work by +associating a thread mutex with a specific context and when a function is +called the lock is acquired prior to the call and then released once the +function returns. + +The simplist example of a decorator for synchronization is one where the +lock is explicitly provided when the decorator is applied to a function. By +being supplied explicitly, it is up to the user of the decorator to +determine what context the lock applies to. For example, a lock may be +applied to a single function, a group of functions, or a class. + +As the lock needs to be supplied when the decorator is applied to the +function we need to use a function closure as a means of supplying the +argument to the decorator. + +:: + + def synchronized(lock): + @wrapt.decorator + def _wrapper(wrapped, instance, args, kwargs): + with lock: + return wrapped(*args, **kwargs) + return _wrapper + + import threading + + _lock = threading.RLock() + + @synchronized(_lock) + def function(): + pass + + class Class(object): + + @synchronized(_lock): + def function(self): + pass + +Note that the recursive lock ``threading.RLock`` is used to ensure that +recursive calls, or calls to another synchronized function associated with +the same lock, doesn't cause a deadlock. + +An alternative to requiring the lock be supplied when the decorator is +applied to a function, is to associate a lock automatically with the +wrapped function. That is, rather than require the lock be passed in +explicitly, create one on demand and attach it to the wrapped function. + +:: + + @wrapt.decorator + def synchronized(wrapped, instance, args, kwargs): + # Retrieve the lock from the wrapped function. + + lock = vars(wrapped).get('_synchronized_lock', None): + + if lock is None: + # There was no lock yet associated with the function so we + # create one and associate it with the wrapped function. + # We use ``dict.setdefault()`` as a means of ensuring that + # only one thread gets to set the lock if multiple threads + # do this at the same time. This may mean redundant lock + # instances will get thrown away if there is a race to set + # it, but all threads would still get back the same one lock. + + lock = vars(wrapped).setdefault('_synchronized_lock', + threading.RLock()) + + with lock: + return wrapped(*args, **kwargs) + + @synchronized + def function(): + pass + +This avoids the need for an instance of a lock to be passed in explicitly +when the decorator is being applied to a function, but it now means that +all decorated methods of a class will have a distinct lock. + +A more severe issue in both these approaches is that locks on each method +work across all instances of the class where as what we really want is a +lock per instance of a class for all methods of the class decorated with +the ``@synchronized`` decorator. + +To address this, we can use the fact that the decorator wrapper function +is passed the ``instance`` and so can determine when the function is being +invoked on an instance of a class and that it is not a normal function +call. In this case we can associate the lock with the instance instead. + +:: + + @wrapt.decorator + def synchronized(wrapped, instance, args, kwargs): + # Use the instance as the context if function was bound. + + if instance is not None: + context = vars(instance) + else: + context = vars(wrapped) + + # Retrieve the lock for the specific context. + + lock = context.get('_synchronized_lock', None): + + if lock is None: + # There was no lock yet associated with the function so we + # create one and associate it with the wrapped function. + # We use ``dict.setdefault()`` as a means of ensuring that + # only one thread gets to set the lock if multiple threads + # do this at the same time. This may mean redundant lock + # instances will get thrown away if there is a race to set + # it, but all threads would still get back the same one lock. + + lock = context.setdefault('_synchronized_lock', + threading.RLock()) + + with lock: + return wrapped(*args, **kwargs) + + @synchronized + def function(): + pass + +Now we actually have two scenarios that match for where ``instance`` is not +``None``. One will be where an instance method is being called on a class, +which is what we are targeting in this case. We will also have ``instance`` +being a value other than ``None`` for the case where a class method is +called. For this case ``instance`` will be a reference to the class type. + +Having the lock being associated with the class type for class methods is +entirely reasonable, but a problem presents. That is that +``vars(instance)`` where ``instance`` is a class type, actually returns a +``dictproxy`` and not a ``dict``. As a ``dictproxy`` is effectively read +only, it is not possible to associate the lock with it. + +A similar problem also occurs where ``instance`` is ``None`` but ``wrapped`` +is a class type. That is, if the decorator was applied to a class. The result +is that the above technique will not work in these two cases. + +The only way that it is possible to add attributes to a class type is to use +``setattr``, either explicitly or via direct attribute assignment. Although +this allows us to add attributes to a class, there is no equivalent to +``dict.setdefault()``, so we loose the ability to add the attribute which will +hold the lock atomically. + +To get around this problem, we need to use an intermediary meta lock which +gates the attempt to associate a lock with a specific context. This meta +lock itself still needs to be created somehow, so what we do now is use +the ``dict.setdefault()`` trick against the decorator itself and use it as +the place to store the meta lock. + +:: + + @wrapt.decorator + def synchronized(wrapped, instance, args, kwargs): + # Use the instance as the context if function was bound. + + if instance is not None: + context = instance + else: + context = wrapped + + # Retrieve the lock for the specific context. + + lock = getattr(context, '_synchronized_lock', None) + + if lock is None: + # There is no existing lock defined for the context we + # are dealing with so we need to create one. This needs + # to be done in a way to guarantee there is only one + # created, even if multiple threads try and create it at + # the same time. We can't always use the setdefault() + # method on the __dict__ for the context. This is the + # case where the context is a class, as __dict__ is + # actually a dictproxy. What we therefore do is use a + # meta lock on this wrapper itself, to control the + # creation and assignment of the lock attribute against + # the context. + + meta_lock = vars(synchronized).setdefault( + '_synchronized_meta_lock', threading.Lock()) + + with meta_lock: + # We need to check again for whether the lock we want + # exists in case two threads were trying to create it + # at the same time and were competing to create the + # meta lock. + + lock = getattr(context, '_synchronized_lock', None) + + if lock is None: + lock = threading.RLock() + setattr(context, '_synchronized_lock', lock) + + with lock: + return wrapped(*args, **kwargs) + + @synchronized # lock bound to function1 + def function1(): + pass + + @synchronized # lock bound to function2 + def function2(): + pass + + @synchronized # lock bound to Class + class Class(object): + + @synchronized # lock bound to instance of Class + def function_im(self): + pass + + @synchronized # lock bound to Class + @classmethod + def function_cm(cls): + pass + + @synchronized # lock bound to function_sm + @staticmethod + def function_sm(): + pass + +This means lock creation is all automatic, with an appropriate lock created +for the different contexts the decorator is used in. + +Specifically, when the decorator is used on a normal function or static +method, a unique lock will be associated with each function. For the case +of instance methods, the lock will be against the instance. Finally, for +class methods and a decorator against an actual class, the lock will be +against the class type. \ No newline at end of file diff --git a/docs/index.rst b/docs/index.rst index 54c78cc..086f787 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -30,10 +30,10 @@ Contents quick-start decorators + examples benchmarks - known-issues - unit-testing + testing + issues -.. proxies .. wrappers -.. examples +.. proxies diff --git a/docs/known-issues.rst b/docs/issues.rst similarity index 97% rename from docs/known-issues.rst rename to docs/issues.rst index 1a66aae..aeaeb8f 100644 --- a/docs/known-issues.rst +++ b/docs/issues.rst @@ -1,5 +1,5 @@ -Known Issues -============ +Issues +====== The following known issues exist. diff --git a/docs/proxies.rst b/docs/proxies.rst index e11229c..a4af7d8 100644 --- a/docs/proxies.rst +++ b/docs/proxies.rst @@ -1,2 +1,2 @@ -Object Proxies -============== +Proxies +======= diff --git a/docs/unit-testing.rst b/docs/testing.rst similarity index 97% rename from docs/unit-testing.rst rename to docs/testing.rst index e4b35f1..d55a563 100644 --- a/docs/unit-testing.rst +++ b/docs/testing.rst @@ -1,5 +1,5 @@ -Unit Testing ------------- +Testing +------- Unit tests are located in the ``tests`` directory. diff --git a/docs/wrappers.rst b/docs/wrappers.rst index 661671c..8586a73 100644 --- a/docs/wrappers.rst +++ b/docs/wrappers.rst @@ -1,2 +1,2 @@ -Function Wrappers -================= +Wrappers +========