327 lines
13 KiB
Markdown
327 lines
13 KiB
Markdown
Maintaining decorator state using a class
|
|
=========================================
|
|
|
|
This is the sixth post in my series of blog posts about Python decorators
|
|
and how I believe they are generally poorly implemented. It follows on from
|
|
the previous post titled [Decorators which accept
|
|
arguments](05-decorators-which-accept-arguments.md), with the very first
|
|
post in the series being [How you implemented your Python decorator is
|
|
wrong](01-how-you-implemented-your-python-decorator-is-wrong.md).
|
|
|
|
In the previous post I described how to implement decorators which accept
|
|
arguments. This covered mandatory arguments, but also how to have a
|
|
decorator optionally accept arguments. I also touched on how one can
|
|
maintain state between invocations of the decorator wrapper function for a
|
|
specific wrapped function.
|
|
|
|
One of the approaches described for maintaining state was to implement the
|
|
decorator as a class. Using this approach though resulted in an unexpected
|
|
error.
|
|
|
|
This post will explore the source of the error when attempting to implement
|
|
our decorator as a class using our new decorator factory and function
|
|
wrapper, and then see if any other issues crop up.
|
|
|
|
Single decorator for functions and methods
|
|
------------------------------------------
|
|
|
|
As described in the previous post, the pattern we were trying to use so as
|
|
to allow us to use a class as a decorator was:
|
|
|
|
```
|
|
class with_arguments(object):
|
|
|
|
def __init__(self, arg):
|
|
self.arg = arg
|
|
|
|
@decorator
|
|
def __call__(self, wrapped, instance, args, kwargs):
|
|
return wrapped(*args, **kwargs)
|
|
|
|
@with_arguments(arg=1)
|
|
def function():
|
|
pass
|
|
```
|
|
|
|
The intent here is that the application of the decorator, with arguments
|
|
supplied, would result in an instance of the class being created. In the
|
|
next phase where that is called with the wrapped function, the
|
|
``__call__()`` method with ``@decorator`` applied will be used as a
|
|
decorator on the function to be wrapped. The end result should be that the
|
|
``__call__()`` method of the class instance created ends up being our
|
|
wrapper function.
|
|
|
|
When the decorated function is now called, the ``__call__()`` method of the
|
|
class would be called with it in turn calling the wrapped function. As the
|
|
``__call__()`` method at that point is bound to an instance of the class, it
|
|
would have access to the state that it contained.
|
|
|
|
When we tried this though we got, at the time that the decorator was being
|
|
applied, the error:
|
|
|
|
```
|
|
Traceback (most recent call last):
|
|
File "test.py", line 483, in <module>
|
|
@with_arguments(1)
|
|
TypeError: _decorator() takes exactly 1 argument (2 given)
|
|
```
|
|
|
|
The ``_decorator()`` function in this case is the inner function from our
|
|
decorator factory.
|
|
|
|
```
|
|
def decorator(wrapper):
|
|
@functools.wraps(wrapper)
|
|
def _decorator(wrapped):
|
|
return function_wrapper(wrapped, wrapper)
|
|
return _decorator
|
|
```
|
|
|
|
The mistake that has been made here is that we are using a function closure
|
|
to implement our decorator factory, yet we were expecting it to work on
|
|
both normal functions and methods of classes.
|
|
|
|
The reason this will not work is due to the binding that occurs when a
|
|
method of a class is accessed. This process was described in a previous
|
|
post in this series and is the result of the descriptor protocol being
|
|
applied. This binding results in the reference to the instance of the class
|
|
being automatically passed as the first argument to the method.
|
|
|
|
Now as the ``_decorator()`` function was acting as a wrapper for the method
|
|
call, and because ``_decorator()`` was not defined so as to accept both
|
|
``self`` and ``wrapped`` as arguments, the call would fail.
|
|
|
|
We could create a special variant of the decorator factory to be used just
|
|
on instance methods, but that goes against the specific complaint expressed
|
|
earlier in regard to how people create multiple variants of decorators for
|
|
use on normal functions and instance methods.
|
|
|
|
To resolve this issue, what we can do is use our function wrapper for the
|
|
decorator returned by the decorator factory, instead of a function closure.
|
|
|
|
```
|
|
def decorator(wrapper):
|
|
def _wrapper(wrapped, instance, args, kwargs):
|
|
def _execute(wrapped):
|
|
return function_wrapper(wrapped, wrapper)
|
|
return _execute(*args, **kwargs)
|
|
return function_wrapper(wrapper, _wrapper)
|
|
```
|
|
|
|
Explicit binding of methods required
|
|
------------------------------------
|
|
|
|
This above change now means we do not have to worry about whether
|
|
``@decorator`` is being applied to a normal function, instance method or
|
|
even a class method. This is because in all cases, any reference to the
|
|
instance being bound to is never passed through in ``args``. Thus any
|
|
wrapper function doesn't need to worry about the distinction.
|
|
|
|
Trying again with this change though, we are confronted with a further
|
|
problem. This time at the point that the wrapped function is called.
|
|
|
|
```
|
|
>>> function()
|
|
Traceback (most recent call last):
|
|
File "<stdin>", line 1, in <module>
|
|
File "test.py", line 243, in __call__
|
|
return self.wrapper(self.wrapped, None, args, kwargs)
|
|
TypeError: __call__() takes exactly 5 arguments (4 given)
|
|
```
|
|
|
|
The issue this time is that when ``@decorator`` is applied to the
|
|
``__call__()`` method, the reference it is passed is that of the unbound
|
|
method. This is because this occurs during the processing of the class
|
|
definition, long before any instance of the class has been created.
|
|
|
|
Normally the reference to the instance would be supplied later when the
|
|
method is bound, but because our decorator is actually a factory there are
|
|
two layers involved. The target instance is available to the upper factory
|
|
as the 'instance' argument, but that isn't being used in any way when the
|
|
inner function wrapper object is being created which associates the
|
|
function to be wrapped with our wrapper function.
|
|
|
|
To solve this problem we need for the case where we are being bound to an
|
|
instance, to explicitly bind the wrapper function ourselves against the
|
|
instance.
|
|
|
|
```
|
|
def decorator(wrapper):
|
|
def _wrapper(wrapped, instance, args, kwargs):
|
|
def _execute(wrapped):
|
|
if instance is None:
|
|
return function_wrapper(wrapped, wrapper)
|
|
elif inspect.isclass(instance):
|
|
return function_wrapper(wrapped, wrapper.__get__(None, instance))
|
|
else:
|
|
return function_wrapper(wrapped, wrapper.__get__(instance, type(instance)))
|
|
return _execute(*args, **kwargs)
|
|
return function_wrapper(wrapper, _wrapper)
|
|
```
|
|
|
|
So what we are using here is the feature of our function wrapper that
|
|
allows us to implement a universal decorator. That is, one which can change
|
|
its behaviour dependent upon the context it is used in.
|
|
|
|
In this case we had three cases we needed to deal with.
|
|
|
|
The first is where the instance was ``None``. This corresponds to a normal
|
|
function, a static method or where the decorator was applied to a class
|
|
type.
|
|
|
|
The second is where the instance was not ``None``, but where it referred to a
|
|
class type. This corresponds to a class method. In this case we need to
|
|
bind the wrapper function to the class type by calling the ``__get__()``
|
|
method of the wrapper function explicitly.
|
|
|
|
The third and final case is where the instance was not ``None``, but where
|
|
it was not referring to a class type. This corresponds to an instance
|
|
method. In this case we again need to bind the wrapper function, this time
|
|
to the instance.
|
|
|
|
With these changes, we are now all done with addressing this issue, and to
|
|
a large degree with filling out our new decorator pattern.
|
|
|
|
Do not try and reproduce this
|
|
-----------------------------
|
|
|
|
So the complete solution we now have at this point is:
|
|
|
|
```
|
|
class object_proxy(object):
|
|
|
|
def __init__(self, wrapped):
|
|
self.wrapped = wrapped
|
|
try:
|
|
self.__name__ = wrapped.__name__
|
|
except AttributeError:
|
|
pass
|
|
|
|
@property
|
|
def __class__(self):
|
|
return self.wrapped.__class__
|
|
|
|
def __getattr__(self, name):
|
|
return getattr(self.wrapped, name)
|
|
|
|
class bound_function_wrapper(object_proxy):
|
|
|
|
def __init__(self, wrapped, instance, wrapper, binding, parent):
|
|
super(bound_function_wrapper, self).__init__(wrapped)
|
|
self.instance = instance
|
|
self.wrapper = wrapper
|
|
self.binding = binding
|
|
self.parent = parent
|
|
|
|
def __call__(self, *args, **kwargs):
|
|
if self.binding == 'function':
|
|
if self.instance is None:
|
|
instance, args = args[0], args[1:]
|
|
wrapped = functools.partial(self.wrapped, instance)
|
|
return self.wrapper(wrapped, instance, args, kwargs)
|
|
else:
|
|
return self.wrapper(self.wrapped, self.instance, args, kwargs)
|
|
else:
|
|
instance = getattr(self.wrapped, '__self__', None)
|
|
return self.wrapper(self.wrapped, instance, args, kwargs)
|
|
|
|
def __get__(self, instance, owner):
|
|
if self.instance is None and self.binding == 'function':
|
|
descriptor = self.parent.wrapped.__get__(instance, owner)
|
|
return bound_function_wrapper(descriptor, instance, self.wrapper,
|
|
self.binding, self.parent)
|
|
return self
|
|
|
|
class function_wrapper(object_proxy):
|
|
|
|
def __init__(self, wrapped, wrapper):
|
|
super(function_wrapper, self).__init__(wrapped)
|
|
self.wrapper = wrapper
|
|
if isinstance(wrapped, classmethod):
|
|
self.binding = 'classmethod'
|
|
elif isinstance(wrapped, staticmethod):
|
|
self.binding = 'staticmethod'
|
|
else:
|
|
self.binding = 'function'
|
|
|
|
def __get__(self, instance, owner):
|
|
wrapped = self.wrapped.__get__(instance, owner)
|
|
return bound_function_wrapper(wrapped, instance, self.wrapper,
|
|
self.binding, self)
|
|
|
|
def __call__(self, *args, **kwargs):
|
|
return self.wrapper(self.wrapped, None, args, kwargs)
|
|
|
|
def decorator(wrapper):
|
|
def _wrapper(wrapped, instance, args, kwargs):
|
|
def _execute(wrapped):
|
|
if instance is None:
|
|
return function_wrapper(wrapped, wrapper)
|
|
elif inspect.isclass(instance):
|
|
return function_wrapper(wrapped, wrapper.__get__(None, instance))
|
|
else:
|
|
return function_wrapper(wrapped, wrapper.__get__(instance, type(instance)))
|
|
return _execute(*args, **kwargs)
|
|
return function_wrapper(wrapper, _wrapper)
|
|
```
|
|
|
|
Take heed though of what was said in prior posts though. The object proxy
|
|
implementation given here is not a complete solution. As a result, do not
|
|
take this code and try and use it yourself as is. If you do you will find
|
|
that some aspects of performing introspection on the wrapped function will
|
|
not work as indicated they should.
|
|
|
|
In particular, access to the function ``__doc__`` string will always yield
|
|
``None``. Various attributes such as ``__qualname__`` in Python 3 and
|
|
``__module__`` are not propagated either.
|
|
|
|
Handling an attribute such as ``__doc__`` string correctly is actually a
|
|
bit of a pain. This is because you cannot use a ``__doc__`` property in an
|
|
object proxy base class that returns the value from the wrapped function
|
|
and have it then work when you derive another class from it. This is
|
|
because the separate ``__doc__`` string attribute from the derived class,
|
|
even if no documentation string were specified in the derived class, will
|
|
override that of the base class.
|
|
|
|
So the object proxy as shown here was intended to be illustrative only of
|
|
what was required.
|
|
|
|
In some respects all the code here is meant to be illustrative only. It is
|
|
not here to say use this code but to show you the general path to
|
|
implementing a more robust decorator implementation. It is to provide a
|
|
narrative from which you can learn. If you were expecting a one line TLDR
|
|
summary on how to do it, then you can forget it, things just aren't that
|
|
simple.
|
|
|
|
Introducing the wrapt decorator module
|
|
--------------------------------------
|
|
|
|
If I am telling you not to use this code, what are you supposed to do then?
|
|
|
|
The answer to that already exists in the form of the wrapt module on PyPi.
|
|
|
|
The wrapt package has been available for a number of months already, but
|
|
isn't widely known of at this point. It implements all of what is described
|
|
but also more. The module has a complete implementation of the object proxy
|
|
required to make all this work correctly. The module also provides a range
|
|
of other features related to the decorator factory as well as other
|
|
separate features related to monkey patching in general.
|
|
|
|
Although I am now finally pointing out that this module exists, I will not
|
|
be stopping the blog posts at this point as there is a range of topics I
|
|
still want to cover. These include examples of how a universal decorator
|
|
can be used, enabling/disabling of decorators, performance issues, the
|
|
remaining parts of the implementation of the object proxy, monkey patching
|
|
and much more.
|
|
|
|
In the next post in this series I will look at one specific example of
|
|
using a universal decorator by posing the question of if Python decorators
|
|
are so wonderful, why does Python not provide a ``@synchronized`` decorator?
|
|
|
|
Such a decorator was held up as a bit of a poster child as to what could be
|
|
done with decorators when they were first introduced to the language, yet
|
|
all the implementations I could find are half baked and not very practical
|
|
in the real world. I believe that a universal decorator can help here and
|
|
we can actually have a usable ``@synchronized`` decorator. I will therefore
|
|
explore that possibility in the next post.
|