427 lines
15 KiB
Markdown
427 lines
15 KiB
Markdown
How you implemented your Python decorator is wrong
|
|
==================================================
|
|
|
|
The rest of the Python community is currently doing lots of navel gazing
|
|
over the issue of Python 3 adoption and the whole unicode/bytes divide. I
|
|
am so over that and gave up caring when my will to work on WSGI stuff was
|
|
leached away by how long it took to get the WSGI specification updated for
|
|
Python 3.
|
|
|
|
Instead my current favourite gripe these days is about how people implement
|
|
Python decorators. Unfortunately, it appears to be a favourite topic for
|
|
blogging by Python developers. It is like how when WSGI was all the rage
|
|
and everyone wanted to write their own WSGI server or framework. Now it is
|
|
like a rite of passage that one must blog about how to implement Python
|
|
decorators as a way of showing that you understand Python. As such, I get
|
|
lots of opportunity to grumble and wince. If only they did truly understand
|
|
what they were describing and what problems exist with the approach they
|
|
use.
|
|
|
|
So what is my gripe then. My gripe is that although one can write a very
|
|
simple decorator using a function closure, the scope it can be used in is
|
|
usually limited. The most basic pattern for implementing a Python decorator
|
|
also breaks various stuff related to introspection.
|
|
|
|
Now most people will say who cares, it does the job I want to do and I
|
|
don't have time to care whether it is correct in all situations.
|
|
|
|
As people will know from when I did care more about WSGI, I am a pedantic
|
|
arse though and when one does something, I like to see it done correctly.
|
|
|
|
Besides my overly obsessive personal trait, it actually does also affect me
|
|
in my day job as well. This is because I write tools which are dependent
|
|
upon being able to introspect into code and I need the results I get back
|
|
to be correct. If they aren't, then the data I generate becomes useless as
|
|
information can get grouped against the wrong thing.
|
|
|
|
As well as using introspection, I also do lots of evil stuff with monkey
|
|
patching. As it happens, monkey patching and the function wrappers one
|
|
applies aren't much different to decorators, it is just how they get
|
|
applied which is different. Because though monkey patching entails going in
|
|
and modifying other peoples code when they were not expecting it, or
|
|
designing for it, means that when you do go in and wrap a function that how
|
|
you do it is very important. If you do not do it correctly then you can
|
|
crash the users application or inadvertently change how it runs.
|
|
|
|
The first thing that is vitally important is preserving introspection for
|
|
the wrapped function. Another not so obvious thing though is that you need
|
|
to ensure that you do not mess with how the execution model for the Python
|
|
object model works.
|
|
|
|
Now I can in my own function wrappers that are used when performing monkey
|
|
patching ensure that these two requirements are met so as to ensure that
|
|
the function wrapper is transparent, but it can all fall in a heap when one
|
|
needs to monkey patch functions which already have other decorators
|
|
applied.
|
|
|
|
So when you implement a Python decorator and do it poorly it can affect me
|
|
and what I want to do. If I have to subsequently work around when you do it
|
|
wrong, I get somewhat annoyed and grumpy as more often than not that
|
|
entails a lot of pain.
|
|
|
|
To cover everything there is to know about what is wrong with your typical
|
|
Python decorators and wrapping of functions, plus how to fix it, will take
|
|
a lot of explaining, so one blog post isn't going to be enough. See this
|
|
blog post therefore as just part one of an extended discussion.
|
|
|
|
For this first instalment I will simply go through the various ways in
|
|
which your typical Python decorator can cause problems.
|
|
|
|
Basics of a Python decorator
|
|
----------------------------
|
|
|
|
Everyone should know what the Python decorator syntax is.
|
|
|
|
```
|
|
@function_wrapper
|
|
def function():
|
|
pass
|
|
```
|
|
|
|
The ``@`` annotation to denote the application of a decorator was only
|
|
added in Python 2.4. It is actually though only fancy syntactic sugar. It
|
|
is actually equivalent to writing:
|
|
|
|
```
|
|
def function():
|
|
pass
|
|
|
|
function = function_wrapper(function)
|
|
```
|
|
|
|
and what you would have done prior to Python 2.4.
|
|
|
|
The decorator syntax is therefore just a short hand way of being able to
|
|
apply a wrapper around an existing function, or otherwise modify the
|
|
existing function in place, while the definition of the function is being
|
|
setup.
|
|
|
|
What is referred to as monkey patching achieves pretty much the same
|
|
outcome, the difference being that when monkey patching the wrapper isn't
|
|
being applied at the time the definition of the function is being setup,
|
|
but is applied retrospectively from a different context after the fact.
|
|
|
|
Anatomy of a function wrapper
|
|
-----------------------------
|
|
|
|
Although I mentioned using function closures to implement a decorator, to
|
|
understand how the more generic case of a function wrapper works it is more
|
|
illustrative to show how to implement it using a class.
|
|
|
|
```
|
|
class function_wrapper(object):
|
|
def __init__(self, wrapped):
|
|
self.wrapped = wrapped
|
|
def __call__(self, *args, **kwargs):
|
|
return self.wrapped(*args, **kwargs)
|
|
|
|
@function_wrapper
|
|
def function():
|
|
pass
|
|
```
|
|
|
|
The class instance in this example is initialised with and records the
|
|
original function object. When the now wrapped function is called, it is
|
|
actually the ``__call__()`` method of the wrapper object which is invoked.
|
|
This in turn would then call the original wrapped function.
|
|
|
|
Simply passing through the call to the wrapper alone isn't particularly
|
|
useful, so normally you would actually want to do some work either before
|
|
or after the wrapped function is called. Or you may want to modify the
|
|
input arguments or the result as they pass through the wrapper. This is
|
|
just a matter of modifying the ``__call__()`` method appropriately to do
|
|
what you want.
|
|
|
|
Using a class to implement the wrapper for a decorator isn't actually that
|
|
popular. Instead a function closure is more often used. In this case a
|
|
nested function is used as the wrapper and it is that which is returned by
|
|
the decorator function. When the now wrapped function is called, the nested
|
|
function is actually being called. This in turn would again then call the
|
|
original wrapped function.
|
|
|
|
```
|
|
def function_wrapper(wrapped):
|
|
def _wrapper(*args, **kwargs):
|
|
return wrapped(*args, **kwargs)
|
|
return _wrapper
|
|
|
|
@function_wrapper
|
|
def function():
|
|
pass
|
|
```
|
|
|
|
In this example the nested function doesn't actually get passed the
|
|
original wrapped function explicitly. But it will still have access to it
|
|
via the arguments given to the outer function call. This does away with the
|
|
need to create a class to hold what was the wrapped function and thus why
|
|
it is convenient and generally more popular.
|
|
|
|
Introspecting a function
|
|
------------------------
|
|
|
|
Now when we talk about functions, we expect them to specify properties
|
|
which describe them as well as document what they do. These include the
|
|
``__name__`` and ``__doc__`` attributes. When we use a wrapper though, this
|
|
no longer works as we expect as in the case of using a function closure,
|
|
the details of the nested function are returned.
|
|
|
|
```
|
|
def function_wrapper(wrapped):
|
|
def _wrapper(*args, **kwargs):
|
|
return wrapped(*args, **kwargs)
|
|
return _wrapper
|
|
|
|
@function_wrapper
|
|
def function():
|
|
pass
|
|
|
|
>>> print(function.__name__)
|
|
_wrapper
|
|
```
|
|
|
|
If we use a class to implement the wrapper, as class instances do not
|
|
normally have a ``__name__`` attribute, attempting to access the name of
|
|
the function will actually result in an AttributeError exception.
|
|
|
|
```
|
|
class function_wrapper(object):
|
|
def __init__(self, wrapped):
|
|
self.wrapped = wrapped
|
|
def __call__(self, *args, **kwargs):
|
|
return self.wrapped(*args, **kwargs)
|
|
|
|
@function_wrapper
|
|
def function():
|
|
pass
|
|
|
|
>>> print(function.__name__)
|
|
Traceback (most recent call last):
|
|
File "<stdin>", line 1, in <module>
|
|
AttributeError: 'function_wrapper' object has no attribute '__name__'
|
|
```
|
|
|
|
The solution here when using a function closure is to copy the attributes
|
|
of interest from the wrapped function to the nested wrapper function. This
|
|
will then result in the function name and documentation strings being
|
|
correct.
|
|
|
|
```
|
|
def function_wrapper(wrapped):
|
|
def _wrapper(*args, **kwargs):
|
|
return wrapped(*args, **kwargs)
|
|
_wrapper.__name__ = wrapped.__name__
|
|
_wrapper.__doc__ = wrapped.__doc__
|
|
return _wrapper
|
|
|
|
@function_wrapper
|
|
def function():
|
|
pass
|
|
|
|
>>> print(function.__name__)
|
|
function
|
|
```
|
|
|
|
Needing to manually copy the attributes is laborious, and would need to be
|
|
updated if any further special attributes were added which needed to be
|
|
copied. For example, we should also copy the ``__module__`` attribute, and
|
|
in Python 3 the ``__qualname__`` and ``__annotations__`` attributes were
|
|
added. To aid in getting this right, the Python standard library provides
|
|
the ``functools.wraps()`` decorator which does this task for you.
|
|
|
|
```
|
|
import functools
|
|
|
|
def function_wrapper(wrapped):
|
|
@functools.wraps(wrapped)
|
|
def _wrapper(*args, **kwargs):
|
|
return wrapped(*args, **kwargs)
|
|
return _wrapper
|
|
|
|
@function_wrapper
|
|
def function():
|
|
pass
|
|
|
|
>>> print(function.__name__)
|
|
function
|
|
```
|
|
|
|
If using a class to implement the wrapper, instead of the
|
|
``functools.wraps()`` decorator, we would use the
|
|
``functools.update_wrapper()`` function.
|
|
|
|
```
|
|
import functools
|
|
|
|
class function_wrapper(object):
|
|
def __init__(self, wrapped):
|
|
self.wrapped = wrapped
|
|
functools.update_wrapper(self, wrapped)
|
|
def __call__(self, *args, **kwargs):
|
|
return self.wrapped(*args, **kwargs)
|
|
```
|
|
|
|
So we might have a solution to ensuring the function name and any
|
|
documentation string is correct in the form of ``functools.wraps()``, but
|
|
actually we don't and this will not always work as I will show below.
|
|
|
|
Now what if we want to query the argument specification for a function.
|
|
This also fails and instead of returning the argument specification for the
|
|
wrapped function, it returns that of the wrapper. In the case of using a
|
|
function closure, this is the nested function. The decorator is therefore
|
|
not signature preserving.
|
|
|
|
```
|
|
import inspect
|
|
|
|
def function_wrapper(wrapped): ...
|
|
|
|
@function_wrapper
|
|
def function(arg1, arg2): pass
|
|
|
|
>>> print(inspect.getargspec(function))
|
|
ArgSpec(args=[], varargs='args', keywords='kwargs', defaults=None)
|
|
```
|
|
|
|
A worse situation again occurs with the class based wrapper. This time we
|
|
get an exception complaining that the wrapped function isn't actually a
|
|
function. As a result it isn't possible to derive an argument specification
|
|
at all, even though the wrapped function is actually still callable.
|
|
|
|
```
|
|
class function_wrapper(object): ...
|
|
|
|
@function_wrapper
|
|
def function(arg1, arg2): pass
|
|
|
|
>>> print(inspect.getargspec(function))
|
|
Traceback (most recent call last):
|
|
File "...", line XXX, in <module>
|
|
print(inspect.getargspec(function))
|
|
File ".../inspect.py", line 813, in getargspec
|
|
raise TypeError('{!r} is not a Python function'.format(func))
|
|
TypeError: <__main__.function_wrapper object at 0x107e0ac90> is not a Python function
|
|
```
|
|
|
|
Another example of introspection one can do is to use
|
|
``inspect.getsource()`` to get back the source code related to a function.
|
|
This also will fail, with it giving the source code for the nested wrapper
|
|
function in the case of a function closure and again failing outright with
|
|
an exception in the case of the class based wrapper.
|
|
|
|
Wrapping class methods
|
|
----------------------
|
|
|
|
Now, as well as normal functions, decorators can also be applied to methods
|
|
of classes. Python even includes a couple of special decorators called
|
|
``@classmethod`` and ``@staticmethod`` for converting normal instance
|
|
methods into these special method types. Methods of classes do provide a
|
|
number of potential problems though.
|
|
|
|
```
|
|
class Class(object):
|
|
|
|
@function_wrapper
|
|
def method(self):
|
|
pass
|
|
|
|
@classmethod
|
|
def cmethod(cls):
|
|
pass
|
|
|
|
@staticmethod
|
|
def smethod():
|
|
pass
|
|
```
|
|
|
|
The first is that even if using ``functools.wraps()`` or
|
|
``functools.update_wrapper()`` in your decorator, when the decorator is
|
|
applied around ``@classmethod`` or ``@staticmethod``, it can fail with an
|
|
exception. This is because the wrappers created by these, do not have some
|
|
of the attributes being copied.
|
|
|
|
```
|
|
class Class(object):
|
|
@function_wrapper
|
|
@classmethod
|
|
def cmethod(cls):
|
|
pass
|
|
|
|
Traceback (most recent call last):
|
|
File "<stdin>", line 1, in <module>
|
|
File "<stdin>", line 3, in Class
|
|
File "<stdin>", line 2, in wrapper
|
|
File ".../functools.py", line 33, in update_wrapper
|
|
setattr(wrapper, attr, getattr(wrapped, attr))
|
|
AttributeError: 'classmethod' object has no attribute '__module__'
|
|
```
|
|
|
|
As it happens, this is a Python 2 bug and it is fixed in Python 3 by
|
|
ignoring missing attributes.
|
|
|
|
Even when we run it under Python 3, we still hit trouble though. This is
|
|
because both wrapper types assume that the wrapped function is directly
|
|
callable. This need not actually be the case. A wrapped function can
|
|
actually be what is called a descriptor, meaning that in order to get back
|
|
a callable, the descriptor has to be correctly bound to the instance first.
|
|
|
|
```
|
|
class Class(object):
|
|
@function_wrapper
|
|
@classmethod
|
|
def cmethod(cls):
|
|
pass
|
|
|
|
>>> Class.cmethod()
|
|
Traceback (most recent call last):
|
|
File "classmethod.py", line 15, in <module>
|
|
Class.cmethod()
|
|
File "classmethod.py", line 6, in _wrapper
|
|
return wrapped(*args, **kwargs)
|
|
TypeError: 'classmethod' object is not callable
|
|
```
|
|
|
|
Simple does not imply correctness
|
|
---------------------------------
|
|
|
|
So although the usual way that people implement decorators is simple, that
|
|
doesn't mean they are necessarily correct and will always work.
|
|
|
|
The issues highlighted so far are:
|
|
|
|
* Preservation of function ``__name__`` and ``__doc__``.
|
|
* Preservation of function argument specification.
|
|
* Preservation of ability to get function source code.
|
|
* Ability to apply decorators on top of other decorators that are implemented as descriptors.
|
|
|
|
The ``functools.wraps()`` function is given as a solution to the first but
|
|
doesn't always work, at least in Python 2. It doesn't help at all though
|
|
with preserving the introspection of a functions argument specification and
|
|
ability to get the source code for a function.
|
|
|
|
Even if one could solve the introspection problem, the simple decorator
|
|
implementation that is generally offered up as the way to do things, breaks
|
|
the execution model for the Python object model, not honouring the
|
|
descriptor protocol of anything which is wrapped by the decorator.
|
|
|
|
Third party packages do exist which try and solve these issues, such as the
|
|
decorator module available on PyPi. This module in particular though only
|
|
helps with the first two and still has potential issues with how it works
|
|
that may cause problems when trying to dynamically apply function wrappers
|
|
via monkey patching.
|
|
|
|
This doesn't mean these problems aren't solvable, and solvable in a way
|
|
that doesn't sacrifice performance. In my search at least, I could not
|
|
actually find any one who has described a comprehensive solution or offered
|
|
up a package which performs all the required magic so you don't have to
|
|
worry about it yourself.
|
|
|
|
This blog post is therefore the first step in me explaining how it can be
|
|
all made to work. I have stated the problems to be solved and in subsequent
|
|
posts I will explain how they can be solved and what extra capabilities
|
|
that gives you which enables the ability to write even more magic
|
|
decorators than what is possible now with traditional ways that decorators
|
|
have been implemented.
|
|
|
|
So stay tuned for the next instalment. Hopefully I can keep the momentum up
|
|
and keep them coming. Pester me if I don't.
|