147

Suppose I have written a decorator that does something very generic. For example, it might convert all arguments to a specific type, perform logging, implement memoization, etc.

Here is an example:

def args_as_ints(f): def g(*args, **kwargs): args = [int(x) for x in args] kwargs = dict((k, int(v)) for k, v in kwargs.items()) return f(*args, **kwargs) return g @args_as_ints def funny_function(x, y, z=3): """Computes x*y + 2*z""" return x*y + 2*z >>> funny_function("3", 4.0, z="5") 22 

Everything well so far. There is one problem, however. The decorated function does not retain the documentation of the original function:

>>> help(funny_function) Help on function g in module __main__: g(*args, **kwargs) 

Fortunately, there is a workaround:

def args_as_ints(f): def g(*args, **kwargs): args = [int(x) for x in args] kwargs = dict((k, int(v)) for k, v in kwargs.items()) return f(*args, **kwargs) g.__name__ = f.__name__ g.__doc__ = f.__doc__ return g @args_as_ints def funny_function(x, y, z=3): """Computes x*y + 2*z""" return x*y + 2*z 

This time, the function name and documentation are correct:

>>> help(funny_function) Help on function funny_function in module __main__: funny_function(*args, **kwargs) Computes x*y + 2*z 

But there is still a problem: the function signature is wrong. The information "*args, **kwargs" is next to useless.

What to do? I can think of two simple but flawed workarounds:

1 -- Include the correct signature in the docstring:

def funny_function(x, y, z=3): """funny_function(x, y, z=3) -- computes x*y + 2*z""" return x*y + 2*z 

This is bad because of the duplication. The signature will still not be shown properly in automatically generated documentation. It's easy to update the function and forget about changing the docstring, or to make a typo. [And yes, I'm aware of the fact that the docstring already duplicates the function body. Please ignore this; funny_function is just a random example.]

2 -- Not use a decorator, or use a special-purpose decorator for every specific signature:

def funny_functions_decorator(f): def g(x, y, z=3): return f(int(x), int(y), z=int(z)) g.__name__ = f.__name__ g.__doc__ = f.__doc__ return g 

This works fine for a set of functions that have identical signature, but it's useless in general. As I said in the beginning, I want to be able to use decorators entirely generically.

I'm looking for a solution that is fully general, and automatic.

So the question is: is there a way to edit the decorated function signature after it has been created?

Otherwise, can I write a decorator that extracts the function signature and uses that information instead of "*kwargs, **kwargs" when constructing the decorated function? How do I extract that information? How should I construct the decorated function -- with exec?

Any other approaches?

1
  • 1
    Never said "out of date". I was more or less wondering what inspect.Signature added to dealing with decorated functions. Commented Jul 23, 2015 at 1:36

9 Answers 9

108
  1. Install decorator module:

    $ pip install decorator 
  2. Adapt definition of args_as_ints():

    import decorator @decorator.decorator def args_as_ints(f, *args, **kwargs): args = [int(x) for x in args] kwargs = dict((k, int(v)) for k, v in kwargs.items()) return f(*args, **kwargs) @args_as_ints def funny_function(x, y, z=3): """Computes x*y + 2*z""" return x*y + 2*z print funny_function("3", 4.0, z="5") # 22 help(funny_function) # Help on function funny_function in module __main__: # # funny_function(x, y, z=3) # Computes x*y + 2*z 

Python 3.4+

functools.wraps() from stdlib preserves signatures since Python 3.4:

import functools def args_as_ints(func): @functools.wraps(func) def wrapper(*args, **kwargs): args = [int(x) for x in args] kwargs = dict((k, int(v)) for k, v in kwargs.items()) return func(*args, **kwargs) return wrapper @args_as_ints def funny_function(x, y, z=3): """Computes x*y + 2*z""" return x*y + 2*z print(funny_function("3", 4.0, z="5")) # 22 help(funny_function) # Help on function funny_function in module __main__: # # funny_function(x, y, z=3) # Computes x*y + 2*z 

functools.wraps() is available at least since Python 2.5 but it does not preserve the signature there:

help(funny_function) # Help on function funny_function in module __main__: # # funny_function(*args, **kwargs) # Computes x*y + 2*z 

Notice: *args, **kwargs instead of x, y, z=3.

Sign up to request clarification or add additional context in comments.

9 Comments

Yours wasn't the first answer, but the most comprehensive so far :-) I would actually prefer a solution not involving a third party module, but looking at the source for the decorator module, it's simple enough that I'll be able to just copy it.
@MarkLodato: functools.wraps() already preserves signatures in Python 3.4+ (as said in the answer). Do you mean setting wrapper.__signature__ helps on earlier versions? (which versions have you tested?)
@MarkLodato: help() shows the correct signature on Python 3.4. Why do you think functools.wraps() is broken and not IPython?
@MarkLodato: it is broken if we have to write code to fix it. Given that help() produces the correct result, the question is what piece of software should be fixed: functools.wraps() or IPython? In any case, manually assigning __signature__ is a workaround at best -- it is not a long-term solution.
Looks like inspect.getfullargspec() still doesn't return proper signature for functools.wraps in python 3.4 and that you must use inspect.signature() instead.
|
34
+25

This is solved with Python's standard library functools and specifically functools.wraps function, which is designed to "update a wrapper function to look like the wrapped function". It's behaviour depends on Python version, however, as shown below. Applied to the example from the question, the code would look like:

from functools import wraps def args_as_ints(f): @wraps(f) def g(*args, **kwargs): args = [int(x) for x in args] kwargs = dict((k, int(v)) for k, v in kwargs.items()) return f(*args, **kwargs) return g @args_as_ints def funny_function(x, y, z=3): """Computes x*y + 2*z""" return x*y + 2*z 

When executed in Python 3, this would produce the following:

>>> funny_function("3", 4.0, z="5") 22 >>> help(funny_function) Help on function funny_function in module __main__: funny_function(x, y, z=3) Computes x*y + 2*z 

Its only drawback is that in Python 2 however, it doesn't update function's argument list. When executed in Python 2, it will produce:

>>> help(funny_function) Help on function funny_function in module __main__: funny_function(*args, **kwargs) Computes x*y + 2*z 

2 Comments

Not sure if it's Sphinx, but this doesn't seem to work when the wrapped function is a method of a class. Sphinx continues to report the call signature of the decorator.
functools.wraps is incomplete. inspect.getfullargspec(func) still returns the signature of the decorator function instead of the wrapped function.
9

There is a decorator module with decorator decorator you can use:

@decorator def args_as_ints(f, *args, **kwargs): args = [int(x) for x in args] kwargs = dict((k, int(v)) for k, v in kwargs.items()) return f(*args, **kwargs) 

Then the signature and help of the method is preserved:

>>> help(funny_function) Help on function funny_function in module __main__: funny_function(x, y, z=3) Computes x*y + 2*z 

EDIT: J. F. Sebastian pointed out that I didn't modify args_as_ints function -- it is fixed now.

Comments

8

Take a look at the decorator module - specifically the decorator decorator, which solves this problem.

Comments

6

Second option:

  1. Install wrapt module:

$ easy_install wrapt

wrapt have a bonus, preserve class signature.

 import wrapt import inspect

@wrapt.decorator def args_as_ints(wrapped, instance, args, kwargs): if instance is None: if inspect.isclass(wrapped): # Decorator was applied to a class. return wrapped(*args, **kwargs) else: # Decorator was applied to a function or staticmethod. return wrapped(*args, **kwargs) else: if inspect.isclass(instance): # Decorator was applied to a classmethod. return wrapped(*args, **kwargs) else: # Decorator was applied to an instancemethod. return wrapped(*args, **kwargs) @args_as_ints def funny_function(x, y, z=3): """Computes x*y + 2*z""" return x * y + 2 * z >>> funny_function(3, 4, z=5)) # 22 >>> help(funny_function) Help on function funny_function in module __main__: funny_function(x, y, z=3) Computes x*y + 2*z

Comments

4

As commented above in jfs's answer ; if you're concerned with signature in terms of appearance (help, and inspect.signature), then using functools.wraps is perfectly fine.

If you're concerned with signature in terms of behavior (in particular TypeError in case of arguments mismatch), functools.wraps does not preserve it. You should rather use decorator for that, or my generalization of its core engine, named makefun.

from makefun import wraps def args_as_ints(func): @wraps(func) def wrapper(*args, **kwargs): print("wrapper executes") args = [int(x) for x in args] kwargs = dict((k, int(v)) for k, v in kwargs.items()) return func(*args, **kwargs) return wrapper @args_as_ints def funny_function(x, y, z=3): """Computes x*y + 2*z""" return x*y + 2*z print(funny_function("3", 4.0, z="5")) # wrapper executes # 22 help(funny_function) # Help on function funny_function in module __main__: # # funny_function(x, y, z=3) # Computes x*y + 2*z funny_function(0) # observe: no "wrapper executes" is printed! (with functools it would) # TypeError: funny_function() takes at least 2 arguments (1 given) 

See also this post about functools.wraps.

1 Comment

Also, the result of inspect.getfullargspec is not kept by calling functools.wraps.
3
from inspect import signature def args_as_ints(f): def g(*args, **kwargs): args = [int(x) for x in args] kwargs = dict((k, int(v)) for k, v in kwargs.items()) return f(*args, **kwargs) sig = signature(f) g.__signature__ = sig g.__doc__ = f.__doc__ g.__annotations__ = f.__annotations__ g.__name__ = f.__name__ return g @args_as_ints def funny_function(x, y, z=3): """Computes x*y + 2*z""" return x*y + 2*z >>> funny_function("3", 4.0, z="5") 22 

I wanted to add that answer (since this shows up first in google). The inspect module is able to fetch the signature of a function, so that it can be preserved in decorators. But that's not all. If you want to modify the signature, you can do so like this :

from inspect import signature, Parameter, _ParameterKind def foo(a: int, b: int) -> int: return a + b sig = signature(foo) sig._parameters = dict(sig.parameters) sig.parameters['c'] = Parameter( 'c', _ParameterKind.POSITIONAL_OR_KEYWORD, annotation=int ) foo.__signature__ = sig >>> help(foo) Help on function foo in module __main__: foo(a: int, b: int, c: int) -> int 

Why would you want to mutate a function's signature ?

It's mostly useful to have adequate documentation on your functions and methods. If you're using the *args, **kwargs syntax and then popping arguments from kwargs for other uses in your decorators, that keyword argument won't be properly documented, hence, modifying the signature of the function.

Comments

1
def args_as_ints(f): def g(*args, **kwargs): args = [int(x) for x in args] kwargs = dict((k, int(v)) for k, v in kwargs.items()) return f(*args, **kwargs) g.__name__ = f.__name__ g.__doc__ = f.__doc__ return g 

this fixes name and documentation. to preserve the function signature, wrap is used exactly at same location as g.__name__ = f.__name__, g.__doc__ = f.__doc__.

the wraps itself a decorator. we pass the closure-the inner function to that decorator, and it is going to fix up the metadata. BUt if we only pass in the inner function to wraps, it is not gonna know where to copy the metadata from. It needs to know which function's metadata needs to be protected. It needs to know the original function.

def args_as_ints(f): def g(*args, **kwargs): args = [int(x) for x in args] kwargs = dict((k, int(v)) for k, v in kwargs.items()) return f(*args, **kwargs) g=wraps(f)(g) return g 

wraps(f) is going to return a function which will take g as its parameter. And that is going to return closure and will assigned to g and then we return it.

Comments

0

As already answered functools.wraps is your way to go. This sometimes has downside that in IDE type-checkers/linters that use typeshed the type-hint in your IDE of even a simple function changes, changes in a not so nicely readable _Wrapped type. You need to cast or overwrite that type hint again, e.g. with the signature of a decorator that that is used alongside.

@wraps(int) def foo(x: str): return int(x) reveal_type(foo) # foo: _Wrapped[(x: ConvertibleToInt = ..., /), int, (x: str), int] 

The longer the function signature the uglier it gets. _Wrapped is an indicator that functools.update_wrapper was used which sets in this case foo.__wrapped__ = int as well as other attributes from int to foo. More precisely, but depending on the python version: '__module__', '__name__', '__qualname__', '__doc__', '__annotations__' and all entries from __dict__.

This is sometimes nice to know, but sometimes inconvenient and you would like to keep the original signature.


In practice you have two different cases 1) keep the signature identical 2) inject additional arguments.

For 1) You can use a bound Callable TypeVar or the new 3.12+ syntax:

Python <3.12
from typing import TypeVar, Callable from functools import wraps CT = TypeVar("CT", bound=Callable[..., Any]) def args_as_ints(f : CT) -> CT: """ This tells the type-checker that the returned value is equivalent to the input value. """ @wraps(f) def g(*args, **kwargs): args = [int(x) for x in args] kwargs = dict((k, int(v)) for k, v in kwargs.items()) return f(*args, **kwargs) return g ... 
Python 3.12+
from typing import Any, Callable from functools import wraps # Any is narrowed by the int from funny_function. def args_as_ints[T: Callable[..., Any]](func: T) -> T: @wraps(func) def wrapper(*args, **kwargs): args = [int(x) for x in args] kwargs = dict((k, int(v)) for k, v in kwargs.items()) return func(*args, **kwargs) return wrapper ... reveal_type(funny_function) # funny_function" is "(x: Any, y: Any, z: Any = 3) -> int" 
2 ) Extend the signature

An alternative to keep the type-hint identical simple to use the decorator a bit differently and use a ParamSpec und functools.update_wrapper

from typing import TypeVar, Callable, ParamSpec, Concatenate from functools import update_wrapper T = TypeVar("T") P = ParamSpec("P") def args_as_ints(f : Callable[P, T]): # -> Callable[Concatenate[int, P], T]: # In this case you can also replace it with "int" to be explicit and not use # the signature of f, i.e. args: int def g(special_value=1, *args: P.args, **kwargs: P.kwargs) -> T: args = [int(x) for x in args] kwargs = dict((k, int(v)) for k, v in kwargs.items()) return f(*args, **kwargs) update_wrapper(g, f) return g ConvertibleToInt : TypeAlias = Any # you can do this properly @args_as_ints def funny_function(x : "ConvertibleToInt", y: "ConvertibleToInt", z: "ConvertibleToInt"=3) -> int: """Computes x*y + 2*z""" return x*y + 2*z reveal_type(funny_function) # "funny_function" is "(special_value: int = ... x: Any, y: Any, z: Any = ...) -> int" 

At runtime this is equivalent to @wraps.

However there is one Pro and one con argument:
Pro: This is proper explicit-typing, it helps to understand the code and you can avoid errors.
Con: You likely loose your default arguments, as you see z has no default value anymore. An explicit annotation with -> Callable[Concatenate[int, P], T]: removes the information that the first parameter is named special_value. Implicit type/signature tracing can keep this information and also does not use _Wrapped. You loose the implicit tracing when you annotate f or the return type in the decorator.

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.