Evaluation order of positional and keyword arguments

Question

Consider this contrived^* example:

def count(one, two, three): print one print two print three

Three shall be the number thou shalt count, and the number of the counting shall be three.

>>> x = [1, 2, 3] >>> count(*map(int, x), three=x.pop()) 1 2 3

Four shalt thou not count,

>>> x = [1, 2, 3, 4] >>> count(*map(int, x), three=x.pop()) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: count() got multiple values for keyword argument 'three'

neither count thou two, excepting that thou then proceed to three.

>>> x = [1, 2] >>> count(*map(int, x), three=x.pop()) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: count() takes exactly 3 arguments (2 given)

Five is right out.

>>> x = [1, 2, 3, 4, 5] >>> count(*map(int, x), three=x.pop()) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: count() takes exactly 3 arguments (5 given)

After having read this question, I would actually have thought that x = [1, 2] is the only one that works, because

first, map(int, x) would be evaluated, one set to 1 and two set to 2
then, x still [1, 2], x.pop() would be evaluated and three set to 2, too.

My expectation for x = [1, 2, 3] was to get the error that I actually saw for x = [1, 2, 3, 4].

What is going on here? Why are the arguments seemingly not evaluated from left to right? Are keyword arguments evaluated first?

^*actually my real code corresponds to x = [1, 2, 3], which works, but I wasn't sure it was safe, and after reading the other question I thought it shouldn't actually work.

I'm using Python 2.7, if that matters.

You pop an element off so you are left with two args in the first example — Padraic Cunningham
– Padraic Cunningham, Commented Oct 20, 2015 at 19:18
Because reasons. Honestly there's no obvious expected behavior here. If any of these worked I would consider it entirely luck and wouldn't rely on it. Write clearly and avoid mutation where possible otherwise it's very difficult to reason about what should happen. — Chad S.
– Chad S., Commented Oct 20, 2015 at 19:29
The statement and the example following it below the "CPython implementation detail" seems relevant here: Python Language Reference » Expressions » Calls: "although the *expression syntax may appear after some keyword arguments, it is processed before the keyword arguments" — Lukas Graf
– Lukas Graf, Commented Oct 20, 2015 at 20:10

Community · Accepted Answer · 2020-06-20 09:12:55Z

Python 2.7

If we look at the CPython source related to creating the AST(ast_for_call) for a function call the order of argument evaluation turns out to be:

return Call(func, args, keywords, vararg, kwarg, func->lineno, func->col_offset, c->c_arena);

ie. args --> keywords --> vararg --> kwarg

So, in your case the keyword argument is evaluated first and then the star based expression(vararg) is evaluated.

Byte code:

>>> dis.dis(lambda: func(1, 2, *('k', 'j', 'l'), z=1, y =2, three=x.pop(), **{kwarg:1})) 1 0 LOAD_GLOBAL 0 (func) 3 LOAD_CONST 1 (1) # arg 6 LOAD_CONST 2 (2) # arg 9 LOAD_CONST 3 ('z') # keyword 12 LOAD_CONST 1 (1) 15 LOAD_CONST 4 ('y') # keyword 18 LOAD_CONST 2 (2) 21 LOAD_CONST 5 ('three') # keyword 24 LOAD_GLOBAL 1 (x) 27 LOAD_ATTR 2 (pop) 30 CALL_FUNCTION 0 33 LOAD_CONST 9 (('k', 'j', 'l')) #vararg 36 BUILD_MAP 1 39 LOAD_CONST 1 (1) 42 LOAD_GLOBAL 3 (kwarg) #kwarg 45 STORE_MAP 46 CALL_FUNCTION_V

Hence in your case the pop() call will happen first followed by the varargs evaluation.

So, if three is a part of kwargs then we will get an error with map:

>>> x = [1, 2, 3] >>> count(*map(float, x), **{'three': x.pop()}) Traceback (most recent call last): File "<ipython-input-133-e8831565af13>", line 1, in <module> count(*map(float, x), **{'three': x.pop()}) TypeError: count() got multiple values for keyword argument 'three'

It will work if we do it *lazily:

>>> x = [1, 2, 3] >>> count(*(float(y) for y in x), **{'three': x.pop()}) 1.0, 2.0, 3

_{*The reason why generator works and map or list comprehension fails is explained at the end.}

Python 3.5

The ast_for_call function here only maintains two lists: args and keywords.

Here the varargs are inserted into the args list and kwargs go to the keywords list. So, in the end the call looks like:

return Call(func, args, keywords, func->lineno, func->col_offset, c->c_arena);

Byte code:

>>> dis.dis(lambda: func(1, 2, *('k', 'j', 'l'), z=1, y =2, three=x.pop(), **{kwarg:1})) 1 0 LOAD_GLOBAL 0 (func) 3 LOAD_CONST 1 (1) 6 LOAD_CONST 2 (2) 9 LOAD_CONST 9 (('k', 'j', 'l')) 12 LOAD_CONST 6 ('z') 15 LOAD_CONST 1 (1) 18 LOAD_CONST 7 ('y') 21 LOAD_CONST 2 (2) 24 LOAD_CONST 8 ('three') 27 LOAD_GLOBAL 1 (x) 30 LOAD_ATTR 2 (pop) 33 CALL_FUNCTION 0 (0 positional, 0 keyword pair) 36 LOAD_GLOBAL 3 (kwarg) 39 LOAD_CONST 1 (1) 42 BUILD_MAP 1 45 CALL_FUNCTION_VAR_KW 770 (2 positional, 3 keyword pair) 48 RETURN_VALUE

Now things can get a little exciting if the expression yielding the varargs is lazy:

>> def count(one, two, three): print (one, two, three) ... >>> x = [1, 2, 3] >>> count(*map(float, x), three=x.pop()) # map is lazy in Python 3 1.0 2.0 3 >>> x = [1, 2, 3] >>> count(*[float(y) for y in x], three=x.pop()) Traceback (most recent call last): File "<ipython-input-25-b7ef8034ef4e>", line 1, in <module> count(*[float(y) for y in x], three=x.pop()) TypeError: count() got multiple values for argument 'three'

Byte code:

>>> dis.dis(lambda: count(*map(float, x), three=x.pop())) 1 0 LOAD_GLOBAL 0 (count) 3 LOAD_GLOBAL 1 (map) 6 LOAD_GLOBAL 2 (float) 9 LOAD_GLOBAL 3 (x) 12 CALL_FUNCTION 2 (2 positional, 0 keyword pair) 15 LOAD_CONST 1 ('three') 18 LOAD_GLOBAL 3 (x) 21 LOAD_ATTR 4 (pop) 24 CALL_FUNCTION 0 (0 positional, 0 keyword pair) 27 CALL_FUNCTION_VAR 256 (0 positional, 1 keyword pair) 30 RETURN_VALUE >>> dis.dis(lambda: count(*[float(y) for y in x], three=x.pop())) 1 0 LOAD_GLOBAL 0 (count) 3 LOAD_CONST 1 (<code object <listcomp> at 0x103b63930, file "<ipython-input-28-1cc782164f20>", line 1>) 6 LOAD_CONST 2 ('<lambda>.<locals>.<listcomp>') 9 MAKE_FUNCTION 0 12 LOAD_GLOBAL 1 (x) 15 GET_ITER 16 CALL_FUNCTION 1 (1 positional, 0 keyword pair) 19 LOAD_CONST 3 ('three') 22 LOAD_GLOBAL 1 (x) 25 LOAD_ATTR 2 (pop) 28 CALL_FUNCTION 0 (0 positional, 0 keyword pair) 31 CALL_FUNCTION_VAR 256 (0 positional, 1 keyword pair) 34 RETURN_VALUE

The lazy call works because unpacking(aka actual evaluation of the generator) doesn't happen until the function is actually called, hence in this case pop() call will remove the 3 first and then later on map will only pass 1, 2.

But, in the case of list comprehension the list object already contains 3 items and then even though pop() removed 3 later on we are still passing two values for the third argument.

Thanks for the detailed insight! So to summarize: It happens to work in CPython 2.7, but not in general in 3.5, except when using map; but it's an implementation detail and cannot be relied on?
@mkrieger1 It's not an implementation detail but a version detail, Python 2 and Python 3 are handling things differently. But across different implementation the output should match, for example I tested on PyPy 2.7.9 and it works same as CPython 2.7.10. Things have changed between Python 3.4 and 3.5 as well, for example now something like count(*(1,), *(2, 3)) is valid(because of PEP-448) in Python 3.5 but not in older versions of Python 3.x.

Collectives™ on Stack Overflow

Evaluation order of positional and keyword arguments

1 Answer 1

Python 2.7

Python 3.5

2 Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Python 2.7

Python 3.5

2 Comments

Linked

Related