How does a generator function work internally?

Question

Below is a generator function.

def f(): x=1 while 1: y = yield x x += y

Does this generator function (f) get implemented, internally, as shown below?

class f(collections.Iterable): def __init__(self): self.x = 1 def __iter__(self): return iter(self) def __next__(self): return self.x def send(self, y): self.x += y return self.next()

Edit:

This is the answer for my question.

You can test whether they behave the same for yourself. Going into the internal implementation details seems too broad for a SO question. — jonrsharpe
– jonrsharpe, Commented Aug 16, 2017 at 22:25
This post may provide some info: aosabook.org/en/500L/a-web-crawler-with-asyncio-coroutines.html — Chris
– Chris, Commented Aug 16, 2017 at 22:27
of course they aren't equivalent, one is a class and one is a function — wim
– wim, Commented Aug 16, 2017 at 22:29
The short answer is that generators are not implemented internally as shown in your pure python class. Instead, they share most of the same logic as regular functions. — Raymond Hettinger
– Raymond Hettinger, Commented Aug 17, 2017 at 6:36

Raymond Hettinger · Accepted Answer · 2017-08-17 06:10:00Z

Internally, a generator works about the same as a regular function call. Under-the-hood, running generators and running functions use mostly the same machinery.

When you call either a function or a generator, a stackframe is created. It has the local variables (including the arguments passed into the function), a code pointer to the active opcode, and a stack for pending try-blocks, with-blocks, or loops.

In a regular function, execution begins immediately. When return is encountered, the final result is kept and the stackframe is freed along with everything it referenced.

In a generator function, the stackframe is wrapped in a generator-iterator object and returned immediately. The code in the generator function only runs when called by next(g) or g.send(v). Execution is suspended when yield is encountered.

One way to think of generators is that they are like functions that can be paused with yield and resumed with g.next(). The stackframe is kept alive, so resuming a running generator is much cheaper than making a new function call which has to build a new frame on every invocation.

@overexchange Execution of both generators and functions involves running the current opcode and updating the code pointer. Pausing just means to stop doing that. Resuming means to continue doing that. Count with me, 1, 2, 3, and now talk about something else, and continue couting 4, 5, 6, ... All you need to know is the last count. Likewise, the stackframe keeps the state of the function and you can resume stop updating it and resume updating it at any time.
Building block for async is about task capable to suspend/resume. For server side programming, generally task is suspended on performing IO. When u say, pausing is to stop, are we not pausing because we are waiting for an IO(say http get request)?
@overexchange I think you're imagining this to be more complex than it is. The code for next(g) boils down to basically PyEval_EvalFrameEx(gen->gi_frame, exc). That's it, it just execs the current state of the frame (see Objects/genobject.c which is clear and simple in Python 2.7). A yield simply returns from that call. In contrast, a function call runs PyEval_EvalCodeEx(...) which creates a new frame and execs it as shown above. See Objects/funcobject.c.

Collectives™ on Stack Overflow

How does a generator function work internally?

1 Answer 1

4 Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Linked

Related