Yet another TL;DR

Iterator on list: next() returns the next element of the list

Iterator generator: next() will compute the next element on the fly (execute code)

You can see the yield/generator as a way to manually run the control flow from outside (like continue loop one step), by calling next, however complex the flow.

Note: The generator is NOT a normal function. It remembers the previous state like local variables (stack). See other answers or articles for detailed explanation. The generator can only be iterated on once. You could do without yield, but it would not be as nice, so it can be considered 'very nice' language sugar.

Here's a simple yield based approach, to compute the fibonacci series, explained:

def fib(limit=50): a, b = 0, 1 for i in range(limit): yield b a, b = b, a+b 

When you enter this into your REPL and then try and call it, you'll get a mystifying result:

>>> fib() <generator object fib at 0x7fa38394e3b8> 

This is because the presence of yield signaled to Python that you want to create a generator, that is, an object that generates values on demand.

So, how do you generate these values? This can either be done directly by using the built-in function next, or, indirectly by feeding it to a construct that consumes values.

Using the built-in next() function, you directly invoke .next/__next__, forcing the generator to produce a value:

>>> g = fib() >>> next(g) 1 >>> next(g) 1 >>> next(g) 2 >>> next(g) 3 >>> next(g) 5 

Indirectly, if you provide fib to a for loop, a list initializer, a tuple initializer, or anything else that expects an object that generates/produces values, you'll "consume" the generator until no more values can be produced by it (and it returns):

results = [] for i in fib(30): # consumes fib results.append(i) # can also be accomplished with results = list(fib(30)) # consumes fib 

Similarly, with a tuple initializer:

>>> tuple(fib(5)) # consumes fib (1, 1, 2, 3, 5) 

A generator differs from a function in the sense that it is lazy. It accomplishes this by maintaining it's local state and allowing you to resume whenever you need to.

When you first invoke fib by calling it:

f = fib() 

Python compiles the function, encounters the yield keyword and simply returns a generator object back at you. Not very helpful it seems.

When you then request it generates the first value, directly or indirectly, it executes all statements that it finds, until it encounters a yield, it then yields back the value you supplied to yield and pauses. For an example that better demonstrates this, let's use some print calls (replace with print "text" if on Python 2):

def yielder(value): """ This is an infinite generator. Only use next on it """ while 1: print("I'm going to generate the value for you") print("Then I'll pause for a while") yield value print("Let's go through it again.") 

Now, enter in the REPL:

>>> gen = yielder("Hello, yield!") 

you have a generator object now waiting for a command for it to generate a value. Use next and see what get's printed:

>>> next(gen) # runs until it finds a yield I'm going to generate the value for you Then I'll pause for a while 'Hello, yield!' 

The unquoted results are what's printed. The quoted result is what is returned from yield. Call next again now:

>>> next(gen) # continues from yield and runs again Let's go through it again. I'm going to generate the value for you Then I'll pause for a while 'Hello, yield!' 

The generator remembers it was paused at yield value and resumes from there. The next message is printed and the search for the yield statement to pause at it performed again (due to the while loop).

yield is similar to return. The difference is:

yield makes a function iterable (in the following example primes(n = 1) function becomes iterable).
What it essentially means is the next time the function is called, it will continue from where it left (which is after the line of yield expression).

def isprime(n): if n == 1: return False for x in range(2, n): if n % x == 0: return False else: return True def primes(n = 1): while(True): if isprime(n): yield n n += 1 for n in primes(): if n > 100: break print(n) 

In the above example if isprime(n) is true it will return the prime number. In the next iteration it will continue from the next line

n += 1 

yield yields something. It's like somebody asks you to make 5 cupcakes. If you are done with at least one cupcake, you can give it to them to eat while you make other cakes.

In [4]: def make_cake(numbers): ...: for i in range(numbers): ...: yield 'Cake {}'.format(i) ...: In [5]: factory = make_cake(5) 

Here factory is called a generator, which makes you cakes. If you call make_function, you get a generator instead of running that function. It is because when yield keyword is present in a function, it becomes a generator.

In [7]: next(factory) Out[7]: 'Cake 0' In [8]: next(factory) Out[8]: 'Cake 1' In [9]: next(factory) Out[9]: 'Cake 2' In [10]: next(factory) Out[10]: 'Cake 3' In [11]: next(factory) Out[11]: 'Cake 4' 

They consumed all the cakes, but they ask for one again.

In [12]: next(factory) --------------------------------------------------------------------------- StopIteration Traceback (most recent call last) <ipython-input-12-0f5c45da9774> in <module> ----> 1 next(factory) StopIteration: 

and they are being told to stop asking more. So once you consumed a generator you are done with it. You need to call make_cake again if you want more cakes. It is like placing another order for cupcakes.

In [13]: factory = make_cake(3) In [14]: for cake in factory: ...: print(cake) ...: Cake 0 Cake 1 Cake 2 

You can also use for loop with a generator like the one above.

One more example: Lets say you want a random password whenever you ask for it.

In [22]: import random In [23]: import string In [24]: def random_password_generator(): ...: while True: ...: yield ''.join([random.choice(string.ascii_letters) for _ in range(8)]) ...: In [25]: rpg = random_password_generator() In [26]: for i in range(3): ...: print(next(rpg)) ...: FXpUBhhH DdUDHoHn dvtebEqG In [27]: next(rpg) Out[27]: 'mJbYRMNo' 

Here rpg is a generator, which can generate an infinite number of random passwords. So we can also say that generators are useful when we don't know the length of the sequence, unlike list which has a finite number of elements.

In Python generators (a special type of iterators) are used to generate series of values and yield keyword is just like the return keyword of generator functions.

The other fascinating thing yield keyword does is saving the state of a generator function.

So, we can set a number to a different value each time the generator yields.

Here's an instance:

def getPrimes(number): while True: if isPrime(number): number = yield number # a miracle occurs here number += 1 def printSuccessivePrimes(iterations, base=10): primeGenerator = getPrimes(base) primeGenerator.send(None) for power in range(iterations): print(primeGenerator.send(base ** power)) 

All of the answers here are great; but only one of them (the most voted one) relates to how your code works. Others are relating to generators in general, and how they work.

So I won't repeat what generators are or what yields do; I think these are covered by great existing answers. However, after spending few hours trying to understand a similar code to yours, I'll break it down how it works.

Your code traverse a binary tree structure. Let's take this tree for example:

 5 / \ 3 6 / \ \ 1 4 8 

And another simpler implementation of a binary-search tree traversal:

class Node(object): .. def __iter__(self): if self.has_left_child(): for child in self.left: yield child yield self.val if self.has_right_child(): for child in self.right: yield child 

The execution code is on the Tree object, which implements __iter__ as this:

def __iter__(self): class EmptyIter(): def next(self): raise StopIteration if self.root: return self.root.__iter__() return EmptyIter() 

The while candidates statement can be replaced with for element in tree; Python translate this to

it = iter(TreeObj) # returns iter(self.root) which calls self.root.__iter__() for element in it: .. process element .. 

Because Node.__iter__ function is a generator, the code inside it is executed per iteration. So the execution would look like this:

1. root element is first; check if it has left childs and for iterate them (let's call it it1 because its the first iterator object)
2. it has a child so the for is executed. The for child in self.left creates a new iterator from self.left, which is a Node object itself (it2)
3. Same logic as 2, and a new iterator is created (it3)
4. Now we reached the left end of the tree. it3 has no left childs so it continues and yield self.value
5. On the next call to next(it3) it raises StopIteration and exists since it has no right childs (it reaches to the end of the function without yield anything)
6. it1 and it2 are still active - they are not exhausted and calling next(it2) would yield values, not raise StopIteration
7. Now we are back to it2 context, and call next(it2) which continues where it stopped: right after the yield child statement. Since it has no more left childs it continues and yields it's self.val.

The catch here is that every iteration creates sub-iterators to traverse the tree, and holds the state of the current iterator. Once it reaches the end it traverse back the stack, and values are returned in the correct order (smallest yields value first).

Your code example did something similar in a different technique: it populated a one-element list for every child, then on the next iteration it pops it and run the function code on the current object (hence the self).

Can also send data back to the generator!

Indeed, as many answers here explain, using yield creates a generator.

You can use the yield keyword to send data back to a "live" generator.

Example:

Let's say we have a method which translates from english to some other language. And in the beginning of it, it does something which is heavy and should be done once. We want this method run forever (don't really know why.. :)), and receive words words to be translated.

def translator(): # load all the words in English language and the translation to 'other lang' my_words_dict = {'hello': 'hello in other language', 'dog': 'dog in other language'} while True: word = (yield) yield my_words_dict.get(word, 'Unknown word...') 

Running:

my_words_translator = translator() next(my_words_translator) print(my_words_translator.send('dog')) next(my_words_translator) print(my_words_translator.send('cat')) 

will print:

dog in other language Unknown word... 

To summarise:

use send method inside a generator to send data back to the generator. To allow that, a (yield) is used.

The yield keyword in Python used to exit from the code without disturbing the state of local variables and when again the function is called the execution starts from the last point where we left the code.

The below example demonstrates the working of yield:

def counter(): x=2 while x < 5: yield x x += 1 print("Initial value of x: ", counter()) for y in counter(): print(y) 

The above code generates the Below output:

Initial value of x: <generator object counter at 0x7f0263020ac0> 2 3 4 

yield in python is in a way similar to the return statement, except for some differences. If multiple values have to be returned from a function, return statement will return all the values as a list and it has to be stored in the memory in the caller block. But what if we don't want to use extra memory? Instead, we want to get the value from the function when we need it. This is where yield comes in. Consider the following function :-

def fun(): yield 1 yield 2 yield 3 

And the caller is :-

def caller(): print ('First value printing') print (fun()) print ('Second value printing') print (fun()) print ('Third value printing') print (fun()) 

The above code segment (caller function) when called, outputs :-

First value printing 1 Second value printing 2 Third value printing 3 

As can be seen from above, yield returns a value to its caller, but when the function is called again, it doesn't start from the first statement, but from the statement right after the yield. In the above example, "First value printing" was printed and the function was called. 1 was returned and printed. Then "Second value printing" was printed and again fun() was called. Instead of printing 1 (the first statement), it returned 2, i.e., the statement just after yield 1. The same process is repeated further.

Simple answer

When function contains at least one yield statement, the function automaticly becomes generator function. When you call generator function, python executes code in the generator function until yield statement occur. yield statement freezes the function with all its internal states. When you call generator function again, python continues execution of code in the generator function from frozen position, until yield statement occur again and again. The generator function executes code until generator function runs out without yield statement.

Benchmark

Create a list and return it:

def my_range(n): my_list = [] i = 0 while i < n: my_list.append(i) i += 1 return my_list @profile def function(): my_sum = 0 my_values = my_range(1000000) for my_value in my_values: my_sum += my_value function() 

Results with:

Total time: 1.07901 s Timer unit: 1e-06 s Line # Hits Time Per Hit % Time Line Contents ============================================================== 9 @profile 10 def function(): 11 1 1.1 1.1 0.0 my_sum = 0 12 1 494875.0 494875.0 45.9 my_values = my_range(1000000) 13 1000001 262842.1 0.3 24.4 for my_value in my_values: 14 1000000 321289.8 0.3 29.8 my_sum += my_value Line # Mem usage Increment Occurences Line Contents ============================================================ 9 40.168 MiB 40.168 MiB 1 @profile 10 def function(): 11 40.168 MiB 0.000 MiB 1 my_sum = 0 12 78.914 MiB 38.746 MiB 1 my_values = my_range(1000000) 13 78.941 MiB 0.012 MiB 1000001 for my_value in my_values: 14 78.941 MiB 0.016 MiB 1000000 my_sum += my_value 

Generate values on the fly:

def my_range(n): i = 0 while i < n: yield i i += 1 @profile def function(): my_sum = 0 for my_value in my_range(1000000): my_sum += my_value function() 

Results with:

Total time: 1.24841 s Timer unit: 1e-06 s Line # Hits Time Per Hit % Time Line Contents ============================================================== 7 @profile 8 def function(): 9 1 1.1 1.1 0.0 my_sum = 0 10 11 1000001 895617.3 0.9 71.7 for my_value in my_range(1000000): 12 1000000 352793.7 0.4 28.3 my_sum += my_value Line # Mem usage Increment Occurences Line Contents ============================================================ 7 40.168 MiB 40.168 MiB 1 @profile 8 def function(): 9 40.168 MiB 0.000 MiB 1 my_sum = 0 10 11 40.203 MiB 0.016 MiB 1000001 for my_value in my_range(1000000): 12 40.203 MiB 0.020 MiB 1000000 my_sum += my_value 

Summary

The generator function needs a little more time to execute, than function which returns a list but it use much less memory.

A simple use case:

>>> def foo(): yield 100 yield 20 yield 3 >>> for i in foo(): print(i) 100 20 3 >>> 

How it works: when called, the function returns an object immediately. The object can be passed to the next() function. Whenever the next() function is called, your function runs up until the next yield and provides the return value for the next() function.

Under the hood, the for loop recognizes that the object is a generator object and uses next() to get the next value.

In some languages like ES6 and higher, it's implemented a little differently so next is a member function of the generator object, and you could pass values from the caller every time it gets the next value. So if result is the generator then you could do something like y = result.next(555), and the program yielding values could say something like z = yield 999. The value of y would be 999 that next gets from the yield, and the value of z would be 555 that yield gets from the next. Python get and send methods have a similar effect.

yield allows you to write smarter for-loops by factoring out the looping part to a separate method for easy reuse.

Suppose you need to loop over all non-blank rows of a spreadsheet and do something with each row.

for i, row in df.iterrows(): #from the panda package for reading excel if row = blank: # pseudo code, check if row is non-blank... continue if past_last_row: # pseudo code, check for end of input data break #### above is boring stuff, below is what we actually want to do with the data ### f(row) 

If you need to call g(row) in a similar loop, you might find yourself repeating the for statement plus the checks for valid rows, which is boring, complex and error-prone. We don't want to repeat ourselves (DRY principle).

You want to separate the code for checking each record from the code that actually process the rows, like f(row) and g(row) .

You could make a function that takes f() as input parameter, but it is much simpler to use yield in a method that does all the boring stuff about checking for valid rows in preparation for calling f():

def valid_rows(): for i, row in df.iterrows(): # iterate over each row of spreadsheet if row == blank: # pseudo code, check if row is non-blank... continue if past_last_row: # pseudo code, check for end of input data break yield i, row 

Note that each call of the method will return one next row, except that if all rows are read and the for finishes, the method will return normally. The next call will start a new for loop.

Now you can write iterations over the data without having to repeat that boring checking for valid rows (which is now factored out to its own method) like:

for i, row in valid_rows(): f(row) for i, row in valid_rows(): g(row) nr_valid_rows = len(list(valid_rows())) 

That is all there is. Note that I have not used jargon like iterator, generator, protocol, co-routine. I think that this simple example applies to a lot of our day to day coding.

What does the Yield Keyword do ?

yield keyword is used to create a generator function. A type of function that is memory efficient and can be used like an iterator object.

In simple terms, the yield keyword will turn any expression that is given with it into a generator object and return it to the caller. Therefore, you must iterate over the generator object if you wish to obtain the values stored there.

Definition alone cannot exaplain yield so here is a very simple exampleA.py:

# example A def getNumber(): for r in range(1,10): return r 

The above function will return only 1 even when it's called multiple times over a loop. Now if we replace keyword return with yield as in exampleB.py:

# example B def getNumber(): for r in range(1,10): yield r 

It will return 1 when first called 2 when called again then 3,4 and it goes to increment till 10.

Although the example B is conceptually true but to call it in python 3 we have to do the following:

# example B def getNumber(): for r in range(1,10): yield r g = getNumber() #instance print(next(g)) #will print 1 print(next(g)) #will print 2 print(next(g)) #will print 3 and so on 

Usually, it's used to create an iterator out of function. Think 'yield' as an append() to your function and your function as an array. And if certain criteria meet, you can add that value in your function to make it an iterator.

arr=[] if 2>0: arr.append(2) def func(): if 2>0: yield 2 

the output will be the same for both.

The main advantage of using yield is to creating iterators. Iterators don’t compute the value of each item when instantiated. They only compute it when you ask for it. This is known as lazy evaluation.

Generators allow to get individual processed items immediately (without the need to wait for the whole collection to be processed). This is illustrated in the example below.

import time def get_gen(): for i in range(10): yield i time.sleep(1) def get_list(): ret = [] for i in range(10): ret.append(i) time.sleep(1) return ret start_time = time.time() print('get_gen iteration (individual results come immediately)') for i in get_gen(): print(f'result arrived after: {time.time() - start_time:.0f} seconds') print() start_time = time.time() print('get_list iteration (results come all at once)') for i in get_list(): print(f'result arrived after: {time.time() - start_time:.0f} seconds') 

get_gen iteration (individual results come immediately) result arrived after: 0 seconds result arrived after: 1 seconds result arrived after: 2 seconds result arrived after: 3 seconds result arrived after: 4 seconds result arrived after: 5 seconds result arrived after: 6 seconds result arrived after: 7 seconds result arrived after: 8 seconds result arrived after: 9 seconds get_list iteration (results come all at once) result arrived after: 10 seconds result arrived after: 10 seconds result arrived after: 10 seconds result arrived after: 10 seconds result arrived after: 10 seconds result arrived after: 10 seconds result arrived after: 10 seconds result arrived after: 10 seconds result arrived after: 10 seconds result arrived after: 10 seconds 

Function - returns.

Generator - yields (contains one or more yields and zero or more returns).

names = ['Sam', 'Sarah', 'Thomas', 'James'] # Using function def greet(name) : return f'Hi, my name is {name}.' for each_name in names: print(greet(each_name)) # Output: >>>Hi, my name is Sam. >>>Hi, my name is Sarah. >>>Hi, my name is Thomas. >>>Hi, my name is James. # using generator def greetings(names) : for each_name in names: yield f'Hi, my name is {each_name}.' for greet_name in greetings(names): print (greet_name) # Output: >>>Hi, my name is Sam. >>>Hi, my name is Sarah. >>>Hi, my name is Thomas. >>>Hi, my name is James. 

A generator looks like a function but behaves like an iterator.

A generator continues execution from where it is lefoff (or yielded). When resumed, the function continues the execution immediately after the last yield run. This allows its code to produce a series of values over time rather them computing them all at once and sending them back like a list.

def function(): yield 1 # return this first yield 2 # start continue from here (yield don't execute above code once executed) yield 3 # give this at last (yield don't execute above code once executed) for processed_data in function(): print(processed_data) #Output: >>>1 >>>2 >>>3 

Note: Yield should not be in the try ... finally construct.

Key points

• The grammar for Python uses the presence of the yield keyword to make a function that returns a generator.

• A generator is a kind of iterator, which is that main way that looping occurs in Python.

• A generator is essentially a resumable function. Unlike return that returns a value and ends a function, the yield keyword returns a value and suspends a function.

• When next(g) is called on a generator, the function resumes execution where it left off.

• Only when the function encounters an explicit or implied return does it actually end.

Technique for writing and understanding generators

An easy way to understand and think about generators is to write a regular function with print() instead of yield:

def f(n): for x in range(n): print(x) print(x * 10) 

Watch what it outputs:

>>> f(3) 0 0 1 10 2 2 

When that function is understood, substitute the yield for print to get a generator that produces the same values:

def f(n): for x in range(n): yield x yield x * 10 

Which gives:

>>> list(f(3)) [0, 0, 1, 10, 2, 20] 

Iterator protocol

The answer to "what yield does" can be short and simple, but it is part of a larger world, the so-called "iterator protocol".

On the sender side of iterator protocol, there are two relevant kinds of objects. The iterables are things you can loop over. And the iterators are objects that track the loop state.

On the consumer side of the iterator protocol, we call iter() on the iterable object to get a iterator. Then we call next() on the iterator to retrieve values from the iterator. When there is no more data, a StopIteration exception is raised:

>>> s = [10, 20, 30] # The list is the "iterable" >>> it = iter(s) # This is the "iterator" >>> next(it) # Gets values out of an iterator 10 >>> next(it) 20 >>> next(it) 30 >>> next(it) Traceback (most recent call last): ... StopIteration 

To make this all easier for us, for-loops call iter and next on our behalf:

>>> for x in s: ... print(x) ... 10 20 30 

A person could write a book about all this, but these are the key points. When I teach Python courses, I've found that this is a minimal sufficient explanation to build understand and start using it right away. In particular, the trick of writing a function with print, testing it, and then converting to yield seems to work well with all levels of Python programmers.

yield:

• can return a value multiple times from a function by stopping the function.
• can use from with it like yield from.
• is used when returning big data by dividing it into small pieces of data to prevent the big usage of RAM.

For example, test() below can return 'One', 'Two' and ['Three', 'Four'] one by one by stopping test() so test() returns 3 times in total by stopping test() 3 times in total:

def test(): yield 'One' # Stop, return 'One' and resume yield 'Two' # Stop, return 'Two' and resume yield from ['Three', 'Four'] # Stop and return ['Three', 'Four'] 

And, these 3 sets of code below can call test() and print 'One', 'Two', 'Three' and 'Four':

for x in test(): print(x) 

x = test() print(next(x)) print(next(x)) print(next(x)) print(next(x)) 

x = test() print(x.__next__()) print(x.__next__()) print(x.__next__()) print(x.__next__()) 

This is the result:

$ python yield_test.py One Two Three Four 

In addition, when using return with yield, there is no way to get the value from return:

def test(): yield 'One' yield 'Two' yield from ['Three', 'Four'] return 'Five' # 'Five' cannot be got x = test() print(next(x)) print(next(x)) print(next(x)) print(next(x)) print(next(x)) # Here 

So, there is the error below when trying to get 'Five':

$ python yield_test.py One Two Three Four Traceback (most recent call last): File "C:\Users\kai\yield_test.py", line 12, in <module> print(next(x)) ^^^^^^^ StopIteration: Five 

What Is Yield In Python?

The Yield keyword in Python is similar to a return statement used for returning values or objects in Python. However, there is a slight difference. The yield statement returns a generator object to the one who calls the function which contains yield, instead of simply returning a value.

Inside a program, when you call a function that has a yield statement, as soon as a yield is encountered, the execution of the function stops and returns an object of the generator to the function caller. In simpler words, the yield keyword will convert an expression that is specified along with it to a generator object and return it to the caller. Hence, if you want to get the values stored inside the generator object, you need to iterate over it.

It will not destroy the local variables’ states. Whenever a function is called, the execution will start from the last yield expression. Please note that a function that contains a yield keyword is known as a generator function.

When you use a function with a return value, every time you call the function, it starts with a new set of variables. In contrast, if you use a generator function instead of a normal function, the execution will start right from where it left last.

If you want to return multiple values from a function, you can use generator functions with yield keywords. The yield expressions return multiple values. They return one value, then wait, save the local state, and resume again.

Source: https://www.simplilearn.com/tutorials/python-tutorial/yield-in-python

RETURN VS YEILD

OK, let's answer this question a bit differently, we all know about return or I will explain it later on in this answer anyway.

But to understand it, let's see what yield is good for. and why do not use return at the first place which I am sure it is a question for many of you.

Differences from return:

Lazy Evaluation: Yield produces values one at a time, which is memory-efficient for large data (e.g., reading lines of a file).

Continuous Execution: Yield allows a function to resume execution later, whereas return ends the function immediately.

Infinite Sequences: With yield, you can create infinite sequences (e.g., Fibonacci numbers) that wouldn't be possible with return.

Why not use return? return gives back a single value or a complete list, requiring all values to be computed and stored in memory. yield is more efficient when handling large datasets or streams because it computes values only as needed.

1. return gives back a single value or a complete list, requiring all values to be computed and stored in memory.

2. yield is more efficient when handling large datasets or streams because it computes values only as needed.

So yes, it's more memory efficient compared to return as can be iterated lazily (whenever is needed).

What is return itself?

In Python, return is used in a function to:

1. Send Data Back: It specifies the value(s) that the function should give back to the caller.

2. End the Function: When executed, return immediately stops the function's execution and exits it.

Example:

def add(a, b): return a + b # Returns the sum of a and b result = add(10, 5) # result will be 15 print(result)