Generator Functions (yield) — Write generator functions using yield (intro) - Python

SECTION 1

Historical Context & Motivation

Imagine you need to process a list of one million numbers. A regular function would create the entire list in memory at once, potentially eating up hundreds of megabytes of RAM. Early Python programmers ran into this problem constantly when working with large datasets or infinite sequences. The language needed a way to produce values one at a time, only when asked, rather than computing everything up front.

The concept of lazy evaluation — computing values only when they are needed — had existed in functional programming languages like Haskell for years. Python's creator, Guido van Rossum, and the community recognized that bringing this idea into Python would make the language far more memory-efficient and expressive. This led to the introduction of generator functions and the yield keyword.

1994 • Python 1.0 Released

Python 1.0 launches with basic iteration using lists and for-loops. All sequences must be stored fully in memory before iteration can begin.

2001 • PEP 255 — Simple Generators

Python Enhancement Proposal 255 introduces generator functions and the yield keyword, enabling lazy evaluation in Python for the first time.

2002 • Python 2.2 — Generators Arrive

Generators become available as an optional feature using a special import. Developers begin experimenting with memory-efficient iteration patterns.

2006 • PEP 342 — Enhanced Generators

Python 2.5 allows sending values into generators with send(), transforming generators into full coroutines capable of two-way communication.

2014 • Python 3.4+ and async/await

Generator concepts directly inspire Python's async programming model. The yield pattern becomes the foundation for modern asynchronous code.

So the central question that generator functions answer is: How can we produce a sequence of values without computing and storing them all at once? As we will see, the yield keyword provides an elegant answer.

SECTION 2

Core Principles & Definitions

Before diving into code, let's establish the foundational ideas behind generator functions. Understanding these principles will make the syntax feel natural rather than mysterious.

Generator Function

A function that contains at least one yield statement. When called, it does NOT run the body immediately — instead, it returns a generator object that you can iterate over.

The yield Keyword

Works like return, but instead of ending the function, it pauses execution and remembers its state. The next time a value is requested, execution resumes right where it left off.

Lazy Evaluation

Values are produced on demand rather than all at once. This means a generator can represent an extremely large — or even infinite — sequence without running out of memory.

Iterator Protocol

Generator objects automatically implement Python's iterator protocol, meaning they work seamlessly with for-loops, next(), and other iteration tools.

StopIteration

When a generator function reaches the end of its code (or an explicit return), Python automatically raises a StopIteration exception to signal that no more values are available.

✦

KEY TAKEAWAY

Think of a generator function like a bookmark in a book. A regular function reads the whole book from start to finish every time you open it. A generator function, on the other hand, lets you read one page, place a bookmark, close the book, and come back later to pick up exactly where you left off. The yield keyword is that bookmark — it saves your place so nothing is lost between reads.

SECTION 3

Visual Explanation — How yield Pauses and Resumes

The following diagram shows the execution flow of a simple generator function that yields three values. Notice how the function's state is frozen at each yield and resumed each time next() is called.

The left column shows the generator function's code. Each colored yield line is a pause point. The right column shows what happens on the caller's side — each call to next() resumes the generator and receives the yielded value. A for-loop handles this automatically.

The key insight from this diagram is that the generator function's execution is interleaved with the caller's code. Each yield acts like a checkpoint — the function freezes its local variables, its position in the code, and everything else about its state. When next() is called again, the function thaws out and continues running from exactly where it paused. This is fundamentally different from a regular function, which always starts from the top.

SECTION 4

How Generator Functions Work Under the Hood

Let's break down the exact mechanics of what happens when you define and use a generator function. Understanding this process step by step will help you predict generator behavior in any situation.

Step 1 — Defining the Generator Function

A generator function looks almost identical to a regular function, but it uses yield instead of (or in addition to) return. Python recognizes the presence of yield anywhere in the function body and treats the entire function as a generator function. Here is the basic syntax:

GENERATOR FUNCTION SYNTAX

def my_generator(params): # ... code ... yield value # ... more code ... yield another_value

The def keyword is the same as for regular functions. The presence of yield is what makes it a generator.

Step 2 — Calling the Function Creates a Generator Object

This is the part that surprises most beginners. When you call a generator function, none of the function body executes. Instead, Python creates and returns a generator object. This object holds the function's code and its state, ready to execute on demand.

CREATING A GENERATOR OBJECT

gen = my_generator(args) # type(gen) \to <class 'generator'>

At this point, no code inside my_generator has run yet. The generator is in a suspended state at the very beginning of the function.

Step 3 — Advancing with next()

Each call to next(gen) runs the function's code from where it last paused until it hits the next yield. The yielded value is returned to the caller, and the function freezes again. When the function body ends without hitting another yield, Python raises StopIteration to signal completion.

Step 4 — The for-loop Shortcut

In practice, you rarely call next() manually. A for loop automatically calls next() on each iteration and catches the StopIteration exception to know when to stop. This makes generators feel just like iterating over a list.

FOR-LOOP WITH A GENERATOR

for value in my_generator(args): print(value)

This loop creates the generator object, calls next() repeatedly, assigns each yielded value to value, and stops when StopIteration is raised.

⚠

Common Gotcha

A generator object can only be iterated through once. After it raises StopIteration, it is exhausted. If you need to iterate again, you must create a new generator by calling the function again.

SECTION 5

Common Generator Patterns

Generator functions are incredibly flexible. Let's look at the most common patterns you will encounter, from finite sequences to infinite streams and data processing pipelines.

Three common generator patterns: a finite sequence that yields a fixed number of values, an infinite stream that never ends, and a filter pipeline that selectively passes values through.

The finite sequence pattern is the most straightforward — use a for loop or counter inside the generator to yield a known number of values. The infinite stream pattern uses while True to yield values forever; the caller is responsible for breaking out of the loop. The filter pipeline pattern accepts another iterable (even another generator) and yields only the values that meet a condition. You can chain these together to build powerful data processing pipelines where each step processes one item at a time.

💡

Pro Tip: Generator Expressions

For simple generators, Python offers a shorthand called a generator expression. It looks like a list comprehension but uses parentheses instead of brackets: (x * x for x in range(10)). This creates a generator object without needing a full function definition.

SECTION 6

Worked Example — Building a Fibonacci Generator

The Fibonacci sequence (0, 1, 1, 2, 3, 5, 8, 13, …) is a classic sequence where each number is the sum of the two before it. It is infinite, which makes it a perfect use case for generators. Let's build a Fibonacci generator step by step.

Fibonacci Generator Function

1
Step 1 — Define the function with yield

We write a function called fibonacci() that uses yield instead of return. We initialize two variables, a = 0 and b = 1, representing the last two numbers in the sequence.

def fibonacci(): a, b = 0, 1
2
Step 2 — Create the infinite loop

Since the Fibonacci sequence never ends, we use while True to loop forever. On each iteration, we yield the current value of a and then update both variables to advance the sequence.

while True: yield a a, b = b, a + b
3
Step 3 — Complete function definition

Here is the complete generator function. Notice how concise it is — only four lines of code to represent an infinite mathematical sequence.

def fibonacci(): a, b = 0, 1 while True: yield a a, b = b, a + b
4
Step 4 — Use the generator

We create a generator object and pull values from it. We can use next() for individual values, or use a for-loop with a break condition. Here we print the first 10 Fibonacci numbers.

fib = fibonacci() for i in range(10): print(next(fib)) # Output: 0, 1, 1, 2, 3, 5, 8, 13, 21, 34
5
Step 5 — Trace the execution

On the first next(fib) call, Python enters the function, sets a=0 and b=1, enters the while loop, and hits yield a → yields 0 and pauses. On the second call, it resumes after yield, updates a=1, b=1, loops back to yield a → yields 1 and pauses again. This pattern continues indefinitely.

Each call to next() only computes one new value. Memory usage stays constant regardless of how many values we generate.

SECTION 7

Generators vs. Regular Functions and Lists

When should you use a generator instead of a regular function that returns a list? The table below compares the two approaches across several important dimensions.

Feature	Regular Function (return list)	Generator Function (yield)
Memory Usage	Stores entire list in memory at once. O(n) memory for n items.	Stores only one value at a time. O(1) memory regardless of size.
Speed to First Value	Must compute all values before the first one is available.	First value available almost instantly — computation is on demand.
Infinite Sequences	Not possible — a list cannot hold infinite elements.	Fully supported — yield values forever and let the caller stop.
Random Access	Yes — use indexing like my_list[42].	No — values are produced sequentially. Cannot jump to a specific index.
Reusability	Can iterate multiple times over the same list.	Exhausted after one pass. Must recreate the generator to iterate again.
Composability	Must create intermediate lists when chaining operations.	Chain generators together — no intermediate lists needed.

Comparison of regular functions returning lists vs. generator functions using yield

✦

WHEN TO USE GENERATORS

Use a generator when you are dealing with large or infinite sequences, when you only need to process items one at a time, or when you want to chain processing steps without creating big intermediate lists. Use a regular function when you need random access, multiple iterations, or when the dataset is small enough that memory is not a concern.

SECTION 8

Connection to Advanced Topics

The yield keyword you have learned is just the beginning. Generators serve as the foundation for several advanced Python features. Here is a preview of where your knowledge can take you.

What You Learned	Advanced Feature	What It Adds
`yield value`	`value = yield result`	Two-way communication: send values INTO a generator using gen.send(), not just receive values from it.
Simple generator functions	`yield from iterable`	Delegate to a sub-generator, allowing you to split complex generators into smaller, reusable pieces.
Lazy iteration	`async def / await`	Asynchronous programming uses the same pause-and-resume concept for I/O operations like web requests and file reads.
Generator pipelines	itertools module	Python's standard library offers powerful generator-based tools like chain(), islice(), and combinations() for efficient data processing.

How basic generator knowledge connects to advanced Python features

The pause-and-resume pattern that yield introduced is one of the most powerful ideas in modern Python programming. Once you are comfortable with basic generators, exploring coroutines and async/await will feel much more natural, because they build directly on the concepts you have learned here.

SECTION 9

Practice Problems

Question 1: CONCEPTUAL

Explain the difference between a regular function that uses return and a generator function that uses yield. What happens to the function's state in each case when the keyword is reached?

Show answer

When a regular function reaches return, it sends back a value and completely terminates — all local variables are destroyed and the function's state is gone. The next time you call it, execution starts from the very beginning. In contrast, when a generator function reaches yield, it sends back a value but merely pauses execution. All local variables and the current position in the code are preserved. The next call to next() resumes from exactly where the function paused, rather than restarting from the top.

Question 2: BASIC CALCULATION

Write a generator function called countdown(n) that yields integers from n down to 1. For example, list(countdown(5)) should return [5, 4, 3, 2, 1].

Show answer

Here is the solution:

def countdown(n):
    while n > 0:
        yield n
        n -= 1

. The function starts with the value of n and yields it, then decrements by 1. Each time next() is called, the function resumes after the yield, decrements n, checks the while condition, and yields again. When n reaches 0, the loop ends and StopIteration is raised.

Question 3: INTERMEDIATE

Write a generator function called unique(iterable) that yields each value from the iterable only the first time it appears. For example, list(unique([1, 2, 2, 3, 1, 4])) should return [1, 2, 3, 4]. Why is a generator well-suited for this task?

Show answer

Here is the solution:

def unique(iterable):
    seen = set()
    for item in iterable:
        if item not in seen:
            seen.add(item)
            yield item

. The generator maintains a set of seen values across calls, which is preserved between yields. A generator is well-suited here because it can process the input lazily — if the caller only needs the first few unique values from a very large dataset, the generator stops early without processing the entire input. The seen set uses memory proportional to the number of unique elements, not the total number of elements.

Question 4: APPLIED

You are building a log monitoring tool that reads lines from a very large log file (potentially millions of lines). Write a generator function error_lines(filename) that opens the file and yields only the lines that contain the word "ERROR". Explain why using a generator is better than loading all error lines into a list.

Show answer

Here is the solution:

def error_lines(filename):
    with open(filename) as f:
        for line in f:
            if 'ERROR' in line:
                yield line.strip()

. This function reads the file one line at a time (Python file objects are themselves iterators) and only yields lines containing "ERROR". A generator is superior here because a log file could be gigabytes in size. Loading it entirely into memory would be impractical. With the generator approach, only one line is in memory at any time. Additionally, the caller can start processing error lines immediately without waiting for the entire file to be read. If they only need the first 10 errors, the generator stops early.

Question 5: CRITICAL THINKING

Consider these two functions that both produce the first n powers of 2: Function A:

def powers_list(n):
    return [2**i for i in range(n)]

Function B:

def powers_gen(n):
    for i in range(n):
        yield 2**i

For n = 1,000,000, compare the memory usage and behavior of each. Is there a scenario where Function A would actually be the better choice? Explain your reasoning.

Show answer

For n = 1,000,000, Function A creates a list of one million integers all at once, which could consume tens of megabytes of RAM. Function B yields one integer at a time, using essentially constant memory regardless of n. However, Function A would be the better choice if you need to access items by index (e.g., get the 500,000th power directly), iterate over the sequence multiple times, or perform operations like sorting, reversing, or slicing. A list supports all of these operations, while a generator can only be consumed once and only in sequential order. The lesson here is that generators are not universally superior — they are a tool optimized for sequential, single-pass processing of large or infinite sequences. Choosing between them requires understanding how the data will be used downstream.

SUMMARY

Lesson Summary

Summary

A generator function is any Python function that contains the yield keyword. When called, it returns a generator object that produces values lazily — one at a time, on demand. Each yield pauses the function and saves its state; each call to next() resumes execution from exactly where it left off. When the function body ends, StopIteration is raised automatically.

Generators shine when working with large datasets or infinite sequences because they use O(1) memory regardless of how many values they produce. Common patterns include finite sequences, infinite streams, and filter pipelines. The concepts behind yield directly connect to advanced topics like coroutines and async/await in modern Python.