Iterables & Iterators — Use iterables and iterators conceptually; use iter() and next() (intro) - Python

SECTION 1

Historical Context & Motivation

Every time you write a for loop in Python, something invisible happens under the hood. Python doesn't just magically know how to walk through a list or a string one element at a time. Instead, it relies on a powerful protocol — a set of agreed-upon rules — called the iterator protocol. Understanding this protocol unlocks a deeper knowledge of how Python actually processes sequences of data.

The idea of iterating — stepping through items one by one — is as old as programming itself. Early languages like Fortran used explicit index counters, forcing programmers to manually track their position in an array. Over the decades, languages evolved to hide that bookkeeping, and Python took the concept further than most by making iteration a core part of the language design.

1957 • Fortran DO Loops

Fortran introduced the DO loop, requiring programmers to manage index variables manually to step through arrays.

1994 • Design Patterns Published

The "Gang of Four" book formalized the Iterator design pattern, showing how to traverse collections without exposing their internal structure.

2001 • PEP 234 — Python Iterators

Python 2.2 adopted PEP 234, adding the __iter__ and __next__ methods, creating the iterator protocol that Python still uses today.

2001 • PEP 255 — Generators

Generators were introduced, allowing developers to create iterators easily using the yield keyword — building on the iterator protocol.

2008 • Python 3 Refinements

Python 3 cleaned up iteration: range(), map(), and filter() now return iterators instead of lists, saving memory.

So why should you care about the machinery behind a simple for loop? Because once you understand iterables and iterators, you can process enormous files without running out of memory, build custom sequences that generate values on demand, and debug confusing errors like StopIteration with confidence. The question is: what exactly are iterables and iterators, and how do iter() and next() connect them?

SECTION 2

Core Principles & Definitions

To understand this topic, you need to separate two closely related concepts that beginners often confuse. An iterable is any object you can loop over. An iterator is the object that does the looping — it keeps track of where you are. Think of an iterable as a playlist and an iterator as the play button that remembers which song is currently playing.

Iterable

Any object that implements __iter__() — such as lists, strings, tuples, dictionaries, and sets. You can call iter() on it to get an iterator.

Iterator

An object that implements both __iter__() and __next__(). Calling next() on it returns the next item, one at a time.

iter() Function

The built-in iter() function takes an iterable and returns an iterator. It's the bridge that converts a collection into something you can step through.

next() Function

The built-in next() function pulls the next value from an iterator. When no items remain, it raises StopIteration.

StopIteration

A special exception Python raises when an iterator has been fully exhausted — there are no more items to return. A for loop catches this silently.

✦

KEY TAKEAWAY

Think of an iterable like a book sitting on a shelf — it contains all the pages, but it doesn't track which page you're on. An iterator is like placing a bookmark in that book: it remembers your current position and can move forward to the next page when you ask. Calling iter() is like inserting the bookmark at the beginning, and calling next() is like flipping to the next page.

A critical detail: every iterator is also an iterable (because it has __iter__()), but not every iterable is an iterator. A list is iterable, but it doesn't remember a position — you need to create an iterator from it first. Once an iterator is exhausted (all items consumed), it stays empty. You'd need to create a fresh iterator from the original iterable to loop again.

SECTION 3

Visual Explanation — How for Loops Really Work

The diagram below shows exactly what happens behind the scenes when Python executes a for loop. Python first calls iter() on the iterable to get an iterator, then repeatedly calls next() until StopIteration is raised.

The iterable (left, in violet) is converted to an iterator (cyan) via iter(). Each call to next() retrieves one value (amber). When no items remain, StopIteration (red) signals the for loop to stop.

Notice that the iterable (the list) doesn't change — it still holds all three values. The iterator is a separate helper object that tracks a position pointer. Each call to next() advances that pointer forward by one. This separation is powerful because you can create multiple independent iterators from the same iterable, each tracking its own position.

SECTION 4

The Iterator Protocol — How It Works

Python's iteration system is built on two special methods (also called dunder methods because of their double underscores). Every iterable must define __iter__(), and every iterator must define both __iter__() and __next__(). The built-in functions iter() and next() are simply shortcuts that call these dunder methods for you.

The Protocol in Three Rules

RULE 1 — ITERABLE CONTRACT

iter(iterable) → returns an iterator object

Calling iter() on an iterable invokes its __iter__() method, which must return an iterator.

RULE 2 — ITERATOR CONTRACT

next(iterator) → returns next value OR raises StopIteration

Calling next() on an iterator invokes its __next__() method. It returns the next item, or raises StopIteration when exhausted.

RULE 3 — ITERATOR IS ALSO ITERABLE

iter(iterator) → returns the same iterator object (itself)

An iterator's __iter__() method returns self. This means you can pass an iterator directly to a for loop.

Here is the exact equivalent of what Python does when it runs for item in [10, 20, 30]: print(item):

ℹ

Behind the Scenes

Python translates for item in [10, 20, 30]: into roughly this: _it = iter([10, 20, 30]) → then it calls item = next(_it) repeatedly inside a while True loop, and breaks when it catches StopIteration. That's the entire mechanism!

Understanding these three rules means you can predict the behavior of any iteration in Python. You don't need calculus or complex math — it's purely a logical protocol. If an object follows these rules, Python's for loop, list() constructor, sum() function, and many other tools will work with it automatically.

SECTION 5

Common Iterables in Python

Python provides many built-in iterables. Some store all their data in memory (like lists), while others generate values one at a time (like range()). The diagram below classifies the most common ones you'll encounter.

Python iterables fall into two broad families. Sequences and collections (left, amber/pink) hold all their data and can be looped over multiple times. Lazy iterables and generators (right, cyan/green) produce values on demand and can only be traversed once.

Type	Example	Stored in Memory?	Reusable?
`list`	`[1, 2, 3]`	Yes — all elements	Yes
`str`	`"hello"`	Yes — all characters	Yes
`dict`	`{"a": 1}`	Yes — all key-value pairs	Yes
`range()`	`range(1000000)`	No — calculates on the fly	Yes
generator	`(x**2 for x in range(5))`	No — one value at a time	No — one pass only

Common Python iterable types and their memory/reuse characteristics

SECTION 6

Worked Example — Using iter() and next() by Hand

Let's manually walk through the iterator protocol using a simple list of colors. This is exactly what Python does behind the scenes when you write a for loop, but we'll do each step ourselves.

Manual Iteration Over a List

1
Step 1 — Define the Iterable

We start with a list of colors: colors = ["red", "green", "blue"]. This list is an iterable because it has an __iter__() method.

colors = ["red", "green", "blue"]
2
Step 2 — Create an Iterator

Call iter() on the list to get an iterator object. This iterator will track our position, starting before the first element.

it = iter(colors) → returns a list_iterator object
3
Step 3 — Call next() the First Time

Each call to next() retrieves the next value and advances the internal pointer.

next(it) → "red"
4
Step 4 — Call next() Again (Second and Third)

We continue calling next(it) to get the remaining values. The iterator remembers where we left off.

next(it) → "green" then next(it) → "blue"
5
Step 5 — Call next() on an Exhausted Iterator

Now there are no more items. Calling next(it) one more time raises StopIteration. In a for loop, Python catches this exception automatically and exits the loop. But here, since we called next() manually, we see the error.

next(it) → StopIteration
6
Step 6 — Start Fresh with a New Iterator

To loop again, we call iter(colors) to create a brand-new iterator. The original list hasn't changed — only the old iterator was exhausted.

it2 = iter(colors) → fresh iterator, next(it2) → "red" again

💡

Pro Tip

You can provide a default value to next() to avoid the StopIteration error: next(it, "done") returns "done" instead of raising an exception when the iterator is exhausted.

SECTION 7

for Loops vs. Manual Iteration — Strengths & Limitations

You might wonder: if for loops handle all this automatically, why learn iter() and next() at all? The answer is that manual iteration gives you finer control, and some situations genuinely require it. The table below compares the two approaches.

Feature	for Loop	Manual iter()/next()
Ease of use	Very easy — one line	More verbose — requires multiple calls
StopIteration handling	Caught automatically	You must handle it yourself
Partial consumption	Loops through all items	Can stop at any point and resume later
Multiple iterators	Creates one iterator per loop	You can create and interleave multiple iterators
Best for	Most everyday looping tasks	Reading files line by line, peeking at data, custom iteration logic

Comparing automatic for-loop iteration with manual iter()/next() usage

🔑

WHEN TO USE MANUAL ITERATION

Think of a for loop like an automatic car — great for daily driving. Using iter() and next() is like driving a manual transmission — it gives you more control over gear shifts (when to fetch the next value), which matters when you need to pause, peek ahead, or interleave multiple data streams.

SECTION 8

Connection to Generators & Advanced Iteration

The iterator protocol you've just learned is the foundation for more advanced Python features. Once you're comfortable with iter() and next(), you'll be ready to explore generators, generator expressions, and the powerful itertools module from Python's standard library.

Concept	What You Know Now	What Comes Next
Creating iterators	Call `iter()` on a list or string	Write your own iterator class with `__iter__` and `__next__`
Getting values	Call `next()` repeatedly	Use `yield` to build generator functions that produce values lazily
Chaining iterators	Loop one iterable at a time	Use `itertools.chain()`, `itertools.zip_longest()`, and more
Infinite sequences	Not yet possible with lists	Generators and `itertools.count()` can produce infinite streams of values

From beginner iteration to advanced Python patterns

The key insight is that everything in advanced Python iteration is still built on the same two functions: iter() and next(). Generators are simply a convenient shortcut for creating objects that follow the iterator protocol. By mastering the basics now, you're building the mental framework that makes these advanced tools easy to learn later.

SECTION 9

Practice Problems

Question 1: CONCEPTUAL

In your own words, explain the difference between an iterable and an iterator. Give one real-world analogy that isn't the book/bookmark or playlist example from this lesson.

Show answer

An iterable is a collection that can be looped over — it contains or can produce data. An iterator is a helper object that tracks your current position within that data and delivers items one at a time. A good analogy: an iterable is like a deck of cards (the whole collection), and an iterator is like a dealer who holds the deck and hands you one card at a time, remembering which card comes next. The deck itself doesn't track what's been dealt — the dealer does.

Question 2: BASIC CALCULATION

Given the code below, predict the output of each print statement: nums = [5, 10, 15] it = iter(nums) print(next(it)) print(next(it)) print(next(it)) print(next(it, "empty"))

Show answer

The first next(it) prints 5, the second prints 10, and the third prints 15. The fourth call uses a default value, so instead of raising StopIteration, it prints empty. The iterator was exhausted after the third value, and the default argument in next(it, "empty") gracefully handles the empty state.

Question 3: INTERMEDIATE

Write a short Python program that uses iter() and next() (not a for loop) along with a while loop to print every character in the string "Python" on its own line. Your loop should exit gracefully without an unhandled StopIteration error.

Show answer

One solution uses the default argument of next() as a sentinel: it = iter("Python") then while True: with ch = next(it, None) inside, followed by if ch is None: break and print(ch). Alternatively, you could wrap next(it) in a try/except block that catches StopIteration and breaks. Both approaches replicate exactly what a for loop does internally.

Question 4: APPLIED

Imagine you have a very large log file (millions of lines). You want to read only the first 3 lines without loading the entire file into memory. Using the iterator protocol, explain conceptually how you'd accomplish this, and write the key lines of Python code. (Hint: file objects in Python are iterators.)

Show answer

Since a file object returned by open() is already an iterator, you don't even need to call iter() on it. You can simply call next() three times: f = open("log.txt"), then line1 = next(f), line2 = next(f), line3 = next(f), and f.close(). This reads only three lines from disk, keeping memory usage tiny regardless of file size. This is a perfect example of why lazy iteration matters in real applications.

Question 5: CRITICAL THINKING

Consider this code: data = [1, 2, 3] then it = iter(data). After calling next(it) once (getting 1), you pass it into a for loop: for x in it: print(x). What does this print, and why? What does this tell you about how for loops interact with iterators that have already been partially consumed?

Show answer

The for loop prints 2 and then 3 — it does NOT print 1. When the for loop calls iter(it), the iterator returns itself (Rule 3 from the protocol). Since we already consumed the first element, the iterator remembers its position and continues from 2. This demonstrates that iterators carry state — once an item has been consumed via next(), it's gone. This behavior is crucial for patterns like "skip the header" when processing data files.

SUMMARY

Summary — Iterables & Iterators

Summary

Python's iteration system is built on a simple protocol with two key players. An iterable is any object you can loop over — lists, strings, dictionaries, tuples, and more. When you start a for loop, Python calls iter() on that iterable to create an iterator — a stateful object that remembers its current position. The loop then calls next() repeatedly to retrieve one value at a time until StopIteration signals there are no more items.

Key distinctions to remember: every iterator is an iterable, but not every iterable is an iterator. Sequences like lists store all their data and can be looped multiple times, while lazy iterables like generators produce values on demand and are exhausted after a single pass. You can call iter() and next() manually for fine-grained control — reading specific numbers of items, peeking at data, or interleaving multiple streams. This protocol is the foundation for generators and the itertools module, which you'll explore next.