# Iterators & generators

> Understand the iteration protocol, write generators with yield, build lazy pipelines that process huge data in constant memory, and use itertools.

Python roadmap · Core language · 16 min read · https://levelcode.org/roadmaps/python/iterators-and-generators

Every `for` loop in Python runs on one small protocol, and once you understand it you can make anything loopable, process files far larger than your memory, and chain transformations together without building intermediate lists. This lesson explains iterables and iterators, generator functions with `yield`, lazy pipelines, and the `itertools` module that provides the standard building blocks.

## Iterables and iterators

An **iterable** is anything you can loop over: lists, strings, dicts, files, ranges. An **iterator** is the object that actually produces the values, one at a time, and remembers where it is. `for` gets an iterator from the iterable with `iter()`, then calls `next()` until the iterator signals it's done:

```python
colours = ["red", "green"]
it = iter(colours)
print(next(it))     # red
print(next(it))     # green
print(next(it))     # raises StopIteration
```

A `for` loop is exactly this, with the `StopIteration` handled for you:

```python
it = iter(colours)
while True:
    try:
        colour = next(it)
    except StopIteration:
        break
    print(colour)
```

Two consequences matter in practice:

- A list can be iterated many times, because each `for` loop asks it for a fresh iterator.
- An **iterator is consumed** as you use it. Loop over it a second time and you get nothing:

```python
squares = (n * n for n in range(3))   # a generator expression is an iterator
print(list(squares))   # [0, 1, 4]
print(list(squares))   # []  already exhausted
```

That second empty list is a classic source of confusing bugs. If you need the values twice, store them in a list.

## Generator functions

Writing an iterator class by hand (with `__iter__` and `__next__`) works but is verbose. A **generator function** does the same with `yield`:

```python
def countdown(start: int):
    print("starting")
    n = start
    while n > 0:
        yield n
        n -= 1
    print("done")


for x in countdown(3):
    print(x)
```

```text
starting
3
2
1
done
```

Calling `countdown(3)` doesn't run the body. It returns a **generator** object. Each time the loop asks for the next value, the function runs until it hits `yield`, hands over that value, and **pauses** with all its local variables intact. The next request resumes right after the `yield`. When the function returns, iteration stops.

That pausing is what makes generators powerful: they produce values **on demand**, one at a time, so they never need to hold all of them in memory.

```python
def fibonacci():
    a, b = 0, 1
    while True:          # an infinite sequence is fine: values are produced only when asked for
        yield a
        a, b = b, a + b


from itertools import islice
print(list(islice(fibonacci(), 10)))   # [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]
```

Type hint generators with `Iterator[T]` (or `Generator[T, None, None]` when you need the extra details) from `collections.abc`.

## Lazy pipelines: processing big data in constant memory

Suppose you have a 20 GB web server log and want the ten most requested paths that returned errors. Reading it into a list would need 20 GB of RAM. A chain of generators processes one line at a time:

```python
from collections import Counter
from collections.abc import Iterable, Iterator


def read_lines(path: str) -> Iterator[str]:
    with open(path, encoding="utf-8") as f:
        for line in f:             # files are already lazy iterators over lines
            yield line.rstrip("\n")


def parse(lines: Iterable[str]) -> Iterator[dict[str, str]]:
    for line in lines:
        parts = line.split()
        if len(parts) < 9:
            continue               # skip malformed lines
        yield {"path": parts[6], "status": parts[8]}


def errors_only(records: Iterable[dict[str, str]]) -> Iterator[dict[str, str]]:
    return (r for r in records if r["status"].startswith("5"))


records = errors_only(parse(read_lines("access.log")))
top = Counter(r["path"] for r in records).most_common(10)
```

Nothing happens until `Counter` starts pulling values. Then each line flows through the whole pipeline (read, parse, filter, count) before the next line is read. Memory use stays constant whether the file is 2 KB or 20 GB, and each stage is a small function you can test on its own with a list of sample lines.

`yield from` delegates to another iterable, which keeps flattening and recursive generators short:

```python
def read_many(paths: list[str]) -> Iterator[str]:
    for path in paths:
        yield from read_lines(path)     # same as: for line in read_lines(path): yield line
```

> [!NOTE]
> A generator that opens a file inside `with` keeps the file open while it's paused. If the consumer stops early (a `break`), Python closes the generator, and the file, when the generator is garbage-collected, which in CPython is immediately. For long-lived generators, call `.close()` explicitly or wrap the work in a `with contextlib.closing(gen)` block.

## The itertools toolbox

The `itertools` module has fast, lazy building blocks for common iteration patterns. The ones worth knowing:

```python
from itertools import batched, chain, count, groupby, islice, pairwise, product

list(islice(count(start=1), 5))            # [1, 2, 3, 4, 5]  count() is infinite
list(chain([1, 2], (3, 4), "ab"))          # [1, 2, 3, 4, 'a', 'b']
list(pairwise([1, 4, 9, 16]))              # [(1, 4), (4, 9), (9, 16)]   (3.10+)
list(batched(range(7), 3))                 # [(0, 1, 2), (3, 4, 5), (6,)] (3.12+)
list(product("ab", [1, 2]))                # [('a', 1), ('a', 2), ('b', 1), ('b', 2)]
```

`pairwise` is great for differences between neighbours (`[b - a for a, b in pairwise(readings)]`). `batched` sends work in chunks, such as inserting rows 1,000 at a time.

`groupby` groups **consecutive** items with the same key, so sort first unless the data is already ordered:

```python
from itertools import groupby

orders = [("ana", 30), ("ben", 12), ("ana", 5), ("ben", 8), ("cy", 40)]
orders.sort(key=lambda o: o[0])
for customer, group in groupby(orders, key=lambda o: o[0]):
    print(customer, sum(amount for _, amount in group))
```

```text
ana 35
ben 20
cy 40
```

Forgetting to sort before `groupby` is the most common mistake with it: unsorted input produces several separate groups for the same key.

## When to be lazy and when not to

Lazy iteration isn't always better:

| Use a generator or iterator when | Use a list when |
|---|---|
| The data is large or unbounded (files, streams, infinite sequences) | The data is small and you need it several times |
| You might stop early (`break`, `any`, the first match) | You need `len()`, indexing or slicing |
| Each stage is a step in a pipeline | You'll sort it or search it repeatedly |
| Producing each value is expensive and not all are needed | You want to inspect it while debugging |

Many standard functions accept any iterable and return iterators already: `map`, `filter`, `zip`, `enumerate`, `reversed`, `dict.items()`. Wrap in `list(...)` when you need a list.

## Practice

You have a large CSV of sensor readings, one per line, in time order: `timestamp,sensor_id,value`. Write a pipeline of generators that:

1. Reads the file lazily and skips the header and blank lines.
2. Parses each line into a tuple `(timestamp: str, sensor: str, value: float)`, skipping lines whose value isn't a number.
3. Keeps only readings for a given sensor.
4. Yields the change from the previous reading of that sensor (`value - previous_value`), together with the timestamp.
5. Finally, prints the first 5 changes larger than 10 in absolute value, without processing the rest of the file.

Test it on a small file you create, then generate a file with a million lines and confirm memory stays flat.

**Solution**

```python
# readings.py
import random
from collections.abc import Iterable, Iterator
from itertools import islice, pairwise


def read_rows(path: str) -> Iterator[str]:
    with open(path, encoding="utf-8") as f:
        next(f, None)                      # skip the header
        for line in f:
            if line.strip():
                yield line.rstrip("\n")


def parse(lines: Iterable[str]) -> Iterator[tuple[str, str, float]]:
    for line in lines:
        timestamp, sensor, raw = line.split(",")
        try:
            yield timestamp, sensor, float(raw)
        except ValueError:
            continue


def for_sensor(readings: Iterable[tuple[str, str, float]], sensor_id: str) -> Iterator[tuple[str, float]]:
    return ((ts, value) for ts, sensor, value in readings if sensor == sensor_id)


def changes(readings: Iterable[tuple[str, float]]) -> Iterator[tuple[str, float]]:
    for (_, prev), (ts, curr) in pairwise(readings):
        yield ts, curr - prev


def make_sample(path: str, n: int) -> None:
    with open(path, "w", encoding="utf-8") as f:
        f.write("timestamp,sensor_id,value\n")
        value = 20.0
        for i in range(n):
            value += random.uniform(-6, 6)
            f.write(f"t{i},{random.choice(['a', 'b'])},{value:.2f}\n")


if __name__ == "__main__":
    make_sample("readings.csv", 1_000_000)
    pipeline = changes(for_sensor(parse(read_rows("readings.csv")), "a"))
    big = (c for c in pipeline if abs(c[1]) > 10)
    for ts, delta in islice(big, 5):
        print(f"{ts}: {delta:+.2f}")
```

`islice(big, 5)` pulls only until five large changes are found, so the program typically reads a few hundred lines of the million and stops. Each stage holds at most one or two readings in memory, so a 10-million-line file uses the same memory as a 10-line one. `pairwise` turns "compare with the previous item" into a one-liner without managing a `previous` variable by hand.

## Key takeaways

- `for` calls `iter()` on an iterable and `next()` on the iterator until `StopIteration`; iterators are consumed as you use them.
- A generator function with `yield` pauses and resumes, producing values on demand.
- Chains of generators form lazy pipelines that process arbitrarily large data in constant memory, one item at a time.
- `itertools` provides `islice`, `chain`, `pairwise`, `batched`, `groupby` (sort first) and more.
- Use lazy iteration for large, unbounded or early-exit work; use lists when you need the data more than once, its length or random access.

## Quiz

1. gen = (x * 2 for x in range(3)); print(sum(gen)); print(sum(gen)). What is printed?

   A. 6 then 6
   B. 6 then 0
   C. 6 then an error
   D. 0 then 6

   **Answer: B.** A generator expression is an iterator. The first sum consumes it entirely, so the second sum sees no values and returns 0.

2. What happens when you call a generator function such as countdown(3)?

   A. The whole body runs and returns a list
   B. The body runs until the first yield
   C. It returns a generator object without running the body yet
   D. It raises StopIteration

   **Answer: C.** The body starts running only when the first value is requested, and pauses at each yield.

3. Why can a pipeline of generators process a 20 GB log file on a laptop with 8 GB of RAM?

   A. Generators compress data
   B. Each line flows through all stages before the next is read, so only a few lines are in memory at once
   C. Python streams files to disk automatically
   D. Generators run on the GPU

   **Answer: B.** Lazy stages pull one item at a time from the stage before, so memory depends on the size of one item, not the whole file.

4. groupby returns two separate groups for the same customer. What's the likely cause?

   A. The input wasn't sorted by the grouping key
   B. groupby only works with numbers
   C. The key function returned None
   D. groupby needs a list, not a tuple

   **Answer: A.** groupby groups consecutive items with the same key. Sort by the same key first so each key's items are adjacent.