Iterators & generators

Every for loop in Python runs on one small protocol, and once you understand it you can make anything loopable, process files far larger than your memory, and chain transformations together without building intermediate lists. This lesson explains iterables and iterators, generator functions with yield, lazy pipelines, and the itertools module that provides the standard building blocks.

#Iterables and iterators

An iterable is anything you can loop over: lists, strings, dicts, files, ranges. An iterator is the object that actually produces the values, one at a time, and remembers where it is. for gets an iterator from the iterable with iter(), then calls next() until the iterator signals it's done:

colours = ["red", "green"]
it = iter(colours)
print(next(it))     # red
print(next(it))     # green
print(next(it))     # raises StopIteration

A for loop is exactly this, with the StopIteration handled for you:

it = iter(colours)
while True:
    try:
        colour = next(it)
    except StopIteration:
        break
    print(colour)

Two consequences matter in practice:

• A list can be iterated many times, because each for loop asks it for a fresh iterator.
• An iterator is consumed as you use it. Loop over it a second time and you get nothing:

squares = (n * n for n in range(3))   # a generator expression is an iterator
print(list(squares))   # [0, 1, 4]
print(list(squares))   # []  already exhausted

That second empty list is a classic source of confusing bugs. If you need the values twice, store them in a list.

#Generator functions

Writing an iterator class by hand (with __iter__ and __next__) works but is verbose. A generator function does the same with yield:

def countdown(start: int):
    print("starting")
    n = start
    while n > 0:
        yield n
        n -= 1
    print("done")


for x in countdown(3):
    print(x)

starting
3
2
1
done

Calling countdown(3) doesn't run the body. It returns a generator object. Each time the loop asks for the next value, the function runs until it hits yield, hands over that value, and pauses with all its local variables intact. The next request resumes right after the yield. When the function returns, iteration stops.

That pausing is what makes generators powerful: they produce values on demand, one at a time, so they never need to hold all of them in memory.

def fibonacci():
    a, b = 0, 1
    while True:          # an infinite sequence is fine: values are produced only when asked for
        yield a
        a, b = b, a + b


from itertools import islice
print(list(islice(fibonacci(), 10)))   # [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]

Type hint generators with Iterator[T] (or Generator[T, None, None] when you need the extra details) from collections.abc.

#Lazy pipelines: processing big data in constant memory

Suppose you have a 20 GB web server log and want the ten most requested paths that returned errors. Reading it into a list would need 20 GB of RAM. A chain of generators processes one line at a time:

from collections import Counter
from collections.abc import Iterable, Iterator


def read_lines(path: str) -> Iterator[str]:
    with open(path, encoding="utf-8") as f:
        for line in f:             # files are already lazy iterators over lines
            yield line.rstrip("\n")


def parse(lines: Iterable[str]) -> Iterator[dict[str, str]]:
    for line in lines:
        parts = line.split()
        if len(parts) < 9:
            continue               # skip malformed lines
        yield {"path": parts[6], "status": parts[8]}


def errors_only(records: Iterable[dict[str, str]]) -> Iterator[dict[str, str]]:
    return (r for r in records if r["status"].startswith("5"))


records = errors_only(parse(read_lines("access.log")))
top = Counter(r["path"] for r in records).most_common(10)

Nothing happens until Counter starts pulling values. Then each line flows through the whole pipeline (read, parse, filter, count) before the next line is read. Memory use stays constant whether the file is 2 KB or 20 GB, and each stage is a small function you can test on its own with a list of sample lines.

yield from delegates to another iterable, which keeps flattening and recursive generators short:

def read_many(paths: list[str]) -> Iterator[str]:
    for path in paths:
        yield from read_lines(path)     # same as: for line in read_lines(path): yield line

#The itertools toolbox

The itertools module has fast, lazy building blocks for common iteration patterns. The ones worth knowing:

from itertools import batched, chain, count, groupby, islice, pairwise, product

list(islice(count(start=1), 5))            # [1, 2, 3, 4, 5]  count() is infinite
list(chain([1, 2], (3, 4), "ab"))          # [1, 2, 3, 4, 'a', 'b']
list(pairwise([1, 4, 9, 16]))              # [(1, 4), (4, 9), (9, 16)]   (3.10+)
list(batched(range(7), 3))                 # [(0, 1, 2), (3, 4, 5), (6,)] (3.12+)
list(product("ab", [1, 2]))                # [('a', 1), ('a', 2), ('b', 1), ('b', 2)]

pairwise is great for differences between neighbours ([b - a for a, b in pairwise(readings)]). batched sends work in chunks, such as inserting rows 1,000 at a time.

groupby groups consecutive items with the same key, so sort first unless the data is already ordered:

from itertools import groupby

orders = [("ana", 30), ("ben", 12), ("ana", 5), ("ben", 8), ("cy", 40)]
orders.sort(key=lambda o: o[0])
for customer, group in groupby(orders, key=lambda o: o[0]):
    print(customer, sum(amount for _, amount in group))

ana 35
ben 20
cy 40

Forgetting to sort before groupby is the most common mistake with it: unsorted input produces several separate groups for the same key.

#When to be lazy and when not to

Lazy iteration isn't always better:

Many standard functions accept any iterable and return iterators already: map, filter, zip, enumerate, reversed, dict.items(). Wrap in list(...) when you need a list.

#Practice

You have a large CSV of sensor readings, one per line, in time order: timestamp,sensor_id,value. Write a pipeline of generators that:

1. Reads the file lazily and skips the header and blank lines.
2. Parses each line into a tuple (timestamp: str, sensor: str, value: float), skipping lines whose value isn't a number.
3. Keeps only readings for a given sensor.
4. Yields the change from the previous reading of that sensor (value - previous_value), together with the timestamp.
5. Finally, prints the first 5 changes larger than 10 in absolute value, without processing the rest of the file.

Test it on a small file you create, then generate a file with a million lines and confirm memory stays flat.

# readings.py
import random
from collections.abc import Iterable, Iterator
from itertools import islice, pairwise


def read_rows(path: str) -> Iterator[str]:
    with open(path, encoding="utf-8") as f:
        next(f, None)                      # skip the header
        for line in f:
            if line.strip():
                yield line.rstrip("\n")


def parse(lines: Iterable[str]) -> Iterator[tuple[str, str, float]]:
    for line in lines:
        timestamp, sensor, raw = line.split(",")
        try:
            yield timestamp, sensor, float(raw)
        except ValueError:
            continue


def for_sensor(readings: Iterable[tuple[str, str, float]], sensor_id: str) -> Iterator[tuple[str, float]]:
    return ((ts, value) for ts, sensor, value in readings if sensor == sensor_id)


def changes(readings: Iterable[tuple[str, float]]) -> Iterator[tuple[str, float]]:
    for (_, prev), (ts, curr) in pairwise(readings):
        yield ts, curr - prev


def make_sample(path: str, n: int) -> None:
    with open(path, "w", encoding="utf-8") as f:
        f.write("timestamp,sensor_id,value\n")
        value = 20.0
        for i in range(n):
            value += random.uniform(-6, 6)
            f.write(f"t{i},{random.choice(['a', 'b'])},{value:.2f}\n")


if __name__ == "__main__":
    make_sample("readings.csv", 1_000_000)
    pipeline = changes(for_sensor(parse(read_rows("readings.csv")), "a"))
    big = (c for c in pipeline if abs(c[1]) > 10)
    for ts, delta in islice(big, 5):
        print(f"{ts}: {delta:+.2f}")

#Key takeaways

• for calls iter() on an iterable and next() on the iterator until StopIteration; iterators are consumed as you use them.
• A generator function with yield pauses and resumes, producing values on demand.
• Chains of generators form lazy pipelines that process arbitrarily large data in constant memory, one item at a time.
• itertools provides islice, chain, pairwise, batched, groupby (sort first) and more.
• Use lazy iteration for large, unbounded or early-exit work; use lists when you need the data more than once, its length or random access.