Python iterators & generators: yield, genexpr & itertools

Learn what a generator really is and when to reach for one: the iterator protocol, yield vs return, generator expressions, itertools, and chaining small generators into a lazy pipeline that streams a large file with almost no memory.

Understand and use Python generators: the iterator protocol (iter/next), yield vs return, generator expressions vs lists, the itertools staples (islice, chain, groupby, accumulate), chaining small generators into a lazy data pipeline that streams a large file one row at a time, the concrete memory win over a list, and a short note on async generators.

TipKey takeaways

• A generator produces values one at a time, on demand — it never holds them all at once. That is the whole point: you can iterate a billion items, or an endless stream, in a few hundred bytes, because only the current value is alive.
• yield turns a function into a generator. Unlike return, yield pauses the function and remembers where it was — the next value resumes right after the yield, local variables intact. A generator is single-use: once iterated to the end, it is exhausted (re-create it to iterate again).
• A generator expression is a lazy list comprehension. Swap the [...] for (...) — (x*x for x in data) — and nothing computes until you iterate. A million squares held as a list weigh in at ~40 MB; the same values as a generator are a flat 200-byte recipe.
• itertools is the standard toolbox for iterators. islice (slice without materializing), chain (concatenate streams), groupby (runs of equal keys), accumulate (running totals), and the infinite count/cycle/repeat — all lazy, all composable.
• Chain small generators into a data pipeline. read → filter → parse → summarize, each a tiny generator, so rows flow through one at a time — the honest way to process a file too big to load. yield from delegates to a sub-generator in one line.

Introduction

Some data doesn’t fit in memory, and some data never ends. A 40-GB log file, a live sensor feed, every row returned by a database cursor, an infinite sequence of IDs — you can’t load these into a list first and then work on them. You have to process them one item at a time, pulling the next value only when you’re ready for it and letting the previous one go. That is exactly what a generator does, and yield is the keyword that makes one.

This lesson is the practical picture, in the order it’s useful. First the iterator protocol — the iter()/next() machinery that every for loop already runs on, so you understand what a generator is. Then yield: how a generator function pauses and resumes, and why that lets it keep state without storing everything. Then generator expressions — the lazy cousin of a list comprehension — with the memory win shown in real numbers. Then the itertools staples you’ll reach for weekly. Then the payoff: chaining small generators into a lazy pipeline that streams a real file, one row at a time. We close with the concrete memory/speed trade-off and a short look at async generators.

The examples are deliberately small and self-contained — ranges, streams, a few records — because that is how generators are best understood. For the “process a file without loading it all” section we stream the shipped penguins.csv (344 rows of Palmer penguin measurements) row by row, which is the same idiom you’d use on a file a thousand times larger. There’s nothing to download and no network at render. This sits on top of Python Comprehensions — a generator expression is one syntax change away from a list comprehension — and opens the Python Intermediate series. Every result below was produced by the code shown, and the blocks build in order. To experiment, edit the Try it live cell near the end and press Run.

The iterator protocol: what a for loop actually does

Before generators, understand the thing they are. Python draws a line between an iterable (something you can loop over — a list, a string, a file) and an iterator (the object that actually produces the values, one at a time, and remembers its position). A for loop is sugar over two built-ins: iter() gets an iterator from the iterable, and next() pulls the next value until there are none left.

nums = [10, 20, 30]
it = iter(nums)          # get an iterator from the list

print(next(it))          # pull values one at a time
print(next(it))
print("is it its own iterator?", iter(it) is it)

10
20
is it its own iterator? True

iter(nums) handed back an iterator; each next(it) produced the next value — 10, then 20. The last line prints True: an iterator is its own iterator, which is the rule that lets you drop one straight into a for loop. When next() runs past the end it raises StopIteration, and that exception is precisely how a for loop knows to stop — you never see it because the loop catches it for you.

You can build an iterator by hand by implementing the protocol — __iter__ (return the iterator) and __next__ (produce the next value or raise StopIteration). Here is a countdown written the long way:

class Countdown:
    def __init__(self, start):
        self.n = start

    def __iter__(self):
        return self                 # I am my own iterator

    def __next__(self):
        if self.n <= 0:
            raise StopIteration     # signal "no more values"
        value = self.n
        self.n -= 1
        return value

print(list(Countdown(3)))

[3, 2, 1]

list() drove the protocol for us — calling __next__ until StopIteration — and collected [3, 2, 1]. It works, but look how much ceremony it took: a class, two dunder methods, manual state (self.n), and an explicit StopIteration. That is the boilerplate yield erases.

yield: a generator function

Put the word yield anywhere in a function and it stops being an ordinary function: calling it now returns a generator — an iterator that runs your function’s body lazily. Where return sends back one value and ends the function for good, yield sends back a value and pauses, freezing every local variable exactly where it is; the next next() resumes on the line right after the yield. That pause-and-resume is what lets a generator keep state across values without storing them.

The entire Countdown class collapses to three lines:

def countdown(n):
    while n > 0:
        yield n                     # hand back n, then pause here
        n -= 1                      # resumes here on the next call

print(list(countdown(3)))

[3, 2, 1]

Same result — [3, 2, 1] — no class, no __next__, no StopIteration (running off the end of the function raises it automatically). The local n survives across yields because the function is paused, not restarted: it yields 3, pauses; resumes, n becomes 2, yields 2; and so on until the while condition fails.

One property surprises everyone once: a generator is single-use. It produces its values exactly once, and then it’s spent — there’s nothing to rewind.

g = countdown(3)
print("first pass: ", list(g))
print("second pass:", list(g))

first pass:  [3, 2, 1]
second pass: []

The first pass drained it to [3, 2, 1]; the second pass got [], because the generator was already exhausted. This is the number-one generator gotcha (we return to it in Common issues): if you need the values twice, either re-create the generator (countdown(3) again) or materialize them once into a list. The upside of being single-use is exactly the memory win — a generator never has to remember what it already produced.

Generator expressions: a lazy comprehension

You don’t need a full function for simple cases. A generator expression is a list comprehension with round brackets instead of square ones — and that one change makes it lazy: it computes nothing up front, producing each value only as you ask for it.

squares_gen = (x * x for x in range(1_000_000))   # note: (...) not [...]

print("type:", type(squares_gen).__name__)
print("first two:", next(squares_gen), next(squares_gen))

type: generator
first two: 0 1

Despite naming a million squares, the object is just a generator — building it was instant and free, because none of the squaring has happened yet. Only when we call next() does it compute the first value (0), then the next (1). A list comprehension [x*x for x in range(1_000_000)] would have computed and stored all million right there; the generator holds a recipe, not the results. The size difference is not subtle:

import sys

gen = (x * x for x in range(1_000_000))
lst = [x * x for x in range(1_000_000)]

print("generator object:", sys.getsizeof(gen), "bytes")
print("list object:     ", sys.getsizeof(lst), "bytes")

generator object: 200 bytes
list object:      8448728 bytes

The generator is 200 bytes regardless of how many values it will eventually produce — it’s a fixed-size recipe. The list is 8,448,728 bytes (~8 MB): a million integer references, all held at once. When you only need to consume the values in order (sum them, filter them, feed them somewhere), the generator gives the same answer for a rounding error’s worth of memory.

And when a generator expression is the only argument to a function, you can drop the extra parentheses entirely — it reads like plain English:

total = sum(x for x in range(100_000))
print(total)

4999950000

sum(x for x in range(100_000)) streams the numbers straight into sum without ever building the list of 100,000 — the running total is the only thing in memory — and prints 4999950000.

Which one when? Use a list comprehension when you need the results more than once, need to index or slice them (result[5]), or need len(). Use a generator expression when you’ll iterate the values exactly once and want to avoid holding them all — especially when there are a lot of them, or you’re passing them straight to sum/min/max/any/all/"".join.

itertools: the iterator toolbox

The standard-library itertools module is a set of fast, lazy building blocks for iterators. A handful come up constantly:

Tool	What it does	A one-line use
islice(it, n)	Take the first n (or a slice) without materializing	islice(huge_stream, 10) — peek at the first 10 rows
chain(a, b, …)	Concatenate several iterables into one stream	chain(file1, file2) — treat many files as one
groupby(it)	Group consecutive equal keys into runs	Collapse a sorted log into runs per status
accumulate(it)	Emit a running total (or any binary reduction)	A cumulative-sum column
count(start, step)	An infinite counter	Generate IDs 1, 2, 3, …
cycle(it)	Repeat an iterable forever	Round-robin colours across chart series
repeat(x, n)	The same value n times (or forever)	A constant column of length n
tee(it, n)	Split one iterator into n independent ones	Read a stream twice without re-reading the source

Four of them, run together:

import itertools

# islice a slice out of an INFINITE counter — no list is ever built
print("islice:    ", list(itertools.islice(itertools.count(10, 5), 4)))
print("chain:     ", list(itertools.chain([1, 2], [3, 4])))
print("accumulate:", list(itertools.accumulate([1, 2, 3, 4])))
print("groupby:   ", [(k, list(g)) for k, g in itertools.groupby("aabbbc")])

islice:     [10, 15, 20, 25]
chain:      [1, 2, 3, 4]
accumulate: [1, 3, 6, 10]
groupby:    [('a', ['a', 'a']), ('b', ['b', 'b', 'b']), ('c', ['c'])]

Read the outputs:

• islice pulled the first four values from count(10, 5) — an infinite counter starting at 10, stepping by 5 — giving [10, 15, 20, 25] and then stopping. islice is how you take a finite bite out of an endless (or huge) source without ever materializing it.
• chain glued [1, 2] and [3, 4] into a single stream — [1, 2, 3, 4] — lazily, so concatenating two 10-GB files costs nothing until you read.
• accumulate turned [1, 2, 3, 4] into its running totals — [1, 3, 6, 10] (1, 1+2, 1+2+3, 1+2+3+4).
• groupby walked "aabbbc" and grouped consecutive equal characters into runs — [(‘a’, [‘a’, ‘a’]), (‘b’, [‘b’, ‘b’, ‘b’]), (‘c’, [‘c’])]. The word consecutive is the catch: groupby only groups adjacent equal keys, so you almost always sort by the key first (unlike SQL’s GROUP BY, which sorts for you).

The infinite ones — count, cycle, repeat — must always be bounded by something like islice or a break, or they’ll spin forever. That is a feature: a generator can represent an endless sequence precisely because it never tries to build it.

Lazy data pipelines

Here is where generators earn their keep. Real data work is a pipeline: read a source, parse each record, filter to the ones you care about, then summarize. If you write each stage as a small generator, the stages compose, and data flows through the whole chain one record at a time — the source is never fully in memory. That’s how you process a file far bigger than your RAM.

We’ll answer a question about the penguins — what is the mean body mass of Gentoo penguins? — by streaming the CSV instead of loading it. Three tiny generators, each doing one job:

import csv

def read_rows(path):
    """Yield one dict per CSV row — the file stays open, one row in memory."""
    with open(path, newline="") as f:
        yield from csv.DictReader(f)      # delegate to the reader's iterator

def only_species(rows, name):
    """Pass through just the rows for one species."""
    for row in rows:
        if row["species"] == name:
            yield row

def field_as_float(rows, field):
    """Pull one numeric field out of each row, skipping blanks."""
    for row in rows:
        value = row[field]
        if value:                         # skip missing/blank cells
            yield float(value)

Each function takes an iterable and yields an iterable, so they clip together like pipe sections. Note yield from in read_rows: it delegates to csv.DictReader’s own iterator, yielding each of its rows in turn — one line instead of a for row in reader: yield row loop. Now connect them and consume the stream:

rows    = read_rows("penguins.csv")
gentoos = only_species(rows, "Gentoo")
masses  = field_as_float(gentoos, "body_mass_g")

count = 0
total = 0.0
for m in masses:            # nothing has been read yet — this loop pulls it
    count += 1
    total += m

print("Gentoo penguins:", count)
print("mean body mass: ", round(total / count, 1), "g")

Gentoo penguins: 123
mean body mass:  5076.0 g

123 Gentoo penguins with a recorded body mass, averaging 5076.0 g. (A 124th Gentoo has a blank mass, which the field_as_float step skips — streaming lets you drop bad rows as they pass.) The important part is when the work happened: building rows, gentoos, and masses read nothing — generators are lazy, so the file wasn’t touched until the for loop started pulling. Then each iteration drew exactly one row all the way through the chain (read → filter → parse → add), and let it go. At no moment was more than a single row in memory. Swap penguins.csv for a 50-GB file and this code is unchanged and still fits in a few kilobytes — that is the streaming/ETL pattern generators are built for.

To count the whole file the same lazy way, feed the reader straight into sum:

print("total rows:", sum(1 for _ in read_rows("penguins.csv")))

total rows: 344

344 rows, counted by streaming a 1 per row into sum — the file is read once, a row at a time, and never assembled into a list.

Performance and memory

The trade-off is worth stating plainly, because it decides when a generator is the right tool. To see it, sum a million squares two ways and measure the memory footprint of each. For the list we count the container and the million integer objects it points to (sys.getsizeof on the list alone only sees the pointer array); the generator is measured as-is:

import sys

# Eager: a real list holds the container PLUS a million integer objects
squares_list = [x * x for x in range(1_000_000)]
list_total = sys.getsizeof(squares_list) + sum(sys.getsizeof(v) for v in squares_list)

# Lazy: the generator is a fixed-size recipe, whatever the range
squares_gen = (x * x for x in range(1_000_000))
gen_total = sys.getsizeof(squares_gen)

total_eager = sum(squares_list)
total_lazy = sum(x * x for x in range(1_000_000))

print("same answer:", total_eager == total_lazy, "->", total_lazy)
print(f"list  footprint: {list_total:>12,} bytes  ({list_total / 1e6:.1f} MB)")
print(f"generator:       {gen_total:>12,} bytes")
print(f"the list holds ~{list_total // gen_total:,}x more")

same answer: True -> 333332833333500000
list  footprint:   40,317,656 bytes  (40.3 MB)
generator:                200 bytes
the list holds ~201,588x more

Identical result — 333332833333500000 both ways — but the list’s true footprint is 40.3 MB: the ~8 MB pointer array we saw earlier plus the million integer objects those pointers reference. The generator is a flat 200 bytes no matter how many values it will eventually produce — over 200,000× smaller — because it never allocates the values at all; it holds only the paused state that knows how to compute the next one. On this scale the memory difference is the entire story; the compute is essentially the same either way.

So when is each right?

• Reach for lazy (a generator) when the data is large or unbounded, you only need to pass through it once, or you want to start producing results immediately — a generator yields its first value before it has computed the second, so a pipeline can begin emitting output before the source is fully read (great for streaming and responsiveness).
• Reach for eager (a list) when you need random access (data[100]), need to iterate the values more than once, need len(), or the dataset is small and the convenience wins. Materializing is a feature when you’ll reuse the results.

A generator is not automatically faster — per-item overhead is similar, and if you’re going to build a list at the end anyway, a list comprehension is usually a touch quicker. The win is memory and latency, not raw throughput. Choose lazy to fit big or endless data in constant space and to start work sooner; choose eager when you need the whole collection sitting there to reuse.

Async generators

When the source is asynchronous — rows arriving from a network socket, pages from a paginated API, events from a queue — the same idea has an async form. An async generator is an async def function that uses yield, and you consume it with async for instead of a plain for. It lets you await between items (so the program can do other work while waiting for the next value) while still streaming one item at a time.

import asyncio

async def read_stream(n):
    """Pretend to pull from an async source (a socket, an API page)."""
    for i in range(n):
        await asyncio.sleep(0)     # 'await' the next chunk without blocking
        yield i * 10               # ... then yield it, one at a time

async def main():
    async for value in read_stream(3):   # note: async for
        print(value)

asyncio.run(main())                # prints 0, 10, 20

The shape is the ordinary generator you already know — a loop with a yield — with two async/await markers layered on: async def + yield makes it an async generator, and async for drives it. You’d reach for one to stream from an async I/O source without buffering the whole response, exactly like the sync pipeline above but for awaitable data. The concurrency mechanics (asyncio, the event loop, await) are a topic of their own — a later lesson in this series covers asynchronous programming; here it’s enough to recognize the pattern and know that generators extend cleanly into the async world.

Try it live

The blocks above ran at build time. Edit this one and press Run to build a lazy pipeline yourself — it executes in your browser via Pyodide (no install). Generators are pure Python, so this runs instantly with nothing to download.

🟢 With an AI agent

Got a file too big to load, or a loop that builds a giant list just to sum it? Paste your code and ask Prova “rewrite this to stream with a generator instead of loading everything” — or “turn this read/filter/summarize into a chained generator pipeline”. It writes the yield functions, wires the stages together, and swaps the memory-hungry list comprehension for a lazy generator expression. Because the runtime is right here, you run it and watch the memory stay flat, instead of trusting that the rewrite still gives the same answer. The runtime is the judge. Ask Prova →

The same idea in R

Working across both languages? R leans on functional tools rather than a yield keyword. Where Python chains generators, R chains Filter(), Map(), and Reduce() (base R) — or the tidyverse purrr package (map(), keep(), reduce()) — to express the same read → filter → transform → summarize flow. R doesn’t have first-class lazy generators in the same way (its laziness is in promise evaluation of function arguments), so for truly large data R users typically stream with a chunked reader (readr::read_csv_chunked()) or a database connection via dbplyr. The concept carries across: keep each step small, compose them, and avoid materializing the whole dataset when you only need to pass through it once. See the Programming pillar for the R side.

Common issues

A generator is empty the second time you use it. Generators are single-use — once iterated to the end they’re exhausted, and re-iterating yields nothing (you saw list(g) return [] on the second pass). If you need the values more than once, either re-create the generator by calling the function again (countdown(3)), or materialize them once into a list (values = list(gen)) and reuse the list. Watch for the sneaky version: calling sum(gen) then max(gen) gives a wrong max of an empty stream, because sum already drained it.

Nothing happens — your generator “doesn’t run”. A generator (or generator expression) is lazy: defining it or calling the function does zero work; the body only executes as something pulls values with next(), a for loop, list(), sum(), etc. If your prints or side effects never fire, you probably built the generator but never iterated it. Wrap it in list(...) (or loop over it) to force it.

TypeError: object of type 'generator' has no len() — or it won’t index. A generator has no length and no positions, because its values don’t exist yet — so len(gen) and gen[0] both raise TypeError (‘generator’ object is not subscriptable). If you genuinely need a length or random access, materialize first (data = list(gen)), or grab a slice lazily with itertools.islice(gen, start, stop) instead of gen[start:stop].

A return inside a generator stops it — it doesn’t return a value. In a generator function, return doesn’t hand back a result; it just ends the generation (raising StopIteration). So return x mid-generator terminates the stream early and the x is discarded from normal iteration (it’s tucked into the StopIteration value, which you rarely read). Use yield to emit values and a bare return (or falling off the end) only to stop.

Frequently asked questions

NoteWhat is the difference between yield and return in Python?

return ends a function and sends back a single value; call the function again and it starts over from the top. yield turns the function into a generator: it hands back a value and pauses, freezing all local state, and the next request resumes on the line after the yield. So return gives you one result, while yield lets a function produce a whole sequence of results lazily, one at a time, remembering where it left off between them. Inside a generator, a bare return doesn’t return a value — it just stops the generation.

NoteWhen should I use a generator instead of a list?

Use a generator when the data is large or unbounded, when you’ll iterate it only once, or when you want results to start flowing immediately (streaming a file, a network feed, a pipeline). It keeps memory flat — only one value is alive at a time. Use a list when you need the values more than once, need random access (data[i]) or len(), or the dataset is small enough that convenience wins. Rule of thumb: pass-through-once and possibly-huge → generator; reuse or index → list.

NoteWhat is the difference between an iterator and a generator in Python?

An iterator is any object implementing the iterator protocol — __iter__ (returns itself) and __next__ (returns the next value or raises StopIteration). A generator is the easy way to make one: any function containing yield returns a generator, which is an iterator whose __next__ is written for you by the pause/resume of yield. So every generator is an iterator, but not every iterator is a generator (you can also write one by hand as a class). In practice you almost always create iterators with yield or a generator expression rather than the class boilerplate.

NoteWhat is a generator expression in Python?

A generator expression is a comprehension with round brackets instead of square: (x*x for x in data) instead of [x*x for x in data]. It produces a lazy generator — computing each value only when asked — rather than building the whole list up front, so it uses near-constant memory. When it’s the sole argument to a function you can omit the extra parentheses: sum(x*x for x in data). Prefer it over a list comprehension whenever you’ll consume the values once (feeding sum/min/max/any/all/join or a for loop) and don’t need to index or reuse them.

NoteWhat is an async generator in Python?

An async generator is an async def function that uses yield; you consume it with async for instead of a plain for. It lets you await between items — so the program can do other work while waiting for the next value — while still streaming one item at a time. Reach for one to read from an asynchronous source (a network socket, a paginated API, an event queue) without buffering the entire response in memory. It’s the asyncio counterpart of an ordinary generator; the concurrency machinery (the event loop, await) is a broader topic covered in the asynchronous-programming lesson of this series.

Test your understanding

ImportantYour turn — build a streaming pipeline

Write a lazy pipeline that streams numbers, keeps only the ones divisible by 3, squares them, and sums the result — without ever building a list of the intermediate values.

Task 1. Write a generator function multiples_of(step, n) that yields step, 2*step, … up to (but not including) n. Then, using generator expressions or small generator functions, compute the sum of the squares of every multiple of 3 below 1,000. Confirm you never materialize the numbers into a list.

Task 2. Explain in one sentence why the pipeline uses almost no memory even if you raise the limit from 1,000 to 1,000,000,000.

TipHint

For Task 1, multiples_of is a while/yield loop just like countdown. For the sum, feed a generator expression straight into sum: sum(x * x for x in multiples_of(3, 1000)) — the parentheses can be dropped because the genexpr is sum’s only argument. Nothing is stored because sum pulls one value at a time.

TipSolution

def multiples_of(step, n):
    value = step
    while value < n:
        yield value
        value += step

# Stream -> square -> sum, one value alive at a time (no list built)
answer = sum(x * x for x in multiples_of(3, 1000))
print(answer)   # 111277611

multiples_of(3, 1000) yields 3, 6, 9, … 999 one at a time; the generator expression squares each as it arrives; and sum adds them without ever holding the sequence. Task 2: memory stays flat because only the current value exists at any moment — the generator holds a recipe (its paused state), not the values, so a limit of a billion costs the same few hundred bytes as a limit of a thousand.

Quick check. You run g = (x for x in range(5)), then print(sum(g)), then print(max(g)). The sum prints 10, but max(g) raises ValueError: max() iterable argument is empty. Why?

TipShow answer

A generator is single-use. sum(g) iterated g all the way to the end and exhausted it, so by the time max(g) runs there are no values left — hence “empty sequence”. Fix it by materializing once (data = list(g), then sum(data) and max(data)) or by creating a fresh generator for each pass.

Conclusion

You now know what a generator really is and when to reach for one. A for loop runs on the iterator protocol (iter/next); yield gives you an iterator for free by pausing and resuming a function, keeping its state without storing every value; a generator expression is the lazy, near-zero-memory cousin of a list comprehension; itertools supplies the lazy building blocks (islice, chain, groupby, accumulate, and the infinite count/cycle/repeat); and chaining small generators into a pipeline lets you stream a file far larger than memory, one row at a time. Remember the two rules that trip everyone: a generator is single-use (exhausted after one pass), and it is lazy (nothing runs until you iterate). Reach for a generator when the data is big, endless, or pass-through-once; reach for a list when you need to reuse or index it.

Related lessons

• Build on this: Python Comprehensions — a generator expression is one bracket away from the list comprehension you already know · Files, CSV & JSON — reading files is where lazy streaming pays off · Data types & structures — the iterables generators produce from and feed into.
• Go further: later lessons in this Python Intermediate series build on generators — asynchronous programming (async generators and the event loop), and parallel processing (a generator feeds a worker pool cleanly). See the Programming pillar for the full set of R & Python lessons.

Was this page helpful?

Prove you can do it. Master the whole Python Intermediate series — track your path, build projects, and earn a certificate.

Start free →

Go Pro — unlimited Prova on your own data and a verifiable certificate that proves the skill.

from $15/mo billed yearly

Go Pro →

✓ You're Pro — keep going. The runtime is the judge.

Get new R & Python lessons by email

Practical, reproducible, no spam. Unsubscribe anytime.

Double opt-in. We never share your email.

Share this pageXLinkedInRedditHN