Python Build a Lazy CSV Processing Pipeline

Build a lazy CSV processing pipeline to read and aggregate records.

What You Will Learn

csv.DictReader produces a dictionary per input row. A generator can validate and transform one row at a time, reducing peak memory for large input files. Always keep the input file open while consuming its generator.

For production inputs, validate missing columns and handle invalid numeric fields explicitly.

Many iterator tools are lazy. Creating the iterator does not necessarily perform the work; advancing it can consume the original source.

Step-by-Step Example

The example below makes the behavior visible with small input values that you can change safely.

import csv
from io import StringIO
source = StringIO('name,amount\nAva,12\nLeo,18\nMia,20\n')
reader = csv.DictReader(source)
def amounts(rows):
    for row in rows:
        yield int(row['amount'])
print(sum(amounts(reader)))

How the Example Works

The imports provide the standard-library tools used in the demonstration: csv, io.
The function amounts defines reusable behavior; its body executes when it is called, not when it is declared.
The variable source holds the result of StringIO('name,amount\nAva,12\nLeo,18\nMia,20\n'). Later lines use that value or inspect how it changes.
The variable reader holds the result of csv.DictReader(source). Later lines use that value or inspect how it changes.

For the chosen input, the expression sum(amounts(reader)) prints 50. Compare that result with the rule above before you change the argument.

Expected output from the example:

When You Would Use This

This pattern is useful in readers for CSV files, event streams, or paginated API results, where collecting everything in memory is unnecessary. Because iterator state advances when values are consumed, decide whether you need a reusable collection or a one-time pass. Keep files and connections open until their lazy consumers finish.

Practice: Modify the Code

Use csv.DictReader on a real file and calculate totals without loading all rows.

Verify the changed output against the behavior described above, including the relevant invalid or boundary case.

For complete parameters and version details, consult the official Python documentation.

Common Beginner Mistakes

Returning a generator from inside a closed file context and then trying to read from it.
Assuming the operation has the same behavior for missing values, empty inputs and regular inputs.

Key Takeaways

Build a lazy CSV processing pipeline to read and aggregate records.
csv.DictReader produces a dictionary per input row.
For production inputs, validate missing columns and handle invalid numeric fields explicitly.