Python Parse Large Documents Incrementally

Process large structured files incrementally instead of loading everything into memory.

How It Works

Streaming is practical for JSON Lines (one complete JSON document per line), CSV rows and event logs. Ordinary JSON files containing one giant array are not automatically streamable with json.load(). If you need incremental parsing of a huge JSON array, choose a suitable dedicated streaming parser.

Streaming reduces peak memory but does not make untrusted inputs safe; enforce size limits and schema checks.

Step-by-Step Example

import json
from io import StringIO

json_lines = StringIO('{"amount": 4}\n{"amount": 7}\n{"amount": 2}\n')
total = 0
for line_number, line in enumerate(json_lines, start=1):
    record = json.loads(line)
    total += record["amount"]
print(total)

Understanding the Example

The expression 'StringIO(\'{"amount": 4}\\n{"amount": 7}\\n{"amount": 2}\\n\')' supplies the value stored as 'json_lines'. This value is then used or checked by the later statements.
The expression '0' supplies the value stored as 'total'. This value is then used or checked by the later statements.

A successful run of this example prints the following output:

Where This Fits in a Real Program

In a real import pipeline, parsing is only the first stage. Convert each field to the intended type, reject missing or unexpected values and report which record failed. Keep the original source file unchanged until the normalized data has passed validation and an export has been checked.

Practice: Try It Yourself

Skip empty lines and report the line number when a malformed record is found.

For this exercise, predict the result before running the changed code. Then explain the difference between the original and modified cases using the actual values, not just whether the program completed.

Common Beginner Mistakes

Treating JSON Lines as though the entire stream were one valid JSON document.
Reading an unbounded input stream into a single string before attempting to parse it.

Key Takeaways

Process large structured files incrementally instead of loading everything into memory.
Streaming is practical for JSON Lines (one complete JSON document per line), CSV rows and event logs.
Streaming reduces peak memory but does not make untrusted inputs safe; enforce size limits and schema checks.