Python Process Streaming Data Without Reading Everything into Memory

Combine incremental input, filtering and aggregation without loading the source into RAM.

What You Will Learn

Lazy processing is useful for logs, event streams and database cursors. Build a generator that yields validated records, then process each record once. Memory stays bounded by the largest record and any explicit buffers, although sorting or converting the stream to a list would change that behavior.

Many iterator tools are lazy. Creating the iterator does not necessarily perform the work; advancing it can consume the original source.

Step-by-Step Example

The example below makes the behavior visible with small input values that you can change safely.

from io import StringIO

log = StringIO("INFO;started\nERROR;disk\nERROR;network\n")
def error_messages(lines):
    for line in lines:
        level, message = line.rstrip("\n").split(";", 1)
        if level == "ERROR":
            yield message

for message in error_messages(log):
    print(message)

How the Example Works

The imports provide the standard-library tools used in the demonstration: io.
The function error_messages defines reusable behavior; its body executes when it is called, not when it is declared.
The variable log holds the result of StringIO('INFO;started\nERROR;disk\nERROR;network\n'). Later lines use that value or inspect how it changes.
The loop demonstrates when each value is processed or when cleanup and exception handling take effect.

Expected output from the example:

disk
network

When You Would Use This

This pattern is useful in readers for CSV files, event streams, or paginated API results, where collecting everything in memory is unnecessary. Because iterator state advances when values are consumed, decide whether you need a reusable collection or a one-time pass. Keep files and connections open until their lazy consumers finish.

Practice: Modify the Code

Add malformed lines and decide where validation should stop or report bad input.

Verify the changed output against the behavior described above, including the relevant invalid or boundary case.

For complete parameters and version details, consult the official Python documentation.

Common Beginner Mistakes

Calling list() on a huge pipeline and losing the memory benefit of streaming.
Assuming the first error occurs when the generator is created rather than when it is consumed.

Key Takeaways

Combine incremental input, filtering and aggregation without loading the source into RAM.
Lazy processing is useful for logs, event streams and database cursors.
Apply this operation only after checking the assumptions and input conditions shown in the example.