Python Why Lazy Iteration Is Useful for Large Files

Keep large-file processing memory-bounded by reading one line at a time.

What You Will Learn

A file object is iterable, so for line in file consumes lines incrementally instead of building one string containing the entire file. Generator expressions can filter or transform a stream lazily. Keep the file open while consuming it and use a with block for cleanup.

Many iterator tools are lazy. Creating the iterator does not necessarily perform the work; advancing it can consume the original source.

Step-by-Step Example

The example below makes the behavior visible with small input values that you can change safely.

from io import StringIO

source = StringIO("INFO ready\nERROR disk\nINFO retry\nERROR network\n")
errors = (line.strip() for line in source if line.startswith("ERROR"))
for error in errors:
    print(error)

How the Example Works

• The imports provide the standard-library tools used in the demonstration: io.
• The variable source holds the result of StringIO('INFO ready\nERROR disk\nINFO retry\nERROR network\n'). Later lines use that value or inspect how it changes.
• The variable errors holds the result of (line.strip() for line in source if line.startswith('ERROR')). Later lines use that value or inspect how it changes.
• The loop demonstrates when each value is processed or when cleanup and exception handling take effect.

Expected output from the example:

ERROR disk
ERROR network

When You Would Use This

This pattern is useful in readers for CSV files, event streams, or paginated API results, where collecting everything in memory is unnecessary. Because iterator state advances when values are consumed, decide whether you need a reusable collection or a one-time pass. Keep files and connections open until their lazy consumers finish.

Practice: Modify the Code

Change the filter to handle WARN and ERROR prefixes and count matching lines.

Verify the changed output against the behavior described above, including the relevant invalid or boundary case.

For complete parameters and version details, consult the official Python documentation.

Common Beginner Mistakes

• Using readlines() before filtering a huge log file and needlessly storing every line.
• Returning a lazy file-based generator after closing the file it reads.

Key Takeaways

• Keep large-file processing memory-bounded by reading one line at a time.
• A file object is iterable, so for line in file consumes lines incrementally instead of building one string containing the entire file.
• Apply this operation only after checking the assumptions and input conditions shown in the example.