Stream a large log source line by line using a generator.
What You Will Learn
Reading an open text file as an iterator processes one line at a time. A generator can yield only matching messages, leaving irrelevant rows uncollected. The source file must stay open until consumption ends.
A production log parser should handle timestamps, encodings, and partial or malformed lines.
Many iterator tools are lazy. Creating the iterator does not necessarily perform the work; advancing it can consume the original source.
Step-by-Step Example
The example below makes the behavior visible with small input values that you can change safely.
from io import StringIO
logs = StringIO('INFO started\nERROR disk full\nINFO retry\nERROR timeout\n')
def errors(lines):
for line in lines:
if line.startswith('ERROR'):
yield line.strip()
print(list(errors(logs)))How the Example Works
- The imports provide the standard-library tools used in the demonstration: io.
- The function errors defines reusable behavior; its body executes when it is called, not when it is declared.
- The variable logs holds the result of StringIO('INFO started\nERROR disk full\nINFO retry\nERROR timeout\n'). Later lines use that value or inspect how it changes.
- The print() calls make the return values or state visible so you can compare them with the explanation and check your prediction.
For the chosen input, the expression list(errors(logs)) prints ['ERROR disk full', 'ERROR timeout']. Compare that result with the rule above before you change the argument.
Expected output from the example:
['ERROR disk full', 'ERROR timeout']When You Would Use This
This pattern is useful in readers for CSV files, event streams, or paginated API results, where collecting everything in memory is unnecessary. Because iterator state advances when values are consumed, decide whether you need a reusable collection or a one-time pass. Keep files and connections open until their lazy consumers finish.
Practice: Modify the Code
Change the filter to detect WARNING or ERROR records in a sample file.
Verify the changed output against the behavior described above, including the relevant invalid or boundary case.
For complete parameters and version details, consult the official Python documentation.
Common Beginner Mistakes
- Calling readlines() and unnecessarily materializing the entire log before filtering.
- Overlooking this documented constraint: a generator can yield only matching messages, leaving irrelevant rows uncollected.
Key Takeaways
- Stream a large log source line by line using a generator.
- Reading an open text file as an iterator processes one line at a time.
- A production log parser should handle timestamps, encodings, and partial or malformed lines.