Python Stream Large Compressed Files with Limited Memory

Read compressed files in chunks rather than materializing large output all at once.

How It Works

A stream interface lets an application consume a limited number of decompressed bytes per iteration. This reduces memory pressure for ordinary input sizes and gives a place to count total output. Chunking alone does not protect against maliciously huge output unless a hard budget is enforced.

When processing untrusted data, combine output limits with time, disk and CPU budgets.

Step-by-Step Example

import gzip
from io import BytesIO

original = b"report line\n" * 200
compressed = gzip.compress(original)
with gzip.GzipFile(fileobj=BytesIO(compressed), mode="rb") as source:
    total = 0
    while chunk := source.read(128):
        total += len(chunk)
        if total > 10_000:
            raise ValueError("Expanded data exceeds limit")
print(total == len(original))

Understanding the Example

The expression 'b"report line\\n" * 200' supplies the value stored as 'original'. This value is then used or checked by the later statements.
The expression 'gzip.compress(original)' supplies the value stored as 'compressed'. This value is then used or checked by the later statements.

A successful run of this example prints the following output:

True

Where This Fits in a Real Program

Before putting the technique into a larger program, identify the required inputs and which results are returned. Choose what should happen with invalid or missing data, and test a small valid case and a failing case independently.

Practice: Try It Yourself

Change the maximum permitted expanded bytes to 1000 and observe the controlled failure.

For this exercise, predict the result before running the changed code. Then explain the difference between the original and modified cases using the actual values, not just whether the program completed.

Common Beginner Mistakes

Calling read() with no size on a stream of unknown uncompressed length.
Assuming a loop that reads small chunks cannot still decompress unlimited total bytes.

Key Takeaways

Read compressed files in chunks rather than materializing large output all at once.
A stream interface lets an application consume a limited number of decompressed bytes per iteration.
When processing untrusted data, combine output limits with time, disk and CPU budgets.