Reducing Memory Usage in Large Data Processing

Stream and process records incrementally rather than collecting large arrays.

What It Does and When to Use It

Yield one file row at a time instead of calling file() on a large file. Current allocations answer how much PHP holds now; peak allocations expose workload spikes that can exceed the process limit.

Requirements, Inputs and Expected Behavior

Confirm the runtime, extension, environment variables, and permissions before running the examples in production. A generator that secretly calls fetchAll() is not memory efficient.

Example 1: Basic Operation

Yield one file row at a time instead of calling file() on a large file.

<?php
function rows(string $path): Generator { $file=new SplFileObject($path); while (!$file->eof()) { $line=$file->fgets(); if ($line!=='') yield rtrim($line,"
"); } }
foreach (rows(__DIR__.'/large.txt') as $line) { /* Process one line. */ }

Example 2: Behavior and Validation

Measure allocations, distinguishing peak and current consumption.

<?php
$start=memory_get_usage(true);
foreach (range(1,1000) as $n) { $row=['id'=>$n]; unset($row); }
echo 'Allocation delta ',memory_get_usage(true)-$start;

Practical Example: Application Use

Use bounded batch sizes and flush the final partial batch.

<?php
$batch=[]; foreach ($reader as $row) { $batch[]=$row; if (count($batch)===500) { $writer($batch); $batch=[]; } }
if ($batch) $writer($batch);

Important Behavior and Edge Cases

The second example highlights: Measure allocations, distinguishing peak and current consumption. The practical example adds: Use bounded batch sizes and flush the final partial batch.

Comparison with Related Approaches

Common Beginner Mistakes

Key Takeaways

• Stream and process records incrementally rather than collecting large arrays.
• Use bounded batch sizes and flush the final partial batch.
• Measure the representative workload before changing memory_limit or forcing garbage collection.