Using Generators to Process Very Large Datasets

Generators yield records one at a time so a whole dataset need not be allocated in memory.

The examples focus on the operation itself, including the important distinction: A normal array materializes all values; yield produces them incrementally.

What It Does

Use the examples below to observe the behavior on a small dataset before extending it to application code.

Before You Start

Read the input requirements before applying this pattern to production data.

Example 1: Basic Use

The generator allocates one record at a time instead of building a large array.

<?php
function records(int $total): Generator {
    for ($n = 1; $n <= $total; $n++) { yield ["id" => $n]; }
}
foreach (records(5) as $row) {
    if ($row["id"] % 2 === 0) { echo $row["id"] . PHP_EOL; }
}

Output from a successful local PHP 8.4 run:

2
4

Example 2: Checking Behavior

The generator allocates one record at a time instead of building a large array.

<?php
function records(int $total): Generator {
    for ($n = 1; $n <= $total; $n++) { yield ["id" => $n]; }
}
foreach (records(6) as $row) {
    if ($row["id"] % 2 === 0) { echo $row["id"] . PHP_EOL; }
}

Output from a successful local PHP 8.4 run:

2
4
6

Example 3: Practical Example

The generator allocates one record at a time instead of building a large array.

<?php
function records(int $total): Generator {
    for ($n = 1; $n <= $total; $n++) { yield ["id" => $n]; }
}
foreach (records(7) as $row) {
    if ($row["id"] % 2 === 0) { echo $row["id"] . PHP_EOL; }
}

Output from a successful local PHP 8.4 run:

2
4
6

Important Behavior and Edge Cases

A generator does not reduce memory if its consumer immediately calls iterator_to_array on the entire stream.

Comparison with Related Approaches

Common Beginner Mistakes

Key Takeaways

• Generators yield records one at a time so a whole dataset need not be allocated in memory.
• A generator does not reduce memory if its consumer immediately calls iterator_to_array on the entire stream.
• A normal array materializes all values; yield produces them incrementally.

Use Using Generators to Process Very Large Datasets when its documented behavior matches the problem. Test the relevant edge cases in your own runtime before deploying it.