A fair benchmark compares equal outputs on equal inputs, warms up setup paths where relevant and repeats enough times to see variance.
Understanding the Concept
A fair benchmark compares equal outputs on equal inputs, warms up setup paths where relevant and repeats enough times to see variance. Never let the candidate implementations perform different work.
Real applications must also be measured with realistic workloads and input distributions.
How to interpret the measurement
Before optimizing, establish an objective measurement and a representative workload. Memory size, function-call counts, elapsed time and samples are different metrics: knowing one does not determine the others. Repeat observations before drawing a conclusion about a performance regression.
Profiling output is evidence for finding a bottleneck, not a performance guarantee. Check that the program still produces the correct result after any optimization and note version, input size and measurement conditions.
Worked Example
Read the code from the first line to the last. Identify which values are inputs, which operation is being demonstrated and what should happen before running it.
import timeit
values = list(range(1000))
first = lambda: sum(values)
second = lambda: sum(x for x in values)
print(first() == second())
print(timeit.timeit(first, number=200) >= 0)
print(timeit.timeit(second, number=200) >= 0)Walkthrough: What Each Part Does
- The imports make timeit available. These are standard-library building blocks unless the example explicitly names an external tool.
- The printed number or profile is diagnostic data. Repeat the observation and identify the unit before comparing values.
This demonstration focuses on one practical behavior. Before changing several lines, vary one input or one configuration choice and check whether the result changes for the reason you expected.
Example Output
True
True
TrueThe output above was captured by running the example in the available Python environment. The order of concurrent events, durations and platform-dependent values can legitimately differ on your computer.
Important Behavior and Edge Cases
Benchmarking one implementation with data already cached and the other with costly setup included.
Real applications must also be measured with realistic workloads and input distributions.
Practice Exercise
Repeat both methods five times and report ranges rather than declaring a universal winner.
Write down the result you expect before running your changed code. Check at least one normal case and one boundary or failure case. If your example starts workers or opens resources, verify how they are cleaned up.
Further reading: official documentation and API reference.
Common Beginner Mistakes
- Benchmarking one implementation with data already cached and the other with costly setup included.
- Comparing results from different workloads or environments as if the measurements were directly interchangeable.
Key Takeaways
- A fair benchmark compares equal outputs on equal inputs, warms up setup paths where relevant and repeats enough times to see variance.
- Real applications must also be measured with realistic workloads and input distributions.
- Practice: Repeat both methods five times and report ranges rather than declaring a universal winner.