Lighthouse required-runs calculator

Why “just run it five times” isn't proof

Running Lighthouse five times and taking the median is excellent advice — for getting a stable single estimate of where a page sits today. It is not the same as having enough statistical power to prove a small before/after change is real. Those are different jobs, and the second one is governed by the spread of the runs:

n ≥ 2 · (z₁₋α/₂ + z_power)² · σ² ⁄ δ²        (independent, equal n)

at α = 0.05 and 80% power:  n ≈ 15.7 · σ² ⁄ δ²

The dependence on σ²/δ² is what bites. Halving the change you want to detect quadruples the runs you need. With a typical run-level SD of 5 points, proving a 5-point change needs about 16 runs per condition; a 3-point change needs about 44. Median-of-five can only resolve changes of roughly nine points or more — fine for a big regression, useless for the single-digit gains most optimisation work produces.

What the pilot measures — and what it can't. The “measure it for me” pilot runs your page through PageSpeed Insights repeatedly and takes the per-metric standard deviation. That σ includes Google's runner mix (we report the CPU-benchmark spread when it's material), so it's the honest σ for PSI-based before/afters; a pinned local or CI setup is usually tighter, so measure there if that's where your experiment will run. A small pilot's σ̂ is itself uncertain — we show its 90% band and the conservative run count from the upper bound rather than hiding it. And note the pilot yields the run-level σ: a paired design needs the SD of paired differences from your own harness. Lab CLS is often deterministic; when the pilot measures σ = 0, any before/after CLS difference is directly interpretable and there is no sample-size question to answer.

How the upload verdict decides. The before/after comparison never trusts a single number: it takes the median of each side, builds a deterministic bootstrap confidence interval on the median difference from your runs' own spread, and sets a noise floor from the within-arm variation. A change is called real only when the interval excludes zero and the change clears the floor — and an apparent regression larger than the page's own baseline is treated as instability, not as a confident result. It also checks what spreadsheets silently miss: mixed Lighthouse versions, mixed device profiles, mixed throttling, and multiple URLs in one comparison.

The full method, the secondary evidence, and the registered protocol behind this calculator are in the study “How many Lighthouse runs prove a change?”