Lighthouse required-runs calculator

Why “just run it five times” isn't proof

Running Lighthouse five times and taking the median is excellent advice — for getting a stable single estimate of where a page sits today. It is not the same as having enough statistical power to prove a small before/after change is real. Those are different jobs, and the second one is governed by the spread of the runs:

n ≥ 2 · (z₁₋α/₂ + z_power)² · σ² ⁄ δ²        (independent, equal n)

at α = 0.05 and 80% power:  n ≈ 15.7 · σ² ⁄ δ²

this formula is for the difference of two MEANS. Aggregating by the median
instead costs precision (≈π/2 the variance under normality), so budget ~1.5×
the runs — the calculator above already does, and the upload verdict tests the
medians directly.

The dependence on σ²/δ² is what bites. Halving the change you want to detect quadruples the runs you need. On a moderately noisy page — σ ≈ 2.7 points, the middle of the published range the presets above are derived from — proving a 3-point change needs about 13 runs per condition by the mean, and 20 by the median, which is what the calculator above sizes for and the verdict below tests; a 2-point change needs 29 (45 by median). Five runs on that page reach 80% power only around a five-point change by the mean — nearer six by the median, which is the aggregation the advice actually names — and on the noisiest sites in that study (σ ≈ 4.3) it takes 33 runs to prove 3 points (51 by median). Fine for a big regression, useless for the single-digit gains most optimisation work produces. Note what that number is: the effect size this design would usually catch, not a floor below which a measured difference cannot be significant. The threshold a measured difference must clear is smaller (√(π/2) · z₁₋α/₂ · σ · √(2/n) for the median comparison the verdict makes), which is exactly why the upload verdict tests your actual runs instead of comparing them to a planning number.

What the pilot measures — and what it can't. The “measure it for me” pilot runs your page through PageSpeed Insights repeatedly and takes the per-metric standard deviation. That σ includes Google's runner mix (we report the CPU-benchmark spread when it's material), so it's the honest σ for PSI-based before/afters; a pinned local or CI setup is usually tighter, so measure there if that's where your experiment will run. A small pilot's σ̂ is itself uncertain — we show its 90% band and the conservative run count from the upper bound rather than hiding it. And note the pilot yields the run-level σ: a paired design needs the SD of paired differences from your own harness. Lab CLS is often deterministic; when the pilot measures σ = 0, any before/after CLS difference is directly interpretable and there is no sample-size question to answer.

How the upload verdict decides. The before/after comparison never trusts a single number: it takes the median of each side, builds a deterministic bootstrap confidence interval on the median difference from your runs' own spread, and sets a noise floor from the within-arm variation. A change is called real only when the interval excludes zero and the change clears the floor — and an apparent regression larger than the page's own baseline is treated as instability, not as a confident result. It also checks what spreadsheets silently miss: mixed Lighthouse versions, mixed device profiles, mixed throttling, and multiple URLs in one comparison.

The full method, the secondary evidence, and the registered protocol behind this calculator are in the study “How many Lighthouse runs prove a change?”