Analytic result + secondary evidence + registered protocol

How many Lighthouse runs prove a change?

“We ran it five times and the score went up two points.” That sentence drives sprint reviews, agency invoices and CI gates — and most of the time it isn't evidence. Lighthouse is noisy enough that a small before/after change can be pure run-to-run variance. Median-of-five gives you a stable estimate; it does not give you the statistical power to prove a small improvement is real. Here's the framework that does — with the numbers, a calculator, and the protocol to settle it at scale.

Published June 22, 2026 Result: n ≈ 15.7 · σ²/δ² (independent, α = 0.05, power = 0.8) Tool: required-runs calculator Data + protocol · CC BY 4.0

What's settled here, and what isn't

This article delivers three things and is careful to keep them apart. (1) A completed analytic result — the confidence-interval and power tables that convert a run-level standard deviation into the runs you need; these are mathematics, not measurement. (2) Secondary empirical evidence — real repeated-run Lighthouse data from a published study, showing the variance is large and site-specific. (3) A registered protocol for the primary study we have not yet run (a 200-site × 50-run variance census). No empirical variance number is invented; the analytic tables are exact, and the census is specified so it can be executed and audited.

The claim everyone makes — and why it's usually unproven

The advice to run Lighthouse multiple times and aggregate is correct and well-founded. Google's own variability guidance lists the sources of run-to-run noise — network, CPU contention, layout non-determinism, A/B and personalisation — and recommends aggregating runs. The closest academic study, Heričko, Šumak & Brdnik (2021), reaches the same operational rule: single runs are unrepresentative, and five consecutive runs aggregated by the median reduce variability greatly in reasonable time. That is where “median of 5” comes from, and for its purpose — a stable snapshot of one page — it's good guidance.

The trouble is that the same heuristic gets quietly promoted to a job it was never validated for: proving that a change moved the score. Stabilising an estimate and detecting a small difference are different statistical problems. The first asks “where does this page sit?”; the second asks “is this 2-point gap bigger than the noise?” — and the answer to the second depends entirely on how much the runs scatter relative to the change you're claiming.

30%of the Lighthouse score is Total Blocking Time — the most variance-prone metric
≈ 9 ptssmallest change median-of-5 can actually prove (σ = 5)
σ²/δ²what the run count scales with — halving δ quadruples n
0fabricated variance measurements in this release

The variance is real, large, and stubbornly site-specific

Before any statistics, look at what repeated runs actually do. Heričko et al. ran each of ten production sites 1, 5, 10 and 100 times. Aggregation does pull the median in — but the underlying spread is wide for some sites and tiny for others, and it does not vanish even at 100 runs:

Horizontal bars of the performance-score range across 100 Lighthouse runs for ten sites. Wix spans 23 points, AliExpress and Mercari 21, Amazon India 20, Bola 14, Zendesk 13, Freepik 7, IKEA and Shopify 5, Unsplash 4. The range is large for some sites and small for others.
Figure 1 — Performance-score range across 100 runs, per site (Heričko et al., 2021). Even after 100 runs, Wix spans 23 points and Amazon India 20, while Unsplash spans 4. The median stabilises; the distribution doesn't — and you can't know which kind of site you have without measuring.

Two things explain why the score is so jumpy. First, its composition: the performance score is a weighted blend in which Total Blocking Time carries 30% and LCP and CLS 25% each — and TBT, which depends on exactly how main-thread long tasks get scheduled, is the most timing-sensitive of the lot. Second, throttling: PageSpeed Insights and default Lighthouse use simulated (Lantern) throttling, whose modelled timings react differently to noise than applied throttling. The upshot is that a “run-level σ” of 2 points on one site can be 7+ on another in the same lab — which is why a borrowed variance number is unsafe and a pilot is mandatory.

The right question is power, not repetition

Treat a before/after comparison as what it is: a two-sample test on noisy measurements. To detect a true difference δ in the score, with run-level standard deviation σ, at two-sided significance α and power 1−β, the runs per condition follow the standard sample-size formula: 

n  ≥  2 · (z₁₋α/₂ + z_power)² · σ² / δ²        independent, equal n per condition

at α = 0.05 and power = 0.80:   n  ≈  15.7 · σ² / δ²

paired runs (alternated before/after):  drop the factor of 2, use σ of the paired differences

The σ²/δ² term is the whole story. Runs scale with the square of the noise-to-signal ratio, so halving the change you want to detect quadruples the runs you need. That is why “a few more runs” never rescues a small claim — and why the median-of-5 default silently fails the moment the effect you're chasing is small relative to the spread.

Log-scale chart of runs required per condition versus the target before/after change for run-level standard deviations of 2, 5 and 8 points. A median-of-five line shows five runs only suffices for large changes. At SD 5, detecting a 5-point change needs about 16 runs and a 3-point change about 44.
Figure 2 — Runs required per condition (log scale) to detect a target change, by run-level σ. Five runs only lands in the shaded “enough” band for large changes. At a typical σ = 5, a 5-point change needs ~16 runs and a 3-point change ~44 — and a noisy site (σ = 8) needs far more.
Required runs per condition, n ≈ 15.7 · σ²/δ² (α = 0.05, power = 0.8, independent). Reproducible from the formula; full grid in the pack.
Detect δσ = 2σ = 5σ = 8
10 points1411
5 points31641
3 points744112
1 point633931,005

What median-of-5 can and can't prove

Read the same physics the other way: fix the runs and ask how small a change they can resolve. The minimum detectable effect at n runs is (z₁₋α/₂ + z_power) · σ · √(2/n), which shrinks only as 1/√n — brutal diminishing returns. At a typical σ = 5, here is what each run budget actually buys:

Curve of the smallest provable before/after change versus runs per condition at run-level SD of 5 points. Median-of-five can only prove a change of about 9 points; proving a 5-point change needs about 16 runs and a 3-point change about 44. Precision improves only as one over the square root of n.
Figure 3 — The smallest change you can prove at σ = 5, by run count. Median-of-5 resolves only changes of ~9 points or more — enough to catch a big regression, useless for the single-digit gains most optimisation produces. Each halving of the detectable change roughly quadruples the runs.

So median-of-5 isn't wrong — it's a screening tool. It will reliably flag a page that fell off a cliff. It cannot, and was never meant to, certify that your CSS refactor bought 3 points. For that you need either many more runs, a paired design that cancels shared drift, or the honesty to report “within noise.” This is the same lesson the systems-benchmarking literature settled long ago: Mytkowicz et al. showed you can “produce wrong data without doing anything obviously wrong,” and Kalibera & Jones formalised how many repetitions rigorous benchmarking actually requires. Web performance is not exempt.

Put your own numbers in

The tables above assume round numbers. Your site has its own spread, and your claim has its own threshold. The companion calculator turns a pilot-measured σ and a target δ into the runs you need — and, for any candidate run count, the smallest change you could honestly claim and the confidence interval you'd report.

Lighthouse required-runs calculator

Enter your run-level SD, the change you want to prove, and your confidence and power. It returns the required runs per condition, a precision sweep, and a clear red/green read on whether your planned run budget can actually support the claim.

Open the calculator

The protocol to settle σ empirically

The analytic tables need one input the literature only sketches: the real distribution of run-level σ across the web, by site class and setup. The registered primary study fills that gap, and ships its data schema pre-frozen so results can't be fished:

  1. A 200-site × 50-run census

    200 sites stratified by score band, framework, third-party intensity and content class, each run 50× per condition on a pinned single-tenant runner — enough to estimate per-site σ precisely and to map how σ varies across the web.

  2. Fixed environment, recorded to the byte

    One Lighthouse version, Chrome build, throttling mode, device profile and cache policy, with container digest, CPU model and governor, and cgroup quota logged per run (the raw-observation schema is in the pack). Variance you don't control, you record.

  3. Per-metric, not just score

    σ for FCP, LCP, SI, TBT and CLS separately, because the score's noise is mostly TBT's noise — and a fix that targets LCP should be judged on LCP, not on a TBT-dominated composite.

  4. Bootstrap, not just the normal approximation

    The closed-form n assumes approximate normality of the mean; the study reports bootstrap confidence intervals on the median-of-n as well, since medians are what practitioners actually quote.

  5. A published claim standard

    The deliverable is a per-site-class table of “runs needed to defensibly claim a δ-point change,” turning the calculator's general formula into concrete, citable guidance.

A defensible before/after claim

Pulling it together, a Lighthouse before/after claim is decision-grade only when it clears a short bar. Anything less is a screening signal, and should be labelled as one:

Limitations

  1. σ is an input, not yet a measured distribution

    The analytic tables are exact for a given σ; the spread of σ across the real web is what the registered census exists to supply. Until then, use a pilot.

  2. Normal-approximation closed form

    The n formula assumes approximate normality of the aggregated mean and a known σ. For small n and skewed metrics (TBT, CLS), prefer the bootstrap; the secondary-evidence ranges hint the score is not always symmetric.

  3. Score vs metrics

    This focuses on the 0–100 performance score for continuity with the median-of-5 advice. The same machinery applies per metric, with different σ — often the more useful unit.

  4. Lab, not field

    This is about controlled lab repeatability, not CrUX field data. A change that's lab-significant still has to show up for real users — a separate question we treat in our other studies.

Data, code & sources

Published under CC BY 4.0: the digitised secondary evidence, the analytic CI-multiplier and minimum-detectable-effect tables, and the pre-frozen raw-observation schema for the census. Every analytic number is reproducible from the formulas above.

Full pack (ZIP) Secondary evidence — Heričko 2021 Table 2 (CSV) CI-multiplier tables (CSV) Minimum-detectable-effect table (CSV) Raw-observation schema (CSV)

Key sources

Cite this study

PageSpeedAudit Research (2026). How many Lighthouse runs prove a change?
A confidence-interval and statistical-power framework for before/after
performance-score claims.
https://pagespeedaudit.com/research/how-many-lighthouse-runs-prove-a-change

@misc{pagespeedaudit2026lighthousepower,
  title  = {How many Lighthouse runs prove a change? A confidence-interval
            and statistical-power framework for before/after claims},
  author = {{PageSpeedAudit Research}},
  year   = {2026},
  url    = {https://pagespeedaudit.com/research/how-many-lighthouse-runs-prove-a-change},
  note   = {Analytic tables + secondary evidence + registered protocol, CC BY 4.0}
}

Want a before/after you can actually defend?

A PageSpeedAudit report reports effects with their uncertainty, not a lone point score — so a “+3” comes with the interval that tells you whether it's real. Try the free required-runs calculator, and see our sister study “Do Lighthouse's promised savings materialize?” for whether the predicted milliseconds even show up.

Get your audit Open the calculator