Registered study · pilot

Do Lighthouse's promised savings materialize?

Lighthouse and PageSpeed Insights don't just flag problems — for many they print a number: “Est savings: 1,200 ms.” Those numbers drive sprint priorities, agency reports and automated optimizers. Yet nobody has systematically checked whether they come true. We built the method, ran a controlled pilot, and publish the full protocol to measure it at web scale.

Published June 19, 2026 Pilot: 8 controlled cases · Lighthouse 13.4.0 Finding: simulated-metric delta ≥ prediction in 8/8 (1.0–1.9×) Pilot data (CSV) + protocol · CC BY 4.0

What we measured

The question is one of calibration: when you implement a recommendation exactly, does the predicted saving P equal the observed saving R? To get a first signal we built a controlled harness. For each engineered, single-audit page we read Lighthouse 13's predicted metric saving (the insight's metricSavings map), applied the exact fix, and took R as the difference between the median FCP/LCP of five baseline runs and five fixed runs, interleaved.

Read this before the numbers: what R is

The pilot ran Lighthouse on its default mobile profile, which uses simulated throttling. In that mode Lighthouse loads the page unthrottled and then uses the Lantern model to estimate what FCP and LCP would have been under Slow 4G — the docs are explicit that “the trace values will not match up with Lighthouse's metric results, as the original trace is prior to the simulation.” So R is a change in Lighthouse's own simulated output, and P comes out of the same simulator. The page really was rebuilt between conditions, so the dependency graph fed to Lantern genuinely changed — this is not circular — but it is an internal-consistency test, not an independent stopwatch. What the pilot shows is that Lighthouse's advertised saving is not larger than the move in its own metric when you actually do the fix. Whether those milliseconds appear on a real device, in PSI production, or in field data is not established here, and the registered study below is where that gets tested.

Scatter of predicted saving versus the observed change in Lighthouse's simulated metric. All eight points sit on or above the perfect-calibration diagonal: render-blocking points are clearly above it, image-delivery points sit just above it, and the two render-block-gated image points are also on it.
Figure 1 — Predicted saving (x) vs the change in Lighthouse's simulated metric (y), in milliseconds. The dashed line is perfect calibration. Every point sits on or above it: on these controlled pages Lighthouse 13 never promised more than its own simulator then delivered, and usually promised less.
8 / 8 cases where the simulated metric moved by at least the predicted amount
1.0–1.9× simulated-metric delta ÷ predicted, across all cases
≈ 1.1× image-delivery — near-perfect, tightening as the image grows
≈ 1.7× render-blocking — Lighthouse under-promised here
Bar chart of the observed-over-predicted ratio per case: render-blocking 1.91, 1.53, 1.71; image-delivery 1.30, 1.09, 1.04; the two gated image cases 1.05 and 1.32. All bars meet or exceed the 1.0 line.
Figure 2 — Ratio of the simulated-metric delta to the predicted saving, per case. 1.0 means the simulator moved by exactly what the insight promised; every case met or exceeded it.

Two honest surprises. First, the common cynicism — “Lighthouse over-promises” — did not reproduce on clean pages, at least inside its own simulator. If anything, render-blocking predictions under-promised: a fix Lighthouse valued at 150 ms moved its simulated FCP by 287. Image-delivery was the most accurate, and tightened toward 1.0 as the image got heavier (1.30× at 300 KB → 1.04× at 1.8 MB). Second, we tried twice to construct the classic over-prediction trap — a heavy image whose paint should be gated by a render-blocking stylesheet — and both times the prediction still materialized, because the image stayed on the critical path. Reproducing over-prediction needs the messy real web, not a synthetic page. Which is precisely the point of the study below.

Why this needs measuring at all

A forecast can be useful even when imperfect — but only if its error is known. Weather forecasts, credit models and medical risk scores are all judged by calibration. Lighthouse's savings number is a forecast too, and as far as we can find, no published study compares predicted savings against realized improvement after an exact fix. The pieces that look like validation aren't:

  • Lantern validates the wrong thing. Lighthouse's Lantern accuracy doc reports Spearman's ρ and MAPE for FCP/FMP/TTI on 300 URLs, comparing the median of nine runs in one environment to nine in another. That tests whether simulated metrics resemble lab metrics — not whether fixing a flagged opportunity yields the predicted delta. Those are different inferential problems.
  • Lighthouse 13 changed the surface, not the validation. In 2025 Google replaced legacy performance audits with “insights”; Lighthouse 13 made the switch permanent (the score is metric-based, so it's unchanged). The release notes document the migration — they don't publish a per-insight calibration study.
  • The maintainers themselves flag the failure modes. A 2019 issue (#8740) modeled image-optimization impact while admitting load-time gains can “fall to zero if JS rendering takes ages.” A 2022 issue (#14107) shows a user summing five savings and asking which metric the total maps to — a common, and usually invalid, move.
  • Recent automated-fix work measures something else. The MSR 2026 study on LLM DOM-level performance fixes reports reductions in audit incidence, not whether the predicted milliseconds appeared. An audit can vanish without the saving arriving; a metric can improve while the audit stays flagged.

So the gap is real and specific: the savings number is decision-grade only if we know its error, by audit type. The pilot says “on clean pages, decent and conservative.” The study below says how to learn whether that survives contact with real sites.

The estimand

For each page i, audit a and metric m, Lighthouse reports a predicted saving P. After implementing exactly that fix, the observed saving is the difference between the two condition medians over nine runs each: 

R = median(metric, baseline) − median(metric, intervention)
calibration error   E = R − P          (negative ⇒ over-prediction)
delivery ratio      Q = R / P          (1.0 ⇒ the metric moved by exactly the promise)

Note what that R is and isn't. It is a difference of two condition medians under randomized alternation — not a paired statistic. A paired estimator works on the within-pair differences, median(Bj − Ij), and the two are not the same quantity: for B = [0, 100, 101] against I = [1, 2, 102] the difference of medians is +98 while the median paired difference is −1. Interleaving baseline and intervention runs is a real control on host drift, and the pilot used it; it does not make the estimator paired. The registered protocol below reports both, so the pairing the design creates is actually used.

A forecast can fail in distinct, useful-to-separate ways. It can be magnitude-miscalibrated (a 900 ms prediction that yields 100 ms), yet still rank opportunities correctly (the biggest predicted fix is the biggest real one) — in which case it's fine for triage but not forecasting. And savings are usually non-additive: five fixes rarely sum, because they compete for the same critical path. The study estimates calibration and ranking utility per audit, not as one global number, because a single slope would hide exactly the differences a practitioner needs — and estimates non-additivity only on the subset of pages that get the factorial arm described below, because a one-audit-at-a-time design cannot identify an interaction at all.

The full study (registered protocol)

The pilot proves the harness works. To answer the question for the web as practitioners meet it, the registered design scales it to real pages and adds a field arm. It is written to be executed, audited and extended — the companion workbook ships the full data schema with result fields left deliberately blank.

  1. Sample 100–200 real pages

    Stratified across rank, page type, framework, resource profile and baseline speed, from the public HTTP Archive / CrUX frame — calibration should hold for the messy web, not just clean lab pages.

  2. One audit at a time, by a fixed grammar

    “Implement exactly what Lighthouse says” is ambiguous, so each fix follows a pre-registered, minimal patch grammar applied at the HTTP boundary via a proxy. A resolution gate requires the post-fix report to clear the audit (or cut its predicted saving ≥ 90%); a semantic gate (visual diff, DOM and console checks) guards against fixing the number while breaking the page.

  3. Median-of-nine, reported two ways

    Nine baseline and nine intervention runs, alternated in randomized blocks, on a pinned single-tenant runner — because Lighthouse's own variability guidance and Lantern validation both rely on medians, not single before/after runs. Both estimators are reported: the difference of condition medians (comparable to the pilot) and the median of the within-block paired differences, which is the one the blocking actually earns.

  4. A factorial arm, or no non-additivity claim

    One-audit-at-a-time yields the no-fix, A-only and B-only conditions — and those three can never identify an interaction, because the combined condition is missing: RAB − RA − RB needs YAB. Two worlds with identical one-at-a-time results (Y₀ 100, YA 90, YB 90) can hide a perfectly additive YAB = 80 or a strongly antagonistic 95. So commonly co-occurring pairs get a 2×2 sub-study — none/A/B/A+B — starting with render-blocking × LCP-discovery, image-delivery × LCP-discovery, and document-latency × render-blocking. Non-additivity is an estimand only for pages that get the combined arm.

  5. Audit-level calibration cards

    Per insight: median predicted vs realized, signed error, materialization rate, SMAPE, the calibration slope, ranking correlation and the non-resolution rate — so teams can see that, say, image-delivery is decision-grade while another family is only directional.

  6. A field arm for ecological validity

    Join monthly HTTP Archive Lighthouse transitions to CrUX deltas (the History API offers up to 40 overlapping 28-day windows): when an audit flag disappears in the wild, do real-user metrics improve versus matched controls? Observational and secondary — it can't replace the controlled estimand, but it tests whether the lab result has a real-world correlate.

The fixes are deterministic by audit family. A sample of the grammar:

Intervention grammar (excerpt). Each rule is the smallest proxy transformation that satisfies the recommendation while preserving page semantics.
Audit / insight familyMinimal fixPrimary metric
Render-blockingInline critical CSS / defer non-critical stylesheets and eligible scriptsFCP, LCP
Image deliveryRe-encode/resize flagged images to recommended bytes; preserve rendered dimensionsLCP, bytes
LCP discoveryRemove lazy-load from the LCP image; add preload / fetchpriority=highLCP
Document request latencyServe flagged text with gzip/br; collapse redirect chains; fix TTFB at a controlled edgeFCP, LCP
Network dependency treeAdd preconnect/preload only for named origins — no speculative preloadingFCP, LCP
Legacy / duplicated JavaScriptServe the modern/de-duplicated bundle when a functionality gate passesTBT

Method — the pilot

Limitations — read before quoting

  1. The predictor and the outcome share a simulator

    Both P and R are Lantern products: Lighthouse's default mode estimates FCP and LCP from an unthrottled load rather than timing a throttled one. The intervention genuinely changed the page, so this is a real test of whether the promise survives doing the work — but it is Lighthouse checking its own arithmetic. An independent outcome needs applied (not simulated) throttling with external traffic shaping, WebPageTest or equivalent, target hardware, or field data. Read the pilot as internal consistency, and do not quote it as evidence that the milliseconds reach users.

  2. Clean pages, not the real web

    The pilot's whole limitation is its cleanliness. Synthetic single-audit pages isolate the effect perfectly — which is exactly why predictions look well-calibrated. Real pages have JS that gates paint, third parties, A/B tests and field-vs-lab gaps that the full study exists to capture.

  3. The over-prediction case is unproven

    We could not synthesize a confound where Lighthouse over-predicts; the optimized resource stayed on the critical path. So the pilot shows predictions can be conservative, not that they never over-promise. Don't read it as “Lighthouse is always right.”

  4. Small N, two families

    Eight cases across render-blocking and image-delivery. The calibration map needs many audit families at n ≥ 30 each — descriptive only below that.

  5. Pilot used median-of-five

    To keep runtimes bounded; the registered protocol prescribes median-of-nine.

  6. Run-level records were not kept

    Eight cases × five runs × two conditions is eighty observations, and the harness that produced this pilot wrote only the per-case medians — the individual run values were never persisted, so the published CSV cannot contain them and nobody, including us, can recompute a median, inspect an outlier, or form the paired differences. The harness now emits a run-level record per run; a future execution will publish it. Until then, treat the eight rows as aggregates you must take on trust, which is exactly the standard this study exists to raise.

  7. One pinned version

    Lighthouse 13.4.0. Predictions change across versions; the calibration card is versioned and meant to be re-run at major releases.

Data, protocol & sources

The pilot dataset and the full study workbook (data model, dictionary, calibration formulas and SQL templates, with empirical fields blank) are published under CC BY 4.0 — run the harness, disagree with the design, or execute the full study and publish the cards. The pilot CSV holds eight per-case aggregate rows — one per case, not one per run — for the reason given in the limitations.

Pilot results (CSV) Study workbook (XLSX)

Key sources

Cite this study

PageSpeedAudit Research (2026). Do Lighthouse's promised savings materialize?
A registered calibration study of PageSpeed Insights and Lighthouse predicted
metric savings. https://pagespeedaudit.com/research/do-lighthouse-savings-materialize

@misc{pagespeedaudit2026lighthousecalibration,
  title  = {Do Lighthouse's promised savings materialize? A registered
            calibration study of Lighthouse predicted metric savings},
  author = {{PageSpeedAudit Research}},
  year   = {2026},
  url    = {https://pagespeedaudit.com/research/do-lighthouse-savings-materialize},
  note   = {Pilot dataset + protocol workbook, CC BY 4.0}
}

Want to know which of your savings are real?

A PageSpeedAudit report doesn't just forward Lighthouse's numbers — it reads your real critical path and tells you which fixes will actually move LCP and INP, and by how much. Our sister study “The cost of a kilobyte” measures what a fix is worth per resource type, and the free cloaking test checks whether a “95+” score is even real.

Get your audit Read “The cost of a kilobyte”