Registered study · pilot
Do Lighthouse's promised savings materialize?
Lighthouse and PageSpeed Insights don't just flag problems — for many they print a number: “Est savings: 1,200 ms.” Those numbers drive sprint priorities, agency reports and automated optimizers. Yet nobody has systematically checked whether they come true. We built the method, ran a controlled pilot, and publish the full protocol to measure it at web scale.
What we measured
The question is one of calibration: when you implement a recommendation exactly,
does the predicted saving P equal the realized saving R? To get a first signal we
built a controlled harness. For each engineered, single-audit page we read Lighthouse 13's predicted
metric saving (the insight's metricSavings map), applied the exact fix, and measured the
realized saving as a paired median-of-five run delta — the same shape of comparison Google's own
Lantern validation uses.
Two honest surprises. First, the common cynicism — “Lighthouse over-promises” — did not reproduce on clean pages. If anything, render-blocking predictions under-promised: a fix Lighthouse valued at 150 ms delivered 287. Image-delivery was the most accurate, and tightened toward 1.0 as the image got heavier (1.30× at 300 KB → 1.04× at 1.8 MB). Second, we tried twice to construct the classic over-prediction trap — a heavy image whose paint should be gated by a render-blocking stylesheet — and both times the prediction still materialized, because the image stayed on the critical path. Reproducing over-prediction needs the messy real web, not a synthetic page. Which is precisely the point of the study below.
Why this needs measuring at all
A forecast can be useful even when imperfect — but only if its error is known. Weather forecasts, credit models and medical risk scores are all judged by calibration. Lighthouse's savings number is a forecast too, and as far as we can find, no published study compares predicted savings against realized improvement after an exact fix. The pieces that look like validation aren't:
- Lantern validates the wrong thing. Lighthouse's Lantern accuracy doc reports Spearman's ρ and MAPE for FCP/FMP/TTI on 300 URLs, comparing the median of nine runs in one environment to nine in another. That tests whether simulated metrics resemble lab metrics — not whether fixing a flagged opportunity yields the predicted delta. Those are different inferential problems.
- Lighthouse 13 changed the surface, not the validation. In 2025 Google replaced legacy performance audits with “insights”; Lighthouse 13 made the switch permanent (the score is metric-based, so it's unchanged). The release notes document the migration — they don't publish a per-insight calibration study.
- The maintainers themselves flag the failure modes. A 2019 issue (#8740) modeled image-optimization impact while admitting load-time gains can “fall to zero if JS rendering takes ages.” A 2022 issue (#14107) shows a user summing five savings and asking which metric the total maps to — a common, and usually invalid, move.
- Recent automated-fix work measures something else. The MSR 2026 study on LLM DOM-level performance fixes reports reductions in audit incidence, not whether the predicted milliseconds appeared. An audit can vanish without the saving arriving; a metric can improve while the audit stays flagged.
So the gap is real and specific: the savings number is decision-grade only if we know its error, by audit type. The pilot says “on clean pages, decent and conservative.” The study below says how to learn whether that survives contact with real sites.
The estimand
For each page i, audit a and metric m, Lighthouse reports a predicted saving
P. After implementing exactly that fix, the realized saving is the paired median-of-nine delta:
R = median(metric, baseline) − median(metric, intervention)
calibration error E = R − P (negative ⇒ over-prediction)
materialization Q = R / P (1.0 ⇒ fully delivered)A forecast can fail in distinct, useful-to-separate ways. It can be magnitude-miscalibrated (a 900 ms prediction that yields 100 ms), yet still rank opportunities correctly (the biggest predicted fix is the biggest real one) — in which case it's fine for triage but not forecasting. And savings are usually non-additive: five fixes rarely sum, because they compete for the same critical path. The study estimates all three — calibration, ranking utility, and non-additivity — per audit, not as one global number, because a single slope would hide exactly the differences a practitioner needs.
The full study (registered protocol)
The pilot proves the harness works. To answer the question for the web as practitioners meet it, the registered design scales it to real pages and adds a field arm. It is written to be executed, audited and extended — the companion workbook ships the full data schema with result fields left deliberately blank.
Sample 100–200 real pages
Stratified across rank, page type, framework, resource profile and baseline speed, from the public HTTP Archive / CrUX frame — calibration should hold for the messy web, not just clean lab pages.
One audit at a time, by a fixed grammar
“Implement exactly what Lighthouse says” is ambiguous, so each fix follows a pre-registered, minimal patch grammar applied at the HTTP boundary via a proxy. A resolution gate requires the post-fix report to clear the audit (or cut its predicted saving ≥ 90%); a semantic gate (visual diff, DOM and console checks) guards against fixing the number while breaking the page.
Paired median-of-nine
Nine baseline and nine intervention runs, alternated in randomized blocks, on a pinned single-tenant runner — because Lighthouse's own variability guidance and Lantern validation both rely on medians, not single before/after runs.
Audit-level calibration cards
Per insight: median predicted vs realized, signed error, materialization rate, SMAPE, the calibration slope, ranking correlation and the non-resolution rate — so teams can see that, say, image-delivery is decision-grade while another family is only directional.
A field arm for ecological validity
Join monthly HTTP Archive Lighthouse transitions to CrUX deltas (the History API offers up to 40 overlapping 28-day windows): when an audit flag disappears in the wild, do real-user metrics improve versus matched controls? Observational and secondary — it can't replace the controlled estimand, but it tests whether the lab result has a real-world correlate.
The fixes are deterministic by audit family. A sample of the grammar:
| Audit / insight family | Minimal fix | Primary metric |
|---|---|---|
| Render-blocking | Inline critical CSS / defer non-critical stylesheets and eligible scripts | FCP, LCP |
| Image delivery | Re-encode/resize flagged images to recommended bytes; preserve rendered dimensions | LCP, bytes |
| LCP discovery | Remove lazy-load from the LCP image; add preload / fetchpriority=high | LCP |
| Document request latency | Serve flagged text with gzip/br; collapse redirect chains; fix TTFB at a controlled edge | FCP, LCP |
| Network dependency tree | Add preconnect/preload only for named origins — no speculative preloading | FCP, LCP |
| Legacy / duplicated JavaScript | Serve the modern/de-duplicated bundle when a functionality gate passes | TBT |
Method — the pilot
- Lighthouse 13.4.0, default mobile profile (simulated Slow 4G, 4× CPU), Chrome for Testing 148, headless — the same lab profile PageSpeed Insights uses.
- Predicted saving from the insight's
metricSavings. Lighthouse 13 surfaces predicted savings as a per-metric map (e.g.render-blocking-insight → {FCP: 250, LCP: 250}), replacing the legacyoverallSavingsMsopportunity field. We read the target metric for the case's audit. - Single-audit synthetic pages. A local server emits a baseline (problem present) and a fixed variant that changes essentially one thing — a render-blocking stylesheet vs the same CSS inlined; an oversized hero PNG vs a right-sized one at the same display box. Byte-exact, incompressible payloads (the kilobyte-study harness),
Cache-Control: no-store. - Paired median-of-five, interleaved baseline/fixed to spread host-load drift. Realized = median FCP/LCP(baseline) − median(fixed). Two families × three severities, plus two render-block-gated image cases — eight cases total.
- Scope. This is a proof-of-concept on controlled pages, not the 100–200-site census. The full protocol prescribes median-of-nine and real pages; the pilot's job is to prove the method and produce a first, honest signal.
Limitations — read before quoting
Clean pages, not the real web
The pilot's whole limitation is its cleanliness. Synthetic single-audit pages isolate the effect perfectly — which is exactly why predictions look well-calibrated. Real pages have JS that gates paint, third parties, A/B tests and field-vs-lab gaps that the full study exists to capture.
The over-prediction case is unproven
We could not synthesize a confound where Lighthouse over-predicts; the optimized resource stayed on the critical path. So the pilot shows predictions can be conservative, not that they never over-promise. Don't read it as “Lighthouse is always right.”
Small N, two families
Eight cases across render-blocking and image-delivery. The calibration map needs many audit families at n ≥ 30 each — descriptive only below that.
Pilot used median-of-five
To keep runtimes bounded; the registered protocol prescribes median-of-nine. All runs are retained for the published dataset.
One pinned version
Lighthouse 13.4.0. Predictions change across versions; the calibration card is versioned and meant to be re-run at major releases.
Data, protocol & sources
The pilot dataset and the full study workbook (data model, dictionary, calibration formulas and SQL templates, with empirical fields blank) are published under CC BY 4.0 — run the harness, disagree with the design, or execute the full study and publish the cards.
Key sources
- Google Chrome (n.d.). Lighthouse performance scoring & Lighthouse variability.
- GoogleChrome/lighthouse. Lantern model validation (300 URLs, median-of-nine, FCP/FMP/TTI).
- Google Chrome Developers (2025). Lighthouse is moving to performance insight audits & What's new in Lighthouse 13.
- Google. PageSpeed Insights API.
- GoogleChrome/lighthouse issues #8740, #14107, #16769 and discussion #16462.
- HTTP Archive (2025). Web Almanac methodology. Chrome UX Report. CrUX on BigQuery & History API.
- Peters, G., Khatoonabadi, S.H. & Shihab, E. (2026). Evaluating the Use of LLMs for Automated DOM-Level Resolution of Web Performance Issues. arXiv:2601.05502 (MSR 2026).
Cite this study
PageSpeedAudit Research (2026). Do Lighthouse's promised savings materialize?
A registered calibration study of PageSpeed Insights and Lighthouse predicted
metric savings. https://pagespeedaudit.com/research/do-lighthouse-savings-materialize
@misc{pagespeedaudit2026lighthousecalibration,
title = {Do Lighthouse's promised savings materialize? A registered
calibration study of Lighthouse predicted metric savings},
author = {{PageSpeedAudit Research}},
year = {2026},
url = {https://pagespeedaudit.com/research/do-lighthouse-savings-materialize},
note = {Pilot dataset + protocol workbook, CC BY 4.0}
}Want to know which of your savings are real?
A PageSpeedAudit report doesn't just forward Lighthouse's numbers — it reads your real critical path and tells you which fixes will actually move LCP and INP, and by how much. Our sister study “The cost of a kilobyte” measures what a fix is worth per resource type, and the free cloaking test checks whether a “95+” score is even real.