Noise & Silence

The framework in one view

Abstract

Automated performance tools such as Google PageSpeed Insights are screening instruments: they are engineered for sensitivity, and consequently they fail in two directions at once. They produce noise — flags that are false, immaterial, or unfixable for the specific site — and they produce silence — real, user-experienced bottlenecks that appear nowhere in the report. The Noise & Silence framework treats the automated report itself, not the web page, as the primary object of the audit. Its method is the Four Verdicts: every flag the tool raises is adjudicated, with documented evidence, as Real, Wrong, Trivial, or Locked. The adjudication yields two falsifiable quantities — the Noise Rate (the share of flags that should receive no engineering effort) and the Silent Bottleneck (the binding field constraint absent from the flag list) — and every surviving recommendation must then pass a three-step verification ladder: predict, prove, confirm. The framework imports its foundations from screening theory, signal detection, medical alarm-fatigue research, and the actionability literature of automated static analysis; its application to web-performance reporting is, to our knowledge, novel.

1. Two wrong responses to the same report

Web performance remains materially unsolved: in the most recent HTTP Archive Web Almanac, only 48% of mobile origins and 56% of desktop origins achieved “good” Core Web Vitals (Jariyal et al., 2025), while the business stakes are well documented — a Google-commissioned study conducted by Deloitte and 55 across 37 brand sites found that a 0.1-second improvement in mobile speed metrics was associated with, among other effects, an 8.4% increase in retail conversions (Deloitte, 2020). We have quantified that stake for a single page in our own research on the cost of a millisecond.

Into this gap Google ships a free, authoritative diagnostic: PageSpeed Insights (PSI), which wraps Lighthouse lab analysis and Chrome User Experience Report (CrUX) field data. Practitioner responses to its output are strikingly bimodal. One camp treats the report as a work order and burns engineering quarters chasing a 100/100 score. The other camp, exhausted by recommendations that never seem to move anything, concludes the tool should be disregarded altogether — an entire genre of practitioner literature exists to argue that the score “does not matter” or “does not reflect reality” (Lyrical Host, n.d.; HelpfulCrowd, n.d.). The software-engineering research community observed the same two poles over a decade ago with static analysis tools: developers either drown in warnings or abandon the tool, with false positives and warning overload chief among the reasons they cite (Johnson et al., 2013).

Both responses share the same mistake: they assign the report a truth status — all true or all worthless — instead of assigning each claim in it a verdict. A screening instrument's output is neither a work order nor garbage. It is input for a confirmatory diagnosis that has not yet happened.

The Noise & Silence framework is that confirmatory step, made explicit, procedural, and measurable.

2. Foundations: screening instruments and their two errors

None of the framework's ingredients is new. That is deliberate: each pillar is imported from a field that has already spent decades — in one case, nearly a century — learning the lesson web performance has not yet institutionalized.

Screening is not diagnosis. Since Wilson and Jungner's canonical WHO monograph, medicine has separated the high-sensitivity screening test from the confirmatory diagnostic work-up that must follow it — and made the availability of follow-up diagnosis an explicit precondition for screening at all (Wilson and Jungner, 1968). A screening instrument is designed to over-flag: for its maker, a false alarm costs little, while a miss is catastrophic. PSI has exactly this economic structure — a false alarm costs the site owner's time, not Google's — so its over-flagging is a design property, not a defect.

Every detector commits two errors. Statistical decision theory formalized the trade-off: a test can raise a false alarm (Type I error) or miss a real signal (Type II error), and tuning a detector toward sensitivity necessarily buys more of the first (Neyman and Pearson, 1933; Green and Swets, 1966). Silence is the second of these errors — the missed signal — in plain language. Noise, as this framework uses the word, is broader than the statistician’s false alarm: it is every flag owed no effort — the factually false ones (the true Type I errors) and the true-but-unactionable ones the SOC literature later identified (Section 2, final paragraph; Section 5.3 keeps the two shares separable). The framework's contribution is to insist that an audit must measure both — and the literature on diagnostic testing adds a warning that transfers directly: a positive result's usefulness — its predictive value — collapses when the condition it flags is rare in the population under test, which is why a flag's existence is never evidence of its importance (Altman and Bland, 1994).

Unmanaged noise destroys the operator, not the instrument. Medicine calls the failure mode alarm fatigue: between 72% and 99% of clinical alarms are false, clinicians desensitize under the load, and patient deaths have been attributed to the resulting missed alarms (Sendelbach and Funk, 2013). Security operations centers rediscovered the phenomenon at industrial scale, with a finding that matters enormously for this framework: the seminal qualitative study of SOC analysts — titled “99% False Positives” — found that most alarms practitioners dismiss are not technically incorrect but are “true alarms, explained by legitimate behavior”: right about the event, worthless about the action (Alahmadi, Axon and Martinovic, 2022). Web teams that have “fixed everything PageSpeed said” twice and seen nothing move are exhibiting the same syndrome — not laziness, but a rational loss of trust in an unadjudicated alarm stream.

Actionability, not correctness, is the operative standard. The deepest import comes from Google's own static-analysis program. The team that built Tricorder, Google's internal analysis platform, measures what they call effective false positives: “We consider an issue to be an ‘effective false positive’ if developers did not take positive action after seeing the issue” — regardless of whether the flag is technically correct (Sadowski et al., 2018). A decade earlier, a systematic literature review had already consolidated “actionable alert identification” as its own research problem: separating, out of everything an automated analyzer raises, the subset a rational engineer should act on (Heckman and Williams, 2011). The Four Verdicts are a domain-specific actionability classifier in exactly this tradition — applied to a class of report, the web-performance audit, where no such discipline currently exists.

One more import deserves credit. The insight that page-load time is governed by a dependency structure — a critical path, such that optimizing off-path activities yields zero improvement — was established for web pages by the WProf work at the University of Washington (Wang et al., 2013). It grounds the framework's treatment of the binding constraint (Section 5.3), which is the theory-of-constraints principle that a system improves only when its constraint improves (Goldratt and Cox, 1984).

3. The instrument: where the noise comes from

Calling PSI's output noisy is not a criticism of Google's engineering; it is a description of structural properties that Google itself documents. Adjudication is possible precisely because the falsifying evidence is usually in the report's own data. Six mechanisms account for most of the noise we adjudicate in practice.

4. The silence: what the report cannot say

The second failure direction is structural absence — and it is more dangerous than noise, because an absence cannot annoy anyone into skepticism. Google's own documentation states the division plainly: “The lab data is based on a simulated load of a page on a single device and fixed set of network conditions”, and lab data “may not capture real-world bottlenecks” (Google, n.d. a). Four silences recur.

Origin-proximity blindness. The lab fetches from datacenter infrastructure that is frequently network-adjacent to the site's origin. We have recorded lab TTFB near 1 ms — the server-response-time audit passing with a perfect score — on an origin whose real users' 75th-percentile TTFB was 1.6 seconds: double the 0.8-second guideline Google's own documentation sets for server response (web.dev, n.d. b). On that site, the number-one field bottleneck appeared nowhere in the flag list. No amount of diligence applied to the 21 flags that were present could have found the problem, because the report's silence is not visible inside the report.

Interactivity blindness. INP, the responsiveness Core Web Vital, is measurable only when interactions occur; Google's documentation notes that lab tools observing “the loading of a page without any interactions” will not report INP, and offers TBT as “a reasonable proxy metric for INP, but... not a substitute” (web.dev, n.d. a). A site failing INP in the field can present a clean lab interactivity story, and vice versa — we have recorded a simulated “Time to Interactive: 19.4 s” on a page whose real-user INP was a near-good 219 ms. When lab and field diverge like this is itself a researchable question, and we have mapped when lab actually predicts field.

Population blindness. CrUX field data itself samples only opted-in Chrome users on desktop and Android; Chrome on iOS, WebViews, and all other browsers are excluded (Chrome Developers, n.d. b). For a Safari-heavy audience, even the “ground truth” half of the report is silent about most of the audience — a limitation the adjudicator must carry into every field-anchored verdict rather than discovering once and forgetting.

Variant blindness. The lab audits one URL, once, from one place, in one state: logged out, no consent choices made (or a bot-served variant of the page entirely), no A/B assignment, one geography. Whatever real users experience across templates, regions, consent states, and sessions is aggregated invisibly into field percentiles the lab run cannot decompose. The bot-served-variant case is not hypothetical — we measured how many sites serve PageSpeed a different page than their users, and you can check your own with the free PageSpeed cloaking test.

Noise wastes the effort you spend; silence wastes the effort you don't know you should be spending. A framework that only de-noised the report would still fail its user. This is why the framework has two words in its name.

5. The framework

5.1 The stance: audit the report

The unit of analysis shifts. A conventional audit examines the page and produces a second report beside Google's; the Noise & Silence audit examines Google's report about the page and produces verdicts on it — then goes looking, separately and field-first, for what the report omitted. PSI's output is neither competitor nor gospel: it is the case file. And the case file deserves respect without deference: PSI applies the same heuristics on every run — including the same wrong-for-this-site heuristics, every time. Consistency is not validity.

5.2 The Four Verdicts

Every flag the instrument raises leaves the audit decided — with exactly one of the four verdicts — or, when the discriminating evidence cannot be obtained, held Unresolved with the concrete test that would decide it. Unresolved is a decision status, not a fifth verdict: the framework’s own rules (no artifact, no verdict; weak-refutation survivors stay hypotheses) forbid forcing an unknown through binary gates. The verdicts are reached in a fixed order of tests, so that any two competent adjudicators confront the same decisions in the same sequence:

Each flag runs the three gates in order; the first gate it fails decides the verdict. Pick a recurring claim and watch it move:

Read top-down: each test either exits with a verdict or passes the flag to the next. A gate that cannot be answered with obtainable evidence exits sideways instead: Unresolved, with the named test that would answer it.

Real flags — typically a minority — are the only ones that proceed to root-cause analysis, a platform-native fix, and an impact prediction. Wrong flags are published with their falsifying evidence, because an unexplained dismissal teaches the reader nothing and would not survive scrutiny. Trivial flags are, wherever possible, argued from Google's own numbers — reading a flag's metricSavings: 0 back at it is the shortest opening argument in the genre, the model's own confession that it found no network-path benefit; the verdict is then completed with the intent-and-objective check of Section 3c, never rested on the zero alone. Locked flags convert wasted effort into redirected effort: the verdict names the controllable equivalent (for a fully hosted platform, the merchant-side lever) or explicitly licenses the reader to stop trying.

Three rules of evidence bind every verdict. First, verbatim evidence: the actual header, the actual byte count, the actual markup — a verdict with no artifact is not a verdict. Second, a documented falsification attempt: before a claim is asserted, the adjudicator records the check that could have disproved it and did not — the audit's epistemology is conjecture under attempted refutation, not accumulation of confirmations (Popper, 1959). Third, a confidence grade: findings that survive weak refutation attempts are published as hypotheses with a concrete next test, never as conclusions.

The kinship with security-operations triage is intentional and acknowledged: the SOC literature's central finding is that most dismissed alarms are “true alarms, explained by legitimate behavior” — unactionable rather than incorrect (Alahmadi, Axon and Martinovic, 2022). Trivial and Locked are the performance-audit names for exactly that category — true, yet owed no effort — and they are the two verdicts conventional web audits do not have words for.

5.3 The two metrics

The adjudication table compresses into two numbers, each falsifiable by re-running the same procedure.

Noise Rate. For a report whose decided flags number F, of which W are Wrong, T Trivial, and L Locked:

NR = (W + T + L) / F

— the recommended-no-action share: the fraction of decided flags an adjudicated report says should receive no engineering effort. It is a per-site, per-run quantity: the same flag can be Real on one site and Trivial on the next, which is precisely why site-blind flag lists cannot be work orders. Three companion figures keep it honest. W / F is the technical false-discovery share — the flags that are factually wrong, the only component that is a false positive in the statistician's sense. The unresolved share — Unresolved ÷ (decided + unresolved) — is disclosed beside NR, giving a lower and upper bound on the true no-action rate. And a claim-weighted variant weights each flag by the tool's own estimated savings, measuring how much of the report's promised improvement evaporates under adjudication.

What F counts. The number is reproducible only if the counting rule is frozen, so here it is: F is the number of unique failed performance audit or insight IDs raised by one PSI run — one strategy, one Lighthouse version, keyed to the final URL. Passed, not-applicable, manual, informative and errored results are excluded; resource rows inside an audit are evidence within one flag, not separate flags; where Lighthouse lists a legacy audit as replaced by a newer insight, the pair counts once. Every published rate carries the Lighthouse version, strategy and collection timestamp that produced it.

NR's intellectual home is the actionability tradition of Sadowski et al. (2018), whose team measures effective false positives — “developers did not take positive action after seeing the issue” — and holds shipped checks to an effective-false-positive rate under 10%. The kinship is deliberate; the difference is disclosed just as deliberately: Sadowski's quantity is behavioral, observed from what developers did, while NR is adjudicative, an auditor's evidence-backed recommendation. The under-10% bar is the standard of the tradition NR lives in — the sting is that performance reports face no such gate at all — but an 81% NR is not the same measured quantity and is never scored against that bar as if it were.

The Silent Bottleneck. Independently of the flag list, the audit decomposes the failing field metric toward its binding constraint — the phase or factor that dominates the user-experienced number, in the critical-path sense of Wang et al. (2013) and the constraint sense of Goldratt and Cox (1984). One honesty rule governs the decomposition: an aggregate percentile can be a mixture — cohort A server-bound, cohort B discovery-bound — and CrUX's own subpart data is aggregated per-subpart, so subpart p75s indicate relative importance and do not sum to the metric. The legitimate outputs are therefore: one dominant constraint with evidence; several cohort-specific constraints; or an explicitly unresolved constraint naming the segmentation or RUM attribution that would decide it. The Silent Bottleneck is the finding that the dominant constraint (or every candidate constraint) is not in the flag list: the report's Type II error, named. Its canonical instance is the passing server-response-time audit over a 1.6-second field TTFB; its verdict-line form is one sentence: “PSI raised 21 flags — and the number-one field bottleneck appears in none of them.”

A report can be mostly noise and still miss the point; the two metrics are deliberately of opposite polarity so that neither can masquerade as the other.

5.4 The procedure

Select a stage — its operational rule appears below it:

5.5 The verification ladder: predict, prove, confirm

Every recommendation that survives adjudication owes the reader three successively stronger tests.

Confirm. After deployment, confirm in field percentiles — subject to the 28-day window rule: field data aggregates a trailing 28-day collection window (Google, n.d. a), which a deployment's traffic replaces only gradually. But the reported number is a percentile of that window, not an average — so p75 may drift smoothly, sit flat and then step once post-deploy traffic crosses the percentile's share, or move while only the good/poor fractions shift. Two consequences follow. Declaring a fix failed — or victorious — from a partially washed-out window is misreading the instrument's clock. Collection needs 28 days to wash out — but you cannot see that window the moment it closes: the CrUX API “is approximately two days behind today's date since it waits for completed data for the day, and there is some processing time involved” (Chrome Developers, n.d. c). So the first fully clean read is not a fixed calendar day; it is the first report whose collectionPeriod.firstDate falls on or after the deploy, which in practice lands nearer day 30 than day 28. Read the dates the instrument returns rather than counting on your own calendar, and interim reads must name their contamination. And the distribution shares are watched alongside the headline percentile, because they move when p75 cannot. Claiming victory from a lab re-run alone is the same misreading from the other side: lab proof and field confirmation answer different questions, and neither substitutes for the other.

6. A worked example

The table below is an illustrative composite assembled from adjudications recorded during the framework's development; it compresses the recurring row types into one view, with audit labels preserved as recorded under the Lighthouse 12-era interface. (PSI has since moved to Lighthouse 13, which consolidates several of these audits into Insights; the verdicts attach to the underlying evidence, not the label, and every live report records the engine version it adjudicated. Individual client reports carry the full table with verbatim evidence per row.)

The verdict distribution of the example below — only the green segment receives engineering effort.

And then the row that is not in the table — because it was never a flag:

Silent Bottleneck. Initial server response time: passing (lab TTFB ~1 ms from the datacenter). Field p75 TTFB: 1.6 s — double Google's 0.8 s guideline. The number-one field problem produced zero flags. It is found by decomposing the field metric, not by reading the report harder.

Aggregated to the verdict line a client sees first: “PSI raised 21 flags: 4 Real, 6 Wrong, 9 Trivial, 2 Locked — a Noise Rate of 81% — and the number-one field bottleneck (TTFB, p75 1.6 s) appears in none of them.” Adjudication compresses the roadmap: what arrived as 21 tickets of equal apparent urgency leaves as 4 fixes worth shipping, one discovered bottleneck worth escalating, and 17 documented permissions to do nothing — each carrying the evidence a stakeholder needs to believe it.

Apply the Four Verdicts to your own warning list

This worksheet runs entirely in your browser: it stores nothing and sends nothing. Add your report’s warnings, assign evidence-backed verdicts — or hold a flag Unresolved when the deciding evidence is out of reach — and it computes the Noise Rate over decided flags, with the unresolved share reported beside it, exactly as Section 5.3 defines them.

What the professional audit adds: complete artifact collection, causal-chain reconstruction, named script ownership, platform-native fixes, falsification notes, confidence grades and quantified impact ranges — plus the Silent Bottleneck investigation itself. See everything the audit includes →

8. What the framework is not

9. Limitations

Five limitations bound the framework's claims, and stating them is part of the method.

10. Questions we are asked

11. Why these names

Noise and silence descend from the two error directions of detection theory (Neyman and Pearson, 1933), in words a site owner already owns: silence is the classical Type II miss — the report says nothing about what matters most — while noise is deliberately broader than the Type I false alarm, taking in every flag whose evidence-backed disposition is “no action”, true ones included. Verdict — rather than “finding” or “recommendation” — because a verdict closes a case: every flag leaves with a verdict, or with Unresolved and the named test that would decide it; none is left ambient to become next quarter's wasted sprint. The debts are acknowledged deliberately: screening medicine contributed the stance, signal detection the error structure, alarm-fatigue research the cost of ignoring it, and Google's own static-analysis program the standard — actionability, not technical correctness — by which flags deserve to be judged. What is new is where the instrument points: the audit of the report about the page.

Every audit we deliver is this framework executed end to end: every flag adjudicated, both metrics computed, every surviving fix predicted, proven, and confirmed on the field data's own clock.

References