Do WordPress speed plugins work in the field?

The question the market can't answer

WordPress speed plugins make genuine engineering interventions: full-page caching, minification, critical-CSS generation, unused-CSS removal, lazy loading, CDN integration, image optimisation, font optimisation, script deferral, and delaying JavaScript until the first user interaction. NitroPack advertises caching, image and code optimisation, an integrated CDN and lazy loading; LiteSpeed Cache separates page-optimisation features from cache features that depend on a LiteSpeed server or QUIC.cloud; WP Rocket documents delay-JavaScript as holding script execution until interaction, tagging delayed scripts with type="text/rocketlazyloadscript" and replacing src with data-rocket-src.

Those interventions land in two different measurement regimes, and the regimes can disagree. PageSpeed Insights reports both: lab diagnostics from Lighthouse, collected in a controlled synthetic environment, and field data from CrUX — the real experience of Chrome users over a trailing 28-day window. Google's own documentation says the lab number is for debugging and may not capture real-world bottlenecks, while the field number is the true user experience. A plugin can therefore improve Lighthouse audits and still fail to move what users feel — because real visitors arrive on different devices and networks, with warm caches, from different countries, on product and checkout pages rather than the home page, through consent banners, ads and personalisation the lab never sees.

So any honest claim about a plugin has to survive three separate layers — and most marketing collapses them into one:

The design in one picture

Plugin installs are natural experiments staggered across calendar time: different origins adopt in different months. That lets us build an event study. For each treated origin we line up its CrUX outcomes by months-since-adoption, compare against matched origins that have not yet adopted, and read the treatment effect as a function of event time k. The signature of a real win is unmistakable — and so is its absence.

Formally, for plugin or feature p, adoption cohort g (the month a group of origins first adopts) and event time k, the primary estimand is the group-time average treatment effect on the treated:

ATT_p(g, k) = E[ Y_i,g+k(1) − Y_i,g+k(0) | G_i = g, T_i = p ]

Y         a CrUX field outcome at the origin-month-device level
G_i       origin i's first validated adoption month
Y(1),Y(0) potential outcomes with / without the plugin; the untreated
          path is estimated from matched not-yet-treated WordPress origins

The whole credibility of the answer rests on the left half of that picture. If the leads are flat — adopters look just like their controls in the six months before they install — then the jump after adoption is plausibly the plugin. If the leads already slope up, the “effect” is just performance-aware owners who were improving anyway, and the design honestly reports that the effect is not identified.

Data architecture

The analysis dataset is an origin-month panel built entirely from public BigQuery tables — no private crawl, no vendor data. HTTP Archive supplies the technology and resource side: httparchive.crawl.pages carries monthly page rows with rank, page summaries, Lighthouse data and Wappalyzer-detected technologies, and httparchive.crawl.requests carries one row per request, including response headers and text response bodies — the raw material for feature-level markers. CrUX supplies the field side: its monthly materialised tables (device_summary by month, origin and device; country_summary for geography) expose good/needs-improvement/poor fractions and approximate p75 values per origin — exactly the shape an origin-month event study needs.

The subtlety that wrecks naïve before/after comparisons is timing. An HTTP Archive crawl is a single page-load snapshot; a CrUX month is a trailing 28-day field window. The two don't line up, so the crawl month is treated as ambiguous rather than as “post”:

A worked example makes the rule concrete. Suppose an origin first shows the WP Rocket delay-JS markers in the June crawl. June's CrUX entry summarises real users across roughly the trailing 28 days — many of whom visited before the plugin went live. Reading June as “after” would dilute a real effect toward zero (or smear a regression). So June is the transition month, and the first outcome we trust is July (m+1). The pre-period — the placebo leads — is the spring months, which must look flat.

Form factor. Mobile/phone is the primary surface: it's the stricter performance environment, the one these plugins are marketed around, and the one HTTP Archive (mobile crawl) and CrUX (phone dimension) both cover cleanly. Desktop is secondary.

Outcomes & thresholds

The outcome hierarchy is fixed before any result is inspected — the discipline that stops a study from quietly migrating to whichever metric happened to move.

Thresholds follow Google's published Core Web Vitals definitions: Good is LCP ≤ 2,500 ms, INP ≤ 200 ms and CLS ≤ 0.1, assessed at the 75th percentile; the overall assessment passes only when all three are Good where there's sufficient data. p75 units are verified against the live CrUX schema before use.

Estimation strategy

Why not a plain two-way fixed-effects event study? Because adoption is staggered and effects are heterogeneous, and under those exact conditions a naïve TWFE regression uses already-treated units as controls for later-treated ones — the Goodman-Bacon decomposition shows this can flip the sign of a true effect. The primary estimator is therefore a modern group-time estimator — Callaway & Sant'Anna's group-time ATT (with not-yet-treated controls), cross-checked against Sun & Abraham's interaction-weighted event study. Plain TWFE is reported only as a diagnostic, never as the headline.

Effects are aggregated with cohort-size weights and reported dynamically for k = −6…+12, with three focal horizons: a short run (k = +1…+3, after the timing ambiguity clears), a medium run (+4…+6, after caches and image-optimisation queues settle), and persistence (+7…+12). Standard errors cluster at the origin with a block bootstrap; Benjamini-Hochberg false-discovery control spans the plugin × metric × form-factor family, while unadjusted intervals are preserved for effect-size reading. The transparent DiD representation — useful to show, not as the sole estimator — is:

Y_it = α_i + γ_t + Σ_(k ≠ −1) β_k · 1[event_time_it = k] + X_it'θ + ε_it

α_i   origin fixed effects        γ_t   calendar-month fixed effects
k=−1  omitted reference month     β_k   the event-time coefficients (leads + lags)

The coefficient plot must show leads as well as lags. A positive post-adoption estimate is not credible if the leads are already drifting up before the install — which is exactly what the next section is built to catch.

Falsification & robustness

A field-positive verdict has to survive a battery of attempts to break it. These are pre-committed, not chosen after seeing results:

• Pre-trend test. Leads at k = −6…−2 must be jointly near zero. A small wiggle is fine; a monotone pre-adoption climb is disqualifying.
• Placebo adoption. Assign pseudo-adoption months to matched controls and re-estimate — placebo effects should centre on zero.
• Removal symmetry. If a plugin lifts field metrics, validated removals should attenuate or reverse the gain — strongest evidence for caching, CDN and resource-optimisation mechanisms.
• Feature contrast. WP Rocket with delay-JS vs without. If the delay-JS arm improves LCP but not INP, that's the load-versus-interaction trade-off (H3) caught in the data.
• Lab–field contrast. Run the same events against Lighthouse/TBT outcomes. The “lab-only” signature is a clear lab gain with a CrUX ATT indistinguishable from zero.
• Window & encoding sensitivity. Persistence of one/two/three crawls; windows of [−3,+6], [−6,+12], [−12,+12]; good-share vs pass-indicator vs p75 — a result that survives only one fragile encoding is downgraded.
• Cohort restrictions. Re-run excluding host/CDN or theme changes, major WordPress core-release windows, and the top-1,000 rank band where bespoke stacks dominate.

Limitations — read before quoting

1. Page vs origin

   HTTP Archive crawls a root page (and, since 2022, a secondary page for many origins) while CrUX is origin-level. A plugin may help product pages more than the home page, or vice-versa; the origin outcome blends them.

2. Coverage floors

   Low-traffic origins are absent from CrUX, and CrUX reflects eligible Chrome users, not all browsers. Effects generalise to sites big enough to be measured.

3. Detection blind spots

   Wappalyzer and raw markers can miss hidden or aggressively-optimised implementations; brand markers for NitroPack and LiteSpeed start as candidates until validated.

4. Configuration is partly unobservable

   Plugin effects depend on settings (which caches, which exclusions) that response bodies only partly reveal — so estimates are intent-to-treat by detected state, not by optimal configuration.

5. It's still observational

   Matching and pre-trend tests harden the design, but only a randomised install would close every door. The correct response is to validate detection, stratify hard, and refuse to generalise past identified strata — not to retreat to lab scores.