How we test Chrome email tracker blockers

Scope and eligibility

The benchmark covers Chrome extensions whose tested tier provides active receiving-side tracker blocking in webmail. A direct multi-webmail candidate must demonstrate that behavior on at least three separately named webmail families; Outlook, Hotmail and Microsoft 365 count as one Microsoft family rather than three clients. A store description records claimed coverage, not measured coverage. A detector-only free tier remains a reference unless its blocking tier is separately acquired, disclosed and tested.

Freeze the environment

1. Record the UTC test date, Chrome version, operating system and extension version.
2. Acquire the exact current-public package, recompute its SHA-256, extract it in the harness and bind the loaded runtime identity to that package.
3. Use a fresh Chrome profile for every product. Apply a hash-bound, versioned setup driver and record whether protection was the default or configured state and whether that comparable state was free or paid.
4. Record any login, permission prompt, paywall, unavailable client or setup failure instead of treating it as a pass or fail.

Separate ranking evidence from rule conformance

The locked ranking corpus uses independently documented tracker endpoint patterns and observed or controlled benign resources. Rows retain source and family provenance and are macro-averaged by family so aliases from one vendor cannot dominate the result. Mailshade's generated catalog is exercised by a separate source-derived conformance corpus. Because those probes are produced from the same catalog as its rules, they are explicitly ineligible for ranking. Fixture IDs and expected outcomes are versioned before they can affect a later score.

Measure observable outcomes

• Tracker blocking efficacy: a ready Chrome network monitor observes the paired control but not the planted tracker request.
• Benign precision: a benign request reaches the monitor and the calibrated extension UI does not flag it.
• Multi-webmail coverage: real-client evidence counts only when the locked client fixture, tracker request and benign control all run successfully.
• Link protection: a trusted browser gesture must not leak the original tracking URL; an explicit warning or verified direct destination is recorded separately.
• Privacy and permissions: the exact manifest, granted origins and sanitized page, extension and service-worker network capture are reviewed.
• Local evidence and reporting: per-message evidence, retained local history and an inspectable or exportable record are captured from the tested build.

Gmail-proxy behavior is its own locked stratum: a known tracker endpoint embedded in the proxy URL must be blocked, while an ordinary proxy image and a benign sibling path on the same vendor host must both load. The result proves receiver-side request behavior in the tested Chrome page; it does not claim visibility into an earlier Gmail server-side cache fetch.

Hard failures and scoring

An automatically leaked known tracker URL, a blocked or flagged benign control, or undisclosed product-controlled egress of message, identity or secret data is a hard failure for that product. Diagnostic scores remain visible, but a product with a hard failure cannot be named the category winner. Missing evidence is unassessed, never silently converted into a pass.

Publish before concluding

Every comparison page links its sources, exact tested cohort, version record, methodology hash, result status and conflict disclosure. Pending means the shared run has not completed. Inconclusive means a run exists but missing comparable evidence prevents a defensible ranking. Verified requires the test date, a local gate-checked public raw-results artifact and its SHA-256. A winner additionally requires complete rubric evidence, current-public packages for every direct candidate and one uniquely highest eligible score. Corrections create a new dated record rather than silently rewriting history.