Publishers weigh blocking Google as search referrals collapse

By Helen Ross · 2026-07-26

How AI Overviews changed the referral traffic model

Google's search results page looked meaningfully different by mid-2025. AI Overviews and AI Mode had taken up residence at the top of the page, synthesising answers from across the web and presenting them directly to users before a single blue link appeared.verifiedVerified Source: blog.google^[5] Publishers felt the effect immediately: readers who once clicked through to a story now had their question answered without ever leaving Google.

The referral traffic model that sustained online journalism for two decades rests on a simple exchange. Publishers grant Google access to their content; Google sends readers back as reward. AI Overviews broke that arrangement by consuming the content and returning the value directly to the user on Google's own page, with no click required.

The traffic collapse in numbers

Pageviews from Google Search fell 34 per cent between December 2024 and December 2025, according to Chartbeat's Q1 2026 report, with publishers' sites recording sharp referral declines from mid-2025 onwards.verifiedVerified Source: lp.chartbeat.com^[1] The damage was not shared equally. Small publishers faced a 60 per cent year-on-year decline in search traffic, compared with a 22 per cent decline for large sites, underscoring how acutely the shift has hit outlets without diversified audience channels.verifiedVerified Source: lp.chartbeat.com^[1]

Among named outlets, the numbers are starker. Business Insider's Google referral traffic fell more than 85 per cent year-on-year to June 2026. USA Today's national site was down nearly 50 per cent over the same period. CNN and Politico each recorded declines of roughly 25 per cent. These figures represent the systematic erosion of audience pipelines that underpin advertising revenue and subscriber acquisition alike.

Publishers push back: the blocking threat

USA Today CEO Mike Reed said publicly that blocking Google's crawler was under active consideration. Reuters president Paul Bascobert made the same point. Politico, The Economist and Reddit were also reported to be weighing the option by mid-2026.^[6] The argument, said plainly, is that feeding Google content that powers AI Overviews while receiving no traffic in return is not a partnership; it is extraction.

Blocking Googlebot via robots.txt is technically straightforward. A publisher adds a handful of lines to a text file and Google's crawler stops visiting. What follows is less straightforward: the site disappears from Google Search entirely, forfeiting whatever traffic remains. For outlets where Google still delivers 30 or 40 per cent of readers, that is a significant gamble on the proposition that other platforms, direct newsletters and search competitors can fill the gap.

The single-crawler problem

Google operates one crawler, Googlebot, for both its standard search index and its AI features. That architectural fact sits at the centre of the publisher dilemma. A Google deposition witness, Parakh, confirmed in the US v. Google LLC antitrust proceedings that publishers using robots.txt to opt out of AI model training also opt out of Google's entire search index, leaving no middle path between full exposure and full invisibility.^[4]

Google's Ron Eden, Principal of Product Management, said the company was working to change that: "Building on this framework, and working with the web ecosystem, we're now exploring updates to our controls to let sites specifically opt out of Search generative AI features."^[5] On 3 June 2026, Google rolled out a "Search generative AI control" toggle in Search Console, allowing publishers to opt out of appearing in AI Overviews, AI Mode and other generative AI features.^[3] The toggle was a step forward, but it did not resolve the underlying architecture. Because the same crawler underpins both systems, the full opt-out via robots.txt still carries the all-or-nothing consequence confirmed under oath in the antitrust case.

The UK Competition and Markets Authority moved to address the imbalance at the regulatory level. On 3 June 2026, the CMA imposed a publisher conduct requirement on Google, mandating that Google provide publishers with effective controls over the use of their content in generative AI features.^[2] It was the first such requirement imposed on Google anywhere in the world, signalling that regulators had concluded voluntary arrangements were insufficient.

What publishers and advertisers face next

The consequences extend well beyond newsrooms. Any business that has built its customer acquisition model on organic Google traffic faces the same structural exposure. Advertisers who placed premium value on publisher audiences because those audiences arrived via search intent are now buying against a smaller, and structurally different, pool of readers.

Publishers that move first to block Google accept short-term pain in exchange for the argument that they are not subsidising a competitor's AI product. Those that wait hope regulatory pressure from bodies like the CMA, combined with litigation discoveries from US v. Google LLC, will force a technical fix that allows meaningful separation between AI training consent and search index inclusion. As of the time of publication, no such fix existed, and the Chartbeat data for Q1 2026 showed no sign of the decline in Google Search pageviews reversing.

Helen Ross

Helen Ross writes about wealth, inheritance and the human side of money. She is interested in the conversations families struggle to have, and what happens when they don't.