Anthropic blocked crawlers and its pages hit Google anyway

By Jonas Valenti · 2026-07-29

Chat pages surface in search

Shared Claude conversation pages turned up in Google and Bing search results in July 2026, exposing content users had not intended to make permanently discoverable. Google spokesperson Ned Adriance said "Neither Google nor any other search engine controls what pages are made public on the web, and these pages were indexed across many search engines."verifiedVerified Source: wired.com^[1] Adriance placed responsibility squarely on the site owner, not the search engine.

The two-step confusion: crawl blocking versus index blocking

Most site owners treat robots.txt as a privacy fence. It is not. Google's Search Central documentation states that when a page is blocked by robots.txt, Google cannot crawl it and may still index the URL if it discovers links to it elsewhere.verifiedVerified Source: developers.google.com^[2] A Disallow rule tells a crawler not to visit a page; it tells no one to forget the page exists.

Anthropic's own Help Centre makes the same point plainly: a Disallow rule in robots.txt tells crawlers not to fetch those pages, but does not stop search engines listing the blocked URLs if they find them linked elsewhere on the web.^[3] The distinction is technical; the consequence is public.

Why a robots.txt block alone is not enough

Search engines build their index from two signals: what they can crawl, and what other sites link to. Block the crawl and the second signal still operates. Any shared Claude chat link posted to a public forum, a social media thread or a third-party site becomes a reference point a crawler can log even without visiting the destination.

As of 27 July 2026, shared Claude chat pages served an X-Robots-Tag header of none, equivalent to noindex, hidden behind a robots.txt block, which prevented Google from reading the directive until Anthropic removed the crawl block.verifiedVerified Source: searchenginejournal.com^[4] The noindex signal was present but completely invisible to Google, because the robots.txt block stopped the crawler from fetching the page to read it.

The fix: a noindex tag the crawler can actually read

Google's Search Central guidance is unambiguous: only a page-level robots meta noindex tag reliably keeps a page out of Google's index, provided the crawler can fetch the page to read the tag.^[5] The critical condition is access. A noindex instruction buried behind a crawl block is like a no-entry sign on a locked door the inspector cannot open.

Anthropic's Help Centre documentation identifies the same solution for shared content: "The noindex robots meta tag is a rule that tells our partners not to index your content so that they don't send it to us in response to your web search query."^[6] The correct sequence is to allow the crawler to fetch the page, then instruct it via a noindex tag not to include that page in results.

What Australian businesses with share links or portals need to know

Any organisation that publishes one-click share links, client portals, proposal documents or a public-facing copy of its site faces the same exposure Anthropic encountered. A developer who blocks the crawl and considers the job done has only completed half the task. The robots.txt rule protects page content from being read; it does not protect the URL from being catalogued.

The practical checklist is short: confirm the noindex tag or X-Robots-Tag header is present on sensitive pages, confirm the robots.txt file does not block access to those same pages, and verify the configuration using Google's URL Inspection tool. Anthropic corrected its own configuration after the indexed pages were publicly reported on 27 July 2026.

Jonas Valenti

Jonas Valenti writes about search and how businesses get discovered. He has spent years watching what makes a company visible online, and is unsentimental about tactics that no longer work.