Should You Block AI Crawlers? (And The Ones You’re Blocking By Accident)

Everyone wants to get cited by ChatGPT.

We’ve covered how to appear in AI responses, the KPIs worth tracking, and how to actually see AI traffic in GA4.

All of that assumes one thing.

That the bots can get to your site in the first place.

Quite often, they can’t. And nobody notices, because the robots.txt hasn’t been opened since a dev touched it in 2019, and (almost) nobody reads server logs for fun.

Go and check yours. yoursite.com/robots.txt.

Isn’t An AI Crawler Just A Crawler?

No.

There are three completely different jobs being carried out by bots with confusingly similar names.

Training crawlers. GPTBot, ClaudeBot, CCBot, Google-Extended. They collect content that may be used to train models. They send you nothing back.

Search and retrieval crawlers. OAI-SearchBot, Claude-SearchBot, PerplexityBot. These build the index that AI answers get pulled from. These are the ones that get you cited, with a link.

User-triggered fetchers. ChatGPT-User, Claude-User. These fire when a real human asks the assistant to go and look at a specific page. That’s about as close to an actual reader as anything in this list.

Blocking all three because you saw a LinkedIn post about AI stealing your content is a bit like unplugging the router because you’re worried about spam.

Blocking GPTBot Does Not Block ChatGPT

This is the big one, and we’ve now seen it in enough audits to be confident it’s the default state of the web rather than the exception.

GPTBot is OpenAI’s training crawler. OAI-SearchBot is the one that indexes pages for ChatGPT search. Separate user agents. Separate rules. OpenAI documents them separately and explicitly says you can allow one and block the other.

So when someone tells you “we’ve blocked OpenAI”, they could mean two wildly different things, and there’s a decent chance they don’t know which.

If you want out of training but still want to be citable:

That’s it. That’s the whole manoeuvre. The same logic applies to Anthropic, where ClaudeBot does the training and Claude-SearchBot does the searching.

The Google-Extended Misunderstanding

Same family of error, slightly different flavour.

Google-Extended is a training opt-out for Gemini. It is not a search control.

Blocking it does nothing to your rankings. It also does nothing to your presence in AI Overviews or AI Mode, because those are served from Google’s regular index, crawled by Googlebot.

Which cuts both ways, and this is the bit people don’t like hearing. If your client wants out of AI Overviews, Google-Extended won’t do it. The only lever that would is blocking Googlebot, and I don’t think I need to talk you through why that’s a poor trade.

But Look At The Crawl-To-Refer Ratios

Fair. They’re grim, and they’re a big part of why sites started slamming the door in the first place.

Cloudflare Radar tracks the number: pages a crawler takes versus visits its platform sends back. Google sits at roughly 5:1. The AI crawlers have been reported anywhere from the low hundreds to five figures to one.

But here’s my problem with using it as a policy input. Anthropic’s ratio was being quoted above 10,000:1 in the spring and under 2,000:1 by July. OpenAI’s moved by a factor of five inside a single quarter. Different analyses of the same Cloudflare data published in the same month rank the offenders in a different order.

Don’t set a permanent access policy off a number that swings by an order of magnitude every eight weeks. Put it in a client report if it’s useful ammunition. Don’t wire it into your robots.txt.

There’s also evidence the block doesn’t buy what people think it buys. Research from Rutgers and Wharton published at the end of last year found publishers who blocked AI crawlers took a meaningful hit to overall traffic without reliably reducing how often they were cited anyway.

Worst of both, in other words.

The Blocks You Didn’t Choose

This is the part that catches agencies out, and the reason I’d audit before I’d advise.

Your robots.txt can be immaculate and the bots still can’t get in.

• CDNs ship managed bot rules, and some now challenge or block AI crawlers on new zones by default
• Managed hosts run their own bot mitigation on top of that
• Someone on the dev team added a firewall rule during a traffic scare in 2024 and told nobody
• Aggressive rate limiting that reads a crawler as an attack

robots.txt is a polite request. A WAF rule is a wall. When the two disagree, the wall wins, and nothing in Search Console will tell you it happened, because Search Console only cares about Googlebot.

The only way to know is to go and look. Server logs or CDN analytics, filtered by user agent, checking for 403s and 429s.

Do I Need An llms.txt Then?

Almost certainly not.

Google’s Gary Illyes said they don’t support it and aren’t planning to. John Mueller compared it to the keywords meta tag, which should tell you roughly everything you need to know about its future. Google’s own guidance on AI features states plainly that you don’t need to create special machine-readable files to appear in Search or its generative surfaces.

Mark Williams-Cook has even gone so far as to “prove” llms.txt is no more effective than cats.txt (see above).

No major provider has committed to using it in production, and every log study we’ve seen finds the file barely gets requested by the crawlers that matter.

It isn’t harmful. It takes ten minutes. If you publish developer documentation that gets consumed by coding assistants, there’s a genuine case for one. For a plumbing company in Solihull, it’s decoration.

Mueller did make a better point in passing, though: the most basic form of optimising for agents is making sure agents aren’t blocked from reaching your site at all.

Right, What Do I Actually Do?

Five things, in this order:

1. Read your robots.txt. Properly.
2. Check the layer above it. CDN, firewall, host. Go looking for a “block AI bots” toggle somebody flipped.
3. Split by purpose. Block training if you feel strongly about it. Allow retrieval and user fetches unless you’ve got a specific reason not to.
4. Verify in the logs. Filter by user agent, look for 403s, and check requests against published IP ranges – anyone can stick “GPTBot” in a user agent string.
5. Then go back to the real work. Access makes you eligible. It doesn’t make you cited. Getting mentioned on the sites AI already trusts does that.

The Robot-Friendly Summary

Blocking AI crawlers is a legitimate decision with real trade-offs.

Most sites aren’t making it, though – they’re stumbling into it. GPTBot and OAI-SearchBot are different bots doing opposite jobs. Google-Extended has nothing to do with AI Overviews.

Your firewall may be overruling your robots.txt.

And llms.txt isn’t doing anything for you unless you’re publishing developer docs.