Short answer: we tested 74 agency and martech websites that sell AI search visibility. Not one of them disallowed an AI crawler in robots.txt. Fifteen percent of them returned a block anyway when a request arrived carrying an AI crawler’s user agent. The rule doing the blocking is not in robots.txt, so the site owner has no reason to know it exists.
Why we ran this
In September our own site started returning 429 to GPTBot while our robots.txt explicitly welcomed it. Nothing in WordPress or robots.txt said so. The block lived at the hosting layer, in rules nobody at our end had written, and we only found it because we went looking.
That raised an obvious question. If it happened to us while we were actively working on AI visibility, how common is it among everyone else doing the same work? So we tested the people most likely to have it right: agencies and tools that sell answer engine optimization, generative engine optimization and AI search visibility.
Method
Seventy four domains, drawn from the agencies named across the main published best AEO and GEO agency lists, plus the martech tools and publishers in the same space. For each domain we did four things:
- Fetched robots.txt as an ordinary browser.
- Parsed it with a standard robots parser and asked, per crawler, whether the homepage was allowed.
- Fetched the homepage once as an ordinary browser, to confirm the site was up.
- Fetched the homepage once per crawler, sending that crawler’s published user agent string.
Four crawlers: GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot. One request per crawler per domain, a second and a half between domains, read only, homepage only. Sixty six of the 74 were reachable by a normal browser and form the sample below. Run on 17 October 2026.
What we found
Nobody blocks AI crawlers on purpose. Across 66 sites, the number that disallowed GPTBot, ClaudeBot, PerplexityBot or OAI-SearchBot in robots.txt was zero. Every single one of these businesses has decided, in the one file built for the decision, to let AI crawlers in.
Ten of them block anyway. Fifteen percent returned something other than a 200 to a request carrying an AI crawler user agent that their own robots.txt allowed.
| Crawler | Allowed in robots.txt | Returned 200 | Blocked despite being allowed |
|---|---|---|---|
| ClaudeBot | 65 | 57 | 8 |
| GPTBot | 65 | 61 | 4 |
| PerplexityBot | 65 | 64 | 1 |
| OAI-SearchBot | 65 | 64 | 1 |
ClaudeBot gets blocked twice as often as GPTBot. Eight sites versus four. Six of the ten blocking sites blocked ClaudeBot and nothing else, which points at rule sets that were updated for OpenAI’s crawler when it became famous and never revisited for Anthropic’s.
The blocks are split between two causes. Of the fourteen blocked responses, eight were 403 and six were 429. A 403 is a firewall or bot management rule saying no. A 429 is a rate limit treating a single polite request as abuse. One site in the sample blocked all four crawlers with a 403.
The honest caveat, because it matters
We sent the published user agent strings from our own infrastructure, not from the crawlers’ real IP ranges. OpenAI, Anthropic and Perplexity all publish the IP ranges their crawlers use, and a well configured bot manager verifies the IP rather than trusting the user agent.
So some share of these blocks is verification working exactly as designed: an unverified client claiming to be GPTBot got refused, which is correct behaviour. The rest is a rule firing on a crawler the owner meant to allow.
From the outside you cannot tell which is which, and here is the uncomfortable part: neither can the site owner, unless they look at their own server logs. That is the actual finding. Not that fifteen percent of this industry is blocking AI, but that fifteen percent cannot answer the question from their robots.txt alone, and most have never checked anywhere else.
How to check your own site in two minutes
Run these from a terminal. Each one asks your homepage for a response as a specific crawler.
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.1; +https://openai.com/gptbot" https://yourdomain.com/
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" https://yourdomain.com/
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" https://yourdomain.com/A 200 means a request wearing that name gets through. A 403 or 429 means a rule fired. Then do the part most people skip: open your server or CDN logs, filter for those user agents over the last 30 days, and see what real traffic from the verified IP ranges actually received. The logs are the only place the true answer lives.
What to do if you find a block
- Check the bot management layer, not the CMS. On Cloudflare that is Security, then Bots, plus any WAF custom rules and rate limiting rules. On managed WordPress hosts it is often a security module you never configured.
- Look at rate limits before firewall rules. A 429 usually means a generic threshold is catching a crawler that fetches many pages quickly, which is just what crawlers do.
- Allow by verified identity, not by user agent string. Verify against the published IP ranges so you admit the real crawler and still refuse anything merely claiming to be it.
- Decide training and answering separately. GPTBot and ClaudeBot are largely about training. OAI-SearchBot and PerplexityBot are about being retrievable in answers. Plenty of businesses want the second without the first, and that is a legitimate position, but it should be a decision rather than an accident.
- Re test after any change to hosting, CDN, security plugin or WAF, because that is when these rules reappear.
Limitations
Small sample, one industry, one homepage request per crawler, one point in time. Blocks can be intermittent, geographic, or triggered by our requesting IP rather than the user agent. We have not named any site, because the point is the pattern rather than any one company’s configuration, and because a 403 to an unverified request may be correct behaviour. Anyone can reproduce the method above on any sample they like.
AI Crawler Access: Frequently Asked Questions
Does robots.txt control whether AI crawlers can read my site?
It controls whether they are permitted to, not whether they succeed. Robots.txt is a request that well behaved crawlers honour, while firewalls, bot managers and rate limits sit in front of it and can refuse the connection before robots.txt is ever considered. A site can welcome a crawler in robots.txt and still return 403 to it on every request.
Why would my host block an AI crawler I never blocked?
Because most bot management ships with defaults, and many hosts enable rules that treat unfamiliar high volume clients as a threat. Those rules are usually written against categories rather than named crawlers, so a new or less famous crawler gets caught by a rule aimed at scrapers. Nothing in your CMS will show it.
Is a 429 as bad as a 403 for AI visibility?
In practice it is often worse, because it looks temporary. A 403 is an unambiguous refusal that a crawler will usually back off from and record. A 429 invites retries, so a crawler can spend its budget on your site repeatedly without ever retrieving a page, and the pattern is easy to dismiss as noise when you see it in logs.
Should I block AI crawlers to protect my content?
That depends on whether you make money from people finding you or from people paying for your archive. If you sell services and want to be recommended inside AI answers, blocking the crawlers that build those answers removes you from the shortlist. If you are a publisher whose value is the content itself, blocking training crawlers while allowing search crawlers is a reasonable middle position.
How often should I check this?
After any change to hosting, CDN, security plugin or firewall, and otherwise once a quarter. These rules are rarely added deliberately, which means they are rarely announced, and the only reliable signal is testing it yourself.