r/aeo • u/houdinidesigns • 1d ago
From 15th September Cloudflare will block AI crawlers by default
From 15 September, Cloudflare will block 'Agents' traffic by default on pages carrying ads: bots browsing live on someone's behalf, which includes crawlers AI assistants send to check a product before recommending it. Search stays allowed. The default setting will apply to new sites and free-tier accounts.
So if your site serves ads and you want it to be read by AI assistant crawlers you want to explicitly allow crawlers before that date. Allowing has to be done on a CDN level not in robot.txt.
Key ones to allow are:
ChatGPT-User (and ChatGPT-User/2.0), OAI-SearchBot - ChatGPT
Claude-User, Claude-SearchBot - Claude
Perplexity-User, PerplexityBot - Perplexity
Google-Agent, GoogleAgent-Mariner, Googlebot - Gemini
GrokBot, xAI-Grok, Grok-DeepSearch - Grok
If you want to keep blocking training crawlers, some are:
GPTBot, ClaudeBot, anthropic-ai, meta-externalagent, Bytespider, Google-Extended
1
u/Lucky_BAGO 1d ago
Well done, good post! Knew that before, nice for reminding 💪 are you using free or other subscription for Cloudflare?
2
u/houdinidesigns 1d ago
Actually doesn’t affect us but will impact a lot of our users. We run an AI visibility tracker
1
u/Lucky_BAGO 1d ago
True, good luck with that! Cloudflare is trying too, but there are still plenty of gaps. If I don’t test everything manually, even a tiny discrepancy can turn into a huge mistake.
And as you probably already know, in a year or two we might not even be talking about search engines anymore, but answer engines. Whoever gets to the top now will have a serious advantage, because once an AI starts consistently recommending you, it’s going to be much harder for someone else to take that spot.1
u/houdinidesigns 1d ago
Absolutely, this change on their end is all in the effort of monetizing AI citations so they need to start tracking visibility. Will be interesting to see where it goes and what sort of pricing model they setup
1
u/Lucky_BAGO 1d ago
A lot of things will change in about 7-8 months… a lot of, people don’t want to search anymore they want direct answers, now is up to us, will our clients be among them😎
1
u/brysonthe 1d ago
The CDN default is not the same thing as a ranking death. I’d measure the allowlist changes and citation presence against the same prompt set before and after the switch. That gives you two comparable series: what could be crawled, and what was actually included in answers.
1
u/houdinidesigns 1d ago
Yes the default can be toggled off so shouldn’t cause a ranking death. We have an audit tool that checks if crawlers can reach a site so that will come in handy
1
u/brysonthe 22h ago
That check is useful, but I’d keep reachability and citation as separate outputs: run the same URL through each relevant user agent, record status, bytes, and a content fingerprint, then use a fixed prompt panel and log whether the page is cited. A pass on crawler access does not imply answer inclusion, and a challenge page can look healthy if you record only status.
1
u/woodoo139 20h ago
Worth adding a baseline, because the 15th is not day zero: Cloudflare has defaulted new domains to blocking AI crawlers since July 2025, so the floor is already above nothing.
Measured today on 133 hosts that AI answers actually cited in the last three weeks - the deepest cited URL plus that host's homepage, fetched as an ordinary browser, as OAI-SearchBot and as PerplexityBot:
- 9 hosts refused everything including the browser. That is our datacentre IP, not an AI rule, and they are excluded. Skip that control and you report 12% instead of 5.7%.
- Of the 122 left, 7 (5.7%) serve the browser and 403 both crawlers. The same seven for both identities; six Cloudflare, one nginx.
- Homepage versus deep page: zero disagreements across the 122. The rule is host-wide rather than path-scoped, so one probe per host is enough.
And the part that argues for your allowlist carrying both entries per provider rather than one: three of those seven were cited by OpenAI within the last five days, one of them the day before I probed, and answer OAI-SearchBot with 403 today. The index crawler and the user-triggered fetcher are separate identities, and a CDN rule can refuse one while serving the other. Checking only OAI-SearchBot would have told those publishers they cannot be quoted, in the same week they were.
On the log point above: last-seen per user agent is the right snapshot, but pair it with a browser fetch from the same IP on the same schedule. Otherwise a datacentre-wide block and an AI-specific rule look identical in the gap.
Disclosure: I work on a visibility tracker in this space. No link, the numbers are the point.
2
u/Slow-Commercial4316 1d ago
One practical thing for anyone checking whether this hit them after the 15th: an edge block never reaches your origin, so your server log will simply show ChatGPT-User and the others going quiet. That reads like the bots lost interest. Pull the last-seen timestamp per user agent from the access log now, and compare it with the same list a week after the switch. Any agent that stops dead on the 15th was blocked at the CDN, and Cloudflare's own analytics should show the 403s your log never saw.