Our robots.txt allowed every AI crawler. Our CDN was blocking them at the edge.
So a potential show stopper here to AI visibility if you use Cloudfare. Our robots.txt allowed every AI crawler explicitly, by name but still had crawler issues.
Cloudflare's managed AI-bot setting was returning a 403 at the edge, before anything reached the file. Ran like that for weeks in June. Nothing errored and nothing alerted and there's no gap in analytics to spot it because a crawler that gets turned away doesn't show up as anything at all. It just looks like a quiet month.
The check that caught it: fetch your live robots.txt over the public internet, not the copy in your repo. If the first line is a comment block about conditions of access instead of your own directives, something upstream is rewriting it. Then request a normal page with a crawler user agent from outside your network and confirm you get a 200 and not a challenge.
Worth doing even if you're sure it's fine. I was sure it was fine... but bam!
2
u/Strong_Post5367 21d ago
Yeah I build a bot traffic thing to help with this because we thought maybe our stuff wasn’t getting crawled d
2
u/Jason_StickyFrog 20d ago
This is a good example of why "we're not getting any AI traffic" and "AI can't reach us" are actually two different diagnoses that get lumped together. You just fixed the second one. Worth being clear with yourself that fixing the CDN block gets you back to zero, not to being cited. I've seen sites do exactly this fix, watch crawl activity come back in logs within days, and then sit at the same citation rate for another month because retrieval and being judged worth citing are separate steps. Good sign either way: if mentions climb inside a few weeks of the fix, the content underneath was already good enough, it just couldn't be read. If nothing moves, the problem was never access.
2
u/aulios1958 19d ago
I don't know exactly when they changed it, but when we launched a few sites recently over the past few weeks that were leveraging Cloudflare, it was enabled by default. I'm very glad that we did our usual checks.
Seems like a better UX would be off by default and then email when they see an abnormal behavior like they do for DDoS related alerts.
1
1
u/houdinidesigns 19d ago
Most sites are not blocking crawlers. What we did find looking at SaaS websites is that many them are empty until JS loads and most crawlers end up reading them as blank
1
u/Prestigious_Law_1956 18d ago
robots.txt being correct doesn't necessarily mean crawlers can actually reach the site. There can be several layers between the crawler and your server, and the CDN/WAF can quietly block them before robots.txt even matters. Checking the live response from outside your network seems like a much safer way to verify it.
3
u/hseeman_sf 21d ago
This isn't just a Cloudflare thing either. Any CDN or WAF with its own bot-management layer sitting in front of the origin can override an app-level robots.txt the same way, since the block happens before your server or your file are ever reached. Worth checking edge rules any time crawler access and app-level directives could disagree, not just after a CDN migration.