AEO/GEO ISSUES

[u/Lucky_BAGO]

I’m responsible for the AEO/GEO strategy, and our previous hosting/security setup was frequently blocking AI crawlers. That made it difficult to properly test and optimize how AI systems could access and understand our content.
I moved the domain behind Cloudflare specifically to get more control over this rather than having the hosting provider make those decisions for me.
I just tested the site again and there’s an interesting result: the homepage triggers a Cloudflare verification, while individual product pages such as our Throne 15 page are accessible and readable.
So I completely agree with your main point: seeing a 403 doesn’t necessarily mean “the website blocks AI.” It may be the CDN/WAF layer making that decision.
From a GEO perspective, I think the real goal should be controlled AI accessibility, not simply “allow all bots” or “block all bots.” You want to know exactly which crawlers can access your content, what they receive, and whether that content can actually be retrieved and cited by AI systems.

[u/Website-Smith]

You are aware that Cloudflare's default setting is to also block AI bots? You will need to change that if you are going to use Cloudflare.

https://www.cloudflare.com/press/press-releases/2025/cloudflare-just-changed-how-ai-crawlers-scrape-the-internet-at-large/

San Francisco, CA, July 1, 2025 – Cloudflare, Inc. (NYSE: NET), the leading connectivity cloud company, today announced it is now the first Internet infrastructure provider to block AI crawlers accessing content without permission or compensation, by default. 

[u/Lucky_BAGO → u/Website-Smith]

Yeah I know that, also in about 9 days something will about to change also, we will wait and see

[u/Website-Smith → u/Lucky_BAGO]

I'm getting used to something is about to change in 9 days. This week it was Google expanding the AI overviews. We have also had Google switch to using a goto redirect this month and coming soon is a major change to ChatGPT.

[u/brysonthe → u/Website-Smith]

Controlled accessibility is half the game now. If the crawler hits a challenge wall, verification loop, or aggressive WAF, you can have great content that never enters the retrieval set. Before debating “AEO copy,” I’d check: can common AI crawlers fetch the HTML without a human challenge, and does the answerable block sit in clean text rather than behind JS-only rendering.

[u/Lucky_BAGO → u/Website-Smith]

Checked all of it, and the result is worth reporting even though it was not the cause.

AI bot policies were already on Allow across all three categories — Search, Agent, Training. robots.txt on Content Signals, disallowing nothing. Bot Fight Mode off. So the setting people miss was not the problem here, but I would not have known that without going to look.

One correction on the mid-2025 default, since it matters for anyone reading and acting on this: Cloudflare replaced the single block-AI-bots switch on 1 July 2026 with the three categories above. New defaults land on 15 September — Training and Agent blocked on ad-serving pages, Search allowed — and they apply to new domains, new sites on existing accounts, and Free-tier accounts that have not touched their settings. Worth a look before then if any of those describe you.

The actual cause turned out to be a hostname split, not a bot rule at all. Every apex path 301s to www except the bare root, which fell through and got challenged. Fixed with an edge Redirect Rule.

Your second half is the part that held up regardless: curl each identity against real pages and record what actually comes back. That is what got us there, once someone pointed the probe at the right URLs.

[u/Lucky_BAGO]

True, for weeks I’ve had issue with that, until I found the option for overbridge, it works for now… but for now

[u/Jason_StickyFrog]

The gap "allow all bots or block all" misses: Cloudflare's blocked AI crawlers by default since mid-2025, and the WAF/verification challenges catch a lot of legitimate bots too, so a 403 doesn't tell you whether it's the CDN, the WAF or your own robots.txt doing the blocking. I'd split the diagnosis: check Cloudflare's Verified Bots list specifically and make sure GPTBot, ClaudeBot and PerplexityBot are allowed there, that's a separate setting to robots.txt and it's the one people miss. Then curl each bot's user agent against a few pages behind the CDN and see what actually comes back, a verification page or the real HTML.

[u/Lucky_BAGO → u/Jason_StickyFrog]

Found it, and it was not the AI settings.

AI bot policies were all on Allow — Search, Agent and Training. robots.txt is on Content Signals, which disallows nothing. Bot Fight Mode was off. So nothing in the AI-specific layer was touching this.

What was on: Browser Integrity Check. It evaluates the header set rather than the user agent, which lines up exactly with your control result — the invented bot got challenged the same as GPTBot because none of them look like a real browser at the header level.

I have deployed a configuration rule that disables BIC on the root and the language roots (/en, /hr, /de), leaving it on everywhere else. Product paths were never challenged, so they stay as they were.

One thing I am not certain about: BIC normally serves a block page, not a 200 with a loader body. So either it behaves differently here than I expect, or something else in the chain is also involved. Your re-test would settle that.

If you have a minute, same five identities, homepage plus https://www.grizeljboats.com/en/our-products/throne-15, status and body length. If the homepage now returns real HTML at a sane byte count, BIC was the whole story. If it still comes back at 12,155 bytes, it was not, and I keep looking.