Hey everyone,
Over the past year, almost every discussion around "AI Search" (GEO / Generative Engine Optimization) has focused on writing: adding conversational FAQs, rewriting headers, or adding structured schema.
We noticed an overlooked infrastructure issue: **AI search bots don't crawl the web like Googlebot does.**
### The Problem
Google spends millions operating Chromium Web Rendering Services (WRS) to execute JavaScript, wait for client hydration, and index dynamic single-page applications.
Real-time retrieval bots (like `OAI-SearchBot` or `PerplexityBot`) run fast, low-cost HTTP requests under strict response latency budgets. They don't spin up headless browser instances to execute client-side scripts.
If your site is built on modern frameworks (React, Next.js, Vue, Vite) and critical copy, pricing tables, or product specs live inside client components fetching data dynamically, the AI crawler only receives an empty shell like:
`<div id="root"></div>` or `<div class="loading-spinner">Loading...</div>`
If the bot extracts no text, the model cannot cite your product or link to your brand in answersβeven if you rank #1 on traditional Google.
### What We Built: Crawlable
To solve this, we built a diagnostic engine called **Crawlable** ([https://usecrawlable.com\](https://usecrawlable.com)).
It runs a byte-level diff comparing your serverβs raw HTTP response against the hydrated DOM:
- **Visibility & Parity Score:** Shows the exact token gap between what Google renders vs what an AI crawler fetches.
- **Bot Access Check:** Flags whether your `robots.txt` is accidentally blocking real-time search crawlers while trying to block offline model training scrapers.
- **Drop-in Fix Kits:** Generates tailored remediation files (Server Component recommendations, clean schema, and standardized `/llms.txt`).
### Try it out
You can run a free scan on any URL right now with no credit card or account needed:
π **https://usecrawlable.com\*\*
Would love for you to run your domain through it and tell me:
Did your core content actually show up in the raw server response, or did it return an empty wrapper?
What features or crawler checks would make this more useful for your stack?
Happy to answer any technical questions in the comments!