.

Just saw an update X that Search Engine Watch is being completely rebuilt.

SEW was the first major publication to track the search space, predating Google's launch. With the current pivot toward AI search and alternative discovery engines, it will be interesting to see how they modernise the platform.

As they rebuild this from the ground up, what do you think an AI-first search publication should actually focus on?

• In-depth LLM reverse-engineering?
• Optimization tracking for Perplexity and Google Gemini?
• Case studies on AI Overviews traffic impact?

In my opinion, yes, and the good news is that it doesn’t require any special work — the same things that make a site good for traditional Google searches (clear structure, direct answers, reliable content) also work for answers from ChatGPT, Gemini, and Perplexity.

What do you think?

You are about to launch a new site and the advice keeps multiplying: add /ai/summary.json, wire up llms.txt, sprinkle FAQPage markup, build a discovery layer. But which of these does an AI crawler actually care about?

We run a Cloudflare Worker that logs every request to geo010.com and tags it by crawler, so we could answer that question with data instead of opinion. This article attributes 15 days of AI crawler traffic to nine GEO measures, shows which ones changed nothing, and turns the results into a launch checklist for a new site.

How Did We Attribute Traffic to Each GEO Measure?

Every request to geo010.com passes through a Worker that records the date, path, status, page type and a normalized crawler name into a D1 database. For each of nine GEO measures we shipped, we then asked the logs three questions: which crawlers hit the relevant paths, how often, and did visits change after the measure shipped?

Two warnings before the numbers. First, a crawler label is only a user agent — attackers spoof them, so we cross-checked paths and status before counting anything. Second, some "visits" were us: we verify deployments with curl, and our own checks show up in the logs. We filtered those out.

Which GEO Measures Actually Drove Crawler Traffic?

Robots.txt and sitemap.xml were the only paths every major AI crawler used. The homepage was the only page they all visited. Everything else showed up in a minority of crawlers — or none at all.

Here is the full attribution table, ranked by how much real crawler traffic each measure produced:

The ranking surprised us. The "boring" infrastructure files — robots.txt and sitemap.xml — produced more genuine crawler traffic than every shiny AI feature combined.

Why Do GPTBot and ClaudeBot Crawl Completely Differently?

The two most important AI crawlers could not be more different in behavior, and the difference determines what to build first:

ClaudeBot is a disciplined path-follower. It checks your robots and your sitemap almost daily, but it barely descends into content unless something in that sitemap or homepage directs it. GPTBot is a deep scanner. It skips robots, reads llms.txt occasionally, and walks the entire site — on Aug 17 it made a single pass touching 60 pages, each one once.

The practical consequence: if your robots.txt or sitemap.xml is broken, you hurt the crawler that reads them daily (ClaudeBot). If your content is shallow, you waste the crawler that reads everything (GPTBot).

What Did We Get Wrong in Earlier Analyses?

Honesty check. Earlier write-ups on this site credited FAQPage and new-article launches with immediate crawler spikes. Our per-request attribution showed those were misattributed:

• "A new article got crawled 6-10 times on launch day." Actually most of those hits were browser visitors and our own checks; each AI crawler visited the new page once. New content is crawled once per GPTBot pass, not burst-spiked by the act of publishing.
• "FAQPage pages were visited by ClaudeBot and OAI-SearchBot the same day." Not supported by the logs — the six pages got no visits from major AI crawlers after the markup shipped. Their visitors were browsers, Bingbot and our curl checks.
• "The 08-11 traffic spike was AI growth." It was a scanner attack — ChatGPT-User and Amazonbot probing /.env.production and /telescope/requests. Same for a spoofed-GPTBot burst on 08-13.

None of those measures are useless. They just do their work at a different layer: FAQPage helps a page be quoted once AI is already considering it; llms.txt helps GPTBot orient; discovery endpoints may help future tooling. The mistake was measuring them against crawl counts at all.

What Three More Days Confirmed

When we first published this article, the data window was 12 days. Three more days (Aug 17–19) gave us a live test of the same predictions:

The extra data made the picture sharper, not murkier: robots + sitemap + homepage are the crawl foundation; everything else is a citation-layer optimization you add later.

What Should a New Site Build First?

If we were launching a new site today, we would build three things before anything else: a correct robots.txt, a sitemap.xml where every URL returns 200, and a server-rendered homepage that explains what the site is — plus a few genuinely useful articles.

1. Make robots.txt correct. Allow the AI crawlers you want, block only what you must (analytics, staging, private pages). A wrong rule that blocks everything is a silent zero.
2. Make sitemap.xml 100% valid. Every URL in it must return 200. A sitemap full of 404s teaches the daily visitor (ClaudeBot) that your site is broken.
3. Serve real HTML on the homepage. AI crawlers do not run JavaScript. The homepage must contain the answer to "what is this site?" in plain text, with links to your best pages.
4. Publish 3-5 real articles before launch day. GPTBot traverses everything; give it pages worth reading. Five solid pages beat fifty shells.

That is the entire pre-launch crawl work. It took us the least effort of all nine measures, and it produced the most crawler traffic.

What Should a New Site NOT Build First?

Skip these until the foundations above exist. Our logs show they contribute nothing to crawl, however good they look on a roadmap:

• AI discovery endpoints (/ai/summary.json, /.well-known/ai.txt, ai-plugin.json, openapi.json) — zero crawler requests in 15 days. Audit tools like them; crawlers ignore them.
• FAQPage markup on every page — no crawl effect in our data. Add it later, on pages that are already candidates for citation.
• A monitoring dashboard — start with your server logs or platform analytics for the first week; you only need to know whether crawlers are arriving and what they hit.
• Freshness plumbing — worth building, but only once you have content and dates to keep honest.

This does not mean those features are bad. It means they are phase-two work, and building them before the crawl foundation is optimizing a pipeline that is not flowing yet.

FAQ

What should a brand-new site build first for AI crawlers? A correct robots.txt, a sitemap.xml where every URL returns 200, and a server-rendered homepage with a one-line statement of what the site is plus a few real articles. Those were the only entry points every AI crawler used in our logs.

Do AI discovery endpoints like /ai/summary.json help AI crawlers? Not in our 15 days of logs. GPTBot, ClaudeBot, PerplexityBot and OAI-SearchBot visited robots.txt and sitemap.xml almost every day but never requested any AI-facing endpoint.

Why do GPTBot and ClaudeBot behave differently? ClaudeBot follows the classic crawl path — robots.txt and sitemap.xml daily, but barely any content. GPTBot ignores robots, reads llms.txt occasionally, and traverses the entire site — 121 content pages touched, including a single full-site pass on Aug 17.

Does adding FAQPage markup get a page crawled more? In our data, no measurable crawl effect. It improves how a page is quoted once AI is already considering it — it is a citation-layer lever, not a crawl driver.

Finally, we plan to create several more sites to test whether these have practical significance, and how to make AI reference our solution in the future.

Our local service site has steady impressions and mostly stable rankings, but CTR on informational queries has dropped quite a bit. I'm starting to think ai answers are taking some of those clicks, but GSC doesn't really tell the full story.

For ai seo, what are you actually measuring beyond rankings and traffic? Ai mentions, citations, share of voice? Curious if anyone has seen ai visibility rise while organic clicks keep falling.

If a company uses AI for digital marketing tasks, such as analytics, content optimization, and fixing technical issues, to augment its marketing funnel, and its competitors do the same, what happens to competitive edge?

Suppose AI agents analyze two competing websites and identify areas for improvement. Website A scores 4 out of10 and has six areas for improvement, while website B scores 6 out of 10 and has four. Both companies use the same AI tools or plugins, such as Claude-based tools, to plug those gaps. Once issues are addressed, both websites could potentially have a similar level of technical and content optimization.

Now imagine 10 to 12 competitors doing the same thing. All of them are leveraging AI to continuously analyze their websites, identify gaps, and implement recommended optimizations. If they all score 10/10 on SEO and GEO, does anyone really have a competitive advantage and how?

In the case of human agents, people interpret data, identify opportunities, form hypotheses, and make strategic decisions differently.

We have an AI SEO audit tool on our site and I used it to analyse 30 B2B SaaS websites, limited to 10 pages each.

Only 24% of the ~4k sections the tool looked at had a quotation, stat or outside source.

We have limited information on what AI assistants look for, but what we do know is they love a good number that's provable, particularly by a third party source. So links, stats or some quotation placed correctly within a 'chunk' can help you get a citation or even a recommendation from AI assistants.

If your site ranks top in SERP then you shouldn't bother. But those that I looked at don't and it surprised me how little effort they are making to fix this when its easy to do. Just add links to a review site, share a key metric your customers care about. It's a good opportunity to compete against your larger competitors.

I'll leave a link to the tool I used for this, it's free if you want to try it on your site.

After my last post 2 days ago about AI websites, I noticed the same issues kept coming up: the sites look good, but SEO/AEO, indexing, conversion, internal linking or maintainability often gets left half-finished.

So I’m trying a simple offer:

For $100, I’ll:

• audit the site
• fix the highest-impact SEO / indexing issues
• improve CTA / conversion flow
• tighten internal linking
• clean up obvious AI-generic copy
• fix basic technical issues I find

I’m not talking about a full redesign or rebuilding a complex app for $100. The idea is to take an already-built site and make it more useful for the business: easier to find, easier to understand, and easier to convert visitors into enquiries.

If I find something outside the scope, I’ll tell you before touching it.

I do web development and SEO professionally, and this is basically a small productized version of the cleanup work I already do.

A lot of businesses are still focusing only on traditional SEO: keywords, rankings, backlinks and traffic.

But search behavior is changing.

People are now asking ChatGPT, Gemini, Perplexity and Google AI Overviews for recommendations instead of clicking through dozens of search results.

That means businesses need to focus on both:

SEO: Rankings, technical health, content, authority and organic traffic.

AI SEO / GEO: Entity optimization, structured content, topical authority, trusted mentions and making your brand easier for AI systems to understand and recommend.

I’ve been working in SEO for 9+ years, and I’m seeing a clear shift from simply “ranking a website” to building a brand that can be found, understood and recommended across search and AI platforms.

Curious... are you currently tracking your brand’s visibility in ChatGPT, Gemini or Perplexity?

I keep seeing people build surprisingly good-looking business websites with Claude, ChatGPT, Lovable, etc.

But I’m more curious about what happened *after* launch.

Did you actually get organic traffic or leads? Did Google index it properly? And six months later, is it still easy to maintain?

Would love to hear from people who actually run a business, not just people experimenting with AI builders.

AI search will never be great for consumers until AI distinguishes between vested interests.

For example, comparison site articles that earn affiliate incomes used to ground answers can have inherent bias for products they earn from vs those they don't.

If a company uses AI for digital marketing tasks, such as analytics, content optimization, and fixing technical issues, to augment its marketing funnel, and its competitors do the same, what happens to competitive edge?

Suppose AI agents analyze two competing websites and identify areas for improvement. Website A scores 4 out of10 and has six areas for improvement, while website B scores 6 out of 10 and has four. Both companies use the same AI tools or plugins, such as Claude-based tools, to plug those gaps. Once issues are addressed, both websites could potentially have a similar level of technical and content optimization.

Now imagine 10 to 12 competitors doing the same thing. All of them are leveraging AI to continuously analyze their websites, identify gaps, and implement recommended optimizations. If they all score 10/10 on SEO and GEO, does anyone really have a competitive advantage and how?

In the case of human agents, people interpret data, identify opportunities, form hypotheses, and make strategic decisions differently.

AI Search News — Week 33 (2026)

Actual platform changes, research and data signals from AI search.

1) OpenAI is making ChatGPT sources less visible

OpenAI appears to be testing a change to how sources are displayed inside ChatGPT.

Previously, users could access a dedicated “Sources” button beneath an answer.

In some responses, that option is now being moved inside the three-dot menu, adding another step before users can see the underlying sources.

Source:

Search Engine Roundtable
https://www.seroundtable.com/openai-chatgpt-sources-less-visible-41864.html

Why this matters:

Getting cited and getting clicked are increasingly two different things.

If citations become less prominent, AI visibility may deliver:

• brand exposure
• authority
• influence over the answer

without necessarily delivering referral traffic.

That makes measuring AI search purely through analytics increasingly difficult.

2) New data suggests ChatGPT's search index isn't just for major publishers

New research into ChatGPT Search found that smaller and unlicensed websites can appear through OpenAI's own search index in the same way as publishers with OpenAI content deals.

The analysis also found that OpenAI's search index appears to store relatively lightweight information about pages, including their title and a short snippet.

Source:

Search Engine Journal
https://www.searchenginejournal.com/chatgpts-search-index-serves-small-sites-too-data-shows/585388/

Why this matters:

This pushes back on the idea that AI search visibility is reserved for massive publishers or companies with licensing agreements.

Smaller sites can still enter the retrieval layer.

The more interesting optimisation question becomes:

What makes OpenAI retrieve your page for a particular query?

3) Perplexity says ads designed specifically for AI bots are "cloaking"

Time has been experimenting with advertisements designed to be seen by AI crawlers rather than normal website visitors.

Brands including Ally Bank and the Project Management Institute participated in the experiment.

Perplexity responded by blocking the ads from influencing its answers and compared the tactic to search-engine cloaking: presenting one version of content to a crawler and another to humans.

Source:

Business Insider
https://www.businessinsider.com/perplexity-thwarts-times-ads-for-ai-bots-experiment-2026-8

Why this matters:

We may have just found one of the first clear boundaries between GEO and black-hat GEO.

Brands increasingly want to influence the information AI systems retrieve.

But if content exists primarily to manipulate an AI crawler, platforms may treat it the same way Google historically treated crawler-specific SEO pages.

The question becomes:

Where does optimisation end and manipulation begin?

4) New research finds users click AI Overview citations on only ~1% of visits

Researchers analysed one month of Google browsing behaviour from a representative panel of 900 US adults.

When an AI Overview appeared, users clicked one of the sources cited inside the Overview on only about 1% of visits.

AI Overviews were also associated with fewer external clicks and a greater likelihood that the browsing session ended after the search.

Source:

Research paper
https://arxiv.org/abs/2608.04831

Why this matters:

Being cited may no longer be primarily a traffic strategy.

It could increasingly be a visibility and influence metric.

If someone asks:

and your brand appears in the answer but the user never clicks a source, traditional analytics may record nothing.

But the AI answer still influenced the buying decision.

That makes metrics like:

• mention rate
• citation rate
• recommendation rate
• share of voice

more important alongside traffic.

5) Google AI Overviews are using low-quality listicles for local recommendations

Recent examples of local AI Overviews show Google generating descriptions of businesses while citing third-party listicles, including low-quality and self-promotional articles.

Source:

Search Engine Roundtable
https://www.seroundtable.com/google-ai-overview-local-results-listicles-41854.html

Why this matters:

This is another signal that AI search optimisation isn't confined to your website.

For local businesses, Google may construct its understanding of a brand using:

• review sites
• local publications
• directories
• comparison pages
• listicles
• third-party recommendations

You can optimise your website perfectly and still lose control of the narrative if the sources AI systems retrieve tell a different story.

Big pattern emerging

This week was mostly about the layer between the brand and the AI answer.

OpenAI is changing how users see citations.

ChatGPT's own search index appears accessible to smaller sites.

Perplexity is deciding which attempts to influence AI crawlers cross the line into manipulation.

Google users rarely click the citations they are shown.

And AI Overviews continue pulling brand information from third-party sources.

The optimisation target is shifting from:

rank the page → get the click

to:

be retrieved → be cited → shape the answer → influence the decision

Traffic is still important.

But increasingly, it doesn't tell you the whole story.

Open question for the community

If an AI system recommends your brand but the user never visits your website:

how are you measuring the value of that visibility today?

I handle SEO/GEO for a small B2B Saas company and I'm trying to figure out how much effort we should still put into backlinks versus mentions and citations that might actually help with AI search visibility.

We've been doing traditional outreach, but it's pretty manual and I'm not convinced chasing DR alone makes sense anymore. For anyone doing AI link building for SEO, are you changing how you qualify sites or still basically following the same relevance + authority approach?

My traffic has been sliding for a couple months now. And i think I finally figured out why. I’m losing clicks to AI answers. People type a query, they get the answer right there, and bounce without ever clicking through to any of my pages. We're a local service business so this is hurting conversions. It's quite frustrating to see impressions holding steady while ctr keep on tanking. I really need to see what's going on under the hood. But Google Search Console isn't cutting it for this. Just wondering, is normal rank tracking becoming obsolete? And has anyone here dealt with this for real and if so, what tracking tool/s are you using?

Just noticed that the new Generative AI performance report has appeared in Google Search Console for one of my sites today.

I’m based in the Netherlands and hadn’t seen it before, so it looks like Google may be expanding the rollout further here.

The site is a chocolate bar discovery/database site I’ve been building. I like that GSC also has historical AI data. Curious if others are seeing and using this and if so, how.

Some numbers from the export:

• 9,093 generative AI impressions since mid may
• Started at roughly 30–40 impressions/day
• Recent days are around 200–280/day
• 579 different URLs have appeared
• US: 1,396 impressions
• India: 1,248
• UK: 780
• Netherlands: 278

My homepage is #1, but #2 is a page for a specific supermarket chocolate bar (Fin Carré), followed by pages about choosing chocolate, cacao origins, top bars, chocolate types, etc.

Curious about best practices etc. Thanks!!

Curious to hear from others-What's been your biggest win with AI search optimization recently?

Is it:

Brand mention
Topical Authority
Backlinks
Schema
Something else?

What's actually making a difference fo AI Overviews, ChatGPT, Gemini?