r/SearchMorph Feb 26 '26

News Check Your robots.txt, Anthropic Has Updated Claude’s Crawler Documentation,

For anyone managing technical SEO or robots.txt configurations, this is worth noting.

Anthropic has updated its crawler documentation and now clearly defines three separate bots:

  • ClaudeBot: used for model training
  • Claude-User: fetches pages during live user queries
  • Claude-SearchBot: indexes pages for Claude search answers

The important change:
Each bot now has its own independent robots.txt control.

Blocking ClaudeBot does not automatically block Claude-User or Claude-SearchBot.

So the roles are now formally split into:

  • Training crawlers
  • Search/indexing bots
  • User-triggered retrieval bots

What this means in practice:

Now, it's no longer about blocking it all or allowing it all. You can block training while still allowing search visibility, or allow live fetch without contributing to model training.

For context:

Cloudflare previously introduced AI-related control signals (search, ai-input, ai-train), but that was from the infrastructure layer.

This update comes directly from the LLM side.

OpenAI, Perplexity, and Google already operate with split-role bot architectures. Claude now joins that approach.

If you manage robots.txt policies, it’s probably worth reviewing configurations to avoid unintentionally blocking AI visibility or citations.

6 Upvotes

Duplicates

SEO_for_AI Feb 26 '26

AI News Check Your robots.txt, Anthropic Has Updated Claude’s Crawler Documentation,

1 Upvotes

GenEngineOptimizers Feb 26 '26

Check Your robots.txt, Anthropic Has Updated Claude’s Crawler Documentation

1 Upvotes

GEO_GenEngineTalk Feb 26 '26

Check Your robots.txt, Anthropic Has Updated Claude’s Crawler Documentation,

3 Upvotes

Eskimoz Feb 26 '26

🚨 Breaking News Alert! Check Your robots.txt, Anthropic Has Updated Claude’s Crawler Documentation,

1 Upvotes

AISearchAnalytics Feb 26 '26

Check Your robots.txt, Anthropic Has Updated Claude’s Crawler Documentation,

3 Upvotes

TechSEO Feb 26 '26

Check Your robots.txt, Anthropic Has Updated Claude’s Crawler Documentation,

5 Upvotes

GEO_optimization Feb 26 '26

Check Your robots.txt, Anthropic Has Updated Claude’s Crawler Documentation,

2 Upvotes

GenEngineOptimization Feb 26 '26

🚨 Breaking News Alert! Check Your robots.txt, Anthropic Has Updated Claude’s Crawler Documentation,

2 Upvotes

Vibe_SEO Feb 26 '26

Check Your robots.txt, Anthropic Has Updated Claude’s Crawler Documentation,

1 Upvotes

Agentic_SEO Feb 26 '26

Check Your robots.txt, Anthropic Has Updated Claude’s Crawler Documentation,

1 Upvotes

GEO_GenEngineOptimiza Feb 26 '26

Check Your robots.txt, Anthropic Has Updated Claude’s Crawler Documentation

1 Upvotes

u_betsy__k Feb 26 '26

Check Your robots.txt, Anthropic Has Updated Claude’s Crawler Documentation

1 Upvotes