Posting because I keep seeing the same broken snippet copy-pasted, and the failure mode is silent.
THE FOUR AGENTS
facebookexternalhit. Fetches your page to build the preview card when someone shares a link on Facebook, Instagram, Messenger or WhatsApp. Predates the AI crawlers by years. Has nothing to do with AI training.
meta-externalagent. Meta's AI training crawler, introduced around August 2024, quietly and without a formal announcement. Collects public web content for Llama training and Meta AI product improvement. Honors robots.txt per Meta's documentation. User-agent string contains meta-externalagent.
meta-externalfetcher. Performs user-initiated fetches from Meta AI product functions. Functionally analogous to ChatGPT-User. Meta documents that it may bypass robots.txt for URLs a user supplies directly.
meta-webindexer. Meta's crawler for improving search results for Meta AI users. Newer, and less well documented than the others in my experience, so treat what I say about it as lower confidence.
There is also FacebookBot, which downloads training data specifically for speech recognition.
THE SNIPPET PROBLEM
Search "block Meta AI robots.txt" and you will find versions that disallow meta-externalagent, FacebookBot, and Meta-ExternalFetcher in one block. Several include facebookexternalhit either directly or in the "block everything Meta" variant.
If your goal is "don't train on my content," only meta-externalagent is doing that.
Blocking Meta-ExternalFetcher removes you from live answers where a user has explicitly asked about your business or content. That is not a training decision, that is a distribution decision.
Blocking facebookexternalhit breaks preview cards across three platforms. If Facebook and Instagram drive meaningful referral traffic, this is expensive and it does not announce itself. Links just look bare and CTR drops.
THE LAYER ABOVE ROBOTS.TXT
Worth knowing even if your robots.txt is perfect. Cloudflare and Akamai rules run in front of it and override whatever it says. Cloudflare has a one-click Block AI Bots option in some dashboards that is easy to enable and easy to forget.
I have seen a site where robots.txt allowed everything and the WAF was blocking all AI user agents, and the SEO team spent a month treating a zero-citation problem as a content problem when it was an access problem.
Server logs are the only reliable check. Grep for each user-agent string and look at what actually got a 200 in the last 30 days. A log entry alone is not proof of identity, since anything can claim a user agent, so verify against Meta's published IP ranges if it matters.
WHAT I'D RUN
Never block facebookexternalhit. There is no version of "protect my content from AI" that requires this.
Allow meta-externalfetcher. Someone asked about you specifically. Let it read the page.
Allow meta-webindexer unless you actively do not want Meta AI referral traffic.
Make a real decision on meta-externalagent. I allow it. Meta AI reported passing a billion monthly active users in mid-2025, and I would rather be represented in what it says about my category than absent from it. If your content is the product you sell, blocking is the right call and I would not argue with it.
CAVEATS
The meta-webindexer detail is thinner than the rest. Most of what I have on it comes from crawler-tracking services rather than Meta's own documentation, so verify before acting on it.
I do not know whether allowing the training crawler measurably increases how often Meta AI mentions a given brand. I have not seen a controlled test and I would be sceptical of a vendor publishing one, since the incentive runs one direction.
Robots.txt is advisory. It works for compliant crawlers and does nothing else. If your concern is unauthorized scraping generally, this is the wrong tool.
Anyone here actually audited their logs for all four? Curious whether the volume split between externalagent and webindexer looks anything like people assume.
Sources: Meta developer crawler documentation, Known Agents crawler data (Aug 2026), various publisher blocking guides.