Meta-ExternalAgent
Meta-ExternalAgent is the crawler Meta uses, in its own words, “for use cases such as training foundation AI models or improving products by indexing content directly.” Meta runs two other agents next to it, for different things, and each is set on its own.
How it identifies itself
The token in robots.txt is meta-externalagent; tokens are not case-sensitive. In your logs it appears, Meta says, as something like this, sometimes followed by a link to Meta’s crawler documentation in parentheses:
meta-externalagent/1.1
Block it
Meta’s instruction is to add a disallow for the crawler in robots.txt:
User-agent: meta-externalagent Disallow: /
Meta says changes to robots.txt can take up to 24 hours to take effect.
Meta's other two agents
- Meta-ExternalFetcher fetches a single link when a user asks for it. Meta says that, accordingly, it “may bypass robots.txt rules”.
- Meta-WebIndexer reads the web to improve Meta AI’s search results. Meta says allowing it helps Meta AI cite and link your content in its responses. Blocking Meta-ExternalAgent does not block it; it reads its own group.
So a file that keeps training out and lets answers in has, for Meta, a disallow for meta-externalagent and nothing against meta-webindexer.
The rest of the list
Meta is one of several companies with a training crawler: GPTBot for OpenAI, CCBot for Common Crawl’s open archive, and the others in the list of AI crawlers.
Check what your robots.txt tells Meta-ExternalAgent
Paste your file into the robots.txt validator, opened with Meta-ExternalAgent filled in. It runs in your browser and shows, for Meta-ExternalAgent and every other search and AI bot, whether it may come in and the exact line that decides. The list of AI crawlers has the others.
Sources
Read on 14 October 2026; operators change these pages, so we read them again every month.
Last changed 14 October 2026.