Directives Without a Scoreboard: A Measured Approach to llms.txt and AI Crawler Rules
Photo: Groceryheist, CC BY-SA 4.0, via Wikimedia Commons
Few files in the recent history of search infrastructure have moved from proposal to presumed requirement as quickly as llms.txt. Within months of appearing, it began showing up in vendor audits, agency deliverables, and internal roadmaps as though it were a ratified standard with a measurable payoff. It is not. For publishers making capacity decisions for 2026, the useful question is narrower: which of these directives can be observed working, and which are simply plausible?
A Proposal, Not a Protocol
llms.txt was put forward in 2024 as a markdown file at the site root that describes a site's purpose, its canonical resources, and its preferences to large language models. That is the whole of its authority. No standards body has ratified it, no major answer engine has published documentation stating that it governs content selection, and no ranking system is known to score it.
The contrast with robots.txt is instructive. Robots exclusion has been honored informally for three decades and then codified in an actual standard, RFC 9309, with widely documented semantics and a long record of observable behavior. Publishers treat it seriously because violating it has consequences. llms.txt has no equivalent enforcement mechanism and, so far, no confirmed consumer among the platforms that matter most to US traffic.
That does not make the file useless. It makes it provisional, which is a different thing entirely.
Where the Evidence Thins Out
Server logs can tell you whether a crawler fetched /llms.txt. They cannot tell you whether anything downstream changed because of it. A retrieval system that requests the file and then ignores its contents produces the same log line as one that heeds it, and most large language model pipelines cache aggressively, so a single fetch may stand in for months of unexamined behavior.
The frequent claim that models read the file before answering is therefore only half-testable. Access is observable; influence is not, at least not without controlled experiments that most publishers lack the traffic to run. Absence of a declared standard is not proof of futility, but it does mean that investment should be sized accordingly.
Three Artifacts Worth Shipping
A defensible implementation is small. It is deliberately boring, and it costs an afternoon rather than a quarter.
A scoped llms.txt. Resist the urge to generate a sitemap clone. A file of a few hundred lines that states the site's subject, links to a handful of cornerstone pages, identifies the licensing contact, and lists any explicit exclusions will serve better than ten thousand URLs nobody will parse. Maintain an llms-full.txt only if you have a genuine reason and a process to keep it in sync.
An explicit crawler matrix in robots.txt. Group user agents deliberately, annotate each block with a date and an owner, and stop relying on wildcard rules to cover bots that arrived after the file was last touched.
A change log. One dated ledger recording what was changed, why, who approved it, and which signal would indicate the change mattered. This is the artifact most teams skip and the one that makes every future conversation shorter.
The Crawler Matrix, Decoded
The tokens differ in ways that are easy to conflate, and conflating them is how publishers accidentally block a retrieval crawler while intending only to restrict training.
| Token | Operator | What it governs | Practical note |
|---|---|---|---|
| GPTBot | OpenAI | Model training crawl | Blocking limits training use, not live retrieval |
| OAI-SearchBot | OpenAI | Search index for OpenAI products | Separate from training; governs inclusion in answers |
| ChatGPT-User | OpenAI | User-initiated fetch | Typically fires when a person asks for a page |
| ClaudeBot | Anthropic | Training crawl | Distinct from retrieval agents |
| Claude-SearchBot | Anthropic | Retrieval for search features | Governs citation eligibility |
| PerplexityBot | Perplexity | Indexing for answer generation | Blocking removes you from those answers |
| Googlebot | Classic Search indexing | Unrelated to AI training controls | |
| Google-Extended | Training control token | Does not fetch; it is a preference signal in robots.txt | |
| CCBot | Common Crawl | Open corpus | Feeds many downstream models indirectly |
Two rows cause the most confusion. Google-Extended is a control token rather than a crawler, so blocking it changes nothing about fetching and everything about stated training preferences. And OAI-SearchBot or PerplexityBot blocks remove a publisher from retrieval surfaces without touching training permissions at all. Decide those two separately.
Deciding Who Gets In
The matrix becomes a business question quickly, and the honest answer differs by revenue model. A rough starting point:
| Publisher situation | Suggested posture | Reasoning | Residual risk |
|---|---|---|---|
| Active licensing program | Allow retrieval, restrict training until terms are agreed | Preserves citation presence while keeping leverage | Crawler ignores declared preference |
| Ad-supported, no deals | Allow retrieval broadly, allow training selectively | Answer-engine referrals still convert for some verticals | Continued traffic erosion |
| Paywalled or membership | Allow retrieval of indexable previews only | Protects subscription value | Partial answer snippets may substitute for clicks |
| Regulated or legally cautious | Restrict training, log the decision | Creates a defensible record | Reduced brand presence in model outputs |
None of these are permanent. Each should carry a review date and an owner's name.
Treat the Files as Infrastructure, Not Leverage
llms.txt is not a ranking lever, and presenting it as one invites the wrong kind of budget conversation. It belongs in the same operational bucket as redirect maps, canonical tags, and crawl budget rules: low-cost hygiene that reduces ambiguity.
That means it belongs in the same incident record publishers already keep for search shocks. Log the change, log the observed effect, and if there is no observed effect after a reasonable window, say so plainly. A record of experiments that produced nothing is not a failure of the program; it is the program working.
What Would Justify More Investment
Three triggers would change the calculus, and it is worth naming them in advance so the argument is not relitigated every quarter. First, a major answer engine publishing documentation that llms.txt governs selection. Second, log evidence showing a correlated shift in citation frequency after a controlled edit. Third, a licensing or legal requirement that makes the file contractually meaningful.
Until one of those lands, the correct posture is modest: ship a small file, declare your crawler preferences explicitly, write down what you did, and revisit in ninety days. The hype will move on. The change log will still be there, and it will still be more valuable than the file it describes.