What Is llms.txt? Who Reads It, and Whether You Should Publish One

The question arrives at LaCleo in two forms. Clients ask whether they need one because an audit tool flagged its absence, and clients who have one ask why it has not changed their AI visibility. Both have the same answer, and it is worth giving with the evidence rather than an opinion, because the file is cheap to publish and the time spent debating it is not. The GEO audit covers retrievability in two checklist items; this article is the long version of those two lines.

What is llms.txt?

llms.txt is a proposal for a Markdown file at a website's root, /llms.txt, that gives language-model agents a concise description of the site and a curated set of links to its most useful pages. Jeremy Howard, co-founder of fast.ai and Answer.AI, published it on September 3, 2024 at llmstxt.org. The proposal's own framing is practical: language models have limited context windows, HTML pages are padded with navigation and scripts, and an agent that lands on a site benefits from a short, clean map of where the substance is. The stated aim is "to provide information to help agents use a website."

Two things about that framing matter for everything that follows. The file was designed for agents using a site, in the sense of a coding assistant reading a library's documentation, not for search engines indexing it. And it is a convention, in the way a README is a convention: anyone can publish one, and whether anything reads it is up to the reader. It has no relationship to robots.txt, grants and withholds nothing, and is not referenced in the published crawler documentation of Google, OpenAI, Anthropic, Perplexity, Microsoft, Meta or Apple.

The format

One required line, three optional parts. Per the specification, the file is Markdown in this order: an H1 with the project or site name, which is the only required element; a blockquote with a short summary; zero or more sections of notes in any Markdown except headings; and zero or more H2-delimited lists of links, each a Markdown link optionally followed by a colon and a note. An H2 named "Optional" marks secondary links an agent may skip when context is short. The file can sit at the root or at a subpath such as /docs/llms.txt, covering the URLs beneath it. The spec also suggests serving clean Markdown versions of pages at the same URL with .md appended.

# Example Company

> Example Company makes inventory software for independent retailers. Product documentation, API reference and pricing are linked below.

## Documentation
- [Getting started](https://example.com/docs/start.md): Install, connect a store, first sync
- [API reference](https://example.com/docs/api.md): REST endpoints, authentication, rate limits

## Optional
- [Changelog](https://example.com/changelog.md)
- [Pricing](https://example.com/pricing.md)

Note what the spec does not include. It does not define llms-full.txt, a companion file some platforms generate with full page content concatenated; that is a platform convention layered on top. It does not define any directive to allow or block agents. And it carries no signature or verification, which is the basis of the prompt-injection concern Ahrefs raises: an agent that trusts a site's llms.txt is trusting arbitrary text the site owner wrote for it.

Who has adopted it

Adoption is in the tens of thousands of domains and is driven by platforms that generate the file automatically. SE Ranking found 10.13% of roughly 300,000 domains had one in November 2025, with adoption at 9.88% on the lowest-traffic sites and 8.27% on the largest. Ahrefs' June 2026 study found up to 28% of its 137,210-domain sample, a figure the authors call an upper bound because Ahrefs customers skew technical. A Common Crawl analysis of 584,107 files from the July 2026 crawl found 49.9% matched the full spec shape, 22.6% contained no links at all, 41.3% were generated by Wix, and 40.4% were served with a text/html content type rather than plain text.

The adopters named on the spec's own page are documentation platforms and tools: Mintlify, GitBook, Yoast SEO, AIOSEO, Wix, and plugins for VitePress, Docusaurus and Drupal. The AI labs themselves publish llms.txt files for their developer documentation; OpenAI, Anthropic and Google's Gemini docs are listed. That is consistent with the file's purpose and it is also the extent of the labs' endorsement: they publish one for their docs, and none has said its assistants read anyone else's.

Who reads it: the server-log evidence

Two log studies, one small and one large, agree: the file is almost never fetched, and when it is, it is mostly not by the AI assistants it was meant for.

Otterly.AI examined 62,100 AI bot visits to its own site and found 84 requests for /llms.txt, about 0.1%, far fewer than the content pages received. Ahrefs' study, published June 15, 2026, analyzed May 2026 server logs for 137,210 domains with traffic. Of the domains with a valid llms.txt, 97% received no requests for it in the month. The remaining 3%, about 1,100 domains, drew roughly 22,000 requests, 96% from bots. The breakdown of those requests is the useful part:

Who requested llms.txt files in May 2026 (Ahrefs, ~22,000 requests across ~1,100 domains)

Read generously, all four AI categories together account for 19.5% of requests, and Ahrefs notes that a fetch does not prove the file was used. GPTBot was the most frequent AI requester, followed by Claude Code, which is a coding agent and the one category the spec was written for. Googlebot fetched the file about 900 times, which the authors attribute to ordinary crawling of a linked URL. Slackbot requested it more often than PerplexityBot. Requests for llms.txt files that did not exist were 98% human, with zero from AI bots, which means the assistants are not probing for it either.

What Google and the AI labs say

Google Search has said no in three ways; the AI labs have said nothing, which in crawler documentation means no.

John Mueller compared llms.txt to the keywords meta tag in April 2025 and wrote on Bluesky in June 2025 that no AI system currently uses it. Gary Illyes said at Search Central Live in July 2025 that Google does not support the file and is not planning to. Google's AI features guidance, updated July 10, 2026, says site owners can skip "creating unnecessary AI text files (like llms.txt)" and that doing so "will neither harm nor help." The one Google product that acknowledges the file is Chrome's Lighthouse, whose Agentic Browsing audit checks for it and calls it an "emerging convention"; Search Engine Journal covered the inconsistency in May 2026. Lighthouse is a developer tool for agentic browsing; it is not Search, and its audit marks a missing file as not applicable rather than failing.

OpenAI's bot documentation describes GPTBot, OAI-SearchBot and ChatGPT-User and how robots.txt applies to each; it does not mention llms.txt. Anthropic's crawler article describes ClaudeBot, Claude-SearchBot and Claude-User the same way; it does not say any of them reads llms.txt, though Claude will fetch an llms.txt URL if a user pastes one, as it would any URL. Perplexity's bot guide covers PerplexityBot and Perplexity-User and robots.txt only. Microsoft's Bing documentation covers Bingbot. None of the five vendors whose assistants dominate AI search has documented llms.txt as an input, and the log data above is consistent with that.

Does it affect AI citations?

There is no evidence that it does, and one study that looked for the effect did not find it. SE Ranking's November 2025 analysis of about 300,000 domains tested llms.txt presence as a predictor of AI citation frequency and found no statistically significant correlation; removing the variable improved the model's accuracy. Its framing was that the file "doesn't seem to directly impact AI citation frequency. At least not yet." No controlled before-and-after test has been published. Given that the assistants do not document reading the file and the logs show they rarely fetch it, the absence of an effect is the expected result rather than a surprise.

The on-page work that has measured effects is the subject of the LLM SEO article: in the KDD 2024 GEO study, quotations, statistics and cited sources raised generative visibility by 41%, 37% and 30%. Those changes live on the pages the assistants retrieve, not in a side file they do not.

When to publish one, and when not to

When llms.txt is worth publishing

If you do publish one, three rules keep it from doing harm: keep it short, keep it current, and never put anything in it that is not on the live site. Ahrefs' prompt-injection warning is the reason for the third; the file is text an agent may trust, and it should say nothing the pages do not.

What to do instead: the crawler rules that matter

The file that governs whether AI assistants can retrieve your pages is robots.txt, and the decision is not one switch but three. Each vendor runs separate agents for training, for search indexing and for fetching a page a user asked about, and documents which honor robots.txt. A site that wants to be cited in AI answers while declining to feed training data can express that; a site that blocks everything with a wildcard cannot be cited by anyone.

Documented AI user agents by vendor and purpose, October 2026

Sources: each vendor's crawler documentation, linked in the sources section. Tokens and policies change; confirm against the vendor page before editing a live robots.txt.

A robots.txt that allows AI search retrieval while declining training use looks like this. It is a starting point, not a recommendation on the training question, which is a commercial and legal decision for each site:

# Search and retrieval agents: allow
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Bingbot
User-agent: Applebot
Allow: /

# Training crawlers and training-control tokens: site decision
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
Disallow: /

# Googlebot is not listed: blocking it removes the site from Search and AI Overviews.

Beyond robots.txt, five checks decide retrievability, and they are the ones the GEO audit scores:

1. CDN and WAF bot rules. The most common block is not in robots.txt; it is a bot-mitigation rule at the CDN that challenges or drops the AI user agents alongside scrapers. Check the firewall's bot list for every token in the table above and confirm the search agents are on the allow side.
2. Server logs for the search agents. If OAI-SearchBot, Claude-SearchBot, PerplexityBot and Bingbot are not in last month's logs on the pages you want cited, the pages are not being retrieved, whatever robots.txt says. Promptwatch measured OpenAI's crawlers at 79.8% of verified AI crawler requests in September 2026, so the others are now a meaningful share.
3. Content in the HTML. Retrieval agents read the initial response. Text that arrives only after JavaScript runs is text they may not see. Check the raw HTML for the passage you want cited.
4. A dated, maintained page. Ahrefs found AI-cited URLs about 25% newer than Google organic URLs for the same queries; a visible modification date and a changelog of updates are read.
5. Schema that names the entity. Organization, Article and FAQ markup gives the assistant the entity and the question-answer pairs in a form it parses. Whether schema lifts citation on its own is contested this month; whether it hurts is not, and it is part of the structure the LLM SEO article describes.

How we do it

LaCleo's position is in the table above: llms.txt is published for documentation and left alone for marketing sites, and the retrievability hour goes to the crawler and structure checks. In a program:

1. Week one, crawler audit. robots.txt and the CDN bot list are read against the vendor table, the server logs are checked for each search agent on the pages that should be cited, and any block is fixed before content work starts. This is part of the 25-point GEO audit.
2. llms.txt by site type. Sites with documentation get a maintained file at the docs path, generated from the live pages so it cannot drift. Marketing sites that already have one via a plugin get it checked for content type, links and accuracy, and otherwise it is not touched.
3. Retrievability in the implementation pack. robots.txt changes, CDN rule changes and schema are delivered as files and instructions the client's developers paste in, reviewed by a person before delivery.
4. Re-check in the logs. After the pack ships, the search agents' requests to the target pages are confirmed in the next month's logs, and the citation tracking described in the AI Overviews tracker article shows whether retrieval turned into citation.

Statistics with sources

Figures used in this article, with the source and the caveat that applies

Frequently asked questions

What is llms.txt?

A proposed convention, published by Jeremy Howard on September 3, 2024, for a Markdown file at a site's root that gives language-model agents a short summary of the site and a curated list of links. The only required element is an H1 with the site name. It is a proposal, not a standard adopted by any search engine.

Does Google use llms.txt?

No. John Mueller compared it to the keywords meta tag and wrote in June 2025 that no AI system currently uses it. Gary Illyes said in July 2025 that Google does not support it and is not planning to. Google's AI features guidance, updated July 2026, says creating AI text files like llms.txt will neither harm nor help. Chrome's Lighthouse audits for the file, but Lighthouse is a developer tool, not Search.

Do ChatGPT, Claude or Perplexity read llms.txt?

None documents doing so; their crawler pages cover robots.txt only. Ahrefs' May 2026 analysis of 137,210 domains found 97% of published files received zero requests, and AI retrieval bots made 1.1% of the requests to the rest. GPTBot was the most frequent AI requester, followed by Claude Code, a coding agent.

Does llms.txt help AI search visibility or SEO?

There is no evidence that it does. SE Ranking's study of about 300,000 domains found no statistically significant correlation between having the file and AI citation frequency, and Google says it neither harms nor helps. The retrievability work that affects citation is in robots.txt, CDN and WAF rules, clean HTML and schema.

Should I publish an llms.txt file?

If you run developer documentation or a product coding agents use, yes, at the docs path; that is where it is read. For a marketing site seeking AI search visibility, it is optional and low priority. Publish one only if it will be kept accurate, because a stale file is a liability with no offsetting gain.

Which AI crawlers should I allow in robots.txt?

For AI search visibility, allow the search and retrieval agents: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot and Googlebot. Training crawlers (GPTBot, ClaudeBot, meta-externalagent) and the Google-Extended and Applebot-Extended tokens are a separate decision about training use. User-triggered fetchers such as ChatGPT-User and Google's agent fetchers may not honor robots.txt, per the vendors' own documentation.

What is the difference between llms.txt and robots.txt?

robots.txt is a decades-old standard that every major crawler honors; it tells agents which paths they may fetch and is the mechanism every AI vendor documents. llms.txt is a 2024 proposal that describes a site's content for agents and grants or withholds nothing. One controls access; the other offers a summary that, as of October 2026, almost nothing requests.

Working with LaCleo

LaCleo runs agentic SEO and GEO as a human-led managed service, from $750 a month. Retrievability is week one: robots.txt and CDN rules read against the vendor crawler table, server logs checked for each AI search agent on the pages that should be cited, llms.txt handled by site type, and the fixes delivered as files the client's developers paste in. The program then tracks 100 or more prompts across AI Overviews, AI Mode and the major LLMs with licenses included, runs the 25-point GEO audit, closes the per-query citation gap against named competitors, and ships page changes, schema and articles as an implementation pack. A named person reviews every deliverable. There is nothing to install.

Connect to know more Book a call Get a free AI audit

Sources

Jeremy Howard, "The /llms.txt file", llmstxt.org, September 3, 2024, revised August 2026. Louise Linehan and Xibeijia Guan, "We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read", Ahrefs, June 15, 2026. SE Ranking, llms.txt adoption and citation correlation study, November 2025, as reported by Search Engine Journal. Common Crawl analysis of llms.txt files, July 2026 crawl, as reported by Hybrid Ranking. Otterly.AI, llms.txt bot visit experiment, updated February 2026. Search Engine Journal, "Google's llms.txt Guidance Depends On Which Product You Ask", May 20, 2026. Google Search Central, AI features guidance, updated July 10, 2026; Google common crawlers and user-triggered fetchers documentation. OpenAI, bot documentation. Anthropic, "Does Anthropic crawl data from the web". Perplexity, bot guide. Microsoft Bing, crawler documentation. Meta, web crawler documentation. Apple, Applebot documentation. Cloudflare on undeclared Perplexity crawling, August 2025. Promptwatch, AI crawler share, September 2026. Ahrefs, citation freshness study. Aggarwal et al., "GEO: Generative Engine Optimization", KDD 2024. All figures verified October 2026.

Crawler tokens and policies change; the vendor table reflects published documentation in October 2026 and should be confirmed against each vendor page before a live robots.txt is edited. Figures from Ahrefs describe its own customer base, which the authors note skews technical.