Why llms.txt does almost nothing for AI search

Does anyone actually fetch these files?

If llms.txt is meant to help AI systems find your best content, the first question is whether AI systems actually fetch it. The best current answer is: usually, no.

Ahrefs analysed the server logs of 137,210 sites. 28% of them published a valid llms.txt file, and 97% of those files received zero requests of any kind in May 2026, no bots, no humans, nothing. Of the small share that were fetched, the AI retrieval bots that answer live questions, OAI-SearchBot, PerplexityBot, and Claude's search crawler, made up just 1.1% of requests. And zero requests came from AI bots for files that did not exist, which means the engines never go looking for one.

One caveat on that adoption figure: Ahrefs' sample skews more technical and SEO-aware than the wider web, so 28% is an upper bound, not a universal rate. That is the base rate the whole conversation should start from. Most files that exist are read by nothing.

What llms.txt was meant to do

llms.txt is a proposed Markdown file, usually served at /llms.txt, that gives large language models and agents a curated map of a site's useful content. Jeremy Howard proposed it in September 2024 as an inference-time aid, and it suits documentation-heavy sites where an agent might want a compact route into API docs, examples, or canonical resources.

It helps to be precise about what it is not:

• x

  Not robots.txt. It controls nothing and blocks nothing.

• x

  Not a sitemap. It is not a discovery inventory for search engines.

• x

  Not structured data. It does not attach machine-readable meaning to visible content for rich results.

• x

  Not proof of AI visibility. Publishing it does not mean answer engines will fetch, trust, or cite it.

The "AI visibility" framing came later, attached by the SEO industry on the bet that platforms would reward the file. That bet has not paid out.

The two lanes: AI search visibility versus agent readiness

This is the distinction the whole topic turns on, and the one most explainers miss. Keep two questions apart and the confusion clears.

Competitor posts tend to collapse the two, then either dismiss the file entirely or sell it as "AI SEO". Both miss. The file is close to useless for the first job and modestly useful for the second. The rest of this piece takes each lane in turn.

What Google, OpenAI, Perplexity, and Anthropic actually say

For AI search visibility, this is documented, not inferred. Google's May 2026 guide on optimising for generative AI features puts it in a section titled "mythbusting":

You do not need to create new machine-readable files, AI text files, markup, or Markdown to appear in Google Search, because Google Search itself does not use them.

Its "AI features and your website" page is just as plain: there are no additional requirements to appear in AI Overviews or AI Mode beyond ordinary Search eligibility. The same guide adds that maintaining the file for other systems will neither help nor harm your rankings, because Search ignores it. And as far back as July 2025, Gary Illyes said Google does not support llms.txt and is not planning to.

The other engines route you the same way. OpenAI's crawler docs point publishers to OAI-SearchBot access and robots.txt; Perplexity's point to PerplexityBot and robots.txt; Anthropic's guidance covers its distinct bots and crawler controls, not an external llms.txt as a citation mechanism.

Here is the fair limit of what can be proven. There is no official statement from OpenAI, Anthropic, Perplexity, or Microsoft saying their answer engines use external llms.txt files as a ranking, retrieval, or citation signal. That is not the same as "no system will ever use it". But today, none has committed to it, and the independent data agrees: a 300,000-domain analysis found no measurable relationship between having the file and how often a site gets cited in AI answers.

What the log studies show

The logs are where the visibility story comes apart. Return to the 137,000-site study: 97% of valid files unread, and AI retrieval bots at just 1.1% of the requests that did happen. The biggest AI consumer was not a search bot at all. AI agents and coding tools, led by Claude Code, took 10.5% of requests, nearly ten times the retrieval bots.

Smaller studies land in the same place. Different methods, same conclusion: as a search-visibility lever, the file is barely consumed.

Most "llms.txt is AI SEO" advice is folklore.

The best argument for shipping one anyway

The fair version of the pro-llms.txt case is not an SEO case, and it deserves a proper hearing rather than a quick dismissal. Here it is, and the rebuttal, on the same evidence.

This is where llms.txt makes most sense: not as an SEO signal, but as a convenience layer for agents that already know to look for it.

The trusted-file problem

There is a cost the "ship it anyway" camp rarely prices in, and it is the reason to be deliberate even about a cheap file.

"Nobody has been robbed yet" is a thin reason to leave a trusted door unlocked. The reconnaissance is already under way; the deliberate move is to either not publish, or to publish under the same controls you would put on code.

What to do instead

This is the part that actually moves AI-search visibility. In order:

1. 01

   Allow the crawlers that matter.

   Googlebot crawl and Search eligibility for Google AI features, OAI-SearchBot for ChatGPT, PerplexityBot for Perplexity, and the separate ClaudeBot, Claude-SearchBot, and Claude-User for Claude.

2. 02

   Make content crawlable and visible.

   AI surfaces run on the same crawl and index as ordinary Search. Do not hide important content behind client-side rendering.

3. 03

   Write answerable pages, not chunks.

   Google says there is no requirement to break content into tiny pieces. Make the right answer easy to extract: clear headings, direct definitions, comparison tables, and specific examples.

4. 04

   Treat structured data as hygiene.

   Useful housekeeping, not a magic AI-citation switch. Schema does not drive citations on its own.

5. 05

   Build off-domain citation surfaces.

   Reddit, Wikipedia where appropriate, YouTube, G2, review sites, credible digital PR, and highly linked industry pages.

6. 06

   Measure actual outcomes.

   Citation appearances, referral sessions, query coverage, and crawl logs matter more than whether a file exists.

The grown-up recommendation

llms.txt is documented as irrelevant to AI search visibility, supported by every dataset we have as barely read, and propped up by a lot of folklore. There is a real but narrow agent-readiness use, and a real security cost. So do not sell it as AI SEO, and do not publish it on autopilot. Ship one only when agents are a defined audience and someone owns its maintenance. The short version, by situation:

Do not sell it as AI SEO, and do not publish it on autopilot.