Why llms.txt does almost nothing for AI search

Does anyone actually fetch these files?

If llms.txt is meant to help AI systems find your best content, the first question is whether AI systems actually fetch it. The best current answer is: usually, no.

Ahrefs analysed the server logs of 137,210 sites. 28% of them published a valid llms.txt file, and 97% of those files received zero requests of any kind in May 2026, no bots, no humans, nothing. Of the small share that were fetched, the AI retrieval bots that answer live questions, OAI-SearchBot, PerplexityBot, and Claude's search crawler, made up just 1.1% of requests. And zero requests came from AI bots for files that did not exist, which means the engines never go looking for one.

One caveat on that adoption figure: Ahrefs' sample skews more technical and SEO-aware than the wider web, so 28% is an upper bound, not a universal rate. That is the base rate the whole conversation should start from. Most files that exist are read by nothing.

Most "llms.txt is AI SEO" advice is folklore.

What llms.txt was meant to do

llms.txt is a proposed Markdown file, usually served at /llms.txt, that gives large language models and agents a curated map of a site's useful content. Jeremy Howard proposed it in September 2024 as an inference-time aid, and it suits documentation-heavy sites where an agent might want a compact route into API docs, examples, or canonical resources.

It helps to be precise about what it is not:

• Not robots.txt. It controls nothing and blocks nothing.
• Not a sitemap. It is not a discovery inventory for search engines.
• Not structured data. It does not attach machine-readable meaning to visible content for rich results.
• Not proof of AI visibility. Publishing it does not mean answer engines will fetch, trust, or cite it.

The "AI visibility" framing came later, attached by the SEO industry on the bet that platforms would reward the file. That bet has not paid out.

AI search visibility and agent readiness are two totally different things

This is the key distinction to get straight when thinking about llms.txt. Helping LLMs and AI agents navigate your website easily and efficiently is completely different to being visible to them in the first place.

Will llms.txt help you get ranked, retrieved, or cited in Google AI Overviews and AI Mode, ChatGPT search, Perplexity, or Claude search? The evidence says no.

Will software agents fetch a curated map of your site at request time, to orient before they act? Potentially yes.

Google's own ecosystem shows the split in miniature. Search says it ignores the file, while Chrome's Lighthouse added an experimental agent-readiness audit that checks whether a site has one. That reads like a contradiction until you separate the functions.

John Mueller reconciled the two on the record, calling llms.txt "not done for search" and more a token-saving aid for AI coding tools reading developer docs.

What other providers say

The other engines route you the same way. OpenAI's crawler docs point publishers to OAI-SearchBot access and robots.txt; Perplexity's point to PerplexityBot and robots.txt; Anthropic's guidance covers its distinct bots and crawler controls, not an external llms.txt as a citation mechanism.

There is no official statement from OpenAI, Anthropic, Perplexity, or Microsoft saying their answer engines use external llms.txt files as a ranking, retrieval, or citation signal. That is not the same as "no system will ever use it". But today, none has committed to it, and the independent data agrees: a 300,000-domain analysis found no measurable relationship between having the file and how often a site gets cited in AI answers.

What the log studies show

Returning to the Ahrefs study: The biggest AI consumer was not a search bot at all. AI agents and coding tools, led by Claude Code, took 10.5% of requests, nearly ten times the retrieval bots.

The log studies:

• 97% of valid files got zero requests of any kind in May 2026
• 1.1% of the fetches that happened came from AI retrieval bots
• 10.5% came from AI agents and coding tools, led by Claude Code

It’s also worth mentioning that "fetched" is not proof anything read or acted on the file. Smaller studies agree: one saw 84 of 62,100 AI-bot visits touch it; another, 408 requests across more than 500 million events.

Smaller studies land on the same conclusion as Ahrefs: as a search-visibility tool, the file is basically useless.

The best argument for shipping one anyway

The case in favour of llms.txt is not an SEO case at all:

• It is cheap to generate, and increasingly produced for you. Wix and Cloudflare already generate it; Framer and Lovable scan for it.
• The most plausible audience in the data is coding agents. If your buyers use tools like Claude Code to source recommendations, it stands a real chance of being read.
• It may future-proof you: if the agentic web matures and agents come to mediate AI search, the file could matter through the agent layer.

However there’s also plenty of reasons to not bother with it at all:

• Most files are never fetched at all
• Agents fetch when linked, instructed, or working in a docs context, not because they look for the file specifically
• If the internal business case is "AI search visibility", that case is not supported

The file is close to useless for the first job, and modestly useful for the second.

This is where llms.txt makes most sense: not as an SEO signal, but as a convenience layer for agents that already know to look for it.

What to do instead

So you want to appear in more AI-powered searches and you’ve realised llms.txt is a dead end. Here’s what you should take a look at instead:

1. Allow the crawlers that matter. Googlebot crawl and Search eligibility for Google AI features, OAI-SearchBot for ChatGPT, PerplexityBot for Perplexity, and the separate ClaudeBot, Claude-SearchBot, and Claude-User for Claude.
2. Make content crawlable and visible. AI surfaces run on the same crawl and index as ordinary Search. Do not hide important content behind client-side rendering.
3. Write answerable pages, not chunks. Google says there is no requirement to break content into tiny pieces. Make the right answer easy to extract: clear headings, direct definitions, comparison tables, and specific examples.
4. Treat structured data as hygiene. Useful housekeeping, not a magic AI-citation switch. Schema does not drive citations on its own.
5. Build off-domain citation surfaces. Reddit, Wikipedia where appropriate, YouTube, G2, review sites, credible digital PR, and highly linked industry pages.
6. Measure actual outcomes. Citation appearances, referral sessions, query coverage, and crawl logs matter more than whether a file exists.