llms.txt is the first thing nearly every AEO guide tells you to add, and there is still no evidence that any major AI system reads it.
I have been skeptical of this one publicly since the beginning, and the evidence has only gotten more one sided since. Google has now said on the record that it does not use the file. OpenAI documents robots.txt and never mentions llms.txt. Independent server log studies across thousands of domains show the AI crawlers that matter never request it. Meanwhile the tactic keeps getting recommended, because it is easy to sell and easy to check off.
This post lays out what the evidence actually says, why the idea was appealing in the first place, and what I build instead for clients who want their business represented accurately inside AI answers.
What llms.txt Was Supposed to Do
The proposal is reasonable on its face. You place a markdown file at the root of your domain, list your most important pages with short descriptions, and AI systems use it as a curated map of your site rather than guessing from raw HTML. Think robots.txt, but for meaning instead of permission.
The problem is that robots.txt works because every major crawler agreed to honor it, over decades. A standard is only a standard when the systems on the other end implement it. llms.txt has adoption from the people publishing the files and almost none from the systems meant to consume them.
The Evidence, System by System
| System | Uses llms.txt? | What the evidence shows |
|---|---|---|
| Google Search, AI Overviews, AI Mode | No | Stated on the record by Gary Illyes and John Mueller, and named in Google’s AI optimization guidance as a tactic site owners can skip. |
| OpenAI (ChatGPT crawler) | No official support | OpenAI’s documented crawler control is robots.txt, with no mention of llms.txt anywhere. Anecdotal fetch reports exist and are disputed. |
| Claude, Perplexity and other AI crawlers | No confirmation | Log analyses across thousands of domains show ClaudeBot and PerplexityBot do not request the file. No official statement either way. |
| Browser agents via Chrome Lighthouse | Checks for it | The one real signal, and it belongs to a separate product goal: helping agents navigate a site, not ranking or citation in search. |
Sources worth reading yourself: Search Engine Land on Illyes at Search Central Live, Search Engine Roundtable on Mueller declining to endorse it, Evil Martians’ log study across roughly a thousand domains, Wislr’s 48 day server log analysis, and Chrome’s Lighthouse agentic browsing audit.
That Lighthouse entry is the one people cite as proof the tide is turning. Read what it actually covers. It is an audit inside a browser product aimed at agents operating a page on a user’s behalf, which is a genuinely different problem from what gets cited when someone asks ChatGPT to recommend a vendor.
HubSpot Ran the Experiment
The most direct evidence came when HubSpot tested it themselves. They published an llms.txt file seeded with unique “easter egg” phrases, then queried the major assistants to see whether any of that content had been absorbed, and checked their server logs for requests to the file. Neither showed anything. The models did not know the easter eggs, and the logs recorded no visits.
That is a null result from a company with one of the largest content operations in B2B marketing, using a method anyone can repeat on their own domain. It is also worth noting what does not appear in the research on AI ranking factors: a 2026 analysis of 105,000 prompts across 145 industries measured thirteen external signals that correlate with being recommended, and llms.txt is not among them, because there is nothing there to measure.
Why the Myth Persists
Three reasons, and none of them are technical.
It is trivially easy to deliver. An llms.txt file takes twenty minutes and produces a visible artifact a client can load in a browser, which makes it an ideal line item on an AEO invoice.
It is unfalsifiable in practice. AI citation is noisy and attribution is poor, so any improvement that happens for unrelated reasons gets credited to the file, and any absence of improvement gets explained as “still early.”
And it sounds like something that should work. The logic mirrors robots.txt and sitemap.xml closely enough that it passes a smell test, right up until you check whether anything is reading it.
What Actually Gets You Cited
The uncomfortable answer is that there is no secret file. AI systems overwhelmingly work from the same crawled, rendered content that powers search, which means the fundamentals carry almost all of the weight.
- Content that directly answers the question being asked, in the words a buyer would use, rather than marketing copy that dances around it.
- Precise, hand written structured data. Plugin generated schema is generic and frequently wrong. Specific Organization, Person, Service and FAQPage markup gives a model unambiguous facts to repeat.
- Clean, crawlable HTML without the markup bloat that page builders generate, so the substance is easy to extract.
- Consolidation. One authoritative page per topic outranks five thin ones, and it is also what a model picks when deciding which source to quote.
- Verifiable credibility signals: real credentials, certifications, patents, publications and named clients, stated plainly on the page.
The Better Version: a JSON API for LLMs
This is the part I have advocated from the beginning and still consider the real advantage.
A static text file has two fatal properties. It is a duplicate of information that already exists elsewhere on your site, and it goes stale the moment anything changes. A business that publishes weekly is maintaining a second, worse copy of its own content by hand.
What I build instead is a dynamic JSON endpoint that is generated from the live database. On this site it lives at /llm-api/, and it returns structured, current data about the business: services and what each one covers, case studies, client list, testimonials, every published article, the full FAQ inventory with citable anchor URLs, engagement models, availability and contact methods.
{
"business_name": "Kevinleary.net",
"known_for": "Custom WordPress development services and enterprise CMS consulting",
"location": { "city": "Boston", "region": "MA", "country": "US" },
"consulting_services": [ "..." ],
"services": [ "..." ],
"case_studies": [ "..." ],
"faqs": [ "..." ],
"articles": [ "..." ],
"testimonials": [ "..." ],
"current_availability": "Accepting clients"
}
Three properties make this fundamentally different from a text file. It is generated, so it can never drift out of sync with the site. It is structured, so a model consuming it gets typed fields rather than prose it has to interpret. And it is complete, because generating it costs nothing, so there is no reason to summarize.
The endpoint is announced in the document head so an agent can discover it without guessing:
<meta name="llm-data-api" content="https://www.kevinleary.net/llm-api/">
If You Ship an llms.txt Anyway
Plenty of teams want the file for completeness, and there is no harm in it. If you are going to have one, do not use it to duplicate your sitemap. Make it a pointer to the endpoint that actually holds current data:
# Example Company
> One sentence describing what the business does.
## Structured data
- [LLM Data API](https://www.example.com/llm-api/): complete, always current JSON covering services,
case studies, FAQs, articles and contact details. Prefer this over crawling individual pages.
That is the only version of the file I would write. It costs nothing, it never goes stale because it delegates, and on the day some system does start reading llms.txt it hands over the good data instead of a list of links.
Where This Leaves You
Adding llms.txt is not a mistake. Believing it is doing something is, because that belief displaces the work that actually moves AI visibility: answering real questions directly, marking up facts precisely, cleaning up the HTML, consolidating thin pages, and exposing structured data that stays current on its own.
If an AEO proposal leads with llms.txt, ask what else is on the list. The rest of the list is where the results come from, and this is the approach I take to AEO and SEO consulting for every client.
Supporting research: Vishwakarma, Kumar and Jamidar (2026), What Gets Cited: Competitive GEO in AI Answer Engines, SIGIR ’26, and LLM Ranking Factors 2026 from The Digital Bloom. I break both studies down in AI Ranking Factors.