llms.txt is a proposed plain-text file, written in Markdown and served at the root of your site (/llms.txt), that lists your most important pages with one-line descriptions so that tools built on language models can find your best content without parsing your whole site. It was proposed by Jeremy Howard of Answer.AI in September 2024. It is not a robots file: it allows and blocks nothing. At the time of writing, none of the major AI answer engines has publicly confirmed that it reads llms.txt when answering questions, and Google has said it does not use it. It is cheap to add, useful for tools that fetch it on demand, and easy to mistake for a ranking lever it is not.
What llms.txt is
The idea is simple. Web pages are built for browsers: navigation, scripts, banners, footers, and the actual content somewhere in the middle. A language model with a limited context window that wants to understand your site has to fetch a lot of pages and throw most of each one away. llms.txt offers a shortcut: a single, small Markdown file that says what the site is and links to the pages that matter, with a sentence about each.
The proposed format is strict enough to be parsed and loose enough to be written by hand:
- An H1 with the name of the site or project. This is the only required element.
- A blockquote with a short summary of what the site is and what a reader will find.
- Optional paragraphs or lists with context that helps a reader interpret the links.
- H2 sections that group links. Each link is a Markdown link followed by a colon and a one-line description.
- An optional H2 called "Optional" for secondary pages that a tool may skip when context is short.
A companion file, llms-full.txt, can hold the complete text of the listed pages in one Markdown document, for tools that prefer a single request over many.
An example
A short example for a fictional accounting-software site (example content, not a real company):
# Example Accounting
> Online accounting software for freelancers and small businesses in the Netherlands. Invoicing, expenses, VAT returns and bank connections in one tool.
## Product
- [Features](https://example.com/features): What the software does, per module, with screenshots.
- [Pricing](https://example.com/pricing): Plans, what each includes, and how billing works.
- [Bank connections](https://example.com/banks): Which Dutch and Belgian banks are supported and how syncing works.
## Guides
- [VAT returns for freelancers](https://example.com/guides/vat-returns): How quarterly VAT returns work and how to file them from the software.
- [Invoicing requirements in the Netherlands](https://example.com/guides/invoice-requirements): The fields a Dutch invoice must contain.
## Optional
- [Changelog](https://example.com/changelog): Release notes per month.
- [Careers](https://example.com/careers): Open roles.
That is the whole thing. Curated, factual, short.
What llms.txt is not
| File | Purpose | Who reads it | Effect |
|---|---|---|---|
| robots.txt | Tells crawlers which paths they may fetch | Every well-behaved crawler, including AI crawlers | Allows or blocks access |
| sitemap.xml | Lists every canonical URL for discovery | Search engine crawlers | Helps crawlers find pages; no curation |
| llms.txt | Points language-model tools to the pages that explain the site best | Tools that fetch it on demand; adoption by engines unconfirmed | Guidance only; no access control, no ranking effect |
The most common confusion is treating llms.txt as "robots.txt for AI". It is closer to a curated reading list. Access is still decided by robots rules per crawler (GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, Google-Extended and the rest), and a page you list in llms.txt but block in robots.txt is a page the tool cannot read.
Who reads it today
Honesty is useful here. Adoption is real among documentation tools and developer-facing products: many API and framework sites publish one, and some coding assistants and agents fetch it when a user points them at a site. Adoption among the big answer engines is unconfirmed. OpenAI, Perplexity and Anthropic have not documented llms.txt as part of how their search or answer products select sources, and Google has said it does not use the file. Answers in those engines are driven by regular crawling, indexing and the quality of the pages themselves.
So the practical value today is with tools that fetch the file on demand, and with the curation exercise it forces on you: deciding which twenty to a hundred pages explain your business best is work that improves those pages.
Should you create one?
Yes, if it costs you an hour and you can keep it current. It is especially sensible for:
- SaaS products and developer tools with documentation.
- Sites with a help centre or knowledge base that already answers customer questions.
- Businesses that want a single place to state, in plain language, what they are and for whom.
Think twice if the only way to maintain it is a manual list that will be stale in three months, or if you expect it to change your rankings or citations. It will not do that on its own.
How to create llms.txt
- Pick the pages. Twenty to a hundred pages that answer real questions about your product, service, pricing, and how things work. Not every URL; that is what the sitemap is for.
- Write one-line descriptions. Factual, specific, no slogans. "Which banks are supported and how syncing works" beats "Discover our powerful integrations".
- Group into sections. Product, pricing, guides, documentation, company. Three to six sections is plenty.
- Add an Optional section for pages that are useful but not essential.
- Serve it at the root as /llms.txt with a text content type, UTF-8, no HTML wrapper.
- Generate it from your CMS if you can, so that new guides and pages are added on publish and removed pages disappear.
- Optionally add llms-full.txt with the full Markdown text of the listed pages. Keep an eye on its size; a file of several megabytes helps nobody.
- Check it against robots.txt. Every listed page must be fetchable by the crawlers you allow.
Common mistakes
- Listing every URL on the site. That is a sitemap in the wrong format.
- Marketing descriptions. A tool reading "the best solution for modern teams" learns nothing.
- A stale file that links to pages that moved or were deleted.
- Publishing llms.txt while blocking the crawlers that would read the pages.
- Expecting it to change what ChatGPT, Perplexity or AI Overviews cite. Those decisions are made on the pages themselves: direct answers, clear entities, structure, freshness.
llms.txt and your AI crawler policy
llms.txt is a pointer; access is policy. Decide per crawler what you allow: training crawlers, search crawlers, user-triggered fetches. Where a vendor separates those (OpenAI does, with GPTBot for training and OAI-SearchBot for search), you can allow search and block training. Make the robots rules explicit, check that your CDN's bot protection does not undo them, and then make llms.txt consistent with the result. A tidy pair of files says the same thing twice: here is what we are, here is what you may read.
Conclusion
llms.txt is a small, sensible file with an honest limitation: it helps tools that ask for it, and it does not, by itself, change how AI answer engines cite you. Create one if you can keep it current, write it as a curated, factual index, keep it consistent with your robots rules, and put your real effort into the pages it points to. If you want the file maintained automatically alongside the crawler policy and a weekly measurement of whether ChatGPT, Perplexity, Gemini and AI Overviews actually cite you, see how AI visibility works in Traze.