llms.txt: what it is, whether you need it, and how to write one

llms.txt is a Markdown file placed at the root of a website, for example /llms.txt, that gives AI tools a short summary of the site and a list of its most useful pages. It is a proposal from 2024, not an official standard, and no major search engine has said it uses the file for ranking or for choosing sources. It is cheap to write, but don’t rely on it.

Where llms.txt came from

The idea was published in September 2024 by Jeremy Howard of Answer.AI. The problem it tries to solve is real: web pages are full of menus, scripts, cookie banners and adverts, and a language model working within a limited context has to dig through all that to find the content. A small, clean Markdown file could point it straight to the pages that matter.

The proposal also suggests offering Markdown versions of important pages, at the same URL with .md added, so tools can read them without the clutter.

It is worth being clear about what llms.txt is not:

• It is not an access rule. It doesn’t allow or block anything. That is still the job of robots.txt; see robots.txt rules for AI bots.
• It is not a sitemap replacement. Search engines use your XML sitemap, not llms.txt, to discover pages.
• It is not a ranking factor that anyone has confirmed.

Who actually reads it?

This is the honest part. As of 2026:

• Google representatives have said publicly that Google Search doesn’t use llms.txt, and that it isn’t needed to appear in AI Overviews.
• OpenAI, Anthropic and Perplexity have not announced that their crawlers or search products use llms.txt files on third-party websites to choose sources.
• Some developer documentation sites, including those of a few AI companies, publish an llms.txt so that coding assistants and similar tools can load their docs easily. That is the use case where the file is most clearly helpful.
• Some tools and plugins read the file when a user points them at a site directly.

You can check your own logs. If AI crawlers were relying on the file, you would expect to see regular requests for /llms.txt. Many site owners report seeing few or none from the big crawlers, though this varies and may change. The AI bot directory lists each crawler’s published purpose.

Do you need one?

A quick way to decide:

If you are short on time, your effort is better spent on being crawlable, indexed and clearly written. The AI SEO basics guide covers those.

The format

The proposal defines a simple structure, in this order:

1. An H1 with the name of the site or project. This is the only required part.
2. A blockquote with a one or two sentence summary.
3. Optional paragraphs or lists with more context.
4. H2 sections, each with a list of links in the form - [Title](URL): short note.
5. An optional section called Optional, for links a tool can skip if it is short of space.

An example

Here is a short llms.txt for a made-up bakery:

# Northgate Bakery

> Independent bakery in Leeds selling sourdough, pastries and
> celebration cakes. Open Tuesday to Sunday, with local delivery.

Prices are in GBP and include VAT. Cake orders need 72 hours’ notice.

## Main pages

- [Menu and prices](https://example.com/menu): full list with allergens
- [Order a celebration cake](https://example.com/cakes): sizes, flavours, lead times
- [Delivery area](https://example.com/delivery): postcodes and charges

## Help

- [Allergen information](https://example.com/allergens): how we handle the 14 major allergens
- [FAQ](https://example.com/faq): opening hours, parking, payment

## Optional

- [Our story](https://example.com/about)
- [Blog](https://example.com/blog)

How to write a good one

• Keep it short. The point is to be smaller and cleaner than your website. A few dozen links is plenty for most sites.
• Pick your best pages, not all of them. Think: which ten pages would answer most questions about you?
• Write useful notes after each link. “Pricing” says less than “Prices for all three plans, billed monthly or yearly”.
• Use absolute URLs and make sure every link works.
• Keep it in sync. An llms.txt pointing at deleted pages or old prices does more harm than good. Add it to the checklist you follow when pages change.
• Serve it as plain text at /llms.txt, with a normal 200 response.

Some people also publish llms-full.txt, a single file containing the full text of their documentation. That is a convention used by some docs tools, not part of the original proposal, and it only makes sense for documentation-style sites.

What to do instead (or as well)

If your goal is to be cited by AI tools, these matter far more:

• Allow the AI search crawlers in robots.txt and your firewall.
• Put your main content in the HTML, not only in JavaScript.
• Get indexed by Google and Bing, with a working XML sitemap.
• Answer questions directly and keep facts current.
• Check which AI bots really visit, and whether they are genuine, with the free bot checker.

In short

• llms.txt is a 2024 proposal, not a standard.
• No major search engine has said it uses the file; Google has said it doesn’t.
• It is most useful for developer documentation.
• If you write one: H1, summary, a short list of your best pages with clear notes, kept up to date.
• Don’t expect it to change your AI visibility on its own.