What Is llms.txt and Should Your Site Have One?
Published · Updated · 6 min read
Your sitemap tells search engines which pages exist. Your robots.txt tells crawlers which pages they can access. But neither file tells an AI system what your site is, who it is for, or which pages are most worth reading. That is what llms.txt does.
What Is llms.txt?
llms.txt is a plain-text Markdown file published at the root of your website — accessible at yourdomain.com/llms.txt. It provides a structured, human-readable summary of your site that AI language models can consume directly:
- A one-line description of what your site does and who it serves
- Links to your key pages with short descriptions
- Optional topic-organized content map
The convention was proposed in 2024 by Jeremy Howard of Answer.AI and has since been adopted by hundreds of sites including major documentation platforms, SaaS products, and developer tools. It is not a W3C standard — it is a widely adopted community convention. The format is pre-1.0 and may evolve, but publishing one is low-risk and low-effort.
How AI Systems Use llms.txt
When an AI system encounters your domain — through a user query, a crawl, or a direct fetch — it can request /llms.txt to get a structured overview before deciding which pages to crawl or include in a response. Instead of inferring your site structure from your homepage, the AI gets a curated map you wrote yourself.
This matters most for sites with large or complex content libraries — documentation, technical guides, multi-product SaaS — where a crawling AI might otherwise miss the most relevant pages or misidentify what the site is about.
llms.txt vs llms-full.txt
The spec defines only llms.txt. A separate community convention, llms-full.txt, has grown up alongside it:
| File | Contents | Use Case |
|---|---|---|
llms.txt | Site overview + page links with one-line descriptions | Discovery — helps AI find and understand your pages |
llms-full.txt | Full text content of key pages in Markdown | Ingestion — AI reads complete content from one file |
Start with llms.txt. Add llms-full.txt if your site has documentation or guides that AI systems should be able to read end-to-end without crawling each page individually.
A Simple llms.txt Template
This is the format the llmstxt.org spec describes. Replace the bracketed values with your own:
# [Your Site Name] > [One sentence: what your site does and who it is for.] ## Key Pages - [Page Title](/path): [One-line description of what the page contains] - [Page Title](/path): [One-line description of what the page contains] - [Pricing](/pricing): Current pricing plans and what each tier includes - [FAQ](/faq): Answers to common questions about [your product/topic] ## Content by Topic ### [Topic 1] - [Guide Title](/docs/guide-1): [Description] - [Guide Title](/docs/guide-2): [Description] ### [Topic 2] - [Article Title](/blog/article): [Description]
What Makes a Good llms.txt?
- Lead with the description line. The
>block under your site name is consumed first. Make it accurate, specific, and jargon-free. “AI search visibility scanner that checks 18 factors across 5 categories” beats “your AI SEO partner.” - Link to pages with real content, not marketing fluff. AI systems are looking for citable facts, not slogans.
- Token estimates are optional. They are a community convention, not part of the spec. Some sites append an approximate token count to long guides so systems can decide whether to fetch the full page or rely on your summary; if you add them, keep them accurate.
- Keep it current. A stale llms.txt that points to deleted pages or outdated content actively harms your AI discoverability. Review it quarterly.
Should Your Site Have One?
Yes, if any of the following apply:
- You want AI systems to correctly understand what your site is about
- You have more than 10–15 pages of substantive content
- Your site serves a specific audience or use case that a homepage scan might miss
- You have documentation, guides, or FAQ content you want AI to cite accurately
The file takes under an hour to write and publish. The robots.txt check and llms.txt check together account for a significant portion of your site’s AI discoverability score. Both are in your direct control and cost nothing to implement.
Frequently Asked Questions
What is llms.txt?
A plain-text Markdown file at your site root that tells AI language models what your site does, where your key pages are, and how to navigate your content. Proposed by Jeremy Howard in 2024, now widely adopted.
Is llms.txt an official standard?
No — it is a widely adopted community convention, not a W3C standard or IETF RFC. The format is pre-1.0 and may evolve, but publishing one today is low-risk and widely supported.
What is llms-full.txt?
A separate community convention, not defined by the llms.txt spec, that includes the full text content of key pages rather than just links and one-line descriptions. It lets AI systems ingest your complete content from a single file. Optional, but useful for documentation-heavy sites.
Does publishing llms.txt guarantee AI citation?
No. It improves discoverability — AI systems can understand your site faster and more accurately. Whether they cite you depends on content quality, topic relevance, and the specific user query.