robots.txt controls access: it tells crawlers which URLs they may fetch. llms.txt describes content: it is a proposed markdown file that points language models and agents to the pages worth reading. If a page is listed in llms.txt but disallowed in robots.txt, a compliant crawler will not fetch it, so the listing does nothing.

Treat them as two files with two jobs, and make sure they never contradict each other.

What each file does

robots.txtllms.txt
PurposeTells crawlers which URLs they can access [2]Gives LLMs and agents a guide to your site [5]
StatusStandards-track specification, RFC 9309 [1]Proposal, not a formal standard [5]
LocationTop-level path, /robots.txt [1]Site root, /llms.txt [5]
FormatUTF-8, served as text/plain [1]Markdown: an H1, an optional short summary, then sections of links [5]
Enforces anythingAsks compliant crawlers to stay out; not authorization [1]No. It is a reading list

robots.txt is mainly for managing crawling traffic, not for hiding pages [2]. RFC 9309 states that its rules are not a form of access authorization, so it is not a security mechanism [1]. A disallow is a request that well-behaved crawlers honor. For what actually enforces access, see robots.txt is not a privacy control.

llms.txt does not grant or deny anything. It lists links with short descriptions. A companion file, llms-full.txt, is described in one implementation guide as holding full markdown bodies, while llms.txt stays an index of links and short descriptions [6].

Who reads llms.txt

As far as public documentation goes, support varies by system. Google says you do not need to create machine-readable AI text files such as llms.txt to appear in Google Search, because Google Search itself does not use them [4]. The notes we worked from contain no vendor document confirming that a specific AI product reads llms.txt, so we make no claim either way. The practical risk is low: the file is cheap to publish, but do not expect it to override anything in robots.txt.

Where the two files conflict

1. A listed URL is disallowed

This is the main failure. Your llms.txt says "read this page," and robots.txt says "do not fetch it."

User-agent: ExampleBot
Disallow: /docs/
- [Quickstart](https://example.com/docs/quickstart): Install and run in five minutes
- [API reference](https://example.com/docs/api): Every endpoint and parameter

For ExampleBot, both links are unreachable. The guide points at pages the crawler has agreed not to open. Either remove the links or remove the disallow, depending on what you want.

2. The llms.txt file itself is blocked

A broad rule can catch /llms.txt along with everything else:

User-agent: *
Disallow: /

Under that rule, a compliant crawler will not fetch your llms.txt either. If you block everything for a given user agent on purpose, there is no point in publishing a guide for it.

3. A listed page carries noindex

Google's documented way to keep a page out of Google Search is noindex or password protection, not robots.txt alone [3]. A noindex tag is only seen if the page can be crawled. Listing a page in llms.txt while telling indexers to drop it sends mixed signals. Decide whether the page is meant to be found, then make both files agree.

Google says a page blocked by robots.txt can still show up in search results, without a description or snippet [2]. So a disallow does not remove a URL from search. Likewise, listing a URL in llms.txt does not make it crawlable. The two files do not cancel or reinforce each other. Each answers a different question.

5. Two sources of truth for llms.txt

Yoast's documentation notes that its plugin will not overwrite an existing llms.txt [7]. If a plugin, a static file and a CMS setting can each produce /llms.txt, one of them wins and the others are silently ignored. Pick one owner for the file.

These are not robots.txt conflicts, but they have the same effect: the guide points somewhere that does not work.

  • Links that return 404 or redirect through several hops
  • Links to staging or old hostnames
  • Links whose canonical points to a different URL

For canonical handling, see the canonical tags guide.

A page not listed in llms.txt is fine

Absence from llms.txt does not block anything. It only means you did not point to the page. Access is decided by robots.txt and by your server. Keep llms.txt short and curated; it does not need to mirror your sitemap.

Consistency checklist

  1. Fetch both files. Confirm /robots.txt and /llms.txt return HTTP 200. robots.txt should be UTF-8 text/plain [1].
  2. Check that llms.txt is not disallowed for any user agent you want to reach it.
  3. Check every listed URL against the robots.txt rules for those user agents.
  4. Check every listed URL's status code. Remove or fix 404s and redirect chains.
  5. Check indexing directives. No listed page should carry noindex unless you meant it.
  6. Check hostnames. Every link should use your production hostname and the canonical URL.
  7. Find the file's owner. Only one system should generate /llms.txt [7].
  8. Re-check after robots.txt edits. A new Disallow can silently orphan links in llms.txt.

Automate the check

Status codes for every URL in llms.txt:

curl -s https://example.com/llms.txt \
  | grep -o 'https\?://[^)[:space:]]*' | sort -u \
  | while read u; do
      echo "$(curl -s -o /dev/null -w '%{http_code}' "$u") $u"
    done

Listed URLs that robots.txt disallows for a given user agent, using Python's standard library:

import re, urllib.request
from urllib.robotparser import RobotFileParser

site = "https://example.com"
agent = "ExampleBot"

rp = RobotFileParser(site + "/robots.txt")
rp.read()

text = urllib.request.urlopen(site + "/llms.txt").read().decode("utf-8")
urls = set(re.findall(r"\]\((https?://[^)\s]+)\)", text))

for u in sorted(urls):
    if not rp.can_fetch(agent, u):
        print("BLOCKED", u)

Python's parser does not match every crawler's implementation exactly, so treat a "BLOCKED" line as a prompt to read the rules, not a final verdict. Run it once per user agent you care about. To see which AI user agents your robots.txt blocks, see how to check if AI can crawl your website.

If you want a quick check of both files together, the free scan at aaivisibility.com reports robots.txt verdicts for 22 AI crawlers and whether an llms.txt file is discoverable. It does not compare the two files link by link, so the script above still has a job.

What to do next

  • Run the checklist above against your live files.
  • Decide, section by section, whether each part of your site should be readable by AI crawlers. Edit robots.txt to match that decision, then write llms.txt only for what remains open.
  • If you do not have an llms.txt yet, follow how to write an llms.txt file.
  • Put the consistency check on a schedule, or in your deploy process, so robots.txt changes do not break your links.