A Disallow line in robots.txt asks crawlers not to fetch a path. It does not stop them. RFC 9309, the robots exclusion standard, says the protocol is not a substitute for valid content security measures [1]. If the content is private, valuable, or legally restricted, you need a control that enforces access, not one that depends on the visitor's good manners.
What robots.txt is
robots.txt is a plain-text file at the root of a host that tells automated clients which paths they are asked to avoid. Compliance is voluntary. Google's documentation says robots.txt instructions cannot enforce crawler behavior and that other crawlers might not obey them [2].
A well-behaved crawler fetches the file, matches its own user-agent token, and skips disallowed paths. A crawler that does not care simply requests the URL, and your server answers as it would for any visitor.
Where a disallow line fails
Some fetchers ignore it
Anyone can write a scraper that never requests robots.txt, or requests it and discards the result. Nothing in the protocol penalizes this. The file only works against operators who chose to honor it.
User-triggered fetches are a different category
Not every AI-related request is a crawl. OpenAI documents ChatGPT-User as used for certain user actions in ChatGPT and Custom GPTs, and says it is not used to crawl the web automatically [3]. OpenAI also says that because these actions are initiated by a user, robots.txt rules may not apply [3].
That means a block aimed at an AI company's training or search crawler may not govern a fetch that happens because a person pasted your URL into a chat. Vendors split their agents differently, so read each vendor's own documentation. For a worked example from another vendor, see PerplexityBot vs Perplexity-User.
OpenAI also separates purposes: OAI-SearchBot is the agent for managing Search opt-outs and automatic crawl, and ChatGPT-User is not used to decide whether content may appear in Search [3]. Blocking the wrong token can leave you with a different outcome than you intended.
Changes are not instant
RFC 9309 lets crawlers cache robots.txt and says they should not use a cached copy for more than 24 hours unless the file is unreachable [1]. After you add a disallow, a compliant crawler may keep acting on the old rules for up to a day.
A disallowed URL can still surface
Google says a page disallowed in robots.txt can still be indexed if other sites link to it [2]. Disallowing stops fetching, not the existence of the URL in other people's pages.
The file itself is public
Anyone can read https://yourdomain.com/robots.txt. A line like this tells every reader exactly where to look:
User-agent: *
Disallow: /internal-pricing-2025/
If a path is secret, listing it in robots.txt discloses it.
What actually controls access
Google's guidance is direct: to keep information secure from web crawlers, password-protecting private files is a better method than robots.txt [2]. The controls below sit on the server or at the network edge, where requests can be refused rather than politely discouraged.
| Control | Enforces access? | What it is good for | Limits |
|---|---|---|---|
| robots.txt | No | Telling cooperative crawlers what to skip | Ignored by non-compliant clients; public; cached |
| Authentication (login, HTTP auth, signed URLs) | Yes | Private, paid, or personal content | Content behind it is not publicly discoverable by design |
| WAF custom rules | Yes | Blocking or challenging requests by pattern | Rules need maintenance; user-agent strings can be spoofed |
| Bot management / bot scoring | Yes, probabilistic | Filtering likely-automated traffic | Scores are estimates, so expect false positives |
| Rate limiting | Yes | Capping request volume per client | Slows scrapers rather than stopping a patient one |
| Terms of use | Legal, not technical | Setting out what reuse is permitted | Requires enforcement after the fact |
Authentication
If a page should only be seen by certain people, require a login or a signed, expiring URL. A crawler cannot read what the server refuses to send. This is the only option on the list that makes the content unavailable by default, and it applies equally to every client, including user-triggered fetchers.
The trade-off is that the content is also invisible to search engines and AI answers. That is the point, but it means you cannot have both "private" and "discoverable" for the same URL.
WAF rules and bot management
Cloudflare's documentation treats WAF custom rules, rate limiting rules, and bot-management controls as separate mechanisms [5]. That separation is the useful part: none of them is robots.txt, and each acts on the request itself. Most CDN and WAF products offer comparable features, so the pattern matters more than the vendor.
Cloudflare documents that Bot Management generates a bot score from 1 to 99 for every request, with lower scores indicating more automated traffic, and that custom rules can reference bot-score fields [5]. In practice you would write a rule that challenges or blocks requests below a score you choose, perhaps scoped to specific paths. Choose the threshold from your own traffic, and review blocked requests, since scoring is an estimate.
User-agent matching alone is weak. The user-agent header is self-reported, so a scraper can claim to be a browser. Rules that combine user-agent, IP reputation, and behavior hold up better than a string match.
Rate limiting
Cloudflare describes rate limiting as a way to control the rate of actions performed by automated sources, and recommends using it together with Bot Management [4]. Rate limits protect your servers and make bulk copying slower and costlier. They do not decide who may read a page.
Terms of use
Published terms can state that automated collection or reuse of your content is not permitted. That is a legal position, not a technical barrier. How enforceable it is depends on jurisdiction, on whether the visitor agreed to the terms, and on the facts. Ask a lawyer before relying on it. It complements technical controls and does not replace them.
Match the control to the goal
| Your goal | Use | Do not rely on |
|---|---|---|
| Keep content private | Authentication | robots.txt |
| Ask cooperative AI crawlers to skip public pages | robots.txt, per-agent rules | robots.txt as the only measure for sensitive pages |
| Stop abusive scraping volume | Rate limiting plus bot management | User-agent rules alone |
| Refuse a specific client or pattern | WAF custom rule | Terms alone |
| Set legal expectations on reuse | Terms of use, reviewed by counsel | Terms as a technical barrier |
robots.txt remains worth maintaining for the case it fits: public content where you want reputable crawlers to follow your preference. Keep the file small as well. RFC 9309 sets a minimum parsing limit of 500 kibibytes [1], so anything beyond what a parser reads cannot be relied on.
Verify what your server actually does
Test the control, not the intention. Request a protected URL as an anonymous client and check the status code:
curl -I https://example.com/members/report.pdf
You want a 401, 403, or a redirect to a login page, not 200. To see how a user-agent rule behaves, send a request with a different user-agent and compare:
curl -I -A "ExampleBot/1.0" https://example.com/private/
Keep in mind that a custom user-agent proves only that your rule matches the string. It does not prove the rule stops a client that lies about it.
For the robots.txt side, How to Check If AI Can Crawl Your Website covers the tokens and how to read the result. Our free scan at aaivisibility.com reports a robots.txt verdict for 22 AI crawlers, but it only reads what the file says. It cannot tell you whether any crawler obeys it.
What to do next
- List the content you consider private or restricted, and check whether any of it is reachable without a login.
- Move that content behind authentication, and remove its path from robots.txt so the file does not advertise it.
- Decide which public sections you want cooperative AI crawlers to skip, and add per-agent rules. Read each vendor's documentation, since user-triggered fetchers may not follow them.
- Add rate limiting and bot-management rules for traffic you want refused, and review the blocked requests regularly.
- Have terms of use reviewed by counsel if reuse of your content matters commercially.
- Fold these checks into your regular technical SEO checklist so crawl rules and access controls stay in sync.