noindex is documented for Google Search, and OpenAI documents it for ChatGPT Search. For Perplexity, we found no official page that documents noindex or any other page-level directive. The noai meta tag has no standard and no vendor documentation we could find. Treat it as having no effect.
The practical rule: use robots.txt to control which crawlers fetch your site, and use page-level directives only where a vendor says it reads them.
What each vendor documents
| Directive | Documented behavior | Source |
|---|---|---|
noindex (meta tag) | Google: keeps content out of Google Search results. OpenAI: use it if you do not want content surfaced in ChatGPT Search results. | [2], [4] |
nosnippet, max-snippet, data-nosnippet | Google: supported controls for snippets and previews. | [1] |
X-Robots-Tag header | Google: any rule usable in a robots meta tag can be set here. | [1] |
noindex for Perplexity | No official page found. | none |
noai, noimageai | No standard and no vendor documentation found. | none |
The sections below cover each row.
noindex
Google says the noindex robots meta tag tells it not to index content or let it appear in Google Search results [2]. OpenAI says that if you do not want content to appear in ChatGPT Search results, you can use the noindex meta tag [4].
The tag is a meta element with name="robots" and content="noindex", placed in the page head.
noindex is about surfacing in search results. OpenAI's documentation ties it to ChatGPT Search. It does not describe noindex as a training control. For training, OpenAI says to disallow GPTBot in robots.txt for the sites and pages you want excluded [4].
The robots.txt conflict
Google says that if robots.txt blocks crawling, Google cannot discover or follow the indexing and serving rules in meta tags or X-Robots-Tag. URLs that need those rules must not be disallowed [1]. Google also says a blocked URL may still appear in search results, and that to reliably keep it out you should use noindex and allow crawling [5].
OpenAI does not spell out this interaction in the pages we reviewed. The logic is the same, though: a crawler that is not allowed to fetch the page cannot read a tag inside it. If you want noindex honored, don't also disallow that crawler on the same URL. If you want the crawler to skip the URL entirely, use robots.txt and don't rely on the tag.
Choose one mechanism per crawler and URL. Don't stack them.
Perplexity
As far as public documentation goes, Perplexity's published material covers robots.txt behavior for its crawlers, not page-level directives. We found no official statement that PerplexityBot reads noindex, nosnippet or X-Robots-Tag. It may do so. It is simply not documented in what we reviewed.
Treat a noindex tag as unconfirmed for Perplexity. If you need to keep content from that crawler, use a robots.txt rule, and read PerplexityBot vs Perplexity-User for what that rule does and does not stop. For anything that must stay private, a robots.txt rule is not enough. See robots.txt is not a privacy control.
nosnippet, max-snippet and data-nosnippet
These are Google controls for how a page is previewed in results. Google documents nosnippet, max-snippet and data-nosnippet as supported controls, and says data-nosnippet can hide specific text from featured snippets and regular snippets [1].
nosnippetapplies to the whole page.max-snippetlimits how long the preview can be.data-nosnippetis an HTML attribute you put on an individual span, div or section element to exclude just that text.
The documentation we reviewed ties these to Google results, not to ChatGPT or Perplexity. Don't assume they limit how another engine quotes you. They also trade off against visibility: a page with nosnippet gives Google less to show.
X-Robots-Tag
Google says X-Robots-Tag goes in the HTTP response headers and that any rule usable in a robots meta tag can also be specified there [1]. That makes it the way to apply noindex to files with no HTML head, such as PDFs.
Example for nginx:
location ~* \.pdf$ {
add_header X-Robots-Tag "noindex";
}
Example for Apache:
<FilesMatch "\.pdf$">
Header set X-Robots-Tag "noindex"
</FilesMatch>
Check the result with a header request:
curl -I https://example.com/files/report.pdf
Look for X-Robots-Tag: noindex in the response. A CDN or proxy can add, strip or duplicate headers, so test the public URL rather than the origin.
User-agent tokens in meta tags
Google says the supported robots meta user-agent tokens are googlebot and googlebot-news, and other values are ignored [1]. So a meta tag named gptbot or perplexitybot has no documented effect in Google. Neither OpenAI nor Perplexity documents reading bot-specific meta tags in the pages we reviewed. Use robots.txt for per-crawler rules. See how to check AI crawler access in robots.txt.
The noai tag
You will see advice to add a meta tag with name="robots" and content="noai, noimageai", or similar. We found no standard that defines these values and no vendor document that says a crawler honors them. Some site platforms and publishers add them, but that is adoption, not specification.
Adding it costs little, but it should not be what you rely on. If your goal is exclusion from training or search, use the mechanisms vendors document: robots.txt rules for named crawlers [4], and noindex where the vendor says it is read [2][4]. For Google's AI-related crawler token, see Google-Extended: what blocking it does and does not do.
What to do next
- Decide per page what you want: stay out of search results, stay out of training, or only limit previews. These are different goals with different controls.
- For ChatGPT Search, keep OAI-SearchBot allowed in
robots.txtif you want visibility [4]. Usenoindexon pages you want left out of ChatGPT Search results, and don't disallow the crawler on those same URLs. - For training exclusion, disallow GPTBot in
robots.txt[4]. - Apply
X-Robots-Tagto non-HTML files that neednoindex, and confirm withcurl -I. - Change one URL first, confirm the tag or header is served, then roll it out. A sitewide
noindexadded by mistake removes pages from results, so review the template before deploying. - For Perplexity, don't count on page-level tags. Use
robots.txt, and use real access controls for private content.