AI crawlers & robots.txt
Which bots the AI engines run, what each one does, and how robots.txt, meta robots and headers control them.
5 articles
noindex and AI Crawlers: What ChatGPT and Perplexity Respect
· 5 min read
Which page-level directives AI retrieval honors: documented noindex, nosnippet and X-Robots-Tag behavior, and why the noai meta tag has no standard.
How to Verify GPTBot and Other AI Crawlers in Your Server Logs
· 5 min read
Tell real GPTBot hits from spoofed user agents: pull IPs from your logs, match them to published ranges, run reverse DNS, and handle vendors with no list.
Google-Extended: What Blocking It Does and Does Not Do
· 4 min read
Google-Extended controls Gemini training and grounding, not Search. See the robots.txt rule, the trade-offs, and which controls manage Search display.
robots.txt Is Not a Privacy Control: What Actually Blocks AI Crawlers
· 7 min read
A robots.txt disallow is a request, not a lock. Where it fails for AI crawlers, and which controls actually enforce access: auth, WAF rules, rate limits, terms.
PerplexityBot vs Perplexity-User: Why Blocking One Doesn't Stop Fetches
· 5 min read
PerplexityBot indexes pages for search; Perplexity-User fetches them on a user's request. What each does, what robots.txt can control, and what to do.