A user-agent string in your logs proves nothing, because any client can send "GPTBot" in its headers. To verify GPTBot, match the request's source IP against the ranges OpenAI publishes. For crawlers whose vendor documents reverse DNS, resolve the IP and confirm the hostname. For vendors that publish neither, such as Anthropic for ClaudeBot today, you cannot prove identity from a log line alone.
Why the user agent is not evidence
Google states that the Googlebot user agent is often spoofed, and recommends a reverse DNS lookup on the source IP or a match against its published IP ranges instead [1]. The same logic applies to every crawler that announces itself in a header. Cloudflare describes three honest ways for a bot to identify itself: a cryptographic Web Bot Auth signature, a published IP list with a stable user agent, or reverse DNS [5]. The user-agent string appears in only one of them, and only alongside a published IP list.
What each vendor documents
| Crawler | What the vendor publishes | Verification method |
|---|---|---|
| GPTBot (OpenAI) | IP list at openai.com/gptbot.json [3] | Match source IP to the list |
| Googlebot (Google) | IP ranges and reverse DNS domains [1][2] | Reverse DNS, or match ranges |
| ClaudeBot (Anthropic) | No IP ranges; uses service-provider public IPs [4] | None from logs alone |
For other crawlers, check the vendor's own crawler documentation. If it lists IP ranges or reverse DNS domains, use them. If it lists neither, treat the vendor like Anthropic below.
Step 1: Pull the source IPs from your logs
In a standard combined-format access log, the client IP is the first field. Count the IPs that claim to be GPTBot:
grep -i "gptbot" /var/log/nginx/access.log \
| awk '{print $1}' | sort | uniq -c | sort -rn > claimed-gptbot.txt
If your site sits behind a CDN or load balancer, the first field may be the proxy's address. Use the real client IP field your stack logs, such as X-Forwarded-For or CF-Connecting-IP, and only trust that header when it is set by infrastructure you control.
Step 2: Match IPs against the published ranges
OpenAI says GPTBot's published IP addresses are listed in openai.com/gptbot.json, and that these published addresses should be used when allowing the crawler [3]. OpenAI also says the user-agent string may include versioning [3], so match on the substring, not an exact string.
Download the list and extract the ranges. Look at the file first, since this command assumes IPv4 CIDR blocks appear as plain text in it:
curl -s https://openai.com/gptbot.json -o gptbot.json
grep -oE '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+/[0-9]+' gptbot.json > gptbot-ranges.txt
wc -l gptbot-ranges.txt
If the file also contains IPv6 ranges, extend the extraction to cover them. Then test each claimed IP:
import ipaddress
nets = [ipaddress.ip_network(l.strip()) for l in open("gptbot-ranges.txt") if l.strip()]
for line in open("claimed-gptbot.txt"):
count, ip = line.split()
addr = ipaddress.ip_address(ip)
ok = any(addr in n for n in nets if n.version == addr.version)
print(("VERIFIED " if ok else "UNVERIFIED ") + ip + " (" + count + " hits)")
Any IP printed as UNVERIFIED that sent a GPTBot user agent is, by this test, not OpenAI's crawler. Re-download the list on a schedule. Published ranges can change, and an old copy will flag real traffic as fake.
Step 3: Use reverse DNS where the vendor documents it
Reverse DNS is the second method. The standards basis is the PTR record, which RFC 7208 describes as a DNS reverse-mapping test for an IP address [8]. Cloudflare's bot validation treats a result as valid when the reverse DNS resolves to one of the vendor's provided domains [6]. AWS likewise says bot confirmation commonly uses reverse DNS lookups and other objective checks [7].
Google documents this for its crawlers: run a reverse lookup on the accessing IP and confirm the domain is googlebot.com, google.com, or googleusercontent.com [2]. Then do a forward lookup on the returned hostname and confirm it points back to the same IP. The forward step is general practice, and it matters because whoever controls an IP's reverse zone can put any hostname in it.
host 203.0.113.45
# 45.113.0.203.in-addr.arpa domain name pointer crawl-203-0-113-45.googlebot.com.
host crawl-203-0-113-45.googlebot.com
# crawl-203-0-113-45.googlebot.com has address 203.0.113.45
The addresses above are documentation placeholders. Both lookups must agree, and the domain must be on the vendor's documented list. Only use this method when the vendor documents its domains. For GPTBot, the documented method in the notes is the IP list, so use that rather than guessing at hostnames.
Step 4: When the vendor publishes nothing
Anthropic says it does not currently publish IP ranges for ClaudeBot and uses service-provider public IPs [4]. A reverse DNS lookup on those addresses will likely return a cloud provider's hostname, which tells you nothing about who rents the machine. In our reading, there is no log-only test that proves a hit is ClaudeBot.
Anthropic also warns that blocking ClaudeBot by IP address may not reliably guarantee an opt-out, because it can interfere with reading robots.txt [4]. So IP blocking is a poor fit for this crawler even if you could identify its addresses.
What you can do instead:
- Label these hits "claimed," never "confirmed." Keep them separate in reports so spoofed traffic does not inflate your AI crawler numbers.
- Check behavior. A real crawler fetching under a declared policy requests
/robots.txt. A client that sends the ClaudeBot user agent and never touches robots.txt, or hammers one endpoint, is suspect. This is a heuristic, not proof. - Rate-limit by behavior, not identity. Apply limits to any client that exceeds a request rate you set, whatever it calls itself.
- Use your CDN's verification where it exists. Cloudflare, for example, maintains verified-bot status based on the methods above [5]. Rely on the verdict there rather than reproducing it.
- Do not allow-list by user agent alone. If a rule grants a crawler special access, such as skipping a paywall or a WAF challenge, tie it to a verified IP, never to a header.
- Keep policy in robots.txt. It only binds crawlers that honor it, which is the point of robots.txt is not a privacy control.
A note on user-triggered fetchers
Some vendors run separate agents that fetch a page when a user asks. They may use different user agents and different IP ranges than the indexing crawler, so a verified list for one agent does not cover the other. PerplexityBot vs Perplexity-User covers that split. Check each user agent against the documentation for that specific agent.
What to do next
- Run the Step 1 command for each AI user agent you care about and save the IP counts.
- For GPTBot, download the range list, run the match script, and note how many hits come from outside the ranges.
- For vendors with documented reverse DNS domains, run both lookups on a sample of IPs.
- For vendors with no published method, relabel those hits as "claimed" in your reporting.
- Confirm your robots.txt says what you intend. How to check if AI can crawl your website walks through that, and the technical SEO checklist covers log and crawl checks more broadly.