Verify AI Crawlers With Reverse DNS (Spot Fake Bots)

Your server log shows a hit from "Googlebot." It is polite, it respects your robots.txt, and it appears at a sensible pace. Or is it? Plenty of scrapers, spam networks, and aggressive AI crawlers set their user-agent to "Googlebot" because site owners instinctively trust that name and leave the door open. The reliable way to know who is really visiting is reverse DNS verification: take the visitor's IP address, look up its hostname, then resolve that hostname back to the IP and confirm the two match. It takes under a minute by hand, and it works exactly the way Google documents it.

In this guide you will learn the step-by-step method Google recommends, the copy-paste commands for Linux, macOS, and Windows, which bots publish verifiable hostnames (Googlebot and Bingbot do; OpenAI's GPTBot does not — it needs a different check), and what to do with the impostors you find.

Why user-agent strings are worthless evidence

A user-agent is just an HTTP header. Anyone can send any header they like. Running curl -A "Googlebot" from a laptop makes your request look like Google's crawler, and every spam network on earth knows this trick.

That matters because fake crawlers are not rare. Site operators routinely find that 20 to 50 percent of their "Googlebot" or "Bingbot" traffic fails verification — these are scrapers harvesting content for spam sites, SEO tools gone rogue, and aggressive AI data collectors hiding behind a trusted name. Some follow your robots.txt. Many do not.

This is also why the reverse-DNS check exists. Google does not publish a list of its crawler IP addresses as the primary method (ranges change, and stale lists cause trouble). Instead it gives you a way to verify any single IP on the spot: if the visitor really is Googlebot, the DNS records will prove it, because only Google controls the reverse DNS for its own IP space.

The verification method: reverse DNS plus forward DNS

Google's documented method has two halves, and both matter. Here is the full process.

Step 1: Get the visitor's real IP from your logs

Find the suspicious request in your server log. In Apache the access log usually lives at /var/log/apache2/access.log; in nginx at /var/log/nginx/access.log. The first field on each line is the client IP:

66.249.66.1 - - [19/Oct/2026:10:15:32 +0000] "GET /blog/post HTTP/1.1" 200 4123 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Grab the IP and the user-agent together. You need the IP — the user-agent only tells you which identity the visitor is claiming.

One warning before you go further: if your site sits behind Cloudflare, a load balancer, or any reverse proxy, the IP in your log might be the proxy's address, not the visitor's. Fix your logging first (for example with Apache's mod_remoteip or nginx's real_ip module) so you are checking the true client IP. Verifying a proxy IP and then blocking it would be a costly mistake.

Step 2: Reverse DNS lookup — IP to hostname

Ask DNS which hostname belongs to that IP. On Linux or macOS:

host 66.249.66.1

or the equivalent with dig:

dig -x 66.249.66.1 +short

On Windows, use nslookup:

nslookup 66.249.66.1

or in PowerShell:

Resolve-DnsName 66.249.66.1

A real Google crawler returns something like crawl-66-249-66-1.googlebot.com. Notice the pattern: the hostname ends with googlebot.com. That is the important part. But do not stop here — this step alone is not proof.

Step 3: Forward DNS lookup — hostname back to IP

Now resolve the hostname back to an IP and check it matches the original address:

host crawl-66-249-66-1.googlebot.com

If the forward lookup returns 66.249.66.1 — the same IP you started with — the crawler is genuine. Both directions agree.

Skipping this step is the classic mistake. Anyone can configure reverse DNS (PTR records) for their own IPs to point at a googlebot.com hostname. That costs nothing and proves nothing. What a spoofer cannot do is make the real crawl-x-x-x-x.googlebot.com hostname resolve to their server's IP, because Google controls the forward DNS for googlebot.com. The round trip is what makes the check trustworthy.

Step 4: Match the hostname to the right bot

Different crawlers use different domain suffixes. Check the hostname against the documented suffix for the bot named in the user-agent:

Googlebot and most Google crawlers — hostname ends in googlebot.com or google.com (for example rate-limited-proxy-66-249-90-77.google.com).
Bingbot — hostname ends in search.msn.com (for example msnbot-207-46-13-5.search.msn.com).
Other search crawlers — DuckDuckGo, Yandex, and Baidu each document their own hostname patterns; check each vendor's webmaster docs before trusting a suffix.

If the user-agent says "Googlebot" but the verified hostname ends in some random domain, or resolves to a residential ISP, you are looking at an impostor.

Which AI crawlers can you verify this way?

Not every bot publishes verifiable DNS. Here is the honest split as of 2026.

Verifiable by reverse DNS: Googlebot, Bingbot

The two big search crawlers both support the round-trip method. Bing goes one step further and offers a Verify Bingbot tool where you paste an IP and get a yes-or-no answer, plus published IP ranges you can compare against your logs. When in doubt, run both checks: the tool is quick, and the DNS round trip gives you independent confirmation.

Not verifiable by reverse DNS: GPTBot and most AI crawlers

OpenAI does not document a reverse-DNS verification path for GPTBot. If a guide tells you to reverse-resolve GPTBot to an openai.com hostname, treat it as unverified — that method is not how OpenAI identifies its crawler. Instead, OpenAI publishes its crawler IP ranges as machine-readable JSON files:

curl -s https://openai.com/gptbot.json

The rule is simple: a request claiming to be GPTBot from an IP outside those published CIDR ranges is spoofed. OpenAI keeps separate files for its other agents (searchbot.json, chatgpt-user.json), and the lists change over time, so fetch them on a schedule rather than hard-coding today's ranges into a firewall rule.

Other AI crawlers follow the same pattern: they hand you a published IP list rather than a DNS suffix. The principle stays the same — check the connecting IP against the operator's own published data, never trust the user-agent string on its own.

Automating the check: a small script for your logs

Checking one IP by hand is fine. Checking the thousand suspicious hits from last night is a job for a script. Here is a compact Python snippet that performs the full round trip:

import socket

def verify_googlebot(ip):
    try:
        hostname = socket.gethostbyaddr(ip)[0]
    except socket.herror:
        return False, "no reverse DNS record"
    if not (hostname.endswith(".googlebot.com") or
            hostname.endswith(".google.com")):
        return False, f"hostname {hostname} is not a Google crawler domain"
    try:
        resolved = socket.gethostbyname(hostname)
    except socket.gaierror:
        return False, "forward lookup failed"
    if resolved != ip:
        return False, "forward lookup does not match original IP"
    return True, hostname

print(verify_googlebot("66.249.66.1"))

Run the suspicious IPs from your logs through this on a schedule, write the failures to a file, and you have an impostor list you can act on. Swap the suffix check for .search.msn.com to verify Bingbot, or swap the whole DNS approach for CIDR matching against openai.com/gptbot.json to verify GPTBot.

What to do with the impostors you find

Finding fake bots is only half the job. Here is how to act on the results without shooting your own SEO in the foot.

Verify first, block second

Never block an IP just because it "looks odd" or because the reverse lookup returned nothing. Missing PTR records are common — plenty of legitimate services never set reverse DNS. Block only when there is a clear mismatch: the visitor claims a trusted identity and fails that identity's documented verification.

Block at the right layer

A robots.txt disallow means nothing to an impostor — a bot already lying about its identity is not going to honor your text file. Block at the server or firewall layer instead:

Apache (.htaccess) — block a single confirmed fake IP with mod_rewrite:
```
RewriteEngine On
RewriteCond %{REMOTE_ADDR} ^203\.0\.113\.77$
RewriteRule ^ - [F]
```
Build your blocklist in a file and generate the .htaccess rules from it, or use our .htaccess generator to create clean access-control rules without hand-editing.
Firewall or CDN — if the fake crawler is aggressive (hundreds of requests per minute), block the IP in your firewall or Cloudflare firewall rules so the traffic never reaches your origin server.
Rate limit, don't just block — for borderline cases (unverifiable but not obviously fake), rate limiting the IP is safer than a hard block.

If you want to tighten your bot rules at the source first, use our robots.txt generator to build a correct file, then run it through the robots.txt tester before publishing. Honest crawlers obey it; the fakes you catch with DNS verification get blocked at the server level, where they cannot ignore you.

Pitfalls that trip people up

Only doing the reverse lookup. Worth repeating: without the forward lookup back to the same IP, the check proves nothing.
Checking the wrong IP. Behind a proxy or CDN, your log's first IP field may be Cloudflare's or your load balancer's. Verify the true client IP or you will "verify" (and possibly block) your own infrastructure.
Trusting X-Forwarded-For. That header is set by the client and trivially forged. Only trust it when it comes from your own known proxy.
DNS caching during an attack. If you are verifying many IPs quickly, be aware your resolver caches answers; a TTL of a few minutes is fine, but do not treat a cached result as permanent evidence for an IP you plan to block long-term.
IPv6. Google crawls over IPv6 too. The same method works (dig -x handles IPv6 addresses), but make sure your script and firewall rules handle both address families.
Confusing fetch services with crawlers. A request with Google's hostname but a weird user-agent might be a legitimate Google service (Translate, Web Light, a rich-result tester) rather than Googlebot. Verify the specific service's documented user-agent before deciding what to do with it.

Make verification a habit, not a one-off

Here is a practical routine: once a month, pull the top crawler IPs from your logs, run them through the round-trip check (or the Bing tool and the OpenAI JSON files), and update your blocklist. Pair that with a clean robots.txt — test it with our robots.txt tester — so honest bots get clear instructions and only the dishonest ones need the DNS treatment. If you are new to bot management, our guide on blocking AI crawlers and bad bots with .htaccess walks through the enforcement side in detail, and the robots.txt directives guide explains what each rule actually does.

FAQ

Can I just publish a list of Googlebot IPs and block everything else?

No. Google's crawler IP ranges change, and maintaining your own list means it will go stale and you will eventually block the real Googlebot — which directly hurts your rankings. Google itself recommends per-IP DNS verification instead of hard-coded lists for exactly this reason.

What does it mean if a "Googlebot" IP has no reverse DNS at all?

It means the request is not verifiable as Googlebot, so treat it as an ordinary unknown visitor: it gets no special trust and no special access. Do not automatically block it, though — some legitimate services simply never configure PTR records.

Does reverse DNS verification work for GPTBot?

No. OpenAI does not document a reverse-DNS method for its crawlers. Verify GPTBot by checking the source IP against the CIDR ranges in https://openai.com/gptbot.json. A claim of GPTBot from an IP outside those ranges is spoofed.

How do I verify Bingbot?

Use the same round-trip method (hostname should end in search.msn.com and resolve back to the same IP), or paste the IP into Microsoft's Verify Bingbot tool for an instant answer. Microsoft also publishes Bingbot's IP ranges as JSON for log comparison.

Will blocking fake Googlebots hurt my rankings?

Blocking an IP that failed verification does not affect the real Googlebot, because the real Googlebot comes from different, verifiable IPs. That is the whole point of verifying first: you only block visitors that provably are not who they claim to be.

Should I verify every single bot hit in real time?

Real-time DNS verification on every request adds latency and can slow your site under load. The practical approach is to log first, verify in batches (a script or cron job), and then block the confirmed fakes. Reserve real-time checks for high-value or high-risk endpoints.

Bottom line: the user-agent is a name tag anyone can print. Reverse DNS plus a forward lookup is the ID check. Run it on your suspicious "crawler" traffic, block the confirmed impostors at the server level, and keep your robots.txt clean for the honest bots that actually follow it.

How to Verify AI Crawlers With Reverse DNS (Stop Fake Googlebots)