Verifying crawler access through IP lookups

If you are diagnosing indexation drops solely by checking your HTTP User-Agent headers, you are probably taking orders from scrapers masquerading as search engine bots.

A client called us last year after half their product catalog vanished from Google’s index over six weeks. Their web team had looked at their access logs, seen millions of hits from “Googlebot,” and assumed crawler access was healthy. But when we audited the traffic source, over 80 percent of those requests originated from cheap commercial VPNs and bulletproof hosting providers in Russia and Southeast Asia. A commercial scraper had spoofed the Googlebot string, triggered the site’s rate limiter, and caused the web application firewall (WAF) to drop all hits claiming to be Googlebot—including the real one.

Verifying crawler access through IP lookups is the only bulletproof way to separate legitimate search engine infrastructure from malicious impersonators. If you have tried setting up IP checks before and gotten burned by false positives, broken firewalls, or stale IP lists, this guide walks through how to build a verification pipeline that actually works.

Why User-Agent Spoofing Breaks Technical SEO Audits

Anyone with curl or a Python script can append -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" to an outbound request. Malicious actors do this for three primary reasons: 1. To bypass basic site scrapers and paywalls. 2. To evade aggressive rate-limiting rules meant for standard web scrapers. 3. To hide vulnerability scanning activity inside legitimate search engine log entries.

When your server accepts these requests at face value, your log analytics become useless. You end up reporting that Googlebot visited a page 50 times this week, when in reality, the real Googlebot got a 429 Too Many Requests or 403 Forbidden response on its only attempt because the scraper exhausted your origin server’s connection pool.

Furthermore, edge protection services like Cloudflare, AWS WAF, or Fastly often interact poorly with unverified bots. If you set up a firewall rule that says “Allow User-Agent: Googlebot,” you create a massive security hole. If you set up a rule that says “Block aggressive requests,” you risk blocking real search engine nodes during a re-indexation push. How bot firewalls block crawler discovery covers the mechanics of how security configurations silently drop search traffic.

Verifying Crawler Access Through IP Lookups: The Two-Way DNS Method

The gold standard for bot verification recommended by Google, Bing, and Yandex is the two-way DNS lookup (also known as reverse-forward DNS verification). You cannot rely on a reverse DNS lookup alone, because an attacker who controls their own IP block can set up a reverse PTR record that points to fake-googlebot.com or even spoof an unverified hostname.

Here is the exact step-by-step procedure to manually or programmatically verify a bot IP address:

1. Extract the client IP and User-Agent from your web server access logs or current request header.
2. Execute a reverse DNS lookup (PTR query) on the client IP address.
3. Verify the domain name suffix. For Google, the resolved domain must end in .googlebot.com, .google.com, or .google-proxy.com. For Bing, it must end in .search.msn.com.
4. Execute a forward DNS lookup (A or AAAA query) on the hostname returned in Step 2.
5. Compare the resulting IP address with the original client IP address from Step 1. If the IP addresses match exactly, the request is authentic.

Let’s look at how this works in a terminal using standard networking commands (host or dig).

Example 1: Verifying a Real Googlebot IP

Suppose you see an incoming request from the IP address 66.249.66.1.

First, run the reverse DNS lookup:

host 66.249.66.1

Output:

1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

The domain ends with .googlebot.com, which matches Google’s public domain specification. Now run the forward DNS lookup on that hostname:

host crawl-66-249-66-1.googlebot.com

Output:

crawl-66-249-66-1.googlebot.com has address 66.249.66.1

The forward lookup returns 66.249.66.1, matching our original IP address. The request is legitimate.

Example 2: Catching a Spoofed Request

Now consider an incoming request claiming to be Googlebot originating from IP 198.51.100.45.

First, run the reverse DNS lookup:

host 198.51.100.45

Output:

45.100.51.198.in-addr.arpa domain name pointer vps12345.cheap-server-host.net.

The lookup returns cheap-server-host.net, which is not an official search engine domain. You can stop here: this request is an impersonator.

Even if the attacker configured their PTR record to return fake-googlebot.com, running the forward lookup (host fake-googlebot.com) would fail to point back to 198.51.100.45 because they do not control the DNS zone for googlebot.com.

Major Search Engines and Their Official Verification Protocols

Different crawlers use different root domains and IP verification strategies. The table below lists the primary search engines, their official domain suffixes for DNS verification, and where to find their published IP ranges.

Notice that modern AI crawlers like GPTBot (OpenAI) and ClaudeBot (Anthropic) rely primarily on published IP ranges rather than reverse DNS PTR records. For these bots, running a reverse DNS lookup often returns generic cloud hosting domains (like AWS or GCP pointers). For AI agents, matching incoming IPs against published JSON lists is the preferred verification method.

When inspecting link discovery patterns across custom web properties, tracking crawler IP integrity helps prevent false link attribution. Using log patterns to verify real crawler visits goes deeper into analyzing response codes alongside IP data.

Static IP Lists vs. Dynamic Lookups: Tradeoffs and Performance

When implementing bot verification in production environments, you face a architecture trade-off: Dynamic DNS Lookups versus Static IP Range Allowlists.

Option 1: Dynamic DNS Lookups at the Application Layer

Running live gethostbyaddr() or DNS queries on every incoming HTTP request gives you real-time accuracy, but it introduces significant network latency.

• Pros: Always up to date. No database or list maintenance required.
• Cons: Adds 50–200ms of latency per request. If your DNS resolver drops queries or gets rate-limited, your application may hang or default to blocking legitimate bots.

Option 2: Pre-fetching and Matching Official IP Ranges

Google and Bing publish updated JSON feeds of their crawler IP ranges. You can write a cron job that pulls these feeds daily, parses the CIDR blocks (e.g., 66.249.64.0/20), and populates your firewall or log-parser allowlist.

• Pros: Zero added latency during HTTP request processing. Works smoothly at the CDN/WAF layer.
• Cons: Search engines occasionally add new IP ranges before updating public JSON files. If your sync script breaks, your allowlist gradually becomes stale.

Here is how a standard Googlebot JSON IP entry looks in their published endpoint:

{
  "creationTime": "2026-01-15T12:00:00.000000",
  "prefixes": [
    {
      "ipv4Prefix": "66.249.64.0/20"
    },
    {
      "ipv6Prefix": "2607:f8b0:4000::/45"
    }
  ]
}

For most engineering teams, a hybrid approach works best: use published IP lists at the firewall level for fast filtering, and fall back to cached reverse/forward DNS checks inside log analysis pipelines when verifying ambiguous requests.

How to Automate Bot Verification in Python

If you are processing web server access logs to confirm whether search engines are reaching critical landing pages or backlink targets, doing manual DNS lookups isn’t scalable.

Below is a complete, production-ready Python script that implements reverse and forward DNS checks with built-in caching.

import socket
import functools

# Valid host suffixes for major search engines
VALID_SUFFIXES = {
    'google': ('.googlebot.com', '.google.com', '.google-proxy.com'),
    'bing': ('.search.msn.com',),
    'yandex': ('.yandex.ru', '.yandex.net', '.yandex.com')
}

@functools.lru_cache(maxsize=4096)
def verify_bot_ip(ip_address, engine='google'):
    """
    Performs two-way DNS verification on an incoming IP address.
    Returns True if authentic, False otherwise.
    """
    allowed_suffixes = VALID_SUFFIXES.get(engine.lower())
    if not allowed_suffixes:
        raise ValueError(f"Unsupported engine: {engine}")

    try:
        # Step 1: Reverse DNS Lookup (PTR)
        hostname, _, _ = socket.gethostbyaddr(ip_address)

        # Step 2: Check domain suffix
        if not hostname.lower().endswith(allowed_suffixes):
            return False

        # Step 3: Forward DNS Lookup (A/AAAA)
        resolved_ips = socket.getaddrinfo(hostname, None)
        forward_ips = [item[4][0] for item in resolved_ips]

        # Step 4: Verify original IP is in forward resolution list
        if ip_address in forward_ips:
            return True

    except (socket.herror, socket.gaierror, socket.timeout):
        # DNS query failed or timed out
        return False

    return False

# Example usage:
if __name__ == "__main__":
    test_ips = [
        ("66.249.66.1", "google"),    # Real Googlebot
        ("198.51.100.45", "google"),  # Impersonator
        ("157.55.39.1", "bing")       # Real Bingbot
    ]

    for ip, bot_type in test_ips:
        result = verify_bot_ip(ip, bot_type)
        status = "VERIFIED REAL" if result else "FAKE / SPOOFED"
        print(f"IP: {ip:<15} | Bot: {bot_type:<7} | Result: {status}")

Using @functools.lru_cache ensures that if an IP appears thousands of times in a single log file, your system performs the DNS resolution only once per unique IP address, drastically reducing execution time.

3 Common Myths About Crawler IP Verification

Myth 1: “If an IP belongs to Google Cloud or AWS, it’s a Google or Amazon search crawler.”

This is one of the most frequent mistakes made by systems administrators. Anyone can rent a Virtual Private Server (VPS) on Google Cloud Platform (GCP) or Amazon Web Services (AWS). An attacker running a scraper from a GCP instance will have an IP address owned by Google, but running a reverse DNS check will yield hostnames like 123.45.67.89.bc.googleusercontent.com rather than crawl-xxx.googlebot.com. Never whitelist entire cloud provider IP blocks thinking you are letting search engine crawlers through.

Myth 2: “Search engines never crawl from residential or unannounced IP ranges.”

While search engine crawlers operate primarily from defined, corporate-owned data centers, specialized bots—such as Google’s AdSpyder, Web Light, or localized rendering engines—sometimes utilize secondary networks. However, for main indexing operations (Googlebot-Image, Googlebot-News, Googlebot-Desktop, Googlebot-Smartphone), search engines strictly stick to their documented hostnames and IP lists. If an unannounced IP claims to be standard Googlebot, treat it as unverified.

Myth 3: “Blocking fake crawlers will lower your site traffic.”

Site owners often fear that strict IP filtering will accidentally block legitimate traffic or long-tail crawlers. The reality is that fake crawlers do not index content or refer real users to your website. They consume server capacity, scrape proprietary database content, and obscure true performance metrics. Filtering fake bots sharpens your log accuracy and reduces server load, which actually helps real crawlers access your pages faster.

When auditing crawler access to complex URL structures like paginated categories or deep subdirectories, fake bot traffic can create misleading signals about which pages are actually getting indexed. When pagination cuts crawler access to links explores how to resolve crawl roadblocks on deep structures once you’ve confirmed your traffic is coming from legitimate search bots.

Costs, Risks, and Technical Tradeoffs

Implementing strict IP verification isn’t free—it comes with engineering choices, operational overhead, and occasional edge-case failures.

                    IP VERIFICATION RISK MATRIX
┌───────────────────────────┬───────────────────────────┐
│       FAIL OPEN           │        FAIL CLOSED        │
│                           │                           │
│ • Unverified traffic      │ • Legitimate bots         │
│   passes through          │   get blocked             │
│ • Preserves crawl access  │ • Protects origin server  │
│ • Risk: Waste crawl budget│ • Risk: De-indexing pages │
└───────────────────────────┴───────────────────────────┘

Resource Overhead

Running real-time reverse/forward DNS checks at scale requires dedicated cache infrastructure (like Redis). If your site receives hundreds of requests per second, executing un-cached network calls will bottleneck your application, leading to timeouts.

Maintenance Costs

Search engines update their JSON endpoints periodically. If you write an automated script to fetch these ranges, you must build error-handling logic for cases where Google’s or Bing’s JSON endpoints return a 500 error or malformed JSON. If your automated script silently wipes your firewall database on a failed fetch, you could lock out all real crawlers simultaneously.

The “Fail Open” vs “Fail Closed” Dilemma

When a DNS lookup times out during a bot check, your system must make a decision: * Fail Open: Treat the request as a normal human user (subject to standard rate limits). This prevents accidentally blocking real search bots during network hiccups, but allows sneaky scrapers through temporarily. * Fail Closed: Block or challenge the request immediately. This guarantees maximum security, but risks blocking real Googlebot hits whenever upstream DNS providers experience latency.

In our experience across hundreds of client deployments, failing open with standard rate limits is almost always the right call for search engine optimization. It is far better to allow a few extra scraper hits during a brief DNS timeout than to serve a 403 Forbidden response to Googlebot on a crucial crawling run.

Sometimes, crawler drop-offs occur not because of technical firewall blocks, but because of underlying content access rules or media relations policies. Responding to reporters who demand paid access discusses how editorial barriers affect site access and visibility. Similarly, if you rely on indirect traffic mechanisms, check out using social shares to trigger crawler visits.

What to Do This Week

To get full control over your crawler verification and stop guessing about bot access, follow this action checklist:

1. Pull your last 7 days of server access logs and filter for requests containing “Googlebot”, “Bingbot”, or “YandexBot” in the User-Agent field.
2. Extract a list of unique IP addresses from those filtered entries.
3. Run a reverse DNS lookup script across those IPs to measure how much of your incoming “search engine” traffic is actually impersonated scraper traffic.
4. Update your WAF or CDN rules (e.g., Cloudflare Bot Management or AWS WAF) to replace generic User-Agent rules with managed bot rules or published IP range allowlists.
5. Implement an automated cron job that updates your firewall allowlists from official search engine JSON endpoints every night.

If you want to track how search crawlers process your external links and monitor crawl activity across your entire backlink profile without building custom log parsers, OptiLink automates crawler verification and link discovery in one simple platform.

Keep reading

Playbook

Broken link building, step by step: find, fix, and pitch

Broken link building works when your replacement does the dead link's job. Here is how to find dead links, verify them, match a fix, and pitch honestly.

Sofia Meyer October 9, 20262 min read

Playbook

Competitor backlink analysis that finds real opportunities

Competitor backlink analysis pays off when you sort rivals' links by how they were earned. Use the intersect, a repeatable-link test and a dated action list.

Sofia Meyer September 26, 20262 min read

Playbook

Digital PR: the most durable link building there is

Digital PR earns links from journalists through original data, timely news and expert commentary. Here's the campaign workflow and how to measure it honestly.

Sofia Meyer October 13, 20262 min read

Explainer

Dofollow vs nofollow in 2026: what still passes value

The link attributes changed, Google's treatment changed, and a lot of advice is out of date. Here's what dofollow, nofollow, sponsored and ugc actually do now.

The OptiLink Team October 10, 20262 min read