Auditing raw crawl patterns on old domains

Last Tuesday, a client came to me with a classic “legacy site” headache. They’d spent months pushing new content, but their rankings remained flat. We pulled their server logs and found Googlebot was spending 70% of its time crawling a defunct legacy directory structure from 2012. They were essentially paying for a Ferrari but driving it in first gear. Auditing raw crawl patterns on old domains is the messy, unglamorous work that separates SEOs who chase myths from those who actually move the needle.

Most people skip the raw logs because they are intimidating. They prefer the clean, visual charts of a crawler tool. But a crawler tool tells you what Google can see; your logs tell you what Google actually does. When you have a domain with a decade of history, the “crawl bloat” is real. You have orphaned pages, abandoned parameter-based URLs, and redirects that chain into oblivion. This is how you reclaim your crawl efficiency.

The difference between discovery and refresh crawling

When you look at your log files, you need to categorize every request. Googlebot isn’t just one person wandering around your site. It is a dual-purpose engine. Discovery crawling is when Googlebot hits new URLs to see what is out there. Refresh crawling is when it revisits known pages to check for updates.

On an old domain, the ratio between these two is often completely skewed. If you see thousands of hits on pages that haven’t changed in five years, you have a refresh crawl problem. This usually stems from poor internal link architecture or outdated sitemaps. A healthy site should have a high refresh rate on core content and a controlled, targeted discovery rate on new content. If your logs show massive spikes in hits on old, static pages, Google is wasting its time checking for updates that don’t exist, which keeps them away from your newer, more important pages.

Distinguishing signal from noise in log exports

You will be tempted to look at every single line in your log file. Don’t. It is a massive waste of time. Instead, focus on the user agent strings and the status codes. Use a tool like Screaming Frog Log File Analyzer or a simple grep command if you are comfortable in the terminal.

Look for the “junk” first. If your server is spitting out 404s for URLs that don’t exist, identify if those URLs are still being linked internally. If they are, you aren’t just wasting crawl budget; you are providing a poor user experience for Googlebot. A common mistake is using robots.txt to block these. If the URL is still linked, Googlebot will still try to visit it, see the block, and then keep the URL in the crawl queue anyway.

When auditing, group your logs by path. If you see a specific folder, like /archives/2014/, getting 500 hits a day, that is a clear signal. You don’t need to look at every hit. You just need to realize that the folder itself is a source of leakage. Once you group by path, the “noise” disappears, and the structural failures become obvious.

Mapping the legacy URL graveyard

Every old domain has a graveyard. These are the categories or tags that were abandoned years ago. When you audit these, look for patterns. Is Googlebot hitting old calendar archives? Are there thousands of URLs generated by an old search feature?

We typically see this on sites that have migrated CMS platforms multiple times. The old URL patterns linger in the database or the sitemap files. If you find a pattern, don’t just fix one URL. Fix the template or the sitemap generation logic. This is the only way to scale the cleanup. You can learn more about this in our guide on why most crawl frequency factors advice is outdated for small teams.

When mapping the graveyard, categorize the pages into three buckets: 1. Redirectable: Pages with genuine backlink value that should move to a new home. 2. Prunable: Dead weight with no value that should 404 or 410. 3. Ghost: Pages that exist only because of a CMS bug or a lingering sitemap entry.

Implementation checklist for log cleanup

Follow this sequence to ensure you aren’t breaking anything while clearing out the noise:

1. Request at least 30 days of raw server access logs from your hosting provider (Apache/Nginx format).
2. Filter the logs to show only requests from Googlebot (verify the IP addresses against official Googlebot ranges).
3. Aggregate the requests by path or directory to identify the top 50 most frequently crawled URLs that provide no business value.
4. Use a crawler like Screaming Frog to check if these high-crawl URLs are internally linked in your current theme, footer, or sidebars.
5. Manually remove those internal links or update them to point to relevant, modern pages.
6. Implement 301 redirects for any high-authority legacy pages that Googlebot is still hitting heavily to preserve the backlink juice.
7. Use the URL Inspection tool in Search Console to verify that the “junk” pages are no longer being prioritized in the index.

Handling the redirect chains

One of the most common ways to break a site’s crawlability is to create a “redirect web.” You have a legacy URL, which redirects to a different legacy URL, which finally goes to the homepage. Googlebot hates this, and it eats through your crawl budget like wildfire.

When you audit your raw crawl patterns on old domains, specifically filter for 301 and 302 status codes. If you see a sequence, flatten it. Every redirect adds a round-trip to the server. If your server is slow, that delay adds up. You can read more about how to manage these in how to google crawl budget (step by step).

Monitoring the aftermath of your audit

After you implement these changes, wait. Crawl patterns don’t shift overnight. Googlebot is a creature of habit. It will continue to try and hit those old URLs for a few weeks, even after you remove the links.

Watch the log files over the next two to three months. You should see a slow decay in the number of requests to the junk directories. If the requests persist, check your XML sitemaps. Sometimes, we find that these old pages are still buried in an ancient sitemap that we forgot to update. For more on this, check out our piece on what nobody tells you about google crawl budget.

If you see a sudden spike in 404s after your cleanup, don’t panic. That is expected. It means Googlebot is finally hitting the dead ends you created. As long as you have removed the internal links, these 404s will eventually drop off as Googlebot updates its internal crawl queue.

Avoiding common traps in log analysis

The biggest trap is over-optimizing. You do not need to aim for a perfect crawl. You just need to ensure the most important content—your money pages—are crawled and refreshed regularly. If you block everything, you might inadvertently block pages that Google finds useful for context.

Be careful with noindex tags as well. While they tell Google not to index the page, they don’t always stop Google from crawling it. A noindex page still consumes crawl budget. If you really want to stop the crawl, the combination of internal link removal and a 404 status code (or a specific robots.txt block if you have removed the internal links) is the gold standard.

Another mistake is ignoring your search parameters. If your site uses filters, Googlebot might be creating infinite combinations of them. Check your log files for URL parameters that look like ?color=red&size=large&sort=price. If you see hundreds of variations of these, implement a canonical tag or use the parameter handling features if they are still relevant, but ideally, you want to stop these from being linked in the first place.

Why structural fixes always beat configuration hacks

Newer SEOs often look for a “quick fix” in the robots.txt file. They think that by adding a few lines of disallow, they can save their crawl budget. In reality, robots.txt is the last resort, not the first line of defense. If your site structure is messy, robots.txt will just hide the symptoms while the disease—your bad internal links—continues to spread.

A clean internal linking architecture acts as a roadmap for Googlebot. When you remove the dead ends and the circular redirects, you are effectively guiding the bot toward your most valuable content. This is why we focus so heavily on the audit phase. By understanding how the bot moves, you can build a structure that naturally encourages it to prioritize your core pages without needing to micromanage every line of your robots.txt file.

Remember that Googlebot is constantly learning from your site’s history. If you have years of content that was never pruned, you are dealing with a bot that expects to find “garbage” on your domain. It takes patience to retrain that expectation. Keep the audit process regular and, more importantly, keep the internal architecture clean.

The reality of auditing these domains is that it is never truly “finished.” You fix one layer of the onion, and you find another beneath it. It is a cycle of discovery and maintenance. If you stay consistent with your log reviews, you will eventually find that your crawl budget is being spent on things that actually drive revenue. It is not about tricking the bot; it is about cleaning up the digital debris so the bot can see the value you have built. If you find the manual process of cross-referencing your sitemaps with log data too slow, look into how OptiLink automates the identification of these wasted crawl paths.

Log file analysis can be overwhelming, but start small by targeting the top 10% of your most-crawled, least-valuable pages. Consistent, incremental improvements are better than a single, massive site overhaul that might break your current rankings. Use these tools to guide your decision-making and stay focused on the user experience. You have the power to control how the search engines see your domain, one URL at a time.

Keep reading

Playbook

Broken link building, step by step: find, fix, and pitch

Broken link building works when your replacement does the dead link's job. Here is how to find dead links, verify them, match a fix, and pitch honestly.

Sofia Meyer October 9, 20262 min read

Playbook

Competitor backlink analysis that finds real opportunities

Competitor backlink analysis pays off when you sort rivals' links by how they were earned. Use the intersect, a repeatable-link test and a dated action list.

Sofia Meyer September 26, 20262 min read

Playbook

Digital PR: the most durable link building there is

Digital PR earns links from journalists through original data, timely news and expert commentary. Here's the campaign workflow and how to measure it honestly.

Sofia Meyer October 13, 20262 min read

Explainer

Dofollow vs nofollow in 2026: what still passes value

The link attributes changed, Google's treatment changed, and a lot of advice is out of date. Here's what dofollow, nofollow, sponsored and ugc actually do now.

The OptiLink Team October 10, 20262 min read