By the DenverSEO.io team · 2026-10-10
Log file analysis for SEO means reading your web server's access logs to see which URLs search engine and AI crawlers request, how often, and what status codes they receive. It shows wasted crawling on parameters and redirects, errors that only bots hit, and important pages that crawlers rarely visit. Always verify Googlebot by reverse DNS, because user agents are easy to fake.
Search Console tells you what Google decided. Your access log tells you what Google did. Those are different things, and the gap between them is where many stubborn technical problems hide: a faceted filter that quietly generates thousands of crawlable URLs, a redirect chain that only bots follow, a staging subdomain someone forgot to lock, or a product section crawlers visit once a month while they hammer your tag archive every day.
Log analysis used to be reserved for enterprise sites. It no longer has to be. A local business site on decent hosting can export a month of logs and get useful answers in an afternoon with a spreadsheet or a few command-line tools. This walkthrough covers how we do it.
Step 1: get the right logs
You want the raw access logs from the server or CDN that actually answers requests, not an analytics export. Analytics tools run JavaScript in a browser; most crawlers never execute that script, so they are invisible there.
- Shared or managed hosting: look for “raw access logs” in the control panel. Many hosts keep only a few days by default, so turn on log archiving before you need it.
- Your own server: Nginx and Apache write access logs by default, commonly in the “combined” format.
- Behind a CDN: if a CDN serves cached pages, the origin server never sees those hits. Pull logs from the CDN as well, or you will undercount crawling of cached pages.
Aim for at least 30 days. Crawl patterns are lumpy, and a single week can mislead you.
Step 2: know what a log line contains
A combined-format line looks roughly like this:
66.249.66.1 - - [08/Oct/2026:14:02:11 -0600] "GET /services/drain-cleaning/ HTTP/1.1" 200 18342 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"The fields that matter for SEO are the client IP, the timestamp, the requested path, the status code, the response size and the user agent. With those six columns you can answer nearly every question below.
Step 3: verify the bots before you trust them
The user agent string is self-reported. Scrapers routinely claim to be Googlebot to get past firewalls, so a raw count of “Googlebot” lines can be badly inflated. Google documents two ways to check:
- DNS check, for spot checks. Run a reverse DNS lookup on the IP. A genuine Google crawler resolves to a hostname under
googlebot.com,google.comorgoogleusercontent.com. Then run a forward lookup on that hostname and confirm it returns the same IP. Both steps are required; reverse DNS alone can be spoofed. - IP range match, for bulk checks. Google publishes its crawler IP ranges as JSON files, split by crawler type. Matching every IP in your log against those ranges is faster than thousands of DNS lookups.
Bing documents a similar reverse DNS check, with genuine Bingbot hosts under search.msn.com. Major AI crawlers, such as OpenAI's, publish their user agents and IP ranges too. Treat any crawler that will not verify as unknown traffic, not as a search engine.
A quick first filter on a Linux machine:
grep -i "googlebot" access.log | awk '{print $1}' | sort | uniq -c | sort -rn | headThat lists the IPs claiming to be Googlebot, most active first. Verify the top ones, and drop the rest from your Googlebot analysis until they pass.
Step 4: ask five questions
1. Where is crawl activity actually going?
Group verified bot requests by site section (the first folder in the path) and by URL type: pages, images, scripts, parameters, redirects. On healthy small sites, most crawler requests land on real pages and their assets. When a large share goes to URLs with query strings, calendar archives, internal search results or tracking parameters, crawlers are spending effort on pages you never wanted indexed.
2. What status codes do bots receive?
Count status codes for verified crawlers separately from humans. Watch for:
- Lots of 301s and 302s: internal links or sitemaps still pointing at old URLs. Every redirect hop is a wasted request. Update the links to the final destination.
- 404s on URLs you never published: often old spam, a broken template or a script building malformed links. Find the referring source if you can.
- 5xx errors or timeouts: the most urgent. Google slows crawling when a server responds with errors or slowly, and pages that keep failing can drop out of the index.
- 200 responses on pages that should be gone: a “not found” message served with a 200 status is a soft 404, which wastes crawling and confuses indexing.
3. Are important pages being crawled at all?
Take your list of money pages, the services, locations and products that make you money, and check when each was last requested by verified Googlebot. A page Google has not requested in weeks is usually poorly linked internally or buried deep in the site. Our internal link graph tool helps find those orphan and deep pages.
4. Do bots see something users do not?
Compare response sizes for the same URL between bots and browsers. A page that returns 4 KB to Googlebot and 60 KB to a browser may be serving an error, a blocked script or a consent wall to crawlers. On JavaScript-heavy sites, this is often the first sign that content depends on rendering that crawlers do not complete. See our notes on JavaScript SEO.
5. Which AI crawlers visit, and what do they take?
Filter for known AI crawler user agents and verify them against their published ranges. Many AI crawlers fetch raw HTML only and do not run JavaScript, which we cover in what AI crawlers actually read. If they request your pages but receive thin HTML, your content is not reaching them, whatever your robots.txt allows.
Step 5: turn findings into fixes
A log review is only worth doing if it ends in a short, ranked fix list. The common ones:
- Fix server errors first. Anything returning 5xx to crawlers, or responding very slowly, comes before every other task.
- Update internal links that redirect. Point navigation, footers, sitemaps and canonical tags at final URLs.
- Close crawl traps. Stop linking to infinite filter combinations, internal search results and calendar pages. Use robots.txt for URL patterns that should never be crawled, and remember robots.txt prevents crawling, not indexing.
- Strengthen links to neglected money pages. Link to them from pages that crawlers already visit often, such as the homepage and top service hubs.
- Return real 404 or 410 codes for pages that are truly gone, and keep your XML sitemap limited to live, canonical, indexable URLs.
Does crawl budget matter for a small site?
Usually not in the way people fear. Google's own crawl budget guide is written for very large sites, roughly a million or more unique pages, or ten thousand or more pages that change daily, and for sites with many URLs stuck as “Discovered, currently not indexed.” For a typical local business site, Google says keeping the sitemap current and checking index coverage is enough.
Log analysis is still worth it for small sites, just for different reasons: catching errors only bots see, confirming that new pages get found, spotting fake bots, and checking whether AI crawlers can read your content. If pages are being crawled but not indexed, that is a quality and duplication question rather than a crawl budget one, which we cover in crawled, currently not indexed.
A note for Denver businesses
Most of the Denver sites we audit are under a few thousand pages, built on WordPress, Shopify or a custom stack. The problems logs reveal on those sites are rarely exotic. They are leftover redirects from a redesign, plugin-generated archives, and staging copies that were never blocked. Each one is cheap to fix once you can see it. Log review is part of every SEO audit we run, and we will happily tell you if your logs show nothing worth paying us for.