Search Converts LLC BBB Business Review
1-800-481-8638The Commit
Search Converts LLC BBB Business Review
Home / Blog / Noindex vs robots.txt
Blog · 11 min read

Noindex vs robots.txt: block the crawl or block the index

Pick the wrong one and the page you meant to hide stays in Google.

By the DenverSEO.io team · 2026-11-17

Robots.txt controls crawling. Noindex controls indexing. A Disallow rule tells crawlers not to fetch a URL, but Google can still list that URL if other pages link to it. A noindex meta tag or X-Robots-Tag header tells Google to drop the page from results, but only works if Google is allowed to crawl the page and read it. To keep a page out of Google, use noindex and leave it crawlable. Use robots.txt to save crawling on URLs that do not matter.

This is one of the most common mix-ups we find in technical audits, and it is easy to see why. Both tools live in the same mental drawer labeled “keep Google away from this.” They do different jobs, they run at different stages of the pipeline, and when you stack them on the same URL they cancel each other out. The result is the classic Search Console warning: Indexed, though blocked by robots.txt, sitting next to a page you were sure you had hidden.

This guide is written for developers and for the business owners who have to sign off on what developers do. It covers what each directive actually controls, a decision table for the situations we see on real sites, the server code for Apache, Nginx and static hosts, and the three terminal commands that prove your setup works before you trust it.

What does robots.txt actually control?

Robots.txt is a plain text file at the root of a host, for example https://www.example.com/robots.txt. It is defined by the Robots Exclusion Protocol, published by the IETF as RFC 9309 in 2022. Well-behaved crawlers fetch it first and follow the rules for their user agent. Those rules govern one thing: which URL paths the crawler may request.

That is all. Robots.txt does not say “do not show this in results.” Google's documentation is direct about the gap: if a page is blocked by robots.txt, the page can still appear in search results, for example when other pages link to it. In that case Google knows the URL exists and knows the anchor text people use to point at it, but it has never read the page. You end up with a result that has no description, or one that says no information is available for the page.

A few details that trip people up:

  • Rules are per host and per protocol. https://example.com/robots.txt does not govern https://staging.example.com/ or https://shop.example.com/. Every subdomain needs its own file.
  • Paths are case-sensitive. Disallow: /Private/ does not block /private/.
  • The most specific rule wins. Under RFC 9309, when an Allow and a Disallow both match, the longer (more specific) path takes precedence.
  • A user agent group replaces the default, it does not add to it. If you write a User-agent: Googlebot group, Googlebot follows that group only and ignores everything under User-agent: *.
  • Blocking CSS and JavaScript is still a mistake. Google renders pages. If it cannot fetch the files that build your layout, it may misjudge the page.

What does noindex actually control?

Noindex is an indexing directive. It tells a search engine that has already fetched the page: do not include this URL in results. You can deliver it two ways, and Google says they have the same effect:

<!-- In the <head> of an HTML page -->
<meta name="robots" content="noindex">

<!-- Or target Google only -->
<meta name="googlebot" content="noindex">
HTTP/2 200
content-type: application/pdf
x-robots-tag: noindex

The meta tag only works inside HTML. The X-Robots-Tag response header works on anything the server sends, which is why it is the right tool for PDFs, images, video files and other non-HTML resources. It is also handy when you want to apply a rule to a whole directory or subdomain from server config instead of editing templates.

The catch is in the order of operations. A crawler has to request the URL to see either form of noindex. If robots.txt blocks the request, the crawler never sees the directive, so the directive does nothing.

Why does using Disallow and noindex together backfire?

Because the first one hides the second. Picture the sequence from Google's side:

  1. Googlebot finds a link to /thank-you/ on your contact page.
  2. It checks robots.txt and sees Disallow: /thank-you/, so it does not request the page.
  3. Your <meta name="robots" content="noindex"> sits unread in the HTML.
  4. Google still knows the URL exists, so it can still index the bare URL from link signals alone.

The fix is counterintuitive: to get a page out of the index, you usually have to remove the robots.txt block, let Google crawl the page, and let it read the noindex. Once Google has processed the noindex and dropped the URL, keep the page crawlable. If you re-block it, Google stops seeing the directive again.

Which one should you use? A decision table

Here is how we decide on the sites we build and audit. The right answer depends on whether the URL should exist for users, whether it has any search value and whether crawling it wastes server time.

SituationUseWhy
Form thank-you or confirmation pagesnoindex, crawlableUsers need them; searchers do not. Noindex keeps them out of results and keeps conversion tracking clean.
Staging or dev sitePassword protection (HTTP auth), plus noindex header as a backupRobots.txt on staging is the leak we see most. Auth stops everyone, including crawlers that ignore robots.txt.
PDF price sheets, spec sheets, old brochuresX-Robots-Tag: noindexMeta tags cannot live in a PDF. The header is the only clean option.
Internal site search resultsnoindex first; consider Disallow laterSearch result pages are thin and infinite. Noindex removes existing ones; a Disallow can follow once they are gone, to save crawling.
Cart, checkout, account and admin pathsDisallowNo search value, often no public links, and crawling them is pure waste.
Sort and filter parameters (?sort=, ?color=)Canonical to the clean URL; Disallow only for truly endless combinationsCanonicals consolidate signals. Disallow is for crawl traps, not for duplicates you want merged.
Near-duplicate pages you want mergedrel="canonical" or a 301Noindex discards a page's signals; canonical and redirects pass them to the version you keep.
Content you want gone from the webDelete it (404 or 410)If the page should not exist, do not keep serving it with a directive attached.

Notice how little of this table says “robots.txt.” On a typical service business site of a few hundred pages, crawl volume is rarely the problem. Getting the right pages indexed, and the wrong ones out, almost always is.

A worked example: the staging site that showed up in Google

This is a composite of a pattern we see a few times a year, with the details changed. A Denver home services company launches a redesign. During the build, the developer works on staging.example.com and protects it with this robots.txt:

# staging.example.com/robots.txt
User-agent: *
Disallow: /

Someone pastes a staging link into a supplier's forum post. Months later, a search for the company name shows two results: the real home page, and a staging URL with no description. Search Console for the staging host, once someone verifies it, reports the URL as Indexed, though blocked by robots.txt. The robots.txt rule did exactly what it promised. It stopped crawling. It never promised to stop indexing.

The fix has three parts. First, put real access control on staging so nothing outside the team can load it. Here is the Nginx version:

server {
    listen 443 ssl;
    server_name staging.example.com;

    # 1. Nobody gets in without a password, crawlers included
    auth_basic           "Staging";
    auth_basic_user_file /etc/nginx/.htpasswd;

    # 2. Belt and suspenders: every response says noindex
    add_header X-Robots-Tag "noindex, nofollow" always;

    root /var/www/staging;
}

The Apache equivalent goes in the staging virtual host or its .htaccess:

AuthType Basic
AuthName "Staging"
AuthUserFile /etc/apache2/.htpasswd
Require valid-user

Header always set X-Robots-Tag "noindex, nofollow"

Second, delete the Disallow: / line on staging, or replace the whole file with an empty User-agent: * group. With authentication in place, crawlers that request a page get a 401, which tells them nothing is publicly there. Third, use the Removals tool in Search Console for a fast temporary hide while Google catches up. The tool is a short-term measure; the 401 response is what keeps the URL out for good.

The step most teams skip is the launch-day version of this problem: the new site goes live with the staging noindex header or Disallow: / still in place. That one can wipe a site out of results. It is item one on our website migration SEO checklist for a reason.

How do you noindex PDFs and other files?

With the header, set at the server. In Apache:

<FilesMatch "\.pdf$">
    Header set X-Robots-Tag "noindex"
</FilesMatch>

In Nginx, inside the relevant server block:

location ~* \.pdf$ {
    add_header X-Robots-Tag "noindex" always;
}

On Netlify, a _headers file in the publish directory does the same job for a folder:

/files/*
  X-Robots-Tag: noindex

Think before you apply this to every PDF. A well-written spec sheet or buyer's guide can rank on its own and pull in searchers your HTML pages miss. Noindex the stale price sheets, not the useful documents.

What if your host will not let you set headers?

Some static hosts, GitHub Pages among them, do not let you add custom response headers. That leaves you with three options:

  1. For HTML pages, the meta robots tag works everywhere. That covers thank-you pages, private landing pages and anything else built as HTML.
  2. For files, move documents you do not want indexed off the public site, or serve them through a CDN layer that can add headers, such as a Cloudflare transform rule in front of the host.
  3. For anything sensitive, do not publish it on a static host at all. A static site has no access control, and an unlisted URL is not a private one.

How do you prove it works? Three commands

Do not trust a plugin's settings screen or a template variable. Check what the server actually returns. These run on any Mac or Linux terminal, and in Windows PowerShell with curl.exe:

# 1. Does the file send the header? (-I = headers only)
curl -sI https://www.example.com/files/2019-price-sheet.pdf | grep -i x-robots-tag

# 2. Does the page carry the meta tag in the raw HTML?
curl -s https://www.example.com/thank-you/ | grep -io '<meta[^>]*robots[^>]*>'

# 3. What does robots.txt say right now, on this exact host?
curl -s https://www.example.com/robots.txt

If command 2 comes back empty but you see the tag in your browser's inspector, the tag is being injected by JavaScript. Google can often still pick it up after rendering, but the first HTML response is the safer place for it, and the same goes for canonicals. Our JavaScript SEO page covers why.

Then confirm from Google's side with the URL Inspection tool in Search Console. It shows whether indexing is allowed, whether the page was blocked by robots.txt and which directive Google detected. For robots.txt itself, the robots.txt report in Search Console shows the version Google last fetched for each host.

Does robots.txt or noindex affect AI crawlers?

Partly, and differently for each company. Robots.txt is the main lever most AI crawlers document. OpenAI, for example, publishes separate user agents for search (OAI-SearchBot) and for model training (GPTBot), each controllable in robots.txt. Google offers Google-Extended, a robots.txt token that governs use of your content for Gemini models and does not change how your site appears in Google Search.

Noindex is a search-index directive. Do not assume every AI system honors it the way Google's index does, and do not assume blocking a training crawler keeps you out of AI answers that pull from live search. If you are deciding which bots to allow, our free AI crawler checker reads your robots.txt and shows which of the major bots can reach any path.

The mistakes we fix most often

  • Disallow plus noindex on the same URL. Covered above. Pick noindex and leave it crawlable.
  • A staging noindex shipped to production. Usually a WordPress “discourage search engines” checkbox, or an environment variable that defaulted the wrong way on deploy.
  • Noindex on paginated category pages. Page two of a product or blog category is how crawlers reach older items. Noindexing it can strand them.
  • Noindex combined with a canonical pointing elsewhere. One says “drop me,” the other says “merge me into that page.” Choose one. If you want the signals consolidated, use the canonical alone. Our post on Google choosing a different canonical goes deeper.
  • Using nofollow as if it were noindex. Nofollow is about links on the page, not the page itself.
  • Removing a page by blocking it. If the page should be gone, return 404 or 410, or 301 it to its replacement. Our guide to setting up 301 redirects covers the server code.

How long until Google drops a noindexed page?

It drops the page the next time it crawls the URL and processes the directive. For a page Google visits often, that can be days. For a deep page it rarely visits, it can be weeks or longer. You can speed it up by requesting indexing in URL Inspection, which prompts a recrawl, and by keeping the page linked internally until it is gone so crawlers keep finding it. If the noindexed pages are stuck in a different state, our breakdown of crawled, currently not indexed explains what the other Search Console statuses mean.

Where this fits in a technical audit

Indexing controls are one of the first things we inventory in a technical SEO audit: every robots.txt on every host, every URL returning noindex, every header rule at the server and CDN, compared against the list of pages that should be earning traffic. On most sites the fixes are small. They are also some of the highest-value lines of code on the site, because a single wrong directive can silence an entire section.

If you would rather have an outside team check before you change anything, Eye To Ad Media offers a free website SEO review that covers indexing along with the rest of the basics. If you are comfortable in a terminal, the three commands above will tell you most of what you need in under a minute.

Sources

FAQ

Quick answers

What is the difference between noindex and robots.txt?

Robots.txt controls crawling: it tells crawlers which URL paths they may request. Noindex controls indexing: it tells a search engine that has fetched a page not to show it in results. A URL blocked in robots.txt can still be indexed from links, and a noindex directive only works if the page can be crawled.

Can I use robots.txt Disallow and noindex on the same page?

You can, but it backfires. The Disallow rule stops Google from fetching the page, so it never reads the noindex. To remove a page from Google, allow crawling and use noindex alone.

Why does Search Console say indexed, though blocked by robots.txt?

Google found links to the URL and indexed it without being allowed to crawl it. Robots.txt stopped the crawl but not the indexing. Remove the Disallow rule, add noindex or real access control, and let Google recrawl the page.

How do I noindex a PDF?

Send an X-Robots-Tag: noindex HTTP response header for the file. In Apache use a FilesMatch block with Header set, in Nginx use add_header inside a location block, and on Netlify use a _headers file. Meta tags cannot be placed inside a PDF.

What is the best way to keep a staging site out of Google?

Password-protect it with HTTP authentication so crawlers get a 401 response, and add an X-Robots-Tag noindex header as a backup. A robots.txt Disallow alone stops crawling but can still let staging URLs be indexed from links.

How long does it take Google to drop a noindexed page?

Google drops it the next time it crawls the URL and reads the directive. That can be days for frequently crawled pages and weeks or longer for deep ones. Requesting indexing in the URL Inspection tool prompts a recrawl.

60-second project scope

Tell us what you are working on. Get a straight answer.

Four taps and your contact info. An engineer (not a salesperson) reviews it and tells you what we would do, what it costs and whether you even need us.

What are you working on?

What is it built on?

What keeps you up at night?

When does it need to happen?

Where should we send your answer?

Search Converts LLC BBB Business ReviewBBB Accredited Business. A real person reads every request.

Got it. Here is what happens next.

  1. An engineer reviews your answers today.
  2. You get a call or email with what we would fix first, and why.
  3. If you do not need us, we will tell you that too.
Can't wait? Call 1-800-481-8638

Want an engineer to look at yours?

Fifteen minutes on the phone usually tells us whether we can help, and how much.

☎ Call now