By the DenverSEO.io team · 2026-11-17
Robots.txt controls crawling. Noindex controls indexing. A Disallow rule tells crawlers not to fetch a URL, but Google can still list that URL if other pages link to it. A noindex meta tag or X-Robots-Tag header tells Google to drop the page from results, but only works if Google is allowed to crawl the page and read it. To keep a page out of Google, use noindex and leave it crawlable. Use robots.txt to save crawling on URLs that do not matter.
This is one of the most common mix-ups we find in technical audits, and it is easy to see why. Both tools live in the same mental drawer labeled “keep Google away from this.” They do different jobs, they run at different stages of the pipeline, and when you stack them on the same URL they cancel each other out. The result is the classic Search Console warning: Indexed, though blocked by robots.txt, sitting next to a page you were sure you had hidden.
This guide is written for developers and for the business owners who have to sign off on what developers do. It covers what each directive actually controls, a decision table for the situations we see on real sites, the server code for Apache, Nginx and static hosts, and the three terminal commands that prove your setup works before you trust it.
What does robots.txt actually control?
Robots.txt is a plain text file at the root of a host, for example https://www.example.com/robots.txt. It is defined by the Robots Exclusion Protocol, published by the IETF as RFC 9309 in 2022. Well-behaved crawlers fetch it first and follow the rules for their user agent. Those rules govern one thing: which URL paths the crawler may request.
That is all. Robots.txt does not say “do not show this in results.” Google's documentation is direct about the gap: if a page is blocked by robots.txt, the page can still appear in search results, for example when other pages link to it. In that case Google knows the URL exists and knows the anchor text people use to point at it, but it has never read the page. You end up with a result that has no description, or one that says no information is available for the page.
A few details that trip people up:
- Rules are per host and per protocol.
https://example.com/robots.txtdoes not governhttps://staging.example.com/orhttps://shop.example.com/. Every subdomain needs its own file. - Paths are case-sensitive.
Disallow: /Private/does not block/private/. - The most specific rule wins. Under RFC 9309, when an
Allowand aDisallowboth match, the longer (more specific) path takes precedence. - A user agent group replaces the default, it does not add to it. If you write a
User-agent: Googlebotgroup, Googlebot follows that group only and ignores everything underUser-agent: *. - Blocking CSS and JavaScript is still a mistake. Google renders pages. If it cannot fetch the files that build your layout, it may misjudge the page.
What does noindex actually control?
Noindex is an indexing directive. It tells a search engine that has already fetched the page: do not include this URL in results. You can deliver it two ways, and Google says they have the same effect:
<!-- In the <head> of an HTML page -->
<meta name="robots" content="noindex">
<!-- Or target Google only -->
<meta name="googlebot" content="noindex">
HTTP/2 200
content-type: application/pdf
x-robots-tag: noindex
The meta tag only works inside HTML. The X-Robots-Tag response header works on anything the server sends, which is why it is the right tool for PDFs, images, video files and other non-HTML resources. It is also handy when you want to apply a rule to a whole directory or subdomain from server config instead of editing templates.
The catch is in the order of operations. A crawler has to request the URL to see either form of noindex. If robots.txt blocks the request, the crawler never sees the directive, so the directive does nothing.
Why does using Disallow and noindex together backfire?
Because the first one hides the second. Picture the sequence from Google's side:
- Googlebot finds a link to
/thank-you/on your contact page. - It checks robots.txt and sees
Disallow: /thank-you/, so it does not request the page. - Your
<meta name="robots" content="noindex">sits unread in the HTML. - Google still knows the URL exists, so it can still index the bare URL from link signals alone.
The fix is counterintuitive: to get a page out of the index, you usually have to remove the robots.txt block, let Google crawl the page, and let it read the noindex. Once Google has processed the noindex and dropped the URL, keep the page crawlable. If you re-block it, Google stops seeing the directive again.
Which one should you use? A decision table
Here is how we decide on the sites we build and audit. The right answer depends on whether the URL should exist for users, whether it has any search value and whether crawling it wastes server time.
| Situation | Use | Why |
|---|---|---|
| Form thank-you or confirmation pages | noindex, crawlable | Users need them; searchers do not. Noindex keeps them out of results and keeps conversion tracking clean. |
| Staging or dev site | Password protection (HTTP auth), plus noindex header as a backup | Robots.txt on staging is the leak we see most. Auth stops everyone, including crawlers that ignore robots.txt. |
| PDF price sheets, spec sheets, old brochures | X-Robots-Tag: noindex | Meta tags cannot live in a PDF. The header is the only clean option. |
| Internal site search results | noindex first; consider Disallow later | Search result pages are thin and infinite. Noindex removes existing ones; a Disallow can follow once they are gone, to save crawling. |
| Cart, checkout, account and admin paths | Disallow | No search value, often no public links, and crawling them is pure waste. |
Sort and filter parameters (?sort=, ?color=) | Canonical to the clean URL; Disallow only for truly endless combinations | Canonicals consolidate signals. Disallow is for crawl traps, not for duplicates you want merged. |
| Near-duplicate pages you want merged | rel="canonical" or a 301 | Noindex discards a page's signals; canonical and redirects pass them to the version you keep. |
| Content you want gone from the web | Delete it (404 or 410) | If the page should not exist, do not keep serving it with a directive attached. |
Notice how little of this table says “robots.txt.” On a typical service business site of a few hundred pages, crawl volume is rarely the problem. Getting the right pages indexed, and the wrong ones out, almost always is.
A worked example: the staging site that showed up in Google
This is a composite of a pattern we see a few times a year, with the details changed. A Denver home services company launches a redesign. During the build, the developer works on staging.example.com and protects it with this robots.txt:
# staging.example.com/robots.txt
User-agent: *
Disallow: /
Someone pastes a staging link into a supplier's forum post. Months later, a search for the company name shows two results: the real home page, and a staging URL with no description. Search Console for the staging host, once someone verifies it, reports the URL as Indexed, though blocked by robots.txt. The robots.txt rule did exactly what it promised. It stopped crawling. It never promised to stop indexing.
The fix has three parts. First, put real access control on staging so nothing outside the team can load it. Here is the Nginx version:
server {
listen 443 ssl;
server_name staging.example.com;
# 1. Nobody gets in without a password, crawlers included
auth_basic "Staging";
auth_basic_user_file /etc/nginx/.htpasswd;
# 2. Belt and suspenders: every response says noindex
add_header X-Robots-Tag "noindex, nofollow" always;
root /var/www/staging;
}
The Apache equivalent goes in the staging virtual host or its .htaccess:
AuthType Basic
AuthName "Staging"
AuthUserFile /etc/apache2/.htpasswd
Require valid-user
Header always set X-Robots-Tag "noindex, nofollow"
Second, delete the Disallow: / line on staging, or replace the whole file with an empty User-agent: * group. With authentication in place, crawlers that request a page get a 401, which tells them nothing is publicly there. Third, use the Removals tool in Search Console for a fast temporary hide while Google catches up. The tool is a short-term measure; the 401 response is what keeps the URL out for good.
The step most teams skip is the launch-day version of this problem: the new site goes live with the staging noindex header or Disallow: / still in place. That one can wipe a site out of results. It is item one on our website migration SEO checklist for a reason.
How do you noindex PDFs and other files?
With the header, set at the server. In Apache:
<FilesMatch "\.pdf$">
Header set X-Robots-Tag "noindex"
</FilesMatch>
In Nginx, inside the relevant server block:
location ~* \.pdf$ {
add_header X-Robots-Tag "noindex" always;
}
On Netlify, a _headers file in the publish directory does the same job for a folder:
/files/*
X-Robots-Tag: noindex
Think before you apply this to every PDF. A well-written spec sheet or buyer's guide can rank on its own and pull in searchers your HTML pages miss. Noindex the stale price sheets, not the useful documents.
What if your host will not let you set headers?
Some static hosts, GitHub Pages among them, do not let you add custom response headers. That leaves you with three options:
- For HTML pages, the meta robots tag works everywhere. That covers thank-you pages, private landing pages and anything else built as HTML.
- For files, move documents you do not want indexed off the public site, or serve them through a CDN layer that can add headers, such as a Cloudflare transform rule in front of the host.
- For anything sensitive, do not publish it on a static host at all. A static site has no access control, and an unlisted URL is not a private one.
How do you prove it works? Three commands
Do not trust a plugin's settings screen or a template variable. Check what the server actually returns. These run on any Mac or Linux terminal, and in Windows PowerShell with curl.exe:
# 1. Does the file send the header? (-I = headers only)
curl -sI https://www.example.com/files/2019-price-sheet.pdf | grep -i x-robots-tag
# 2. Does the page carry the meta tag in the raw HTML?
curl -s https://www.example.com/thank-you/ | grep -io '<meta[^>]*robots[^>]*>'
# 3. What does robots.txt say right now, on this exact host?
curl -s https://www.example.com/robots.txt
If command 2 comes back empty but you see the tag in your browser's inspector, the tag is being injected by JavaScript. Google can often still pick it up after rendering, but the first HTML response is the safer place for it, and the same goes for canonicals. Our JavaScript SEO page covers why.
Then confirm from Google's side with the URL Inspection tool in Search Console. It shows whether indexing is allowed, whether the page was blocked by robots.txt and which directive Google detected. For robots.txt itself, the robots.txt report in Search Console shows the version Google last fetched for each host.
Does robots.txt or noindex affect AI crawlers?
Partly, and differently for each company. Robots.txt is the main lever most AI crawlers document. OpenAI, for example, publishes separate user agents for search (OAI-SearchBot) and for model training (GPTBot), each controllable in robots.txt. Google offers Google-Extended, a robots.txt token that governs use of your content for Gemini models and does not change how your site appears in Google Search.
Noindex is a search-index directive. Do not assume every AI system honors it the way Google's index does, and do not assume blocking a training crawler keeps you out of AI answers that pull from live search. If you are deciding which bots to allow, our free AI crawler checker reads your robots.txt and shows which of the major bots can reach any path.
The mistakes we fix most often
- Disallow plus noindex on the same URL. Covered above. Pick noindex and leave it crawlable.
- A staging noindex shipped to production. Usually a WordPress “discourage search engines” checkbox, or an environment variable that defaulted the wrong way on deploy.
- Noindex on paginated category pages. Page two of a product or blog category is how crawlers reach older items. Noindexing it can strand them.
- Noindex combined with a canonical pointing elsewhere. One says “drop me,” the other says “merge me into that page.” Choose one. If you want the signals consolidated, use the canonical alone. Our post on Google choosing a different canonical goes deeper.
- Using nofollow as if it were noindex. Nofollow is about links on the page, not the page itself.
- Removing a page by blocking it. If the page should be gone, return 404 or 410, or 301 it to its replacement. Our guide to setting up 301 redirects covers the server code.
How long until Google drops a noindexed page?
It drops the page the next time it crawls the URL and processes the directive. For a page Google visits often, that can be days. For a deep page it rarely visits, it can be weeks or longer. You can speed it up by requesting indexing in URL Inspection, which prompts a recrawl, and by keeping the page linked internally until it is gone so crawlers keep finding it. If the noindexed pages are stuck in a different state, our breakdown of crawled, currently not indexed explains what the other Search Console statuses mean.
Where this fits in a technical audit
Indexing controls are one of the first things we inventory in a technical SEO audit: every robots.txt on every host, every URL returning noindex, every header rule at the server and CDN, compared against the list of pages that should be earning traffic. On most sites the fixes are small. They are also some of the highest-value lines of code on the site, because a single wrong directive can silence an entire section.
If you would rather have an outside team check before you change anything, Eye To Ad Media offers a free website SEO review that covers indexing along with the rest of the basics. If you are comfortable in a terminal, the three commands above will tell you most of what you need in under a minute.
