How to Unindex Pages from Search Engines

How to Keep Pages Out of Search Engines with noindex

Not every page on a site belongs in search results. A search engine will happily list anything it can reach, including pages that only make sense after a visitor has done something first, or pages that were never meant for the public at all.

Typical candidates are:

• Thank-you and download pages - if the page that hands over a file can be found in Google, visitors skip the form, the payment or the donation that was meant to come first.
• Admin, login and test pages - nothing useful to a searcher, and they advertise where your back end lives.
• Duplicate content - printer-friendly versions, filtered or sorted lists, and internal search results that repeat what is already on other pages.
• Files - PDFs, archives and documents that should only be reached through a page.

Crawling is Not Indexing

Before choosing a method it helps to separate two things search engines do:

• Crawling - fetching the page to read it.
• Indexing - storing it so it can appear in search results.

robots.txt controls crawling. noindex controls indexing. They are not interchangeable, and using the wrong one is the most common reason a page refuses to leave the results.

Method 1 - The Robots Meta Tag

For an HTML or PHP page, add a robots meta tag inside <head>:

<meta name='robots' content='noindex'>

The content value can combine rules, separated by commas:

• noindex - do not show this page in results. Links on it are still followed.
• nofollow - do not follow the links on this page. The page itself can still be listed.
• noindex, nofollow - hide the page and ignore its links. Good for a thank-you page that links straight to the file being given away.

To target only Google, use name='googlebot' instead of name='robots'.

If your pages share one head file, the cleanest approach is a flag that each page sets before including it, so the tag is written in one place only:

// in the page
$No_Index = true;
require_once $Lib.'head.php';

// in head.php
if (isset ($No_Index))
   {$Robots = 'noindex';}
else
   {$Robots = 'all, index';}
...
<meta name='robots' content='<?php echo $Robots;?>'>

Method 2 - The X-Robots-Tag Header

A PDF or a RAR file has no <head> to put a meta tag in. For these, send the same rule as an HTTP response header instead. Search engines treat X-Robots-Tag exactly like the meta tag.

With .htaccess (Apache)

Place this in the .htaccess of the folder holding the files. It needs Apache's mod_headers module enabled:

<IfModule mod_headers.c>
  <FilesMatch "\.(pdf|rar|zip|docx?)$">
    Header set X-Robots-Tag "noindex, nofollow"
  </FilesMatch>
</IfModule>

Leave out the FilesMatch lines to apply the header to everything in the folder.

With PHP

If a PHP script delivers the file, send the header before any output:

header ('X-Robots-Tag: noindex, nofollow');

Method 3 - robots.txt, and Why It Is Not Enough

A Disallow line in robots.txt asks crawlers not to fetch a URL:

User-agent: *
Disallow: /test/

That stops the page being read, but not being listed. If other pages link to the URL, Google can still show it in results - usually with no description, because it was never allowed to read the page.

Worse, a blocked page can never be seen carrying a noindex tag. Google states plainly that if a page is blocked by robots.txt, the crawler will never see the noindex rule and the page can still appear in results. So:

• To remove a page from results, use noindex and leave the page crawlable.
• Use robots.txt to save crawl effort on folders that have no business being fetched - images, libraries, scratch areas - not to hide pages.
• If you add noindex to a page that is already disallowed, remove the Disallow line until the page has dropped out.

Pages That Should Not Exist at All

noindex is for pages that stay online but stay out of search. If the page is deleted, return 410 Gone (or 404) instead - search engines drop it on their next visit. If it is private, put it behind a password - a crawler that cannot log in cannot index it. Both are covered in the related articles below.

Speeding Up Removal

Nothing changes until the search engine visits the page again, which can take days or weeks for a page that rarely changes. To hurry it along:

• Google Search Console - URL Inspection - inspect the URL and request indexing, so Google re-reads the page and sees the new rule.
• Google Search Console - Removals - hides a URL from results quickly, but only for about six months. It is a stop-gap while noindex, 404/410 or a password does the permanent job.
• Bing Webmaster Tools - Block URLs - the equivalent temporary block for Bing. Bing honours both the robots meta tag and X-Robots-Tag.

Search engines other than Google and Bing may treat these rules differently, so a page can linger elsewhere for longer.

Testing

• For a page, view the source and confirm the robots meta tag is in <head> with the value you expect.
• For a file, check the response header:

  curl -I https://yoursite.com/files/manual.pdf
  
  X-Robots-Tag: noindex, nofollow
  

• In Search Console, URL Inspection reports whether indexing is allowed and whether the page is blocked by robots.txt.

Summary

• Robots meta tag - HTML and PHP pages that stay online but stay out of search
• X-Robots-Tag header - PDFs, archives and other files, or a whole folder at once
• robots.txt - stops crawling, not listing; never combine it with noindex on the same URL
• 410 / 404 - pages that have been deleted
• Password protection - pages that are private
• Removals / Block URLs - a temporary fast fix while one of the above takes effect

Useful References

• Google - Block Search Indexing with noindex
• Google - Removals Tool
• Bing - Robots Meta Tags Bing Supports
• Apache mod_headers Documentation

If you found this useful, then please consider making a donation.

Please donate if helpful