By: , December 11, 2007 | Starting Tips For Webmasters


Google introduced the new X-Robots-Tag directive in 2007 to allow webmasters to control access to non-web page documents, such as Adobe PDF files, video and audio files.

The X-Robots-Tag is included in the HTTP header of a document. The HTTP header is the initial reply of a server to a query. It contains information about the document that follows, including content type, creation date, character set, encodings, and so forth...

The new X-Robots-Tag allows webmasters to add information about search engine indexing to the HTTP header.


Which commands are supported by the X-Robots-Tag?


  • X-Robots-Tag: NOINDEX (Use this tag if you don't want to show the URL in Google's search results)
  • X-Robots-Tag: NOARCHIVE (Use this tag if you don't want to see a Cache link in the search result pages for the document)
  • X-Robots-Tag: NOSNIPPET (Use this tag if you don't want to display a summary in the search result pages.)
  • X-Robots-Tag: NOFOLLOW (Use this tag if you don't want Googe and Yahoo to index the links in the page.)

The X-Robots-Tag is currently supported by Google and Yahoo. Other search engines don't support the tag yet.

 How to add the X-Robots-Tag to your documents

It depends on your server and on the method you use to create your web pages how you can add additional tags to the HTTP header. If your web server uses Apache, you can use the htaccess file to modify your HTTP headers.

In general, you shouldn't mess with the HTTP header if you're not 100% sure what you're doing. A broken HTTP header can keep search engines away so that no search engine will index your website.

 Instead of using the X-Robots-Tag, you can also add meta tags with the same functionality to your web pages:

<meta name="robots" content="noarchive">
<meta name="robots" content="nosnippet">
<meta name="robots" content="nofollow">
<meta name="robots" content="noindex">

You can edit these tags directly in the head part of your web pages. They have the same effect as the corresponding X-Robots-Tags explained above.

Do you need the X-Robots-Tag or the corresponding meta tags?

 If you don't want to limit access to your documents of if you only have normal web pages on your website then you don't need the X-Robots-Tag or the meta tags.


Search engine robots must be able to easily parse your web pages and they must be able to find the right elements on your web pages. If you optimize the right web page elements with the right keywords, your web page will get high search engine rankings for those keywords.

Share this article:

Featured Articles

Opportunities for Mobile Promotions

The mobile market is undisputedly becoming an integral part of the marketing mix and for many marketers presents abundant opportunities to reach consumers. As adoption of mobile internet continues to increase, so do consumer’s expectations...

Read More
Blame Television For Everything That's Wrong In Your World

Feeling a little out of sorts lately? Not accomplishing your life's goals? Fatter and balder than you wish you were? Don't worry, it's not your fault! You can blame everything that's wrong in your world on.... TELEVISION!

Read More
Providing Anonymity Through Virtual Cash

uKash.com provides a paypal type payment solution for your adult website.

Read More
Cinemix Affiliate Program

Read about a new adult affiliate progam from Sweden called cinemix.

Read More
The "411" on PDA Porn

Pocket-xxx.com offers the background and the essentials for turning your adult site on to the wireless web. A MUST READ for Amateur Webmasters.

Read More

Privacy Policy


©1994-2012 Adult Chamber Webmaster Resources. All rights reserved.

Business Services

Web Design
Print Design