What is a Robots.txt File?

By: , December 11, 2007 | Starting Tips For Webmasters

robots.txt file, google bot, robots.txt

What is a robots.txt file?

The robots.txt file is a simple text file that must be placed in your root directory (http://www.example.com/robots.txt). It tells the search engine spider which web pages on your website should be indexed and which web pages should be ignored.

You can use a simple text editor to create a robots.txt file. The content of a robots.txt file consists of so-called "records".

A record contains the information for a special search engine. Each record consists of two fields: the user agent line and one or more Disallow lines. Here's an example:

             User-agent: googlebot
        Disallow: /cgi-bin/

This robots.txt file would allow the "googlebot", which is the search engine spider of Google, to retrieve every page from your site except for files from the "cgi-bin" directory. All files in the "cgi-bin" directory will be ignored by googlebot.

Share this article:

Featured Articles

Opportunities for Mobile Promotions

The mobile market is undisputedly becoming an integral part of the marketing mix and for many marketers presents abundant opportunities to reach consumers. As adoption of mobile internet continues to increase, so do consumer’s expectations...


Read More
Blame Television For Everything That's Wrong In Your World

Feeling a little out of sorts lately? Not accomplishing your life's goals? Fatter and balder than you wish you were? Don't worry, it's not your fault! You can blame everything that's wrong in your world on.... TELEVISION!


Read More
Providing Anonymity Through Virtual Cash

uKash.com provides a paypal type payment solution for your adult website.


Read More
Cinemix Affiliate Program

Read about a new adult affiliate progam from Sweden called cinemix.


Read More
The "411" on PDA Porn

Pocket-xxx.com offers the background and the essentials for turning your adult site on to the wireless web. A MUST READ for Amateur Webmasters.


Read More

Privacy Policy

RTA / ASACP / FSC

©1994-2012 Adult Chamber Webmaster Resources. All rights reserved.

Business Services

Web Design
Print Design
Programming