The robots.txt file is a set of instructions for visiting robots (spiders) that index the content of your web site pages. For those spiders that obey the file, it provides a map for what they can, and cannot index. The file must reside in the root directory of your web. The URL path (web address) of your robots.txt file should look like this.
http://www.seoconsultants.com/robots.txt
The Robots text file open in Notepad might look like this:
This is a screen shot of an empty (not recommended) robots.txt file
Definition of the above robots.txt file:
User-agent: *
The asterisk (*) or wildcard represents a special value and means any robot.
Disallow:
The Disallow: line without a / (forward slash) tells the robots that they can index the entire site. Any empty value indicates that all URLs can be retrieved. At least one Disallow field needs to be present in a record without the / (forward slash) as shown above.
The presence of an empty "/robots.txt" file has no explicit associated semantics, it will be treated as if it was not present, i.e. all robots will consider themselves welcome.
The Disallow: line without the trailing slash (/) tells all robots to index everything. If you have a line that looks like this:
Disallow: /private/
It tells the robot that it cannot index the contents of that /private/ directory.
Summarizing the Robots Exclusion Protocol - robots.txt file
To allow all robots complete access:
User-agent: *
Disallow:
Important Note:
The above format is the common and acceptable standard for allowing all spiders' access to the site. We've recently learned (2002-06-09), that the practice of having just a User-agent: * and Disallow: without a trailing forward slash (empty robots.txt file) may not be recommended. Some spiders may incorrectly interpret this as blocking all content. You'll notice that we disallow the _private, css, and JavaScript folders in the below example and do not recommend an empty file.
2003-05-13 - Do not disallow your /css/ directory. Recent issues with Google may suggest that disallowing your css directory could be a flag for a manual review to see if you are using css to deceive the indexing robots (spiders).
This is a screen shot of a robots.txt file
To exclude all robots from the server:
User-agent: *
Disallow: /
To exclude all robots from parts of a server:
User-agent: *
Disallow: /private/
Disallow: /images-saved/
Disallow: /images-working/
To exclude a single robot from the server:
User-agent: Named Bot
Disallow: /
To exclude a single robot from parts of a server:
User-agent: Named Bot
Disallow: /private/
Disallow: /images-saved/
Disallow: /images-working/
Note:
The asterisk (*) or wildcard in the User-agent field is a special value meaning "any Robot" and therefore is the only one needed until you fully understand how to set up different User-agents.
If you want to Disallow: a particular file within the directory, your Disallow: line might look like this one:
Disallow: /private/top-secret-stuff.htm
Keep in mind that using the above example excludes that specified page (top-secret-stuff.htm) but will not exclude the entire /private/ directory. If you have files that you do not want indexed, then you should put them in a private folder and Disallow: the entire directory, or put them in a password protected directory, or don't put them on the web at all!
No comments:
Post a Comment