Robots.txt Validator

Robot text file validation

This search engine tool will check for the existence and validate your robot.txt file. You must type in the full path to your robot.txt file which is usually at http://www.yourdomain.com/robot.txt

Robots.txt file is a file that is typically loaded on your server root directory. Each time a search engine spider or crawler indexes your site it first looks for the robots.txt file to see if it is allowed to crawl your site and if so which directories or folders it can collect information from.

To create a robots.txt file just open up notepad and name the file “robots.txt” Then enter the information that allows or disallows the robot to crawl your site.

User-agent: *
The asterisk (*) or wildcard represents a special value and means any robot.

Disallow:
The Disallow: line without a / (forward slash) tells the robots that they can index the entire site.
If you would like to disallow a folder just enter
Disallow: /private/
Enter the location of the robot.txt (eg. http://www.yoursite.com) file of your site or other sites in the field below to check if the file is present and was created correctly.

Below are a few link to other Robots.txt resources.
A Standard for Robot Exclusion
>Robots Exclusion META tag Protocol
Robots.txt Validator
http://www.google.com/remove.html

SEO Forums

Local Online Marketing

WAP

SEO Tools

Robots.txt Validator

Useful Resources
RSS Readers
Search Engine Optimization
CPC Management
CGI Scripts Webmaster Tools
Forum Hosting