Write a Robots.txt File
http://www.seochat.com/c/a/Search-Engine-Optimization-Help/Write-a-Robotstxt-File/

Dixon, Clint
SEOchat
2005

Abstract:

One of the most fundamental steps when optimizing a website is writing a robots.txt file. It helps tell spiders what is useful and public for sharing in the search engine indexes and what is not. It should also be noted that not all search spiders will follow your instructions left in the robots.txt file. In addition, a poorly done robots.txt file can stop the search spiders from crawling and indexing your website properly. In this article I will show you how to be sure everything will work correctly.