http://tc.eserver.org/authors/Dixon,_Clint A bibliography of works by Dixon, Clint in the field of technical communication. en-us Copyright (c) 2005-08 by the EServer. All rights reserved. tclib-editorial@eserver.org (TC Library Editorial Board) webmaster@eserver.org (Geoffrey Sauer) http://tc.eserver.org/images/newlogo.gif http://tc.eserver.org/dir/Dixon,_Clint http://tc.eserver.org/26493.html http://tc.eserver.org/26493.html One of the most fundamental steps when optimizing a website is writing a robots.txt file. It helps tell spiders what is useful and public for sharing in the search engine indexes and what is not. It should also be noted that not all search spiders will follow your instructions left in the robots.txt file. In addition, a poorly done robots.txt file can stop the search spiders from crawling and indexing your website properly. In this article I will show you how to be sure everything will work correctly.