Archive for September, 2006

Creating robots instruction document – Robots.txt

Sunday, September 24th, 2006

In earlier post I have mentioned that using a robot directive file - robots.txt you can control whether a document can be crawled by a search engine or not. Generally if there is no robots.txt file in the root of your website or it has no instructions written in it, it ...