Creating robots instruction document – Robots.txt
Sunday, September 24th, 2006In earlier post I have mentioned that using a robot directive file - robots.txt you can control whether a document can be crawled by a search engine or not. Generally if there is no robots.txt file in the root of your website or it has no instructions written in it, it ...