SmartFAQ is developed by The SmartFactory (, a division of InBox Solutions (

How do you fine tune robots txt?
Requested by WarDick and Answered by FusionKid on 2006/3/29 16:48:58 (6600 reads)
There is an excellent Wikipedia entry here, with links to several articles and websites that specifically discuss robots.txt and how to fine tune its configuration.

The comments are owned by the author. We aren't responsible for their content.