How do you fine tune robots txt?

There is an excellent Wikipedia entry here, with links to several articles and websites that specifically discuss robots.txt and how to fine tune its configuration.

This Q&A was found on XOOPS Web Application System : https://xoops.org/modules/smartfaq/faq.php?faqid=374