4
I've had a lot of trouble with search engines going wild (Yahoo and MSN for starters) on a few of my non XOOPS sites. I would think protector would help that with the high crawler options. You could try removing Slurp from the welcomed crawler list:
[size=x-small]
Welcomed User-Agent:
A perl regex pattern for User-Agent. If it matches, the crawler is never considered as a high loading crawler. eg) /(msnbot|Googlebot|Yahoo! Slurp)/i[/size]
If it is a search engine, you could also try adding a crawl delay to your robots.txt file such as:
User-agent: Slurp
Crawl-delay: 30I believe Yahoo, MSN, and Ask/Teoma will obey this but last I checked Google simply ignores. Yahoo has so many bots that it takes a while for all of them to catch on.
Read more about Yahoo and crawl delay
HERE.
Quote:
I wonder if its yahoo slurp going beserk on my site for some reason
[size=x-small]Working sites:
XOOPS 2.0.16 PHP 5.2.2, MySQL 5.0.24a-standard-log, Apache/2.0.54 (Unix)
XOOPS 2.2.4, PHP 4.3.10, MySQL 3.23.58, Apache/1.3.33 (Unix)[/size]
