This weekend, I had two clients with XOOPS websites go over their allotted bandwidth. And it wasn't even the middle of the month!

Upon further investigation, I found out that GoggleBot was the culprit. Apprarently, it really liked the piCal module.

After researching the XOOPS site, I found this:

In your XOOPS installation the default robots.txt file is fine the way it is for *most* cases. The only time you should ever have to chage that file is if you install piCal or if you just don't want search engines to index certain areas of your site.

The reason you want to disallow access to piCal is because the spiders will each up all your bandwidth searching through the calendars internal links month by month. To disallow piCal, just add the following to robots.txt:

Disallow: /modules/piCal/

(Taken fromhttps://xoops.org/modules/smartfaq/faq.php?faqid=306)

Well, before I found this, I disabled the piCal module on one of the sites because it turned out that they weren't using it. Thought that would solve the problem - but it appears that Googlebot is STILL trying to access it. So I have added the above "disallow" to see if that works. Ugh!

Hopefully this will take care of it. Just wanted to post it here in case someone else has the same problem.

Hey thanx for posting this, got an email from my host today about being close to going over bandwidth limit, I did notice there was a lot of pical spidering going on from the logs. Cheers!

I altered my weblog at www.jonathanspencer.net and last night googlebot used up my bandwidth

I altered the site with a sitemap, search engine friendly URLs and page titles that matched the articles in the weblog module, + turning off the robot follow on print and pdf links (maybe this did something??)

each article seems to have been read by the bot 200 times, can anyone tell me why this would be or what would make it stop doing this.

Hi,

Quote:

How did you do that? I have a rudimentary version with my news items, but it adds < a>section name< /a> to the title bar too, not very attractive. I did it with an include in the theme and haven't figured out a work around. I'd like to do that with both news and weblog if I could. Would appreciate a pointer.

Perhaps googlebot has the bird flu?

Thanks in advance,
David.
P.S. Nice sites, btw.

check my discussion with the excellent and helpful webmaster at XOOPS tips

http://xoops-tips.com/modules/news/article.php?storyid=69&com_id=185&com_rootid=185&#comment185

which explains more

Thanks,

That's where I got some of the other tips with page title too, article #93 or 92 I think. Now to connect the dots....

And a tip of my own, weblog uses ping o matic in it's common ping servers, replace it with this;http://pingoat.com/goat/RPC2 - it pings over 60 servers and does it really fast. This file to edit is commonping_servers.inc.php

Thanks again, all the best.
David.