233766
Hi,
Quote:
tl wrote:
What is the crawler's user agent?
Is it something like the following?
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
The usual Google user agent is
"Googlebot/2.1 (+http://www.google.com/bot.html)"
I suspect the first one is a faked Google crawler - I have been hit continuously for a couple of days now.
Yes, thanks for pointing that out. If it is a faked crawler, the "real" Googlebot is agent:
"Googlebot/2.1 (+http://www.google.com/bot.html)"
whilst the fake one is:
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
however, the IP address of the 'fake' ones resolve to domain google.com, however no doubt they would spoof the IP address.
I'll find out more from searchengineworld and a few other places, and if 100% certain it is a fake bot, then I'll ban the agent.
There were no "404's" in the _real_ Googlebot, they only appeared from the IP addreses of the other one.
Thanks,
Peter