XOOPS 
Forum
Forum Index General Forums Community Support Modules Support Themes Support Development International Support Module Hack Reviews Theme Designer Talk
  1. Board index / 
  2. General XOOPS.org forums / 
  3. XOOPS.org Members Lounge 
  4.  / New Google Bot or Someone Spoofing Google IPs?
Register
1
tl
New Google Bot or Someone Spoofing Google IPs?

  • 2004/9/26 15:08

  • tl

  • Friend of XOOPS

  • Posts: 999

  • Since: 2002/6/23


Has anyone noticed new entries in their logs with the following user agent?

Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

The usual user agent for Google is
Googlebot/2.1 (+http://www.google.com/bot.html)

Google usually sends several crawlers to a site and does not crawl a site continuously. The new one usually would come from a single crawler/IP and behave like a spambot by crawling continuously.

It is very suspicious. Anyone has similar experience?
2
FlySwatter
Re: New Google Bot or Someone Spoofing Google IPs?

  • 2004/9/26 19:59

  • FlySwatter

  • Just popping in

  • Posts: 71

  • Since: 2004/6/2 1


Check out:

Bot Trap

And maybe:

Bad Bot page
3
tl
Re: New Google Bot or Someone Spoofing Google IPs?

  • 2004/9/26 22:04

  • tl

  • Friend of XOOPS

  • Posts: 999

  • Since: 2002/6/23


Quote:

Re: New Google Bot or Someone Spoofing Google IPs?
Check out:

Bot Trap

And maybe:

Bad Bot page


Thank you for your suggestion. Sorry I am just looking for confirmation if anyone has seen similar bots and has info on them.

BTW, unless you don't want Google to crawl your site, you can not stop people with (spoof) Google IPs from crawling your site.
4
FlySwatter
Re: New Google Bot or Someone Spoofing Google IPs?

  • 2004/9/26 23:08

  • FlySwatter

  • Just popping in

  • Posts: 71

  • Since: 2004/6/2 1


Quote:
BTW, unless you don't want Google to crawl your site, you can not stop people with (spoof) Google IPs from crawling your site.


But if Google DOES obey robots.txt, won't Bot Trap ID them for you? Then one can at least know.

Anyway, I had an occassion to question Google about a simialr thing and did receive a reply from a real person. Not just an auto-respond.
5
tl
Re: New Google Bot or Someone Spoofing Google IPs?

  • 2004/9/26 23:47

  • tl

  • Friend of XOOPS

  • Posts: 999

  • Since: 2002/6/23


I am not looking to ban or for ways to ban Google, just information or confirmation on the new log entries. I don’t want spambots pretending to be Google to crawl my sites.

It is not difficult to ban Google or any other bots. But in general for most people, it would be foolish or suicidal to ban Google, unless you have an Intranet or a top-secret site to protect.


6
FlySwatter
Re: New Google Bot or Someone Spoofing Google IPs?

  • 2004/9/27 1:32

  • FlySwatter

  • Just popping in

  • Posts: 71

  • Since: 2004/6/2 1


Sorry for the mis-understanding.

Bot Trap doesn't ban Google; or any other Bot for that matter. It just adds an entry to the Robots.txt file disallowing a certain directory. If a Bot ignores the directive it is logged.

It can be very informative.

Hope this clears things up.
7
Dalton
Re: New Google Bot or Someone Spoofing Google IPs?

  • 2004/9/28 14:21

  • Dalton

  • Just popping in

  • Posts: 12

  • Since: 2004/6/2 1


Yes i´ve some visits of the crawl in my web. But Google does not index it.

Login

Register now!  |  Lost Password?

Search

Advanced Search

Recent Posts

Who's Online

183 user(s) are online (93 user(s) are browsing Support Forums)

Members: 0

Guests: 183

more...

Donat-O-Meter

Stats
Goal: $100.00
Due Date: May 31
Gross Amount: $0.00
Net Balance: $0.00
Left to go: $100.00
Make donations with PayPal!

Latest GitHub Commits