XOOPS 
Forum
Forum Index General Forums Community Support Modules Support Themes Support Development International Support Module Hack Reviews Theme Designer Talk
  1. Board index / 
  2. XOOPS Community Support forums / 
  3. Q and A 
  4.  / RE: Spiders and User info?! Bad News?
Register
1
DobePhat
RE: Spiders and User info?! Bad News?

  • 2003/12/5 19:44

  • DobePhat

  • Friend of XOOPS

  • Posts: 656

  • Since: 2003/4/15


Hello,
It has occured to someone at our site that robots and spiders are indexing user info pages! Blech!

Can we prevent this? Would a "No Robots" etc tag help on that file?

Thanks!
2
divulga
RE: Spiders and User info?! Bad News?

  • 2003/12/5 23:12

  • divulga

  • Just popping in

  • Posts: 13

  • Since: 2003/11/25


I wait that these links can help you:

http://www.webmasterworld.com/forum92/413.htm

http://www.webmasterworld.com/forum92/205.htm

I supose is easy to block using .htaccess file. if you know the user agent of the bot or spyder.



3
DobePhat
RE: Spiders and User info?! Bad News?

  • 2003/12/6 5:12

  • DobePhat

  • Friend of XOOPS

  • Posts: 656

  • Since: 2003/4/15


Thanks Ill look into this. Im surprised it hasn't been brought up before...

4
wtravel
RE: Spiders and User info?! Bad News?

You can upload a file robots.txt in which you have set the directories that you do or do not allow to be spidered by robots.

I do not exactly remember the details but look this up in google and I am sure you will find out.

Regards,

Martijn
5
divulga
RE: Spiders and User info?! Bad News?

  • 2003/12/6 13:32

  • divulga

  • Just popping in

  • Posts: 13

  • Since: 2003/11/25


robots.txt is a INEFFICIENT (without effect, useless ,etc,etc ) method to block email harvest, webcopiers, offline browsers.

many spider soft don't read robots.txt.

But the .htacess use the server system to block.

A webcopier(offline browser) software is a bandwith terror, one user even catch 1GB transfer

Login

Register now!  |  Lost Password?

Search

Advanced Search

Recent Posts

Who's Online

182 user(s) are online (114 user(s) are browsing Support Forums)

Members: 0

Guests: 182

more...

Donat-O-Meter

Stats
Goal: $100.00
Due Date: Apr 30
Gross Amount: $0.00
Net Balance: $0.00
Left to go: $100.00
Make donations with PayPal!

Latest GitHub Commits