Policy Annoucement: Offline Browsers / Website Copiers - XOOPS - XOOPS News - XOOPS Web Application System

XOOPS

News

News World of XOOPS Developers Hacks Themes Archive Submit News

XOOPS: Policy Annoucement: Offline Browsers / Website Copiers

Posted by: ackbarrOn 2004/12/10 14:50:00 7494 reads

There have been a recent rash of users attempting to make offline copies of http://www.xoops.org and wiki.xoops.org.

Our policy: Don't.

Many offline copiers are setup to make many simultaneous requests to a website to improve copying performance. A busy website like XOOPS.org gets a lot of traffic (averages 3.5 GB transfered daily), and these offline copiers can easily eat up connections meant for normal users. If we catch a website copier, their address will be banned at the firewall.

First offense - a 1 hour temp ban
Second offense - a 1 day temp ban
Third offense - a permanent ban

This may sound harsh, but offline copiers negatively affect site performance; giving a false impression on the quality of XOOPS.

If you have a special request or need content on xoops.org for offline use, you may contact one of the webmasters asking for permission. It is entirely at the discretion of the webmaster to approve or deny this request.

The comments are owned by the author. We aren't responsible for their content.

user

Re: Policy Annoucement: Offline Browsers / Website Copiers

Published 12/10/2004 15:32 Joined Comments

Hello,
Bad pepole who don't know what they are doing, they trying to download xoops.org then browse it offline, do u think xoops.org static website ? everyday there are new, and must known xoops.org have a dedicated server, so why you decrease your hard disk space ?
I think must make subdomain then redirect it to another hosting space for each section of xoops.org ( eg: forums.xoops.org ) and limit simultaneous connections and downloads.
God save our XOOPS !

Re: Policy Annoucement: Offline Browsers / Website Copiers

tl Published 12/10/2004 16:20 Friend of XOOPS Joined 06/23/2002 Comments 999

ackbarr:

Banning IPs is pretty "re-active". You may want to consider banning downloaders by user-agent using htaccess.

Also you may consider banning RSS newsreaders as well. It would be too late to ban those personal RSS newsreaders that have already connected and are checking XOOPS site every 5 minutes, but it would definitely not let new one in.

Re: Policy Annoucement: Offline Browsers / Website Copiers

Peekay Published 12/10/2004 16:45 XOOPS is my life! Joined 11/20/2004 Comments 2335

vBulletin split their support resource. They have a general forum at vbulletin.com/forum/ but posts about code hacks and developer stuff generally get referred to their other forum at vbulletin.org. Likewise, templates have a special forum at vbulletintemplates.com. You have to register for each, but it's not a big deal.

user

Re: Policy Annoucement: Offline Browsers / Website Copiers

ackbarr Published 12/10/2004 16:53 Joined 10/02/2002 Comments 1449

we already have rewrite rules for known webcopiers, the problem is that some webcopiers allow for the spoofing of the UA, making them appear as legitimate users. Then there is the "Make Content Available Offline" feature in IE that can bring a dynamic site to its knees if pulling down too many levels simultaneously.

I'm considering placing a web-bug in the theme that would trigger when a website copier hit the link. Though it is still re-active - at least it is re-active automatically.

user

Re: Policy Annoucement: Offline Browsers / Website Copiers

ackbarr Published 12/10/2004 16:56 Joined 10/02/2002 Comments 1449

to do something like that would require getting additional servers, 'cause putting them all on the same server would defeat the purpose of seperating the forums. and if we add additional hardware there are other measures we could take:

moving the mysql db to a dedicated machine
pulling searches (3nd biggest hog after spammers and offline browsers) from a mirrored db instead of the live one

user

Re: Policy Annoucement: Offline Browsers / Website Copiers

limecity Published 12/10/2004 17:47 Friend of XOOPS Joined 07/06/2003 Comments 1602

don't really understand what is is about.

you mean "save page as" is not allow here?
thats all? is this what its all about?

user

Re: Policy Annoucement: Offline Browsers / Website Copiers

Mithrandir Published 12/10/2004 17:51 XOOPS is my life! Joined 06/21/2003 Comments 6320

No, that's not the problem as that just saves what is in the browser (the HTML page that has been served)

It's the "make this page available offline" and similar things that not only saves the current page, but also many of the pages that are LINKED from that page - and some even updates every x minutes.

user

Re: Policy Annoucement: Offline Browsers / Website Copiers

ackbarr Published 12/10/2004 18:34 Joined 10/02/2002 Comments 1449

Mithrandir is correct - I'm not talking about saving a page to your computer. I'm talking about the use of programs that crawl through an entire site making a "mirror" of its content. Those are the ones that can and frequently do negatively impact server responsiveness.

user

Re: Policy Annoucement: Offline Browsers / Website Copiers

Dave_L Published 12/10/2004 18:40 Updated 12/10/2004 18:41 XOOPS is my life! Joined 11/07/2003 Comments 2277

And then there's the "accelerator" feature some browsers have, that prefetches every page linked within the current page, so those pages will already be downloaded in case you decide to click on the links. Or maybe that's the same thing you're talking about.

Re: Policy Annoucement: Offline Browsers / Website Copiers

Ballie Published 12/10/2004 18:47 Updated 12/10/2004 18:55 Just popping in Joined 04/17/2004 Comments 7

Instead of punishing people who don't know what they are doing, you might consider to try to get in touch with them, to understand *why* they are DLing the whole site.

I once considered to copy the whole site offline too (including the wiki, the forums and all the other pages), because that's where all the information is.

(remember ? https://xoops.org/modules/newbb/viewtopic.php?topic_id=18927&forum=27&post_id=80515#forumpost80515 - to be clear: I didn't do it at that time, and I didn't do it now)

Providing a (weekly refreshed ?) package with all the content of the site for offline browsing migth be very helpfull for all those people that don't have 24/7 internet-access.

EDIT: by reading it again, I notice my text above sounds a little bit harsh (which is not my intention). If I can help in providing this package (maybe we can program something to automatically select all changed or new pages, and zip them in one huge package ?) I'm off course willing to help.

user

Re: Policy Annoucement: Offline Browsers / Website Copiers

Herko Published 12/10/2004 18:50 XOOPS is my life! Joined 02/04/2002 Comments 4238

Don't we have the FAQ and the Docs site for this now? I think we're providing enough information to get started, so abusing the server (wittingly or otherwise), should be discouraged.

Herko

Re: Policy Annoucement: Offline Browsers / Website Copiers

tl Published 12/10/2004 19:55 Friend of XOOPS Joined 06/23/2002 Comments 999

Quote:

Instead of punishing people who don't know what they are doing, you might consider to try to get in touch with them, to understand *why* they are DLing the whole site.

How do you track down an IP and the person? sopena the ISP?

People that use downloaders are not welcome by any webmaster and are considerd inconsiderate and plain rude.

user

Re: Policy Annoucement: Offline Browsers / Website Copiers

irmtfan Published 12/11/2004 6:28 Module Developer Joined 12/07/2003 Comments 3419

Quote:

Providing a (weekly refreshed ?) package with all the content of the site for offline browsing migth be very helpfull for all those people that don't have 24/7 internet-access.

is this possible at all?
for example i want zip my forum and put it for public download and i think this help my bandwidth and also my users very pleased about it because nearlly all of them have dial-up conections and low speed.

user

Re: Policy Annoucement: Offline Browsers / Website Copiers

Herko Published 12/11/2004 10:28 XOOPS is my life! Joined 02/04/2002 Comments 4238

the xoops.org forum is about 50 Mb big, add to that the fact that the SQL needs to be imported into a module, I don't see it as an easy accessible solution for our dailup members. That's why we have the search feature and the FAQ: to make usefull content available real easy...

Herko

user

Re: Policy Annoucement: Offline Browsers / Website Copiers

martyboy Published 12/11/2004 17:38 Quite a regular Joined 05/25/2004 Comments 256

This is interesting, whenever someone downloads the site/makes it available offline it uses up bandwidth? I think this could be happening on my site too its relatively small, on shared hosting but i seem to be using most of my 75gb limit each month..would it be fair to say that i could ban a user using more than 300mb bandwidth in the sace of a week or 2 say.

Personally ive never downloaded a page for viewing offline i got broadband no need to if theres info like a tutorial or text info i need and i wasnt to look at it offline i just use good old copy and paste and save it to notepad or something.

user

Re: Policy Annoucement: Offline Browsers / Website Copiers

vinit Published 12/11/2004 17:41 Just can't stay away Joined 01/10/2004 Comments 530

are you sure that its not another search engine spider crawling the entire site and creating a cached system as google... this web spiders and crawlers can actually bring down entire site too...

Last time my mailing list was stuck by googlebot, unfortunately i deleted the robots.txt and google, msn, alexa went inside... took away wooping 5gb of bandwith making in 5000+ hits. in less than 1 month. bring down my mailing list.

Other thing that can be set is number of connections available to an ip, this can sometimes be a problematic if a few members from a single network bedhind a ip are trying to access XOOPS site... but this very few chance.

Other than this a matrix of (ip * link) /time ratio can be used to ban an ip for a period of 5-10 mins, just some web server conifgs.. similar tweaks can come very handy if you have site which has wooping consumption of 3.5GB a day

user

Re: Policy Annoucement: Offline Browsers / Website Copiers

ackbarr Published 12/11/2004 18:18 Updated 12/11/2004 18:20 Joined 10/02/2002 Comments 1449

yes I'm sure that these requests are from regular users prior to banning them. I have kept a very close eye on server utilization for the last 2 months. When I see a load spike I immediately determine which IP address is the cause.

Re: Number of connections per IP - I discussed this exact issue on my site 2 months ago, and IIRC you argued against using such settings.

Quote:

yes mod_trottle will do the trick. But i still feel its not best solution. Many small companies, institutes and local ISPs dont provide real ip to there users, they are generally under NAT, under such a situation if there are 15 or 20 members from a network trying to access the xoops.org than it would keep them out. becaue of limitations.

** Edit - I just realized that vinit might be responding to the previous commenter, and not to me. If that is the case I apologize.

user

Re: Policy Annoucement: Offline Browsers / Website Copiers

martyboy Published 12/11/2004 19:36 Quite a regular Joined 05/25/2004 Comments 256

ackbarr how do you determine which ip adresse(s) cause such increase in load? is it possible for you to post a link to a tutorial on this I would like to learn more so I can root out people doing similar things on my server, i.e being greedy with bandwidth. I tried looking through raw logs and webstats(awstats) for domains/ip with large bandwidth consumption but the lists are huge.

user

Re: Policy Annoucement: Offline Browsers / Website Copiers

Herko Published 12/11/2004 19:45 XOOPS is my life! Joined 02/04/2002 Comments 4238

XOOPS runs on a dedicated server, so we can monitor every apache connection thru a (standard) apache feature.

Herko

Login

Username

Password

Remember me

Register now! | Lost Password?

Search

Advanced Search

Recent Comments

Re: New Admin Theme for XOOPS (Beta)

Publisher liomj 10/28/2024 23:33
Sanford Pharmacy: Exceptional Pharmacy Services Tailored to You

Publisher donaldjohn 10/16/2024 9:53
Re: New Admin Theme for XOOPS (Beta)

Publisher walkero 10/15/2024 18:30
Re: XOOPS 2.5.12 Beta-2 available for Testing

Publisher cadch 08/30/2024 6:33
Re: XOOPS 2.5.12 Beta-2 available for Testing

Publisher cadch 08/30/2024 6:30

Who's Online

411 user(s) are online (6 user(s) are browsing XOOPS News)

Members: 0

Guests: 411

Donat-O-Meter

Stats
Goal:	$100.00
Due Date:	Nov 30
Gross Amount:	$0.00
Net Balance:	$0.00
Left to go:	$100.00

Latest GitHub Commits

{{ record.sha.slice(0, 7) }} - {{ record.commit.message | truncate }}

{{ record.commit.author.name }} {{ record.commit.author.date | formatDate }}

Archives

News archives