xoops forums

wishcraft

Module Developer
Posted on: 2011/8/29 14:36
wishcraft
wishcraft (Show more)
Module Developer
Posts: 3710
Since: 2007/5/18
#1

Spiders 2.7.3 (Review requested)

Resized Image
Spiders 2.7.3
Community Release by - Chronolabs Co-op

Spiders is a robot manager tool, that imports a list of all crawler and scanner robots on the web. It allows you to use XOOPS Permissiveness to control the data that robots list online your site. It will also log the robot in using a post loader and display when the robot is online on you 'Whos Online'.

Do you want your robots like GoogleBot, Yahoo Slurp! and others to log in and identify on your xoops installation? Then spiders is for you!!! The robot text file used is taken from an online resource of Robot data and stores it in your database. Remember to adjust your mainfile.php to include the post loader after the common file is loaded. Robot Manager (Spiders) is a good way to control what your site displays in search engines.

Watch this video to understand more about spiders!




New Features
  • Pre PHP 5.2 Compatibility
  • Search Functions Included


Some of the Features
  • Clean Robots-all.txt with over 200 bots
  • Try Exceptions added to API Calls for seemless entry
  • Complete Restricted Keywords
  • Polls API in SOAP
  • Polls API in cURL JSON
  • Polls API in cURL XML
  • Polls API in cURL Serialised
  • Polls API in wGET JSON
  • Polls API in wGET XML
  • Polls API in wGET Serialised
  • Modification & Live Area
  • Easy Xortify Signup
  • Improved Preloads
  • SEO Advantage Sharer
  • Upgrade Path Maintained
  • SEO URL Rewrites
  • User Interface


Bugs Fixed
  • Warnings Fixed
  • Notices Fixed
  • JSON_Services Duplicated Add
  • wGET Polling
  • CURL Polling
  • Xortify Preferences URI
  • No option of what protocol to use


Spider is only written for XOOPS 2.4 and later.

Download: xoops2.5_spiders_2.73.zip (153Kb)
Sourceforge Mirror: xoops2.5_spiders_2.73.zip
Demo: http://xoops.demo.chronolabs.coop/

Upgrade Instructions:

If you haven't set permission throughout your XOOPS Site for Robots then it is a simple case of uninstalling and re-installing then importing the current robots-all.txt. However if you have set permissions then I suggest you follow the steps as so. a) import robots-all.txt b) change 'Protected Keywords from Useragents' in Robot Managers (spiders) preferences to contain at least these keywords without the carriage return ::

Mozilla/5.0|Mozilla/4.0|Mozilla/3.0|Mozilla/3.01|Mozilla
/2.0|Mozilla|mozilla|Windows|Unix|Linux|OS|Mac|Macintosh|
Compatible|compatible|Yes|yes|no|No|http
Resized Image
www.ohloh.net/accounts/226400

Follow, Like & Read:-

twitter.com/SimonXaies
github.com/Chronolabs-Cooperative
facebook.com/SimonSXaies

wishcraft

Module Developer
Posted on: 2011/8/31 9:41
wishcraft
wishcraft (Show more)
Module Developer
Posts: 3710
Since: 2007/5/18
#2

Re: Spiders 2.7.3 (Review requested)

btw. I have patch the 2.73 Zip just now to stop the inclusion of the Control Panel Icons in its functionality of the Admin.
Resized Image
www.ohloh.net/accounts/226400

Follow, Like & Read:-

twitter.com/SimonXaies
github.com/Chronolabs-Cooperative
facebook.com/SimonSXaies

easyb9

Just popping in
Posted on: 2011/10/17 17:27
easyb9
easyb9 (Show more)
Just popping in
Posts: 41
Since: 2011/8/10
#3

Re: Spiders 2.7.3 (Review requested)

alexa and Yandex bot not included in robot text.
and some robot in list not from most used search engine.


try to block anonymous view but let the bots crawl it
and the problems
Fetch as Googlebot
« Go back

This is how Googlebot fetched the page
.

URLhttp://www.mysiteexample/modules/xnews/article.php?storyid=413

DateMondayOctober 172011 10:21:38 AM PDT

Googlebot Type
Web

Download Time 
(in milliseconds): 480

HTTP
/1.1 302 Moved Temporarily
Date
Mon17 Oct 2011 17:21:34 GMT
Server
Apache
X
-Powered-ByPHP/5.2.17
Expires
Thu19 Nov 1981 08:52:00 GMT
Cache
-Controlno-storeno-cachemust-revalidatepost-check=0pre-check=0
Pragma
no-cache
Set
-CookiePHPSESSID=dfc506708b1e00e1173a42d9d11b7be5path=/
Locationhttp://www.mysite/user.php?from=xnews
VaryUser-Agent,Accept-Encoding
Content
-Encodinggzip
Content
-Length20
Connection
close
Content
-Typetext/html