xoops forums

wishcraft

Module Developer
Posted on: 2008/1/1 3:30
wishcraft
wishcraft (Show more)
Module Developer
Posts: 3710
Since: 2007/5/18
#1

backend.php is good but what about a sitemap.php?

There is one feature that was recently added to search engines, that is a sitemap..

When you are implementing a sitemap they are an XML Document that tells search engine, when and where to scan/crawl on your site.

So far for all my site I have had to build a stand alone sitemap that is in php that does the site's content etc. It would be good if there was a controlling class in the module systems for this..

There are a couple of things you have to do to get a sitemap working properly..

First it has to have only existing URL's in it; you can see the full sitemap this is taken from at http://www.bee-unlimited.co.uk/sitemap.php

for example a section of this is like:

<urlset> <loc>http://www.bee-unlimited.co.uk/modules/content/?id=1</loc>
  
<lastmod>2008-01-01</lastmod>
  <
changefreq>monthly</changefreq>
  <
priority>1</priority>
  </
url>
<
url>
   <
loc>http://www.bee-unlimited.co.uk/modules/content/?id=2</loc>
   
<lastmod>2008-01-01</lastmod>
   <
changefreq>monthly</changefreq>
   <
priority>1</priority>
</
url>
</
urlset>


Loc: URL for crawling
lastmod: The last modification time
changefreq: Daily, Monthly, Yearly, Never - This is the frequency of change
Priority: 1 - 0.1 (1 being the highest).

For sitemap convention check www.sitemap.org, you will also have to change your robots.txt to add in the sitemap for search engines like MSN & Live which only automatically check for them

User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /cache/
Disallow: /class/
Disallow: /images/
Disallow: /include/
Disallow: /install/
Disallow: /kernel/
Disallow: /language/
Disallow: /templates_c/
Disallow: /themes/
Disallow: /uploads/
Sitemap: /sitemap.php


Amicability most modules do not have a all this data, ie the content module, which has no date of creation.

The other change you may have to make is to common.php what I have done is made a database with robots crawler hostnames that allows for crawlers to access section anonymously but not the regular user.

This is a simple few lines of code around 20

and example bot checking tool would be:

function friendlybot(){
        global 
$xoopsDB;
        
$hostname gethostbyaddr($_SERVER['REMOTE_ADDR']);
        
$sql "SELECT bottype FROM ".$xoopsDB->prefix("bot");
        
$ret $xoopsDB->query($sql);
        
$state=false;
        while (
$row $xoopsDB->fetchArray($ret)){
            if (
strpos(" ".$hostname,$row['bottype'])>0){
                
$state=true;
            }
        }
        return 
$state;
    }


Where the change in common.php would be start at lines 336 and would be changed to something like this allowing for crawlers to crawl a page but the security still to be implemented.

if ($xoopsUser) {
            if (!
$moduleperm_handler->checkRight('module_read'$xoopsModule->getVar('mid'), $xoopsUser->getGroups())) {
                if (!
friendlybot()){
                    
redirect_header(XOOPS_URL."/user.php",1,_NOPERM);
                    exit();
                }
            }
            
$xoopsUserIsAdmin $xoopsUser->isAdmin($xoopsModule->getVar('mid'));
        } else {
            if (!
$moduleperm_handler->checkRight('module_read'$xoopsModule->getVar('mid'), XOOPS_GROUP_ANONYMOUS)) {
                if (!
friendlybot()){
                    
redirect_header(XOOPS_URL."/user.php",1,_NOPERM);
                    exit();
                }
            }
        }


Your bottype varchar (128) field would contain things like:

archive.org      
ask
.com          
crawl
.yahoo.net  
cs
.tamu.edu      
cuill
.com        
entireweb
.com    
googlebot
.com    
inktomisearch
.com
looksmart
.com    
msnbot
.msn.com   
picsearch
.com    
search
.live.com  
snarked
.org      
yahoo
.com        
blinklist
.com    
icio
.us          
digg
.com         
furl
.net         
simpy
.com        
spurl
.net        
img
.com          
facebook
.com


We need something like a globalism solution for site maps as it will ensure our websites are indexed correctly and quickly, I have index appear in google in a couple of hours with a sitemap rather than several days since i implemented this.

Sitemap.php looks like this at www.bee-unlimited.co.uk but you would have to customise this for most environment until some form of standardisation occurs in the XOOPS library.

<?php require("mainfile.php");

header('Content-type: text/xml');
$query[0] = "SELECT storyid from _beeu_content";
$query[1] = "SELECT storyid from _beeu_singleclip";
$query[2] = "SELECT storyid from _beeu_vjmixes";
$query[3] = "SELECT lid,cid,date from _beeu_myalbum_photos where status>0 order by date desc limit 5000";
$query[4] = "SELECT storyid,created from _beeu_stories order by created desc limit 5000";

global 
$xoopsDB;

$ret $xoopsDB->query($query[0]);
while (
$row $xoopsDB->fetchArray($ret)){
    
$url[] = array("loc" => XOOPS_URL.'/modules/content/?id='.$row['storyid'],
                   
"lastmod" => date('Y-m-d',time()),
                   
"changefreq" => "monthly",
                   
"priority" => "1");
}

$ret $xoopsDB->query($query[1]);
while (
$row $xoopsDB->fetchArray($ret)){
    
$url[] = array("loc" => XOOPS_URL.'/modules/singleclip/?id='.$row['storyid'],
                   
"lastmod" => date('Y-m-d',time()),
                   
"changefreq" => "monthly",
                   
"priority" => "1");
}

$ret $xoopsDB->query($query[2]);
while (
$row $xoopsDB->fetchArray($ret)){
    
$url[] = array("loc" => XOOPS_URL.'/modules/vjmixes/?id='.$row['storyid'],
                   
"lastmod" => date('Y-m-d',time()),
                   
"changefreq" => "monthly",
                   
"priority" => "1");
}

$ret $xoopsDB->query($query[3]);
while (
$row $xoopsDB->fetchArray($ret)){
    
$url[] = array("loc" => XOOPS_URL.'/modules/myalbum/photo.php?lid='.$row['lid']."&amp;".'cid='.$row['cid'],
                   
"lastmod" => date('Y-m-d',$row['date']),
                   
"changefreq" => "monthly",
                   
"priority" => "1");
}

$ret $xoopsDB->query($query[4]);
while (
$row $xoopsDB->fetchArray($ret)){
    
$url[] = array("loc" => XOOPS_URL.'/modules/news/article.php?storyid='.$row['storyid'],
                   
"lastmod" => date('Y-m-d',$row['date']),
                   
"changefreq" => "monthly",
                   
"priority" => "1");
}

?>
<? 
echo "<?xml version="1.0" encoding="UTF-8"?>n"?>
  <urlset xmlns="http://www.google.com/schemas/sitemap/0.84">

  <?php
   
     
for($f=0;$f<count($url);$f++){
    
?>
   <url>
    <loc><? echo $url[$f]['loc'];?></loc>
    <lastmod><? echo $url[$f]['lastmod'];?></lastmod>
    <changefreq><? echo $url[$f]['changefreq'];?></changefreq>
    <priority><? echo $url[$f]['priority'];?></priority>
   </url>
    <?php
    
?>
  </urlset>
Resized Image
www.ohloh.net/accounts/226400

Follow, Like & Read:-

twitter.com/SimonXaies
github.com/Chronolabs-Cooperative
facebook.com/SimonSXaies

Chappy

Friend of XOOPS
Posted on: 2008/1/1 4:36
Chappy
Chappy (Show more)
Friend of XOOPS
Posts: 456
Since: 2002/12/14
#2

Re: backend.php is good but what about a sitemap.php?

I use the sitemap module from here. It does take a little adaptation but it serves my purposes pretty well. It would be nice if a standardized approach was adopted more as part of the core, however.
MMM...It tastes like chicken! ...

script_fu

Friend of XOOPS
Posted on: 2008/1/1 5:21
script_fu
script_fu (Show more)
Friend of XOOPS
Posts: 1494
Since: 2002/12/27
#3

Re: backend.php is good but what about a sitemap.php?

I have a few questions about this so I understand better. Here is a link to a google search.

Sitepoint

When looking at the result on google for sitepoint you can see how the different sections of the site are broken down.

Is this what your code refers to? Would this be the answer to have our content displayed like this with xoops?


Examples
-----
Marketplace
Forums
Design Contests
Books

Recent Articles
Nifty Navigation Using CSS
Php Tutorials
Kits

script_fu

Friend of XOOPS
Posted on: 2008/1/1 5:40
script_fu
script_fu (Show more)
Friend of XOOPS
Posts: 1494
Since: 2002/12/27
#4

Re: backend.php is good but what about a sitemap.php?

I just wanted to add this search for your style of site thru google.

VJ Clips Previews

Its #1 on google for that search term. Now I do not see where google breaks down your site into sections.

I assume you just applied this hack so we wouldn't see the results in google yet.

I am hoping that your code would produce this feature. We can check in later google results if this is what you are proposing.

I might be wrong about what your trying to do. If so maybe somebody else will come along and let us know how to do this.

Thanks for a very good post!

wishcraft

Module Developer
Posted on: 2008/1/1 10:25
wishcraft
wishcraft (Show more)
Module Developer
Posts: 3710
Since: 2007/5/18
#5

Bot Sitemap vs Person Sitemap

Ok I noticed that some people have a HTML Clicky sitemap confused for a robot sitemap.. if you check www.sitemap.org it will give the specifications for this ISO Standardised document used by search engines.

I think that sitemap module is for a drill down for people to use, the sitemap I am talking about here it for googlebot and yahoo crawler etc, they read this file so they know what index they need to check next and keep records off it.. You may have noticed in google web master tools a function to add a sitemap or in yahoo siteexplorer.

These where invented to reduce crawl times, and do add to your SEO Score btw. You will notice sites with them get a higher ranking than sites without one.

There are alot of tools to build these as XML files for a static content site, but as XOOPS is a portal you will have to have a dynamic php file that set the header to an XML type.

Later Yall!!
Resized Image
www.ohloh.net/accounts/226400

Follow, Like & Read:-

twitter.com/SimonXaies
github.com/Chronolabs-Cooperative
facebook.com/SimonSXaies

wishcraft

Module Developer
Posted on: 2008/1/1 10:37
wishcraft
wishcraft (Show more)
Module Developer
Posts: 3710
Since: 2007/5/18
#6

Re: backend.php is good but what about a sitemap.php?

btw. Script_Fu, that extra content you find in google is added by google staff, you cannot purchase it and it cannot be added.. They put it in when they feel the requirements are met..

But having a sitemap is part of the requirements.
Resized Image
www.ohloh.net/accounts/226400

Follow, Like & Read:-

twitter.com/SimonXaies
github.com/Chronolabs-Cooperative
facebook.com/SimonSXaies

script_fu

Friend of XOOPS
Posted on: 2008/1/1 17:33
script_fu
script_fu (Show more)
Friend of XOOPS
Posts: 1494
Since: 2002/12/27
#7

Re: backend.php is good but what about a sitemap.php?

You have brought up a very important topic about SEO, Sitemaps & Xoops. I have never heard about sitemap.org before. I hope that your code will be looked at for a start to meeting this requirement that XOOPS and we need as users.



Again thank you for the awesome topic you started.

Chappy

Friend of XOOPS
Posted on: 2008/1/2 2:07
Chappy
Chappy (Show more)
Friend of XOOPS
Posts: 456
Since: 2002/12/14
#8

Re: backend.php is good but what about a sitemap.php?

Actually, with a little adaptation, the sitemap module does work with Google Sitemaps. Mine was read by Googlebot 15 hours ago. It creates xml_google.php for you, which Googlebot reads. IT does also give a list of the various pages, module by module on your site (which you can control via preferences).

For whatever reason, it seemed I struggled with getting it working, though, for a while. The sitemap protocol was a little confusing to me.

Having written that, your recent posts have been very helpful and creative.
MMM...It tastes like chicken! ...

chenasona

Just popping in
Posted on: 2008/11/22 14:21
chenasona
chenasona (Show more)
Just popping in
Posts: 11
Since: 2008/8/22
#9

Re: backend.php is good but what about a sitemap.php?

Great Topic, Thanks To All.
But Whats the conclusion?
How to make the google friendly sitemap?

voltan

Theme Designer
Posted on: 2008/11/22 16:25
voltan
voltan (Show more)
Theme Designer
Posts: 724
Since: 2006/12/5
#10

Re: backend.php is good but what about a sitemap.php?

are you see sitemap.php for d3pipes module ?

its very useful in search and google rank