XOOPS 
Forum
Forum Index General Forums Community Support Modules Support Themes Support Development International Support Module Hack Reviews Theme Designer Talk
  1. Board index / 
  2. XOOPS Community Support forums / 
  3. Q and A 
  4.  / xoops robots.txt and google
Register
1
sarahmx
xoops robots.txt and google

  • 2010/6/19 1:05

  • sarahmx

  • Quite a regular

  • Posts: 381

  • Since: 2007/10/28


i tried adding sitemaps in Google Webmaster Tools

modules/sitemap/xml_google.php
modules/publisher/backend.php
modules/study/backend.php


but all received this status
Quote:

We were unable to crawl your Sitemap because we found a robots.txt file at the root of your site but were unable to download it. Please ensure that it is accessible or remove it completely.


i can access the robots.txt in my browser

how to fix this ?

TQ
2
Peekay
Re: xoops robots.txt and google

  • 2010/6/19 1:14

  • Peekay

  • XOOPS is my life!

  • Posts: 2335

  • Since: 2004/11/20


I would Google the pros and cons of sitemaps before you submit one.
A thread is for life. Not just for Christmas.
3
sarahmx
Re: xoops robots.txt and google

  • 2010/6/19 1:37

  • sarahmx

  • Quite a regular

  • Posts: 381

  • Since: 2007/10/28


Hi peekay

do you have any experience regarding sitemap and google ? mind to share...

4
bjuti
Re: xoops robots.txt and google

  • 2010/6/19 9:23

  • bjuti

  • Just can't stay away

  • Posts: 871

  • Since: 2009/1/7 2


Hm.. my experience is that google reads robots.txt. I have deny modules/smartsection/ in case of duplicate content possible when I use url rewrite.

About sitemaps...

Use sitemap generator, there is plenty of it online. Submit it to Google, and then submint rss feed also. But the most elegant solution is to use rss module for it, so include modules/rss/rss.php.

Works for me :)
5
Peekay
Re: xoops robots.txt and google

  • 2010/6/19 13:02

  • Peekay

  • XOOPS is my life!

  • Posts: 2335

  • Since: 2004/11/20


@sarahmx

A few years ago there was some controversy about Google and site maps because in some cases submitting one could actually cause a loss of traffic, rather than an increase.

I would hope that is no longer the case, but from a quick search it appears some SEO experts still hold the opinion that XML sitemaps are only really beneficial if you need spiders to find pages that are not part of the normal site navigation. In most sites, the normal internal links are sufficient to get all the pages indexed.

When I tried it on a site myself it didn't do any harm, but I saw no benefit either.

It just meant I had to do more work!

A thread is for life. Not just for Christmas.
6
sarahmx
Re: xoops robots.txt and google

  • 2010/6/19 13:14

  • sarahmx

  • Quite a regular

  • Posts: 381

  • Since: 2007/10/28


Bjuti:
Google still says my robots.txt unreachable..so it can't verify my rss feed and xml_google.php (sitemap)

Peekay:
thank you for sharing..

I'm just experimenting right now with sitemap and search engine




7
Runeher
Re: xoops robots.txt and google

  • 2010/6/19 13:43

  • Runeher

  • Module Developer

  • Posts: 825

  • Since: 2008/1/24


Could be wrong, but aren't you suppose to put xml_google.php in xoops root?
8
Peekay
Re: xoops robots.txt and google

  • 2010/6/19 13:55

  • Peekay

  • XOOPS is my life!

  • Posts: 2335

  • Since: 2004/11/20


You should have just one robots.txt file on the site and it needs to be in the root directory.

It's just a txt file, so you should be able to open it in a browser. For example, here's the one for Xoops.org

Quote:
User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /cache/
Disallow: /class/
Disallow: /images/
Disallow: /include/
Disallow: /install/
Disallow: /kernel/
Disallow: /language/
Disallow: /templates_c/
Disallow: /themes/
Disallow: /uploads/

I believe runeher is right, the sitemap file also needs to be in the root.
A thread is for life. Not just for Christmas.
9
sarahmx
Re: xoops robots.txt and google

  • 2010/6/19 14:28

  • sarahmx

  • Quite a regular

  • Posts: 381

  • Since: 2007/10/28


i have the xml_google.php in root and also in modules/sitemap/

same problem


i checked in cpanel filemanager that i have two robots.txt

a. the default that comes with xoops installation
b. and in modules/xobile/ - when anyone use handphone to browse my page it redirects to this module


should i delete the robots.txt in xobile folder ?
10
sarahmx
Re: xoops robots.txt and google

  • 2010/6/19 14:31

  • sarahmx

  • Quite a regular

  • Posts: 381

  • Since: 2007/10/28


Quote:

Crawl errors

Issues Google encountered when crawling your site.

Unreachable ‎robots.txt unreachable



Does this mean google bot can't crawl in my site ?
(1) 2 »

Login

Register now!  |  Lost Password?

Search

Advanced Search

Recent Posts

Who's Online

361 user(s) are online (259 user(s) are browsing Support Forums)

Members: 0

Guests: 361

more...

Donat-O-Meter

Stats
Goal: $100.00
Due Date: Nov 30
Gross Amount: $0.00
Net Balance: $0.00
Left to go: $100.00
Make donations with PayPal!

Latest GitHub Commits