3
alexa and Yandex bot not included in robot text.
and some robot in list not from most used search engine.
try to block anonymous view but let the bots crawl it
and the problems
Fetch as Googlebot
« Go back
This is how Googlebot fetched the page.
URL: http://www.mysiteexample/modules/xnews/article.php?storyid=413
Date: Monday, October 17, 2011 10:21:38 AM PDT
Googlebot Type: Web
Download Time (in milliseconds): 480
HTTP/1.1 302 Moved Temporarily
Date: Mon, 17 Oct 2011 17:21:34 GMT
Server: Apache
X-Powered-By: PHP/5.2.17
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Set-Cookie: PHPSESSID=dfc506708b1e00e1173a42d9d11b7be5; path=/
Location: http://www.mysite/user.php?from=xnews
Vary: User-Agent,Accept-Encoding
Content-Encoding: gzip
Content-Length: 20
Connection: close
Content-Type: text/html