1
mariuss
robots.txt is blocking archive.org
  • 2004/3/9 5:48

  • mariuss

  • Just popping in

  • Posts: 99

  • Since: 2002/9/6 5


It seems that the robots.txt file shipped with XOOPS 2 is preventing archive.org (aka Wayback Machine) from archiving your XOOPS site.

I don't know what the problem is but all the sites using XOOPS that I created stopped being archived after XOOPS 2 was installed. After the installation the only thing that was archived for a while is robots.txt, after that nothing.

As an example have a look at the www.xoops.org archives:
http://web.archive.org/web/*/www.xoops.org

As you can see the archiving stopped on June 12, 2003.

The robots.txt of this site:
http://web.archive.org/web/*/www.xoops.org/robots.txt

was archived until November 10, 2003.

For now I renamed robots.txt on all of my XOOPS sites.

Any clues what could be wrong?

Thanks,
Marius


Login

Who's Online

247 user(s) are online (153 user(s) are browsing Support Forums)


Members: 0


Guests: 247


more...

Donat-O-Meter

Stats
Goal: $100.00
Due Date: May 31
Gross Amount: $0.00
Net Balance: $0.00
Left to go: $100.00
Make donations with PayPal!

Latest GitHub Commits