Mailing List Archive: 49091 messages
  • Home
  • Script library
  • AltME Archive
  • Mailing list
  • Articles Index
  • Site search


 [1/3] from: bry:itnisk at: 2-Dec-2003 18:35

I've never had to realy take robots.txt into consideration before, as any site crawling I've ever done has been in secluded sites that I knew the structure of. It seems from my understanding that google uses a system of robots, the first robot checks for robots.txt, makes decisions based on what it finds, contacts other robots to do realy work. This strikes me as a reasonable way to proceed, does anyone have any counter recommendations. Also anyone have a robots.txt to object script out there already?

 [2/3] from: hallvard:ystad:tokerud:gs:oslo:no at: 6-Dec-2003 0:15

Hi You'll find a script to handle robots.txt files on my site: Hope this helps, Hallvard

 [3/3] from: bry:itnisk at: 8-Dec-2003 10:22

thanks :)