robots.txt
[1/3] from: bry:itnisk at: 2-Dec-2003 18:35
I've never had to realy take robots.txt into
consideration before, as any site crawling
I've ever done has been in secluded sites
that I knew the structure of.
It seems from my understanding that google
uses a system of robots, the first robot
checks for robots.txt, makes decisions based
on what it finds, contacts other robots to
do realy work. This strikes me as a
reasonable way to proceed, does anyone have
any counter recommendations.
Also anyone have a robots.txt to object
script out there
already?
[2/3] from: hallvard:ystad:tokerud:gs:oslo:no at: 6-Dec-2003 0:15
Hi
You'll find a script to handle robots.txt files on my site:
http://folk.uio.no/hallvary/rebol/
Hope this helps,
Hallvard
[3/3] from: bry:itnisk at: 8-Dec-2003 10:22
thanks :)