[REBOL] Re: [ANN] (but what should I call it?)
From: hallvard:ystad:helpinhand at: 11-May-2003 15:30
Dixit Andreas Bolka (14.20 11.05.2003):
>does it obey robots.txt? if yes, what is its agend id? if no, why not?
It reads robots.txt, but doesn't obey yet. I plan to of course, but haven't gotten that
far. For the moment, it excludes anything that smells like dynamically created pages
(cgi-bin, .php, .r, .asp, .jsp, .pl, .cfm ...) I just couldn't resist posting this now,
even though some work is yet to be done with it...
Agent ID? I haven't decided on its name yet. Propositions are welcome.