Mailing List Archive: 49091 messages
  • Home
  • Script library
  • AltME Archive
  • Mailing list
  • Articles Index
  • Site search

[REBOL] Re: [ANN] (but what should I call it?)

From: hallvard:ystad:helpinhand at: 11-May-2003 15:30

Dixit Andreas Bolka (14.20 11.05.2003):
>does it obey robots.txt? if yes, what is its agend id? if no, why not?
It reads robots.txt, but doesn't obey yet. I plan to of course, but haven't gotten that far. For the moment, it excludes anything that smells like dynamically created pages (cgi-bin, .php, .r, .asp, .jsp, .pl, .cfm ...) I just couldn't resist posting this now, even though some work is yet to be done with it... Agent ID? I haven't decided on its name yet. Propositions are welcome. ~H