[REBOL] Re: [ANN] (but what should I call it?)
From: andreas:bolka:gmx at: 11-May-2003 13:20
Sunday, May 11, 2003, 11:18:30 AM, Hallvard wrote:
> We don't really need this - there are enough search engines for the
> web as it is. Still - I made a little robot that searches the web
> for pages containing the word "rebol".
does it obey robots.txt? if yes, what is its agend id? if no, why not?
(see eg. http://www.searchengineworld.com/robots/robots_tutorial.htm
or http://www.robotstxt.org/wc/norobots.html)
--
Best regards,
Andreas mailto:[andreas--bolka--gmx--net]