Mailing List Archive: 49091 messages
  • Home
  • Script library
  • AltME Archive
  • Mailing list
  • Articles Index
  • Site search
 

[REBOL] Re: [ANN] (but what should I call it?)

From: andreas:bolka:gmx at: 11-May-2003 13:20

Sunday, May 11, 2003, 11:18:30 AM, Hallvard wrote:
> We don't really need this - there are enough search engines for the > web as it is. Still - I made a little robot that searches the web > for pages containing the word "rebol".
does it obey robots.txt? if yes, what is its agend id? if no, why not? (see eg. http://www.searchengineworld.com/robots/robots_tutorial.htm or http://www.robotstxt.org/wc/norobots.html) -- Best regards, Andreas mailto:[andreas--bolka--gmx--net]