Mailing List Archive: 49091 messages
  • Home
  • Script library
  • AltME Archive
  • Mailing list
  • Articles Index
  • Site search

[REBOL] Re: Rebol indexer

From: SunandaDH::aol::com at: 25-Aug-2003 23:19

> The robots.txt standard is unclear about one thing. Is this allowed: > user-agent: someAgent > user:agent: someOtherAgent > user-agent: someOtherAgentsAunt > disallow: / > > Or must it be like this: > user-agent: someAgent > disallow: / > > user:agent: someOtherAgent > disallow: /
Both formats are acceptable according to the robots standard. The first is good as it reduces bloat -- the file is half the size. The second is good as some robots are badly written and won't handle the first format. If your code can handle it either way, then you have a good robot, Sunanda.