[REBOL] Re: Rebol indexer
From: SunandaDH::aol::com at: 25-Aug-2003 23:19
Hallvard:
> The robots.txt standard is unclear about one thing. Is this allowed:
> user-agent: someAgent
> user:agent: someOtherAgent
> user-agent: someOtherAgentsAunt
> disallow: /
>
> Or must it be like this:
> user-agent: someAgent
> disallow: /
>
> user:agent: someOtherAgent
> disallow: /
Both formats are acceptable according to the robots standard.
The first is good as it reduces bloat -- the file is half the size.
The second is good as some robots are badly written and won't handle the
first format.
If your code can handle it either way, then you have a good robot,
Sunanda.
