[REBOL] Re: Rebol web presence statistics
From: hallvard:ystad:oops-as:no at: 20-Mar-2004 20:40
Dixit Carl Read (02.42 20.03.2004):
>On 20-Mar-04, Hallvard Ystad wrote:
>> Oh yes. And it all started with http://www.rebol.com/ in the
>> beginning. The bot checks all links _from_ pages that contain the
>> word "rebol". Pages that do not contain the word does not get their
>> links checked. It's amazing how many URLs have lined up on the
>> checklist. 5000 records per day is my choice. If the bot ran all the
>> time, it could reach about 15000-25000, I guess, so maybe that's
>> worth a try for a period of time.
>
>But a bit rough on your bot! :-)
Actually, no. The load on the bot server is nothing much. I'm more concerned for the
servers that it visits. I make sure no server is visited more often than every 20 seconds,
(but I have no check for virtual hosts...), but places like codeur.org, agora-dev.org
and compkarori.com really fill up the list, so I suspect I'd bug them if I were to index
at high speed.
>Perhaps the "submitted by hand" URLs could be put to the top of the
>list? It's unlikely you'll get many a day and so it shouldn't make
>much of an impact on your searching, but it would be a bit of
>encouragement for the people who take the trouble to add an URL.
Yes, that might just be a good idea. If not on top of search results, they at least could
be given som extra visibility, like with the text "User recommended site" or something
similar. I already try to register which links are followed by users, but haven't yet
decided if I want to use that information for search results (could be ranked higher?).
The only use I make of it for now, is the displaying of the five last clicked links on
the rix start page: http://www.oops-as.no/rix
Thanks for ideas. They don't exactly give me more sleep, but more will be just as welcome
anyway.
HY