Mailing List Archive: 49091 messages
  • Home
  • Script library
  • AltME Archive
  • Mailing list
  • Articles Index
  • Site search

[REBOL] Re: Rebol web presence statistics

From: hallvard:ystad:oops-as:no at: 20-Mar-2004 20:40

Dixit Carl Read (02.42 20.03.2004):
>On 20-Mar-04, Hallvard Ystad wrote: >> Oh yes. And it all started with in the >> beginning. The bot checks all links _from_ pages that contain the >> word "rebol". Pages that do not contain the word does not get their >> links checked. It's amazing how many URLs have lined up on the >> checklist. 5000 records per day is my choice. If the bot ran all the >> time, it could reach about 15000-25000, I guess, so maybe that's >> worth a try for a period of time. > >But a bit rough on your bot! :-)
Actually, no. The load on the bot server is nothing much. I'm more concerned for the servers that it visits. I make sure no server is visited more often than every 20 seconds, (but I have no check for virtual hosts...), but places like, and really fill up the list, so I suspect I'd bug them if I were to index at high speed.
>Perhaps the "submitted by hand" URLs could be put to the top of the >list? It's unlikely you'll get many a day and so it shouldn't make >much of an impact on your searching, but it would be a bit of >encouragement for the people who take the trouble to add an URL.
Yes, that might just be a good idea. If not on top of search results, they at least could be given som extra visibility, like with the text "User recommended site" or something similar. I already try to register which links are followed by users, but haven't yet decided if I want to use that information for search results (could be ranked higher?). The only use I make of it for now, is the displaying of the five last clicked links on the rix start page: Thanks for ideas. They don't exactly give me more sleep, but more will be just as welcome anyway. HY