[REBOL] Having fun with msnbot (was: REBOL.org Outage)
From: hallvard:ystad:oops-as:no at: 30-Jun-2004 13:16
Dixit [SunandaDH--aol--com] (00.14 27.06.2004):
>One problem any site may be having right now is the overly-aggressive msnbot
>-- it can drain a month's worth of bandwidth from an innocent site in a single
>day by compulsive repeated spidering of the same page.
>And all to no useful effect -- there is no publicly available search engine
>for msnbot results as yet.
The msnbot works hard on indexing the whole of www.oops-as.no. We've had the msnbot around
every 10 seconds for about two months now. This is no problem for the server, but could
turn out to be one for the bot. The oops server has only got about 10 different documents.
The stuff that the bot fetches, is parsed through the distorter (www.oops-as.no/roy/dis).
And so it seems msn is aiming at downloading the whole internet through the distorter.
Maybe I ought to forbid the msnbot throught the robots.txt file. But then again...