Mailing List Archive: 49091 messages
  • Home
  • Script library
  • AltME Archive
  • Mailing list
  • Articles Index
  • Site search

[REBOL] Having fun with msnbot (was: Outage)

From: SunandaDH::aol::com at: 2-Jul-2004 6:22

> The msnbot works hard on indexing the whole of We've had
> msnbot around every 10 seconds for about two months now. This is no problem > for the server, but could turn out to be one for the bot. The oops server
> only got about 10 different documents. The stuff that the bot fetches, is > parsed through the distorter ( And so it seems msn
> aiming at downloading the whole internet through the distorter.
MSNbos isn't the only misbehaved entity out there that might come and suck your website and its bandwidth dry. has just been attacked (it seems the appropriate word) by HTTRACK -- an off-line viewer for websites. It decided to download nearly half-a-meg a minute for two and a half hours. I guess the good news is that it didn't crash -- so the zombie bug is not *entirely* related to volume of activity. Like the Altme recycle bug, it's exact etiology remains a mystery.
> Maybe I ought to forbid the msnbot throught the robots.txt file. But then > again...
At least you can now preview MSNbot's search pages: It looks like a throwback to the days before Google, when SEs indexed strictly according to the words on the page. Can only get better, I hope. Sunanda.