Mailing List Archive: 49091 messages
  • Home
  • Script library
  • AltME Archive
  • Mailing list
  • Articles Index
  • Site search

[REBOL] Re: [] Outage

From: SunandaDH::aol::com at: 26-Jun-2004 18:14

> I think the real problem is that only one of the processes > actually terminates. Wild guess --- this could be related to the > sysport bug, one of the processes hangs after getting a signal. > (REBOL starts two processes on Unix.)
Thanks. I didn't know about the two processes.....It certainly looks like one of them (at least) doesn't terminate under some conditions. But most days, it's fine. Theories vary about why, but a common theme is that failure is triggered when the server is under a heavier than usual load. On our failure day, we had twice as many messages as on an average day. So, maybe, the problem is (also) related to having two or more independent REBOL CGIs running at once -- maybe they deadlock on trying to access some common resource. If so, it puts an uncomfortably low ceiling on the number of messages we can handle in one day -- at least until the problem is fixed. Our ISP tech support say that the unterminated processes were doing *something* -- not just sitting there hogging memory and resources, but also eating lots of CPU cycles. I've attempted to cure the symptom by banning bots that have been too aggressive. One problem any site may be having right now is the overly-aggressive msnbot -- it can drain a month's worth of bandwidth from an innocent site in a single day by compulsive repeated spidering of the same page. And all to no useful effect -- there is no publicly available search engine for msnbot results as yet. msnbot supports the non-standard robots.txt command, crawl-delay. I'd recommend anyone with a website to add something like this to their robots.txt, lest msnbot comes and over-spiders you: User-Agent: msnbot Crawl-Delay: 20 The robots.txt bans many time-wasting bots. You can see a copy here: Sunanda.