[REBOL] Re: [REBOL.org] Outage
From: SunandaDH::aol::com at: 26-Jun-2004 18:14
Gabriele:
> I think the real problem is that only one of the processes
> actually terminates. Wild guess --- this could be related to the
> sysport bug, one of the processes hangs after getting a signal.
> (REBOL starts two processes on Unix.)
Thanks. I didn't know about the two processes.....It certainly looks like one
of them (at least) doesn't terminate under some conditions. But most days,
it's fine.
Theories vary about why, but a common theme is that failure is triggered when
the server is under a heavier than usual load.
On our failure day, we had twice as many messages as on an average day.
So, maybe, the problem is (also) related to having two or more independent
REBOL CGIs running at once -- maybe they deadlock on trying to access some
common resource.
If so, it puts an uncomfortably low ceiling on the number of messages we can
handle in one day -- at least until the problem is fixed.
Our ISP tech support say that the unterminated processes were doing
*something* -- not just sitting there hogging memory and resources, but also eating
lots of CPU cycles.
I've attempted to cure the symptom by banning bots that have been too
aggressive.
One problem any site may be having right now is the overly-aggressive msnbot
-- it can drain a month's worth of bandwidth from an innocent site in a single
day by compulsive repeated spidering of the same page.
And all to no useful effect -- there is no publicly available search engine
for msnbot results as yet.
msnbot supports the non-standard robots.txt command, crawl-delay. I'd
recommend anyone with a website to add something like this to their robots.txt, lest
msnbot comes and over-spiders you:
User-Agent: msnbot
Crawl-Delay: 20
The REBOL.org robots.txt bans many time-wasting bots. You can see a copy here:
http://www.rebol.org/robots.txt
Sunanda.
