Rugby performance starting to fade ...
[1/4] from: petr:krenzelok:trz:cz at: 19-Feb-2002 10:38
Hi,
as there is some problem with tcp performance on W9x machines (tests for
100x call to echo function is some 44 sec regardless of machine being
P300 or P650), I gave Rugby a test on W2K, 1.4 Athlon XP machine ....
Could some of you, help here with some profiling, please?
console1:
do %rugby.r
serve/with [echo] tcp://:8005
console2:
do %rugby.r
do get-rugby-service http://localhost:8005
s: now/time/precise loop 100 [echo "test"] print now/time/precise - s
I received the performance of some 8 - 9 sec, while older Rugby 4.3
gives 0.59 sec on the some machine! Now, - I am doing something wrong,
or just new version of Rugby have to use some more complexity .... but
the performance drop is, imo, too big to be acceptable, if changes are
not significant ...
So, actually, Maarten - have you performed any benchmarking yourself? I
looked at some sources, and while you refused my proposed unified
/secure refinement usage, to have simply secure and non-secure (default)
behavior separated, you seem to mix those two levels in 'chain function
(too much 'either function usage), and you also add complexity to
compose/decompose message functions, because of your new dialect.
Most ppl using Rugby surely do regard Rugby being a cool product, but we
should be probably more carefull with sudden changes.
So, - what is the reason of a slowdown?
I also performed following simple tests, so, could too much function
call granularity or conditions cause any performance hit in practice? (I
performed each test twice, to avoid random result times)
->> huh: "ble"
== "ble"
->> s: now/time/precise loop 1'000'000 [c: huh] now/time/precise - s
== 0:00:00.77
->> s: now/time/precise loop 1'000'000 [c: huh] now/time/precise - s
== 0:00:00.77
->> s: now/time/precise loop 1'000'000 [either huh [c: huh][c: huh]]
now/time/precise - s
== 0:00:02.96
->> s: now/time/precise loop 1'000'000 [either huh [c: huh][c: huh]]
now/time/precise - s
== 0:00:02.96
->> s: now/time/precise loop 1'000'000 [either huh [c: huh][c: huh]]
now/time/precise - s
== 0:00:02.97
->> f: does [c: huh]
->> s: now/time/precise loop 1'000'000 [f] now/time/precise - s
== 0:00:01.54
->> s: now/time/precise loop 1'000'000 [f] now/time/precise - s
== 0:00:01.59
btw: is there enought ppl intereste in/using Rugby, to create separate
yahoo group for it? Maybe we could discuss it more deeply in there. But
then our main Rugby guru would have to agree he has good intentions to
keep Rugby going ;-)
Anyone?
-pekr-
[2/4] from: brett:codeconscious at: 20-Feb-2002 23:32
Hi Petr, Maarten and other ladies and gentlemen,
> as there is some problem with tcp performance on W9x machines (tests for
> 100x call to echo function is some 44 sec regardless of machine being
> P300 or P650), I gave Rugby a test on W2K, 1.4 Athlon XP machine ....
>
> Could some of you, help here with some profiling, please?
I've run some tests with a Rugby I downloaded on 18-Feb-02. Be warned though
these tests were not rigourous.
Tests were run with View which was stupid (memory concerns), but I started
that way and was not going to go over
it again. Besides I'm not well and should be in bed :).
Conclusion: Win9x throttled output of Rugby to normally less than 2 requests
per second, *but* was capable of serving more if more clients attached. The
2 requests per second is close to what Petr reported.
Petr you might want to try doing a multiple client script like I have to see
if this is reflected on your machine.
Tests performed using Petr's server script with a modified client script
(both included below). The tests took time so
I did not run many. The numbers therefore jump around a bit. The machine IPs
were used for the network tests.
I tried this stuff on three machines - four environments (the p300 was dual
booted). Here are the configurations and the approximate test results in
requests per second in the brackets:
===test: localhost / single server / single client (Req/Sec)
The telling result here is the dual booted p300...
486 laptop 20Mb ?mhz W95 [1.7 - 1.73]
P150 48Mb W98 [1.67 - 1.7]
P300 96Mb W98 [1.67 - 1.7]
P300 96Mb WNT4 [38 - 41]
===test: machine-to-machine / single server / single client (Req/Sec)
These machine to machine tests were carried over a quiet network through a
switch. Client listed first. Unfortunately,
and I don't know why, I couldn't get NT to serve the W9x clients (the
clients could not find it) - probably a screw up
in the networking configuration. Could invalidate the results.. but oh
well...
These tests were not repeated.
486W95 -> P150W98 [2.26]
P300Nt4 -> P150W98 [1.93]
P150W98 -> 486W95 [1.7]
P300Nt4 -> 486W95 [1.76]
===test: single server / multiple clients (Req/Sec)
This is a more interesting test. It shows that the W9x machines could server
more requests but are
effectively throttled when talking to a particular client. I basically
started multple client sessions
on the p300Nt4 box.
Starting the clients involves bashing the enter key multiple times quickly
so the timing is not brilliant but using a test duration of 1 minute should
be ok (which is what I did).
---Network test.
10x P300Nt4 -> 486W95 [0.9 - 0.95]
Multiply up and the 486 was really delivering about 9 requests per second.
Go the 486!
---Network test.
13x P300Nt4 -> P150W98 [1.3 - 1.5]
(1 client failed Vxd error on server)
Multiply up and the P150 was really delivering over 15 requests per second.
---Localhost tests
5x P300Nt4 -> P300Nt4 [7.3 - 8.1]
Multiply up and the P300Nt4 was really delivering about 35 - 40 requests per
second which is consistent with the single server localhost test. I tried
this particular test with other number of clients (11, 13). The results were
consistent with
about 35 - 40 requests per second once you took all the clients together.
10x P300W98 -> P300W98 [1.63]
I suspect that all the clients would be limited to about 1.65 in this
environment unless they number more than 20. But
I could not test this because I could not practically manage to set off over
20 clients!
===Scripts
---Server script
REBOL []
do %rugby.r
serve/with [echo] tcp://:8005
---Client script
REBOL []
do %rugby.r
do get-rugby-service http://localhost:8005
end-time: add 0:00:30 start-time: now/precise
req: 0
until [req: req + 1 echo "test" greater? now end-time]
print ["Duration (s)" duration: to-decimal (end-time/time -
start-time/time)]
print ["Requests" req]
print ["Requests/seq" req / duration]
halt
===Regards :)
If no one blows my tests out of the water, then I feel like I've learnt
something.
I hope it is of some use.
Regards
Brett.
[3/4] from: petr:krenzelok:trz:cz at: 20-Feb-2002 14:56
Thanks Brett, appreciated! One small observation btw:
W9x to W9x machine - 40 sec
W9x to W2K machine - 20 sec
There is really something wrong with W9x, that's my conclusion ...
-pekr-
[4/4] from: koopmans:itr:ing:nl at: 20-Feb-2002 15:46
Hi Brett,
What you are reporting is (IMO) that clients seem to be the limiting factor
here (as is also the case when you go though a proxy).
Thanks,
Maarten
On Wednesday 20 February 2002 13:32, you wrote: