World: r3wp

Ported my atom functions to Redis.. this thing is BLAZING  100,000 
reads/ sec   100,000 writes/sec   95% of my functions work at O(1)

Now all we need is a Cheyenne protocol :)

I'm thinking about porting  Redis to Rebol as a datastore for Cheyenne, 
but I'm not sure of performance.. 

Redis is a key/value db like CouchDB, with a number of nice features 
(persistence etc.)


Now in Redis, I can create a db with 120,000 key / value pairs.. 
keys are integers from 1 - 120,000 (in this example) with a 10 byte 
string as the value

In redis, i can do 10,000 GETS (return the value of the key) in 1.39 
seconds or 7200 / second.


Now trying to model this in Rebol, i used R3 on a newer machine (i7 
/ 6gb ram) with a block of integers from 1 - 120,000

PICK is blazing:  10, 000 picks in .011 seconds
but with SELECT it's 10,000 GETS in 2.23 seconds


My question is..  is there a faster way in Rebol to select a value 
from a key?

did you try using a map?

nope

that uses a hash, and should be MUCH faster for selects.

but I'm not sure if you can use integers for the keys.  ':-/

any map example code kicking around?

don't need int

don't know... haven't played with many new features of R3...

me neither

http://reboltutorial.com/blog/map-reduce-functions-in-rebol-towards-massive-parallel-functional-programming-part-i/

MAP is fairly easy.

So is brain surgery if you've been studying it for 10 years

Do you have an map-each example to try the problem above Henrik?

oh, you want examples?

Just one will do :)

that shows key/values pairs using map-each

>> a: make map! []
== make map! [
]

>> a/test: true
== true

>> a
== make map! [
    test true
]

>> a/test
== true

>> a/test: none
== none

>> a
== make map! [
]

that's basically it

you can of course use any value instead of true.

not sure how to apply that to map-each, though.

ok.. loaded up my map.. anyway to probe this thing?

(should talk elsewhere guys)

moving to core

(back from core :)

some nice results with map!


>> length? n  ; n is a map containing random keys, and a 25 byte 
string value
== 2000000


loadit 1000000  ; loads in an additional 1m key value pairs (same 
as others, and returns execution time
== 0:00:9.27  


test3 1000000   ; test3 is a function that GETS the value of a particular 
key 1M times, returns the last iteration's value (!string),  returns 
execution time
this is a bit of a string
0:00:01.616

That is blazing, nearly 10 times faster than Redis

NoSQL for Cheyenne!

wrote the data of that example to a text file.. 117mb

Be aware that the keys for Map! are case insensitive so that "Terry", 
"TERRY" and "terry" wil all refer to the same entry in the Map!

If case really matters to you for maps, you might want to use binary! 
keys - it's the most efficient way.

If you are serving multiple virtual sites and using stunnel, can 
you have a certificate for each virtual domain?

AFAIK, a certificate is only valid for one FQDN (http://en.wikipedia.org/wiki/Fully_qualified_domain_name).

Terry: map! uses a hash map, you would have similar results using 
hash! in R2. A hash map is as far from a NoSQL database as, IMHO, 
a NoSQL database is from a RDBMS ;-).

Yeah, was seeing if R3 was any faster.

A hash map is far from a NoSQL DB.. unless you're magic. I have a 
method that's working fine, now looking for the fastest key/value 
datastore i can find that's easy to work with.

Rebol is promising here.. 10x faster than Redis, but Redis is ready 
to go, gaining popularity, and open source.

<rant>One thing Cheyenne (and Rebol) needs to do is grow up, and 
lose Altme as the primary source of communication</rant>


AltMe is convenient because almost all Cheyenne users come here for 
realtime chatting, but I agree on your comment. Adding a web forum 
to the Cheyenne web site would be a good thing.

Doc, how do you think Cheyenne would stack up against Nodejs in benchmarks.. 
particularly  handling 1000s of simulaneous clients?

Terry, what Redis usage are you comparing to?

Same-process R3 map! vs Redis over the network?

redis you're forced over the network, if you coupled map! with Cheyenne, 
there's no such issue. Which is why Im curious as to Cheyennes benchmarks.

in other words, would Cheyenne + map! be faster than say Nodejs + 
Redis?  .. without having to spend the day testing

Nodejs: I'm not sure if JS in V8 executes much faster than R2, I 
guess that V8's JIT would give it a significant raw speed advantage 
other Cheyenne. On the scalability side, Nodejs will scale much better 
than Cheyenne for a high number of concurrent connections, just because 
it can use polling and kernel queues instead of the non-scalable 
select( ) used in WAIT by REBOL. This is an important aspect if long 
lasting connections are used.

by "long lasting" would that include websockets?

Yes.

But for a hundred of concurrent connections, I'm not sure that it 
would matter that much, both JS and REBOL have their own internals 
overheads, so I guess that the select( ) bottleneck would really 
start to show up when reaching several thousands concurrent connections 
(cf the "C10k problem").

Kaj: "not exists? incoming-dir/:out" => thanks, that was a nasty 
one! I've commited your fix in svn.

Can Cheyenne handle 10K clients?

Kaj: "Cheyenne doesn't seem to load its MIME definitions from the 
standard location on Unix systems, %/etc/mime.types, loading an internal 
list %misc/mime.types instead"


Yes, it was done that way to ensure a common base of mime-types across 
all OSes. Anyway, I have in my todo list an entry for such a feature. 
I'm not yet sure if it should be by default, or controled by an explicit 
option.

Kaj: "From earlier messages here, it was my understanding that Cheyenne 
on Linux can run its UniServe taskmaster processes under a non-root 
user account by enabling 'userdir in the modules and 'user and 'group 
in the globals. However, when I do that, all of Cheyenne is still 
running as root. The designated user and group exist. What else do 
I need to do?"


It was working last time I've checked. What Cheyenne version are 
you using? Have you remove all the log files generated by Cheyenne 
that are own by root before testing? Did you tried with other combinations 
of user and group?

Terry: short anwser: yes and no. Long answer: you should be more 
precise in your question. I guess that you're asking if a *single* 
Cheyenne instance can handle 10k concurrent connections on long lasting 
communications (HTTP keepalive or web sockets), the answer is : I 
don't know, I never tested that case. My guess is that if all clients 
are not sending requests simultaneously, Cheyenne should be able 
to handle it, but with a probably significant latency in the responses. 
For a real world web site, when dealing with short lasting requests, 
having a continuous load of 10k connections would mean that you're 
having a few million hits per hour making your web site one of the 
top 50 sites in the world, giving you enough money to invest in a 
huge cluster of servers to nicely handle the load with dozens of 
Cheyenne's instances ;-).