The REBOL messaging engine

[1/7] from: maarten:vrijheid at: 11-Aug-2003 20:25

Hi All, I have been working on a messaging engine lately. It always seemed strange to me that REBOL was designed for semantic exchange but didn't have a send-message function on board. I have some cool stuff working, but I have some questions to my potential user-base ;-) I expect to have a beta out in a few weeks. Right now there is a server and a client part. To support a message type on the server-side you call one function: Add-message-type: func [ 'w [word!] handler [any-function!]][...] So you'd get: add-message-type echo func [ b [block!]][b] which would be an echo handler. Note that the handler function must expect a block (containing.... a message = dialect). The server engine is completely based on async I/O. The client side is one function as well: Send-message: func [ 'w [word!] u [url!] msg [block!] /deliver f [any-function!]][...] So you'd get: Send-message echo tcp://localhost:8000 [{Echo this}] This will wait for the answer and then return. Optionally you can do it asynchronously when your in an event loop (e.g. View): Send-message/deliver echo tcp://localhost:8000 [{Echo this}] :print This will continue execution immediately and when the result is completely back it will be delivered to the print function, much like events in View. Asynchronously! The idea is that you can easily implement simple protocols as different message types, with each type having its own dialect. For example, a P2P set of messages would be 'publish 'locate 'delete 'get with their handler functions. Or you could have a Rugby handler, or.... Now some questions: - Should I add (optionally) message encryption or leave that to implementors of messages, as their needs may vary? Pro: Ecnryption Con: only works with the SDK, Command, ... - Should I work a bit on the message format so we can port it to other languages as well (.NET, Python, etc.) or keep it REBOL-only and enhance it, for example integrating a new Rugby on top of this? - Is this API with server based handler functions a good concept or is it too hard for most programmers? - Should I add meta-data so you can query an engine to see what messages it udnerstands. If so, what should the meta-data look like? - Should I add authentication, so you can demand that a message must be properly authenticated? If so, I'll probably keep it pluggable, meaning that you can add your own authentication methods as well. - Should I transparently propagate errors back from server to client or just return none when an error occurs? It may be annoying to do a try with each send-message, OTOH seeing what goes wrong is worth something as well. - Any other suggestions? --Maarten

[2/7] from: andreas::bolka::gmx::net at: 11-Aug-2003 21:18

Monday, August 11, 2003, 8:25:43 PM, Maarten wrote:

> I have been working on a messaging engine lately.

Sounds good :) What's the difference to Rugby? AFAICS the underlying metaphor is rather not messaging but still RPC.

> - Should I add (optionally) message encryption or leave that to > implementors of messages, as their needs may vary? Pro: Ecnryption > Con: only works with the SDK, Command, ...

Certainly a nice feature, but I'd rather not add it now. Leave it to the implementors for a beginning, and if there is a real demand, add it later.

> - Should I work a bit on the message format so we can port it to > other languages as well (.NET, Python, etc.) or keep it REBOL-only > and enhance it, for example integrating a new Rugby on top of this?

+1 for keeping it REBOL-only. I'd love to see high-performance (!) inter-REBOL messaging, so I'd like to have a message format purely optimized for REBOL and optimized for speed :) On the other hand, if you can keep a good REBOL performance and _also_ rework the format to be more friendly for other languages,

> - Is this API with server based handler functions a good concept or > is it too hard for most programmers?

I like it. And as the handler-functions concept closely maps to messaging per se, I'd stay with it.

> - Should I add meta-data so you can query an engine to see what > messages it udnerstands. If so, what should the meta-data look like?

I'd rather implement that as custom protocol (message types). Any reasons for adding it as first-class?

> - Should I add authentication, so you can demand that a message must > be properly authenticated? If so, I'll probably keep it pluggable, > meaning that you can add your own authentication methods as well.

A nice feature too, but as with the encryption feature: keep the underlying protocol extensible and add it later.

> - Should I transparently propagate errors back from server to client > or just return none when an error occurs? It may be annoying to do a > try with each send-message, OTOH seeing what goes wrong is worth > something as well.

I'm undecided here :) Just based on instinct I tend towards not propagating errors ;)

> - Any other suggestions?

As mentioned earlier, I'd love to see this being high-performance stuff ;) -- Best regards, Andreas

[3/7] from: joel:neely:fedex at: 11-Aug-2003 16:06

Hi, Maarten, Nice work! Here are a few $0.02 answers to your questions... Maarten Koopmans wrote:

> Now some questions: > - Should I add (optionally) message encryption or leave that to > implementors of messages, as their needs may vary? Pro: Ecnryption > Con: only works with the SDK, Command, ... >

Leave it to the message implementors. Do the simplest thing that could possibly work. (It's always possible to complicate life at a later date! ;-)

Make basic message interchange language neutral, for the following reasons: - The trend of the industry is away from "lock-in" mechanisms (even if the pace is erratic!) - The most successful 'Net protocols are language neutral (HTTP, FTP, SMTP, ...). - If (as I assume) at least part of the goal is to strengthen the case for REBOL, I believe that it hurts the cause to send a message (pun intended) to the world that says "Here's another closed environment. Use our language so you can use our tools." Most developers won't make that investment without some significant (short-term) payoff. The best position for REBOL would be "Here's a transport that can be used by any language. Here are benefits of using that transport. OBTW, REBOL is demonstrably the best language for doing so." - Chicken and egg (Most developers won't take time to learn a new language/transport unless there's already something there. Most developers won't build something there unless they believe there will be something else later -- including users.)

> - Is this API with server based handler functions a good concept or > is it too hard for most programmers? >

Sounds elegant, clean, and easy.

> - Should I add meta-data so you can query an engine to see what > messages it udnerstands. If so, what should the meta-data look like? >

For a first cut, how about just a list of all the words for which handlers are bound?

> - Should I add authentication, so you can demand that a message > must be properly authenticated? If so, I'll probably keep it pluggable, > meaning that you can add your own authentication methods as well. >

Couldn't that be the responsibility of the handler?

Could there be a refinement to manage the error-handling strategy? For instance (better naming is certainly possible!): send-message ... to attempt and return NONE on failure, but send-message/try ... to attempt and return error on failure (in which case one certainly should wrap in a TRY block...) -jn-

[4/7] from: joel:neely:fedex at: 11-Aug-2003 16:07

[Possible dup, sorry if so, network glitch while sending] Hi, Maarten, Nice work! Here are a few $0.02 answers to your questions... Maarten Koopmans wrote:

Leave it to the message implementors. Do the simplest thing that could possibly work. (It's always possible to complicate life at a later date! ;-)

> - Is this API with server based handler functions a good concept or > is it too hard for most programmers? >

Sounds elegant, clean, and easy.

> - Should I add meta-data so you can query an engine to see what > messages it udnerstands. If so, what should the meta-data look like? >

For a first cut, how about just a list of all the words for which handlers are bound?

Couldn't that be the responsibility of the handler?

[5/7] from: petr:krenzelok:trz:cz at: 11-Aug-2003 23:41

Andreas Bolka wrote:

>Monday, August 11, 2003, 8:25:43 PM, Maarten wrote: > >>I have been working on a messaging engine lately. >> >> > >Sounds good :) What's the difference to Rugby? AFAICS the underlying >metaphor is rather not messaging but still RPC. >

good question. Rugby was good. It was not fully async, I didn't liked its way-too-long-func-names-without-refinements, but I had fund using it.

>>- Should I add (optionally) message encryption or leave that to >>implementors of messages, as their needs may vary? Pro: Ecnryption

<<quoted lines omitted: 4>>

>the implementors for a beginning, and if there is a real demand, add >it later.

what about rugby way once again? But please, this time - no more additional function names, use refinements, please "send-message/secure" :-)

>>- Should I work a bit on the message format so we can port it to >>other languages as well (.NET, Python, etc.) or keep it REBOL-only >>and enhance it, for example integrating a new Rugby on top of this? >> >> > >+1 for keeping it REBOL-only. >

+1 for Rebol only too. I don't want to sound selfish, but we desperately need rebol engines first, then we can go for further integration ...

>>- Is this API with server based handler functions a good concept or >>is it too hard for most programmers? >> >> > >I like it. And as the handler-functions concept closely maps to >messaging per se, I'd stay with it. >

no problem with handlers ... maybe I would: 1) exchange parameter ordering in following - Send-message echo tcp://localhost:8000 [{Echo this}] to send-message url echo [{Echo this}] .... to be compatible with native rebol ordering ... 2) use different refinementname instead of /deliver .... e.g. /callback, /on-return etc. ... both 1) and 2) are just implementation details though ...

>>- Should I add meta-data so you can query an engine to see what >>messages it udnerstands. If so, what should the meta-data look like? >> >> > >I'd rather implement that as custom protocol (message types). Any >reasons for adding it as first-class? >

well, we should really consider this one. With Rugby, you were able to query server, install remote functions locally etc. Maybe it could really be just a message type, but I would like to have such functionality available somewhere. The point is, that if we will let it upon message-type designers, we will end up with mixture of different aproaches ... it is the same as someone could say - why to build messaging engine at all, if rebol is about messaging in its roots? Well, it is, but we need some more unified way used by rebol developers to take our apps further ...

>As mentioned earlier, I'd love to see this being high-performance >stuff ;) >

Yes, I think I start to understand - new Maarten's thingy, whatever it is gonna be called, is just high-performance async networking message passing mechanism ... Maarten - will it use kind of green threads you introduced in Rugby or how will it divide its time between message-types? What if my message type handler does some more intensive stuff? Hmm, right now I am not even sure what exact usage will that stuff provide us with. I can understand you would like to add Rugby upon it ... but will it be general enough for any kind of protocol, so e.g. someone could create IRC protocol and plug-it into your messaging scheme? -pekr-

[6/7] from: maarten:vrijheid at: 12-Aug-2003 7:27

> -----Original Message----- > From: [rebol-bounce--rebol--com] [mailto:[rebol-bounce--rebol--com]] On Behalf

> Petr Krenzelok > Sent: maandag 11 augustus 2003 23:41

<<quoted lines omitted: 4>>

> >metaphor is rather not messaging but still RPC. > >

That's true, albeit a little less programmer friendly than Rugby. More powerful though.

> > > good question. Rugby was good. It was not fully async, I didn't liked > its way-too-long-func-names-without-refinements, but I had fund using

it.

Yes, but as it turned out the /deferred stuff isn't working (anymore). So I'm doing async now.

> >+1 for keeping it REBOL-only. > > > +1 for Rebol only too. I don't want to sound selfish, but we

desperately

> need rebol engines first, then we can go for further integration ...

OK, but keeping it simple implies that I have time for other implementations as well ;-)

> >I like it. And as the handler-functions concept closely maps to > >messaging per se, I'd stay with it.

<<quoted lines omitted: 3>>

> Send-message echo tcp://localhost:8000 [{Echo this}] > to send-message url echo [{Echo this}] .... to be compatible with

native

> rebol ordering ... > > 2) use different refinementname instead of /deliver .... e.g.

/callback,

> /on-return etc. ... > > both 1) and 2) are just implementation details though ... >

Yes, they can be easily changed. I'll have a look.

> >>- Should I add meta-data so you can query an engine to see what > >>messages it udnerstands. If so, what should the meta-data look like?

<<quoted lines omitted: 8>>

> really be just a message type, but I would like to have such > functionality available somewhere. The point is, that if we will let

> upon message-type designers, we will end up with mixture of different > aproaches ... it is the same as someone could say - why to build > messaging engine at all, if rebol is about messaging in its roots?

Well,

> it is, but we need some more unified way used by rebol developers to > take our apps further ... >

I could add a default message handler, 'meta-data ?

> >As mentioned earlier, I'd love to see this being high-performance > >stuff ;)

<<quoted lines omitted: 5>>

> introduced in Rugby or how will it divide its time between > message-types? What if my message type handler does some more

intensive

> stuff? >

I don't know how well async I/O works in that case, but I expect there are no problems in most cases. We'll have to learn as we go along.

> Hmm, right now I am not even sure what exact usage will that stuff > provide us with. I can understand you would like to add Rugby upon it > ... but will it be general enough for any kind of protocol, so e.g. > someone could create IRC protocol and plug-it into your messaging

scheme?

I don't think so. My idea is to have a block based message format (that's what I'm using right now) so you'd be limited to that. But porting it to C# and python is trivial. Interpretation of the message may e a bit harder there, but that would be the programmer's reason to switch to Rebol.... --Maarten

[7/7] from: antonr:iinet:au at: 13-Aug-2003 14:46

Sounds great, Maarten!

> - Should I add (optionally) message encryption or leave that to > implementors of messages, as their needs may vary? Pro: Ecnryption Con: > only works with the SDK, Command, ...

Encryption is pretty necessary these days. It would be nice to have it built in, though as noted, not necessary. It's pretty easy to add, isn't it? But, don't call the functions like rugby, by just changing first letter (rexec -> sexec). I didn't think these were nice function names, although they were nice and powerful ones.

> - Should I work a bit on the message format so we can port it to other > languages as well (.NET, Python, etc.) or keep it REBOL-only and enhance > it, for example integrating a new Rugby on top of this? > > - Is this API with server based handler functions a good concept or is > it too hard for most programmers?

Seems ok to me. I can't think how to make it simpler. I think "add-message-type" is not a good name? How about "add-message-handler". That's what you are doing after all. (is that in use somewhere else?) Maybe if you make the handler [any-function! block!], and if user passes a block, add-message-type makes a new function using the block as the body.

> - Should I add meta-data so you can query an engine to see what messages > it udnerstands. If so, what should the meta-data look like?

Yes. At least return the help of the handler function. (That is, just send back what rebol help returns for that function.) I suppose the implementor is most likely to use built-in rebol function documentation to show how to use that dialect. Help should be a built-in message handler. For security, some message types should be able to hide their presence on the server.

> - Should I add authentication, so you can demand that a message must be > properly authenticated? If so, I'll probably keep it pluggable, meaning > that you can add your own authentication methods as well.

I think so. Otherwise anyone is running your code.

> - Should I transparently propagate errors back from server to client or > just return none when an error occurs? It may be annoying to do a try > with each send-message, OTOH seeing what goes wrong is worth something > as well.

I think errors should be returned. You want to see what's wrong. Clients, now, can always just use attempt if they want a none on error. Doing it the other way is just confusing.

Notes

Quoted lines have been omitted from some messages.
View the message alone to see the lines that have been omitted