Mailing List Archive: 49091 messages
  • Home
  • Script library
  • AltME Archive
  • Mailing list
  • Articles Index
  • Site search
 

The REBOL async:// tutorial - take 1

 [1/24] from: maarten:vrijheid at: 3-Mar-2004 15:48


See below. Enjoy it. Republish it. ================================================== The REBOL async:// tutorial - Take 1 Maarten Koopmans, code samples and root protocol by Gabriele Santilli What is async:// anyway? It is an asynchronous TCP root protocol that can be used to drive REBOLs hidden async TCP features. This allows network code to be processed automagically upon certain network events, such as getting a connection, ready to read/write, and closing a conection. Once you have set the code to be called upon these events, things will go by themself if you're in an event loop. This is much like GUI events are processed automagically when you call 'view. The first thing you need to do is get the async:// root protocol by Gabriele. Download it at http://www.rebol.it/giesse/async-protocol.r Note that this version supports binary transfers only, /lines support is being worked on. Now let's start with a simple client script, downloading images in the background in REBOL/View. First the code: handler: func [port [port!] state [word! error!] /local tmp cmd] [ if error? :state [print mold disarm state return true] switch state [ connect [ ; do HTTP request insert port {GET /fg/anen.jpg HTTP/1.0^M^JHost: www.3dwallpaper.com^M^J^M^J} false ] read [false] write [false] close [ ; get data data: copy port close port ;print copy/part data find data "^M^J^M^J" data: to binary! find/tail data "^M^J^M^J" other/image: attempt [load data] other/text: "" show other false ] ] ] port: open async://www.3dwallpaper.com:80 port/awake: :handler view layout [ across me: box 100x100 random 255.255.255 0:00:00.5 feel [ engage: func [f a e] [ if a = 'time [ me/color: random 255.255.255 show me ] ] ] other: box 100x100 255.255.255 "Downloading image..." Return Area 208x100 "You can type here while downloading." ] Here is how it works: you open an async port using normal URI syntax in the middle of the script. You can see this in the lines: port: open async://www.3dwallpaper.com:80 port/awake: :handler The second line sets the callback function. This function is the crucial part for async:// based implementations, as it provides code that can be executed for all the events. So, let's take a look a the handler function! It starts with the following piece of code: handler: func [port [port!] state [word! error!] /local tmp cmd] [ if error? :state [print mold disarm state return true] As you can see, we get two parameters: The port that we can read/write to and the state. The state can be an error! or a word!. If it is an error we need to clean up and the above code shows briefly one way how this can be done on the second line. If it is a word it can have any of four values: - 'connect, signalling that we have just acquired a connection (client-side only) - 'read , signalling that we can have data to read - 'write , signalling that we can write data (again) - and 'close , signalling that the other side has closed the connection. So what we do next in our handler is a switch based on the state, executing different code for each possible state. switch state [ connect [ ; do HTTP request insert port {GET /fg/anen.jpg HTTP/1.0^M^JHost: www.3dwallpaper.com^M^J^M^J} false ] read [false] write [false] close [ ; get data data: copy port close port ;print copy/part data find data "^M^J^M^J" data: to binary! find/tail data "^M^J^M^J" other/image: attempt [load data] other/text: "" show other true ] ] As you see, we simply do a HTTP GET for an image, and write it to file and the view it. Upon 'connect we immediately make our request. Why not in the 'write event? The 'write event is triggered everything we have something written and we canw rite more. So the first time we always have to write from another event!!! Now, once the data is inserted we return false. This indicates that we shouldn't be removed from the system/ports/wait-list list, where all ports are stored (at least async:// and event:// ones). As you can see we did return true on error. The read and write states COULD be used to do some reading from data or write more, but as this is a very simple request-response, that's not necessary, so they just return false. So... the other side returns the data and closes the connection. Upon close we simple read it ALL (again this could have been done in 'read using a temporary buffer) and close the port. We update the image and.... done! Now for a simple server: First we add a listening server port to the system/ports/wait-list, like: either error? try [listen: open/no-wait tcp://:8000] [ port: open async://localhost:8000 port/awake: do handler ] [ listen/awake: func [l /local p] [ print "Got connection." p: first listen remove find system/ports/wait-list listen port: make port! [scheme: 'async sub-port: p] open port port/awake: do handler false ] insert tail system/ports/wait-list listen port: none ] As you can see, its awake function convert the accepted port to an async one and sets the handler. So what is the handler then? handler: [ use [ buffer ][ buffer: copy [] func [port [port!] state [word! error!] /local tmp cmd] [ if error? :state [print mold disarm state return true] switch state [ connect [print "Connected." false] read [ append buffer copy port while [tmp: find buffer newline] [ cmd: copy/part buffer tmp remove/part buffer next tmp do-cmd cmd ] false ] write [false] close [print "Peer closed connection." close port true] ] ] ] ] The first thing to notice is the fact that we use 'use to create a context that returns a function! value. This function (and only this particluar value) has access to its buffer. By doing the handler block in the server part above, every accepted port gets a copy of theis function value with its own "static" buffer space. A very simple but effective trick. Now, on to the events. 'Connect does nothing, as we are already connected. This happened when the awake function of our TCP server port was called. We chose 'write and 'close also to do nothing. Why? We only want to receive a command and then do it. The reading is done in 'read, we append what we have read to the buffer. If we see a newline, we execute it and clear the buffer until the newline. And then we return false. The fact that we return false implies that we may read more than one connection. Also note that 'close does nothing, but *if* a port is closed, it also closes. Of course you can use 'close for all sorts of clean-up. Now if you this code do-cmd: func [cmd] [ other/color: attempt [load cmd] show other ] view layout [ across me: box 100x100 random 255.255.255 0:00:01 feel [ engage: func [f a e] [ if a = 'time [ me/color: random 255.255.255 if port [insert port join me/color newline] show me ] ] ] other: box 100x100 random 255.255.255 Return Area 208x100 ] halt to the above, you have a simple GUI with a command server. You are now ready to play with async:// and enhance the code adding substates to 'read or 'write to implement advanced protocols. Its just more of the same. I hope you keep the use [ ] [ ] trick in mind for server handlers, I found it a very easy way to add extra substates this way. Now.... start coding!

 [2/24] from: maarten:vrijheid at: 3-Mar-2004 16:13


Of course you have to do the async-protocol.r first before trying the sample code..... Maarten Koopmans wrote:

 [3/24] from: petr:krenzelok:trz:cz at: 3-Mar-2004 17:58


Maarten Koopmans napsal(a): Hi, that stuff looks really excelent! Strange thing is, that even if the code looks pretty straightforward, I don't understand every detail of it, but that is what there are questions for :-)
> handler: func [port [port!] state [word! error!] /local tmp cmd] [ > if error? :state [print mold disarm state return true]
<<quoted lines omitted: 3>>
> insert port {GET /fg/anen.jpg HTTP/1.0^M^JHost: >www.3dwallpaper.com^M^J^M^J}
That one - raw tcp stream, right? I wonder if some kind of dialect (set of functions) could be produced to handle that ugly MJMJMJ and GET commands etc., as e.g. read/custom allows. Just a theoretical question, if it even would be worth it, nothing more
> false > ]
<<quoted lines omitted: 3>>
> ; get data > data: copy port
OK, so that one does not block right? And it is so just because we are inside handler function, which is being called once some event happens on port, so theoretically some data should be awaiting us. I just wonder it comes in 'close switch part. We get here once other side closed connection? So if I understand it correctly, we read it in parts, but 'read part does nothing, rebol is internally buffering data (how large data is rebol able to buffer easily that way?) and once other side closes connection, we can read it by copy (which will not block, even if no-wait was not used), and whole data is being read out of the port buffer at once? Well, I hope I am still on track :-)
>Now for a simple server: > >First we add a listening server port to the system/ports/wait-list, like: >
Why? Is that needed? Am I right thinking it is because of View? Once View starts, it adds event-port into wait-list and if we want to process all various events properly, we have to go via wait-list?
> either error? try [listen: open/no-wait tcp://:8000] [ > port: open async://localhost:8000 > port/awake: do handler >
Above code somehow escapes my understanding :-) So if we are not able to open listen port (because e.g. we are already listening), we open connection on localhost to that port? What is that good for?
> ] [ > listen/awake: func [l /local p] [ > print "Got connection." > p: first listen > remove find system/ports/wait-list listen > port: make port! [scheme: 'async sub-port: p] >
that is something I never understood. That is why I was not able to further more deeply adapt Sterling's proxy.r script. It contained way too much port subport and proxy (as a port :-) stuff for my brain to swallow :-) 'p is assigned first connected client. It does not contain any sub-port, yet it can communicate. IIRC someone said, that sub-port contains real communication port. But I don't understand the difference, even without sub-port, I am able to send data here and there and I can see it buffered in port/state. What is sub-port then?
> open port > port/awake: do handler
<<quoted lines omitted: 3>>
> port: none > ]
so overall - it is clever - once first event happens on listening port, we remove it from wait-list, reassign handler and insert it back into wait-list. That sounds like nice constructor/init method in OOP :-)
>As you can see, its awake function convert the accepted port to an async >one and sets the handler. So what is the handler then?
<<quoted lines omitted: 24>>
>part above, every accepted port gets a copy of theis function value with >its own "static" buffer space. A very simple but effective trick.
cool! Thanks for your tutorial, very educative! -pekr-

 [4/24] from: ptretter:charter at: 3-Mar-2004 12:27


Thanks Maarten and Gabriele! Paul Tretter

 [5/24] from: maarten:vrijheid at: 3-Mar-2004 19:35


First, realize that we didn't knock this off in two afternoons. So it is OK to hit your head against the wall.
>> handler: func [port [port!] state [word! error!] /local tmp cmd] [ >> if error? :state [print mold disarm state return true]
<<quoted lines omitted: 11>>
>commands etc., as e.g. read/custom allows. Just a theoretical question, >if it even would be worth it, nothing more
You could use the builtin crlf as a replacement.
>> false >> ]
<<quoted lines omitted: 17>>
>no-wait was not used), and whole data is being read out of the port >buffer at once? Well, I hope I am still on track :-)
That copy doesn't block, because it gets diverted to the copy in the async:// handler IIRC. Also, async:// does all low-level input and output buffering. And async opening + async dns://
>>Now for a simple server: >>
<<quoted lines omitted: 6>>
>View starts, it adds event-port into wait-list and if we want to process >all various events properly, we have to go via wait-list?
It can be done many ways indeed. Pick your own example.
>> either error? try [listen: open/no-wait tcp://:8000] [ >> port: open async://localhost:8000
<<quoted lines omitted: 6>>
>open listen port (because e.g. we are already listening), we open >connection on localhost to that port? What is that good for?
Gabriele? I missed it, it looks like setting up a tunnel to me.
>> ] [ >> listen/awake: func [l /local p] [
<<quoted lines omitted: 14>>
>even without sub-port, I am able to send data here and there and I can >see it buffered in port/state. What is sub-port then?
The real connecttion :-) Anyway... this is the way to go. If you take a look at the renewed xml-rpc you will find an add-server function that abstract this away nicely.
>> open port >> port/awake: do handler
<<quoted lines omitted: 11>>
>we remove it from wait-list, reassign handler and insert it back into >wait-list. That sounds like nice constructor/init method in OOP :-)
Go clean your mouth!
>>As you can see, its awake function convert the accepted port to an async >>one and sets the handler. So what is the handler then?
<<quoted lines omitted: 33>>
>> >cool!
I know! That's why I thought I'd share it with you all.
>Thanks for your tutorial, very educative! >
Put it to work. async:// is so clever. Carl explained it Eastern 2003 in a private chat, then I started experimenting and Gabriele turned that into async:// It took some more experimenting, and finally we have a tutorial. --Maarten

 [6/24] from: g:santilli:tiscalinet:it at: 3-Mar-2004 19:58


Hi Maarten, On Wednesday, March 3, 2004, 7:35:46 PM, you wrote:
>>> either error? try [listen: open/no-wait tcp://:8000] [ >>> port: open async://localhost:8000
<<quoted lines omitted: 9>>
>> >>
MK> Gabriele? I missed it, it looks like setting up a tunnel to me. That was a quick demo, first time you launch the script it works as a server, second time it works as a client and connects to the server. You just need to run the script twice... Regards, Gabriele. -- Gabriele Santilli <[g--santilli--tiscalinet--it]> -- REBOL Programmer Amiga Group Italia sez. L'Aquila --- SOON: http://www.rebol.it/

 [7/24] from: maximo:meteorstudios at: 3-Mar-2004 13:58


I find this/last week to be a milestone in the rebol world. we can now start an async xml-rpc server by browsing a remote web page. Our world is getting soooo coool. we now even have a tutorial to understand the async protocol... I am thinking of using it for my liquid net module. -MAx --- You can either be part of the problem or part of the solution, but in the end, being part of the problem is much more fun.

 [8/24] from: atruter:labyrinth:au at: 4-Mar-2004 10:41


Well done guys, and thanks for taking the time and effort to not only release this but *explain* it! ;) Regards, Ashley

 [9/24] from: tbrownell:veleng at: 4-Mar-2004 1:36


Excellent.. finally some light in this dark hole of tcp. I have a couple of questions though.. in the handler we have connect, read, write close.. how can i do this... connect [] read [ (read the buffer, which in this case is "2 + 2", and having processed this to a result string of "4"...)] write [... send the result back?] close [] And my second question.. what does it take to handle multiple clients, and have the server continue to serve, rather than shutting down when the client disconnects? Thanks Terry

 [10/24] from: maarten:vrijheid at: 4-Mar-2004 11:02


Terry Brownell wrote:
>Excellent.. finally some light in this dark hole of tcp. >I have a couple of questions though.. in the handler we have connect,
<<quoted lines omitted: 4>>
>write [... send the result back?] >close []
terminate the string by a newline. inbuf your input buffer from the use context: read [ append inbuf copy port if found? find inbuf newline [ do copy/part inbuf find inbuf newline remove/part inbuf -1 + index? find inbuf newline ] ]
>And my second question.. what does it take to handle multiple clients, >and have the server continue to serve, rather than shutting down when >the client disconnects? >
Nothing. Every connection gets a client port with this handler from the server port. That's the beauty of it. And multiplexing is done by a state machine based on the event per port, hence the name "async". --Maarten

 [11/24] from: maarten:vrijheid at: 4-Mar-2004 11:03


add a false after the read code :-) Terry Brownell wrote:

 [12/24] from: tbrownell:veleng at: 4-Mar-2004 2:21


I was misunderstood.. what I meant was, once I get some stuff FROM read.. and do some magic with it (ie: get the result of 2 + 2).. how do I send the RESULT back to the client via WRITE? I dont have a problem with getting or processing READ. Terry Terry Brownell wrote:
>Excellent.. finally some light in this dark hole of tcp. >I have a couple of questions though.. in the handler we have connect,
<<quoted lines omitted: 4>>
>write [... send the result back?] >close []
terminate the string by a newline. inbuf your input buffer from the use context: read [ append inbuf copy port if found? find inbuf newline [ do copy/part inbuf find inbuf newline remove/part inbuf -1 + index? find inbuf newline ] ]
>And my second question.. what does it take to handle multiple clients, >and have the server continue to serve, rather than shutting down when >the client disconnects? >
Nothing. Every connection gets a client port with this handler from the server port. That's the beauty of it. And multiplexing is done by a state machine based on the event per port, hence the name "async". --Maarten

 [13/24] from: tbrownell:veleng at: 4-Mar-2004 2:24


Using the server example in the doc, the server stops once the client disconnects? Terry

 [14/24] from: tbrownell:veleng at: 4-Mar-2004 2:51


ok, in answer to my own question, you can use "insert port" at anytime while processing script. TB Terry Brownell wrote:

 [15/24] from: maarten:vrijheid at: 4-Mar-2004 12:44


Terry Brownell wrote:
>ok, in answer to my own question, you can use "insert port" at anytime >while processing script. > >TB > >Terry Brownell wrote: >
Yes, but you need to do that once, that will trigger the 'write vent that can take over. Typically I insert the first few chars of an out buffer. --Maarten

 [16/24] from: maarten:vrijheid at: 4-Mar-2004 12:45


Terry Brownell wrote:
>Using the server example in the doc, the server stops once the client >disconnects? > >Terry >
No, it closes the connection with this particular client only. --Maarten

 [17/24] from: Christophe:Coussement:mil:be at: 4-Mar-2004 15:11


Hi Maarten, I do not have very much experience when coming to network protocol, so please excuse the triviality of my question :). The fact is I could for sure use this 'async protocol for a project we are now developing. I tried the following, but without any result. I try to load a %dghr.jpg file situated in the same dir as %async-protocol, and try to connect to localhost. What did I wrong? ;>>>>>>>>>>>>>>>>>>> code --- do %async-protocol.r handler: func [ { your comment here RETURN: } port [port!] "comment" state [word! error!] /local tmp cmd ][ ;$debug/var/stop 'state if error? :state [print mold disarm state return true] switch state [ connect [ insert port {GET logo.jpg HTTP/1.0^M^JHost:127.0.0.1^M^J^M^J} false ] read [false] write [false] close [ data: copy port close port ;$debug/var/stop 'data data: to binary! find/tail data "^M^J^M^J" other/image: attempt [load data] other/text: "" show other false ] ] ] port: open async://127.0.0.1:80 port/awake: :handler view layout [ across me: box 100x100 random 255.255.255 0:00:0.5 feel [ engage: func [f a e][ if a = 'time [ me/color: random 255.255.255 show me ] ] ] other: box 100x100 255.255.255 "Downloading image ..." return area 208x100 "you can type here while downloading" ] ;--- code <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< TIA! ==christophe

 [18/24] from: warp:reboot:ch at: 4-Mar-2004 15:56


I think you need a slash before "logo.jpg" and a space after "Host:" -> insert port {GET /logo.jpg HTTP/1.0^M^JHost: 127.0.0.1^M^J^M^J} On 4 mar 2004, at 15:11, Coussement Christophe wrote:
> insert port {GET logo.jpg HTTP/1.0^M^JHost:127.0.0.1^M^J^M^J}
Will

 [19/24] from: maarten:vrijheid at: 4-Mar-2004 16:09


Hi Christophe,
>I do not have very much experience when coming to network protocol, so please excuse the triviality of my question :). >The fact is I could for sure use this 'async protocol for a project we are now developing. >
Good.
>I tried the following, but without any result. I try to load a %dghr.jpg file situated in the same dir as %async-protocol, and try to connect to localhost. What did I wrong? >
What are you trying to do? Fetch the file from a file or a webserver?
>;>>>>>>>>>>>>>>>>>>> code --- >do %async-protocol.r
<<quoted lines omitted: 19>>
> ;$debug/var/stop 'data > data: to binary! find/tail data "^M^J^M^J"
Do you receive data? What does your webserver log say?
> other/image: attempt [load data] > other/text: ""
<<quoted lines omitted: 18>>
>] >;--- code <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
It looks like you are re-using the exact code I mailed, which (I just checked) works. So it looks like you'll have to debug using your webserver and debugger in the script to see what gets done and what doesn't. Sorry I can't help any more, but the code looks fine (an exact copy of the sample code except for the URI). --Maarten

 [20/24] from: g:santilli:tiscalinet:it at: 4-Mar-2004 16:51


Hi Maarten, On Thursday, March 4, 2004, 12:44:17 PM, you wrote: MK> Yes, but you need to do that once, that will trigger the 'write vent MK> that can take over. Typically I insert the first few chars of an out buffer. It is also possible to insert a big string all at a time (after read or in any other occasion); async:// will care about sending it out a little chunk at a time. Regards, Gabriele. -- Gabriele Santilli <[g--santilli--tiscalinet--it]> -- REBOL Programmer Amiga Group Italia sez. L'Aquila --- SOON: http://www.rebol.it/

 [21/24] from: tbrownell::veleng::com at: 4-Mar-2004 9:41

async:// Close connection


When I use the following example from the tutorial, my server stop whenever a client disconnects with this message... async protocol loaded Got connection. Peer closed connection.
>>
rebol [] do %async-protocol.r either error? try [listen: open/no-wait tcp://:3000] [ port: open async://localhost:3000 port/awake: do handler ] [ listen/awake: func [l /local p] [ print "Got connection." p: first listen remove find system/ports/wait-list listen port: make port! [scheme: 'async sub-port: p] open port port/awake: do handler false ] insert tail system/ports/wait-list listen port: none ] handler: [ use [ buffer ][ buffer: copy [] func [port [port!] state [word! error!] /local tmp cmd] [ if error? :state [print mold disarm state return true] switch state [ connect [print "Connected." false] read [ append buffer copy port while [tmp: find buffer newline] [ cmd: copy/part buffer tmp remove/part buffer next tmp do-cmd cmd ] false ] write [false] close [print "Peer closed connection." close port true] ] ] ] ] do-cmd: func [cmd] [ other/color: attempt [load cmd] show other ] view layout [ across me: box 100x100 random 255.255.255 0:00:01 feel [ engage: func [f a e] [ if a = 'time [ me/color: random 255.255.255 if port [insert port join me/color newline] show me ] ] ] other: box 100x100 random 255.255.255 Return Area 208x100 ] halt Terry Brownell wrote:
>>Using the server example in the doc, the server stops once the client >>disconnects? >> >>Terry >> >> >
No, it closes the connection with this particular client only. --Maarten

 [22/24] from: Christophe:Coussement:mil:be at: 11-Mar-2004 13:33

Re: The REBOL async:// tutorial - take 1


Hi Maarten and Will Sorry for this late answer: I was out of office for a while... Thanks for helping me !
> >I tried the following, but without any result. I try to load > a %logo.jpg file situated in the same dir as %async-protocol, > and try to connect to localhost. What did I wrong? > > > What are you trying to do? Fetch the file from a file or a webserver?
Fetch from the file. I just wanted to try the code running on localhost.
> >;>>>>>>>>>>>>>>>>>>> code --- > >do %async-protocol.r
<<quoted lines omitted: 24>>
> > > Do you receive data? What does your webserver log say?
Well, do I have to work on a webserver? I thought %async-protocol.r serviced a sort of server, listening on specified port ?
> > other/image: attempt [load data] > > other/text: ""
<<quoted lines omitted: 28>>
> (an exact > copy of the sample code except for the URI).
anyway, wet i get is the generated window, but with no image loading. After placing a debug mark at the beginning of 'handler, I can see the function isn't evaluated once. I think I'm confused about the context of the use of this protocol :( Could you provide me with a concrete use case ? TIA ==christophe

 [23/24] from: maarten:vrijheid at: 11-Mar-2004 14:34


The specific example you try is a client that connects to a webserver to fetch the image. If you have no webserver running (no valid url...): no image! async:// is just like tcp:// , but now with calllbacks instead of waiting yourself. Does this help you? --Maarten Coussement Christophe wrote:

 [24/24] from: Christophe:Coussement:mil:be at: 11-Mar-2004 15:12


Hi Maarten, It's all clear now ;) Thanks ! ==christophe

Notes
  • Quoted lines have been omitted from some messages.
    View the message alone to see the lines that have been omitted