Mailing List Archive: 49091 messages
  • Home
  • Script library
  • AltME Archive
  • Mailing list
  • Articles Index
  • Site search
 

hook while downloading from html port

 [1/9] from: moliad::aei::ca at: 14-Apr-2004 0:45


hi, I was wondering, if its possible to put a hook somewhere (even if deep) within the port or more specifically the html port handler, so that we can update something elsewhere, do statistics or print out transfer rate for example... I thought maybe by reducing the buffer size and patching flush buffer, but I'm not sure or where... so I thought, maybe someone has a ready-made tool, patch or its something standard that just needs a little more exposure... If this is documented somewhere, you can just point me to the docs, I'll be happy to read them directly... thanks in advance. -MAx

 [2/9] from: andreas:bolka:gmx at: 14-Apr-2004 20:09


Wednesday, April 14, 2004, 6:45:13 AM, Maxim wrote:
> I was wondering, if its possible to put a hook somewhere (even if > deep) within the port or more specifically the html port handler, so > that we can update something elsewhere, do statistics or print out > transfer rate for example...
I guess you rather mean the http ports, but anyway: I think that should be achievable by using async:// networking. But sorry, haven't done it (yet) and no so I've no example handy. -- Best regards, Andreas

 [3/9] from: Gary:Jones:usap:gov at: 15-Apr-2004 15:38


From: Maxim Olivier-Adlhoch .... I was wondering, if its possible to put a hook somewhere (even if deep) within the port or more specifically the html port handler, so that we can update something elsewhere, do statistics or print out transfer rate for example... .... Hi, Max, Since no one has offered anything any more specific, I can suggest a general way to put in your own hooks, but some of the services are at a lower level than you can access through the standard service. First, you'll need a working copy of the scheme. For example, for http: echo %/path/to/my-http.txt probe system/schemes/http echo off With this text file, first place: system/schemes/http: near the top before the make, like: system/schemes/http: make object! [ scheme: 'HTTP host: none port-id: 80 .... Then go to the end of the file and erase everything after the last "]" and resave the file. This file can be loaded in as a replacement for the http scheme. If reloaded as such, there will be no change. Now, you have to begin to understand how the schemes work. As the port for a scheme is read, it goes through these stages (or functions): init open-proto open close The lowest level functions, accessed more "invisibly," are read and write. By playing with probe or print statements and then accessing a web page, one can begin to get more information at a stage of the process as needed. You will find after a lot of experimentation that much of the reading is done at a very low level, like through 'read-io. This function stores its data in temporary spaces in this protocol, like the port/state object that is used during the access. If you try to access these data, you inadvertently end up botching the operation, so care must be used and there are definitely limits to what can be used. The only way to bypass some of the limits would be to extensively modify the scheme, which can be done with some work and care, or to create your own, which is reinventing the wheel. It may be enough to get you going, but I would not be overenthusiastic about thinking you will be able the scrounge things like transfer rates! Hope this gives you some help. --Scott Jones

 [4/9] from: antonr:lexicon at: 15-Apr-2004 17:51


Scott, I think you are a bit negative about transfer rates. :) Surely there is a way. Here also, is an easier way to save the scheme: write clipboard:// join {rebol [] system/schemes/http:} mold system/schemes/http Now open your favourite editor, paste into a new file and save. Uh oh, now I'm into it... Search for "read:". You have found the read function. Now add a line after the "net-utils/net-log" line: print join "bytes: " [port/state/num " at " now/time/precise] Save the file and do it. Then test the change:
>> read http://www.rebol.com/
connecting to: www.rebol.com bytes: 2048 at 17:39:47.731 bytes: 2048 at 17:39:47.741 bytes: 2048 at 17:39:47.741 bytes: 2048 at 17:39:48.332 bytes: 2048 at 17:39:48.332 bytes: 2048 at 17:39:48.332 bytes: 2048 at 17:39:48.552 == {<HTML> <HEAD> <META HTTP-EQUIV="Content-Type" CONTENT="text/html;CHARSET=iso-8859-1"> <META NAME="keywords" CONTENT="REBOL, pro... With that info it's possible to make up some progress callback so you can make a display showing transfer rate. You would need to add a couple of variables to the http object, eg: total-bytes last-transfer-time and reset these at the appropriate time. Mmm.. while I'm here I can see the /custom refinement. If I ever forget the custom refinement dialect I can look here in the source. Anton.

 [5/9] from: brett:codeconscious at: 15-Apr-2004 19:59


Hi, Have a look at /progress refinement on function Read-net. If this is insufficient, you could take Scott and Anton's approach further and maybe even allow a way to specify a on-read callback function using the /custom refinement - which might be what Anton was implying. Regards, Brett.

 [6/9] from: g:santilli:tiscalinet:it at: 15-Apr-2004 12:00


Hi Andreas, On Wednesday, April 14, 2004, 8:09:45 PM, you wrote:
>> I was wondering, if its possible to put a hook somewhere (even if >> deep) within the port or more specifically the html port handler, so >> that we can update something elsewhere, do statistics or print out >> transfer rate for example...
AB> I guess you rather mean the http ports, but anyway: I think that AB> should be achievable by using async:// networking. But sorry, haven't AB> done it (yet) and no so I've no example handy. If you only need to report in/out speed in bytes/s or something like that, I have an hacked version of async:// with speed measuring in; it's not much tested, was not meant to be of generic use, it's not based on the latest version of async://, and you'd need to implement an HTTP client on top of it... but if you need it, let me know and I can dig it out. Regards, Gabriele. -- Gabriele Santilli <[g--santilli--tiscalinet--it]> -- REBOL Programmer Amiga Group Italia sez. L'Aquila --- SOON: http://www.rebol.it/

 [7/9] from: maximo:meteorstudios at: 15-Apr-2004 10:23


> -----Original Message----- > From: Gabriele Santilli [mailto:[g--santilli--tiscalinet--it]]
<<quoted lines omitted: 7>>
> need to implement an HTTP client on top of it... but if you need > it, let me know and I can dig it out.
what does "implement an HTTP client on top of it involve"? is that completely writing an http scheme or can I borrow the one from within rebol and merge it? maybe I'm completely off track! -MAx

 [8/9] from: g:santilli:tiscalinet:it at: 15-Apr-2004 16:57


Hi Maxim, On Thursday, April 15, 2004, 4:23:51 PM, you wrote: MOA> what does "implement an HTTP client on top of it MOA> involve"? is that completely writing an http scheme or can I MOA> borrow the one from within rebol and merge it? Borrowing the one in REBOL wouldn't work, as async:// works differently than tcp://. (You could reuse some of the code...) I guess it would be much easier to patch around the current HTTP scheme; if you don't need to be accurate just changing the lowlevel read and write functions in the handler would be enough. Regards, Gabriele. -- Gabriele Santilli <[g--santilli--tiscalinet--it]> -- REBOL Programmer Amiga Group Italia sez. L'Aquila --- SOON: http://www.rebol.it/

 [9/9] from: Gary:Jones:usap:gov at: 16-Apr-2004 12:35


From: Anton Rolls AR> Scott, I think you are a bit negative about transfer rates. :) AR> Surely there is a way. .... It was one of those things were as soon as I sent my email, I, too, thought of a way to show progress, but our satellite signals times are limited. My guess is that there will already be 3 more ways to do it by the time this arrives! Not unlike Anton's way in using read, I capture the actual bytes read from the return of read-io and accumulate these in a variable. If the site provides content length, then the patch can compare sent to expected. Unfortunately, not all servers/websites send the content length. At the end of the http scheme, and the variable "bytes-transferred", as in: ... cache-size: 5 user-agent: "REBOL View 1.2.10.3.1" bytes-transferred: 0 ] Then change the read function to (watch for line wrap): read: func [ port "An open port spec" data "A buffer to use for the read" /local bytes-read ][ net-utils/net-log ["low level read of " port/state/num "bytes"] bytes-read: read-io port/sub-port data port/state/num bytes-transferred: bytes-transferred + bytes-read if port/locals/headers/content-length <> none [print [to-integer (bytes-transferred / (to-integer port/locals/headers/content-length) * 100) "percent"]] bytes-read ] Reading a page in which the server gives content-length will allow the calculation of percent received. I realized that the read function needs to return the bytes, read, hence the last line. Expanding on the idea, I decided for fun to try to create a visual feedback method that gives progress bar when the content-length is known, or simple accumulated bytes retrieved, when content-length was unknown. It requires a fair amount of patching. Starting with a fresh copy of the http scheme, one first alters the continue-post function: continue-post: func [/tunnel] [ response-line: system/words/pick port/sub-port 1 net-utils/net-log response-line either none? response-line [do error] [ either none? result: select either tunnel [tunnel-actions] [response-actions] response-code: to-integer second parse response-line none [do error] [do get result] ] if show-progress [ either port/locals/headers/content-length <> none [ progress-message/pane: layout/origin/offset [p: progress] 0x0 0x0 show progress-message ][ progress-message/pane: layout/origin/offset [t: text 200x16] 0x0 0x0 show progress-message ] ] ] Then following the close block and build-port further down, one inserts these lines .... ] build-port ;insert following lines if all [block? port/state/custom post-data: find port/state/custom to-lit-word 'progress block? post-data] [ show-progress: true progress-message: get to-word select port/state/custom to-lit-word 'progress ] Then one changes the close function: close: func [port][ if show-progress [ progress-message/pane: layout/origin/offset [t: text 200x16] 0x0 0x0 show progress-message t/text: "Done!" show t show-progress: none ] system/words/close port/sub-port bytes-transferred: 0 ] One changes the read function: read: func [ port "An open port spec" data "A buffer to use for the read" /local bytes-read ][ net-utils/net-log ["low level read of " port/state/num "bytes"] bytes-read: read-io port/sub-port data port/state/num bytes-transferred: bytes-transferred + bytes-read if show-progress [ either port/locals/headers/content-length <> none [ p/data: bytes-transferred / (to-integer port/locals/headers/content-length) show p ;loop 10000000 [] ;uncomment to see fast pages load ][ t/text: bytes-transferred show t ;loop 10000000 [] ] ] bytes-read ] Finally, one adds variables to the end of the scheme: ..... user-agent: "REBOL View 1.2.10.3.1" bytes-transferred: 0 show-progress: progress-message: p: t: none ] One uses the custom refinement to pass the reference to a VID box: view layout [ f: field 300x24 button "Get Page" [ read/custom trim to-url f/text ['progress :my-pane] ] my-pane: box 200x16 button "Exit" [quit] ] When one types in a url from a website where the server gives the content-length, the box gets a progress bar embedded, which is then updated based on the percentage of bytes read. When the url is served from a server that does not give the content-length, then the bytes retrieved are displayed. After completed, the progress box is converted to text that says "Done!" I have commented out some do nothing loops in case your downloads are too fast, and you are trying to see the changes at work. I was just about to send this when I received the most recent set of emails, including Max's own patch on read-net. Oh well, it still may prove useful to him or someone else in another context and/or to take advantage of all the extra features of the http protocol. --Scott Jones

Notes
  • Quoted lines have been omitted from some messages.
    View the message alone to see the lines that have been omitted