hook while downloading from html port
[1/9] from: moliad::aei::ca at: 14-Apr-2004 0:45
hi,
I was wondering, if its possible to put a hook somewhere (even if deep) within
the port or more specifically the html port handler, so that we can update
something elsewhere, do statistics or print out transfer rate for example...
I thought maybe by reducing the buffer size and patching flush buffer, but I'm
not sure or where... so I thought, maybe someone has a ready-made tool, patch
or its something standard that just needs a little more exposure...
If this is documented somewhere, you can just point me to the docs, I'll be
happy to read them directly...
thanks in advance.
-MAx
[2/9] from: andreas:bolka:gmx at: 14-Apr-2004 20:09
Wednesday, April 14, 2004, 6:45:13 AM, Maxim wrote:
> I was wondering, if its possible to put a hook somewhere (even if
> deep) within the port or more specifically the html port handler, so
> that we can update something elsewhere, do statistics or print out
> transfer rate for example...
I guess you rather mean the http ports, but anyway: I think that
should be achievable by using async:// networking. But sorry, haven't
done it (yet) and no so I've no example handy.
--
Best regards,
Andreas
[3/9] from: Gary:Jones:usap:gov at: 15-Apr-2004 15:38
From: Maxim Olivier-Adlhoch
....
I was wondering, if its possible to put a hook somewhere (even if deep) within
the port or more specifically the html port handler, so that we can update
something elsewhere, do statistics or print out transfer rate for example...
....
Hi, Max,
Since no one has offered anything any more specific, I can suggest a general way to put
in your own hooks, but some of the services are at a lower level than you can access
through the standard service.
First, you'll need a working copy of the scheme. For example, for http:
echo %/path/to/my-http.txt
probe system/schemes/http
echo off
With this text file, first place:
system/schemes/http:
near the top before the make, like:
system/schemes/http: make object! [
scheme: 'HTTP
host: none
port-id: 80
....
Then go to the end of the file and erase everything after the last "]" and resave the
file. This file can be loaded in as a replacement for the http scheme. If reloaded
as such, there will be no change.
Now, you have to begin to understand how the schemes work. As the port for a scheme
is read, it goes through these stages (or functions):
init
open-proto
open
close
The lowest level functions, accessed more "invisibly," are read and write.
By playing with probe or print statements and then accessing a web page, one can begin
to get more information at a stage of the process as needed. You will find after a lot
of experimentation that much of the reading is done at a very low level, like through
'read-io. This function stores its data in temporary spaces in this protocol, like the
port/state object that is used during the access. If you try to access these data, you
inadvertently end up botching the operation, so care must be used and there are definitely
limits to what can be used. The only way to bypass some of the limits would be to extensively
modify the scheme, which can be done with some work and care, or to create your own,
which is reinventing the wheel.
It may be enough to get you going, but I would not be overenthusiastic about thinking
you will be able the scrounge things like transfer rates!
Hope this gives you some help.
--Scott Jones
[4/9] from: antonr:lexicon at: 15-Apr-2004 17:51
Scott, I think you are a bit negative about transfer rates. :)
Surely there is a way.
Here also, is an easier way to save the scheme:
write clipboard:// join {rebol [] system/schemes/http:} mold
system/schemes/http
Now open your favourite editor, paste into a new file and save.
Uh oh, now I'm into it...
Search for "read:". You have found the read function.
Now add a line after the "net-utils/net-log" line:
print join "bytes: " [port/state/num " at " now/time/precise]
Save the file and do it. Then test the change:
>> read http://www.rebol.com/
connecting to: www.rebol.com
bytes: 2048 at 17:39:47.731
bytes: 2048 at 17:39:47.741
bytes: 2048 at 17:39:47.741
bytes: 2048 at 17:39:48.332
bytes: 2048 at 17:39:48.332
bytes: 2048 at 17:39:48.332
bytes: 2048 at 17:39:48.552
== {<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html;CHARSET=iso-8859-1">
<META NAME="keywords" CONTENT="REBOL, pro...
With that info it's possible to make up some progress callback so you
can make a display showing transfer rate.
You would need to add a couple of variables to the http object, eg:
total-bytes
last-transfer-time
and reset these at the appropriate time.
Mmm.. while I'm here I can see the /custom refinement.
If I ever forget the custom refinement dialect I can
look here in the source.
Anton.
[5/9] from: brett:codeconscious at: 15-Apr-2004 19:59
Hi,
Have a look at /progress refinement on function Read-net.
If this is insufficient, you could take Scott and Anton's approach further
and maybe even allow a way to specify a on-read callback function using the
/custom refinement - which might be what Anton was implying.
Regards,
Brett.
[6/9] from: g:santilli:tiscalinet:it at: 15-Apr-2004 12:00
Hi Andreas,
On Wednesday, April 14, 2004, 8:09:45 PM, you wrote:
>> I was wondering, if its possible to put a hook somewhere (even if
>> deep) within the port or more specifically the html port handler, so
>> that we can update something elsewhere, do statistics or print out
>> transfer rate for example...
AB> I guess you rather mean the http ports, but anyway: I think that
AB> should be achievable by using async:// networking. But sorry, haven't
AB> done it (yet) and no so I've no example handy.
If you only need to report in/out speed in bytes/s or something
like that, I have an hacked version of async:// with speed
measuring in; it's not much tested, was not meant to be of generic
use, it's not based on the latest version of async://, and you'd
need to implement an HTTP client on top of it... but if you need
it, let me know and I can dig it out.
Regards,
Gabriele.
--
Gabriele Santilli <[g--santilli--tiscalinet--it]> -- REBOL Programmer
Amiga Group Italia sez. L'Aquila --- SOON: http://www.rebol.it/
[7/9] from: maximo:meteorstudios at: 15-Apr-2004 10:23
> -----Original Message-----
> From: Gabriele Santilli [mailto:[g--santilli--tiscalinet--it]]
<<quoted lines omitted: 7>>
> need to implement an HTTP client on top of it... but if you need
> it, let me know and I can dig it out.
what does "implement an HTTP client on top of it involve"? is that completely writing
an http scheme or can I borrow the one from within rebol and merge it?
maybe I'm completely off track!
-MAx
[8/9] from: g:santilli:tiscalinet:it at: 15-Apr-2004 16:57
Hi Maxim,
On Thursday, April 15, 2004, 4:23:51 PM, you wrote:
MOA> what does "implement an HTTP client on top of it
MOA> involve"? is that completely writing an http scheme or can I
MOA> borrow the one from within rebol and merge it?
Borrowing the one in REBOL wouldn't work, as async:// works
differently than tcp://. (You could reuse some of the code...) I
guess it would be much easier to patch around the current HTTP
scheme; if you don't need to be accurate just changing the
lowlevel read and write functions in the handler would be enough.
Regards,
Gabriele.
--
Gabriele Santilli <[g--santilli--tiscalinet--it]> -- REBOL Programmer
Amiga Group Italia sez. L'Aquila --- SOON: http://www.rebol.it/
[9/9] from: Gary:Jones:usap:gov at: 16-Apr-2004 12:35
From: Anton Rolls
AR> Scott, I think you are a bit negative about transfer rates. :)
AR> Surely there is a way.
....
It was one of those things were as soon as I sent my email, I, too, thought of a way
to show progress, but our satellite signals times are limited. My guess is that there
will already be 3 more ways to do it by the time this arrives!
Not unlike Anton's way in using read, I capture the actual bytes read from the return
of read-io and accumulate these in a variable. If the site provides content length,
then the patch can compare sent to expected. Unfortunately, not all servers/websites
send the content length.
At the end of the http scheme, and the variable "bytes-transferred", as in:
...
cache-size: 5
user-agent: "REBOL View 1.2.10.3.1"
bytes-transferred: 0
]
Then change the read function to (watch for line wrap):
read: func [
port "An open port spec"
data "A buffer to use for the read"
/local bytes-read
][
net-utils/net-log ["low level read of " port/state/num "bytes"]
bytes-read: read-io port/sub-port data port/state/num
bytes-transferred: bytes-transferred + bytes-read
if port/locals/headers/content-length <> none [print [to-integer (bytes-transferred /
(to-integer port/locals/headers/content-length) * 100) "percent"]]
bytes-read
]
Reading a page in which the server gives content-length will allow the calculation of
percent received. I realized that the read function needs to return the bytes, read,
hence the last line.
Expanding on the idea, I decided for fun to try to create a visual feedback method that
gives progress bar when the content-length is known, or simple accumulated bytes retrieved,
when content-length was unknown. It requires a fair amount of patching.
Starting with a fresh copy of the http scheme, one first alters the continue-post function:
continue-post: func [/tunnel] [
response-line: system/words/pick port/sub-port 1
net-utils/net-log response-line
either none? response-line [do error] [
either none? result: select either tunnel [tunnel-actions] [response-actions]
response-code: to-integer second parse response-line none [do error] [do get result]
]
if show-progress [
either port/locals/headers/content-length <> none [
progress-message/pane: layout/origin/offset [p: progress] 0x0 0x0
show progress-message
][
progress-message/pane: layout/origin/offset [t: text 200x16] 0x0 0x0
show progress-message
]
]
]
Then following the close block and build-port further down, one inserts these lines
....
]
build-port ;insert following lines
if all [block? port/state/custom post-data: find port/state/custom to-lit-word 'progress
block? post-data] [
show-progress: true
progress-message: get to-word select port/state/custom to-lit-word 'progress
]
Then one changes the close function:
close: func [port][
if show-progress [
progress-message/pane: layout/origin/offset [t: text 200x16] 0x0 0x0
show progress-message
t/text: "Done!"
show t
show-progress: none
]
system/words/close port/sub-port
bytes-transferred: 0
]
One changes the read function:
read: func [
port "An open port spec"
data "A buffer to use for the read"
/local bytes-read
][
net-utils/net-log ["low level read of " port/state/num "bytes"]
bytes-read: read-io port/sub-port data port/state/num
bytes-transferred: bytes-transferred + bytes-read
if show-progress [
either port/locals/headers/content-length <> none [
p/data: bytes-transferred / (to-integer port/locals/headers/content-length)
show p
;loop 10000000 [] ;uncomment to see fast pages load
][
t/text: bytes-transferred
show t
;loop 10000000 []
]
]
bytes-read
]
Finally, one adds variables to the end of the scheme:
.....
user-agent: "REBOL View 1.2.10.3.1"
bytes-transferred: 0
show-progress: progress-message: p: t: none
]
One uses the custom refinement to pass the reference to a VID box:
view layout [
f: field 300x24
button "Get Page" [
read/custom trim to-url f/text ['progress :my-pane]
]
my-pane: box 200x16
button "Exit" [quit]
]
When one types in a url from a website where the server gives the content-length, the
box gets a progress bar embedded, which is then updated based on the percentage of bytes
read. When the url is served from a server that does not give the content-length, then
the bytes retrieved are displayed. After completed, the progress box is converted to
text that says "Done!" I have commented out some do nothing loops in case your downloads
are too fast, and you are trying to see the changes at work.
I was just about to send this when I received the most recent set of emails, including
Max's own patch on read-net. Oh well, it still may prove useful to him or someone else
in another context and/or to take advantage of all the extra features of the http protocol.
--Scott Jones
Notes
- Quoted lines have been omitted from some messages.
View the message alone to see the lines that have been omitted