open/direct and stream underrun?

[1/2] from: petr::krenzelok::trz::cz at: 5-Jun-2001 23:44

Hi, with great help of Scott I just made my POST http method for downloading large files working satisfactory. Once looking how nicely our 22 MB file downloads and is stored on shared Novell Netware drive, open/direcct draw my attention. I have basically 2 questions: Let's assume following code: while [data: copy/part http-port buffer-size][append file-port data] 1) We are faster than server. What happens? Am I correct if I assume, that in such case, we end up with not fully downloaded file? We don't wait for anything, right? So if there are no data coming from http-port, we quit ... Is it better to use cached port then? 2) Reverse scenario - our server sends us data "too fast", so in comparison our script is "too slow" and before we read out data out of the port, server sends us new data. But we are using /direct refinement, so what happens with older data? Does OS take care of tcp data caching to some degree? Does it have anything in common with TCP stack "window" size? Probably questions for Holger, but noone is prevented from answering them too :-) Thanks, -pekr-

[2/2] from: holger:rebol at: 12-Jun-2001 9:03

On Tue, Jun 05, 2001 at 11:44:55PM +0200, Petr Krenzelok wrote:

> while [data: copy/part http-port buffer-size][append file-port data] > > 1) We are faster than server. What happens? Am I correct if I assume, that > in such case, we end up with not fully downloaded file? We don't wait for > anything, right? So if there are no data coming from http-port, we quit ... > Is it better to use cached port then?

This depends on whether you use the /no-wait refinement in 'open: Without no-wait: 'copy waits until an end-of-file is received, i.e. instead of copy/part you could just as well use copy. With no-wait: 'copy returns an empty string if no data is available, i.e. your while-loop busy-loops until the server returns some data. You should insert a 'wait call into the condition block of your loop.

> 2) Reverse scenario - our server sends us data "too fast", so in comparison > our script is "too slow" and before we read out data out of the port, server > sends us new data. But we are using /direct refinement, so what happens with > older data? Does OS take care of tcp data caching to some degree?

Yes, of course. No data is lost.

> Does it > have anything in common with TCP stack "window" size?

Sort of, indirectly. The TCP "window" size is one of the parameters that determines how much data can be "in transit" in a TCP connection. Regardless of the window size you never lose data though. The worst case is that the connection gets stalled (backlogged) at the server, i.e. that the server cannot send any more data before you read some of the queued data from the port. -- Holger Kruse [holger--rebol--com]