Mailing List Archive: 49091 messages
  • Home
  • Script library
  • AltME Archive
  • Mailing list
  • Articles Index
  • Site search

[REBOL] unbuffered file reads (large files)

From: tim-johnsons::web::com at: 23-Oct-2007 10:46

Hello: I'm processing some large text files - 100,000 lines or more. It would seem to me that using 'open with the 'direct refinement would be the answer, but I'm seeing 1)buffering 2)problems terminating the read loop. What follows is a test rebol file and a little text file to test. I've made this to run as CGI, so that the port dump is a little more readable. I have further comments at the end. ;; rebol file - can run from command line or as CGI #!/usr/bin/rebol -cs REBOL[] print "Content-Type: text/html^/" print <pre> print "Read file with cache" inf: open/lines %test.txt while[not tail? inf][ print first inf inf: next inf ] close inf ;; works fine, but is buffered print "Read file without cache" inf: open/direct/lines %test.txt ;; help open says 'direct should be unbuffered ;; tail test fails immediately ;while[not tail? inf][ ; print first inf ; inf: next inf ; ] ;; use truth test of inf while[inf][ ?? inf ;; look at the 'state members, expecially 'inBuffer print first inf inf: next inf ] close inf ;; here is the input text file line one line two line three line four line five ;; comments follow: 1)the tail test fails in direct mode 2)the truth test for 'inf is not helpful either. 3)It looks to me like direct mode *is* buffered after the first read 4)The termination test could be something like if all[string? inf/states/inBuffer empty? inf/states/inbuffer][break] 5)But we still have buffered input right? What do you all think? Thanks