Mailing List Archive: 49091 messages
  • Home
  • Script library
  • AltME Archive
  • Mailing list
  • Articles Index
  • Site search
 

[REBOL] Re: Problems handling large txt files with open/direct/binary

From: gscottjones:mchsi at: 6-Mar-2002 15:44

Hi, J.M. From: "j m"
> I am processing a large text data file (over 100 MB) > in a Linux RH 7.2 box with 256 MB memory. > > I tried to use open/direct/lines but I had problems with lines above 4096 > chars so I use open/direct/binary and the following function to read
lines:
<snip>
Depending on your needs, you may not need to include either the binary nor lines refinements to your open request. Using your function as a base, try something like the following: read-lines: func [ f [port!] /local line data] [ line: make string! 1000 while [all [ not parse/all line [to "^/" to end] data: copy/part f 1] ][ append line data ] trim/with line "^/" ] my-port: open/direct %my-large-file while [all [ 0 < length? my-line: copy read-lines my-port not-equal? my-line 'none] ] [print my-line] close my-port This algorithm will cut short if an end of line occurs "alone" (meaning two occur together). It may be easier to reverse the organization to something like: my-port: open/direct %my-large-file line: make string! 1000 while [data: copy/part my-port 1] [ append line data if parse/all line [to "^/" to end] [ ;do whatever with the line print trim/with line "^/" ;reset line to empty line: make string! 1000 ] ] close my-port Hope this helps you narrow in on the functionality for which you search. --Scott Jones