Mailing List Archive: 49091 messages
  • Home
  • Script library
  • AltME Archive
  • Mailing list
  • Articles Index
  • Site search

[REBOL] Re: Problems handling large txt files with open/direct/binary

From: gscottjones:mchsi at: 6-Mar-2002 15:44

Hi, J.M. From: "j m"
> I am processing a large text data file (over 100 MB) > in a Linux RH 7.2 box with 256 MB memory. > > I tried to use open/direct/lines but I had problems with lines above 4096 > chars so I use open/direct/binary and the following function to read
Depending on your needs, you may not need to include either the binary nor lines refinements to your open request. Using your function as a base, try something like the following: read-lines: func [ f [port!] /local line data] [ line: make string! 1000 while [all [ not parse/all line [to "^/" to end] data: copy/part f 1] ][ append line data ] trim/with line "^/" ] my-port: open/direct %my-large-file while [all [ 0 < length? my-line: copy read-lines my-port not-equal? my-line 'none] ] [print my-line] close my-port This algorithm will cut short if an end of line occurs "alone" (meaning two occur together). It may be easier to reverse the organization to something like: my-port: open/direct %my-large-file line: make string! 1000 while [data: copy/part my-port 1] [ append line data if parse/all line [to "^/" to end] [ ;do whatever with the line print trim/with line "^/" ;reset line to empty line: make string! 1000 ] ] close my-port Hope this helps you narrow in on the functionality for which you search. --Scott Jones