Mailing List Archive: 49091 messages
  • Home
  • Script library
  • AltME Archive
  • Mailing list
  • Articles Index
  • Site search
 

Problems handling large txt files with open/direct/binary

 [1/2] from: jmalv::hotmail::com at: 6-Mar-2002 14:23


Hi all, I am processing a large text data file (over 100 MB) in a Linux RH 7.2 box with 256 MB memory. I tried to use open/direct/lines but I had problems with lines above 4096 chars so I use open/direct/binary and the following function to read lines: read-lines: func [ f [port!] /local line data ] [ line: copy "" while [ (data: copy/part f 1) <> #{0A} ] [ append line data if data == none [ break ] ] if data == none [ line: none ] line ] Do you have any suggestions for the following problems: 1. I can not get the loop to finish nicely at EOF The loop is ---> while [ not none? line ] [ ] 2. I get the following error but I have no info about it: Invalid datatype during recycle Ocasionally I also get a segmentation fault Any ideas are welcome. Thanks a lot _________________________________________________________________ Join the world’s largest e-mail service with MSN Hotmail.

 [2/2] from: gscottjones:mchsi at: 6-Mar-2002 15:44


Hi, J.M. From: "j m"
> I am processing a large text data file (over 100 MB) > in a Linux RH 7.2 box with 256 MB memory. > > I tried to use open/direct/lines but I had problems with lines above 4096 > chars so I use open/direct/binary and the following function to read
lines:
<snip>
Depending on your needs, you may not need to include either the binary nor lines refinements to your open request. Using your function as a base, try something like the following: read-lines: func [ f [port!] /local line data] [ line: make string! 1000 while [all [ not parse/all line [to "^/" to end] data: copy/part f 1] ][ append line data ] trim/with line "^/" ] my-port: open/direct %my-large-file while [all [ 0 < length? my-line: copy read-lines my-port not-equal? my-line 'none] ] [print my-line] close my-port This algorithm will cut short if an end of line occurs "alone" (meaning two occur together). It may be easier to reverse the organization to something like: my-port: open/direct %my-large-file line: make string! 1000 while [data: copy/part my-port 1] [ append line data if parse/all line [to "^/" to end] [ ;do whatever with the line print trim/with line "^/" ;reset line to empty line: make string! 1000 ] ] close my-port Hope this helps you narrow in on the functionality for which you search. --Scott Jones