Problems handling large txt files with open/direct/binary
[1/2] from: jmalv::hotmail::com at: 6-Mar-2002 14:23
Hi all,
I am processing a large text data file (over 100 MB) in a Linux RH 7.2 box
with 256 MB memory.
I tried to use open/direct/lines but I had problems with lines above 4096
chars so I use open/direct/binary and the following function to read lines:
read-lines: func [ f [port!] /local line data ] [
line: copy ""
while [ (data: copy/part f 1) <> #{0A} ] [
append line data
if data == none [ break ]
]
if data == none [ line: none ]
line
]
Do you have any suggestions for the following problems:
1. I can not get the loop to finish nicely at EOF
The loop is ---> while [ not none? line ] [ ]
2. I get the following error but I have no info about it:
Invalid datatype during recycle
Ocasionally I also get a segmentation fault
Any ideas are welcome. Thanks a lot
_________________________________________________________________
Join the world’s largest e-mail service with MSN Hotmail.
[2/2] from: gscottjones:mchsi at: 6-Mar-2002 15:44
Hi, J.M.
From: "j m"
> I am processing a large text data file (over 100 MB)
> in a Linux RH 7.2 box with 256 MB memory.
>
> I tried to use open/direct/lines but I had problems with lines above 4096
> chars so I use open/direct/binary and the following function to read
lines:
<snip>
Depending on your needs, you may not need to include either the binary nor
lines refinements to your open request. Using your function as a base, try
something like the following:
read-lines: func [ f [port!] /local line data] [
line: make string! 1000
while [all [
not parse/all line [to "^/" to end]
data: copy/part f 1]
][
append line data
]
trim/with line "^/"
]
my-port: open/direct %my-large-file
while [all [
0 < length? my-line: copy read-lines my-port
not-equal? my-line 'none]
] [print my-line]
close my-port
This algorithm will cut short if an end of line occurs "alone" (meaning two
occur together). It may be easier to reverse the organization to something
like:
my-port: open/direct %my-large-file
line: make string! 1000
while [data: copy/part my-port 1] [
append line data
if parse/all line [to "^/" to end] [
;do whatever with the line
print trim/with line "^/"
;reset line to empty
line: make string! 1000
]
]
close my-port
Hope this helps you narrow in on the functionality for which you search.
--Scott Jones