Mailing List Archive: 49091 messages
  • Home
  • Script library
  • AltME Archive
  • Mailing list
  • Articles Index
  • Site search

[REBOL] Re: Large file compare

From: valleyroad:gmx at: 8-Jun-2005 15:48

Hi Gabriele, good hints. So, i now use read/line instead of read and write out the result from the difference operation immediatly and remove it from memory. This drops the actual memory consumption to 140 MB during intersect and 120 MB during difference. But i still think it will become too big when operating on the whole set. As the file content is very trivial like "2348246864;PCINIT2" and can be sorted, i tink of something like stepping through the files line by line and compare the line content. The file which have the lead in the comparison will be alternating, depending, if there is a difference in the first or second column. This only works, when both files are sorted identically. There will be a minimum memory consumption. But, what i don't know is, what commands to use as they must remember the positions in the files. I will think on this further. Perhaps you have good idea how this could be implemented. What i don't know know is, if this will be fast enough. Thanks Thorsten On Wed, 8 Jun 2005 12:31:56 +0200, "Gabriele Santilli" <[gabriele--colellachiara--com]> said:
> Hi Thorsten, > > On Wednesday, June 8, 2005, 11:53:08 AM, you wrote: > > TM> a: read %testfile1.txt > TM> b: read %testfile2.txt > > Did you mean READ/LINES? > > TM> inboth: intersect a b > TM> only_a: difference inboth a > TM> only_b: difference inboth b > > TM> My question is, if there are better ways in rebol to achive the same > with > TM> lesser memory consumption?? > > Yes - don't load the whole files in memory. :-) > > Is the difference going to be big too? If so you may want to avoid > keeping it in memory too. > > OTOH, if you have enough memory for the operation, doing it all in > memory is going to be much faster. > > Regards, > Gabriele. > -- > Gabriele Santilli <[g--santilli--tiscalinet--it]> -- REBOL Programmer > Amiga Group Italia sez. L'Aquila --- SOON: > > -- > To unsubscribe from the list, just send an email to > lists at with unsubscribe as the subject. >
-- Melian Solutions Thorsten Moeller Mail: [tmoeller--fastmail--fm] -- - One of many happy users: -- Geschenkt: 3 Monate GMX ProMail gratis + 3 Ausgaben stern gratis ++ Jetzt anmelden & testen ++ ++