Scan for files through deep directory
[1/4] from: tmoeller::fastmail::fm at: 3-Mar-2006 13:34
Hi, i am playing around with various versions of how to scan through a deep directory tree to find .cmd files. They are basically all working and doing their job, creating a file containing the full path and filename for the results. The problem is, that all these versions consume up to 500 MB of Memory which is too much on my Notebook and even a lot on normal Computers. Is there way to achieve this with less memory consumption??? Perhaps i am missing something. I tried out some scripts i found, looking around on the internet, but they all have the same problem. Maybe some Guru is around to shade some light on me. Thorsten -- http://www.fastmail.fm - The way an email service should be
[2/4] from: SunandaDH::aol::com at: 3-Mar-2006 10:02
> Is there way to achieve this with less memory consumption???
Undoubtable! But it would help having a sample bad script and some knowledge of what is going wrong. Diagnosing blindly, I suspect most such scripts are written to run recursively, so a very deep folder structure could cause them to need a lot of stack space. A non-recursive solution will need far less interim storage. But 500meg sounds a lot. I dusted off the equivalent script I once wrote and tried it on my folder structure, and it needed less than 100K of interim storage. But then may not have so deep a tree as you. Can you post an example of the code you are using? Sunanda
[3/4] from: tmoeller::fastmail::fm at: 5-Mar-2006 19:12
Hi Sunanda, this is the function i used last. It is a modification of a script i found. dir-tree: func [ current-path [file! url!] "directory to explore" /inner tree [block!] "useful to avoid stack overflow" /depth "recursion depth, 1 for current level, -1 for infinite" depth-arg [integer!] /local current-list sub-tree item server ][ if all [not inner not block? tree] [tree: copy ] depth-arg: either all [depth integer? depth-arg] [depth-arg - 1][-1] current-list: read current-path if all [not none? current-list] [ foreach item current-list [ if not find/any to-string item "*log*" [ if any [find/any to-string item ".cmd" find/any to-string item "/"] [ if find/any to-string item ".cmd" [write/append %/c/allcmds.txt join current-path/:item newline] if all [dir? current-path/:item not-equal? depth-arg 0] [ sub-tree: copy  dir-tree/inner/depth current-path/:item sub-tree depth-arg ] ] ] ] ] ] Perhaps you have some advice for me. Thanks Thorsten On Fri, 3 Mar 2006 10:02:14 EST, SunandaDH-aol.com said:
> Thorsten: > > Is there way to achieve this with less memory consumption???
<<quoted lines omitted: 17>>> To unsubscribe from the list, just send an email to > lists at rebol.com with unsubscribe as the subject.
-- http://www.fastmail.fm - IMAP accessible web-mail
[4/4] from: SunandaDH::aol::com at: 5-Mar-2006 20:26
> Perhaps you have some advice for me.
I compared the runstats with my standard folder search routine. Yours seems to need twice the working space. But that's still nowhere near close to 500Meg. If the comment in your code is for real ("useful to avoid stack overflow") then you must have subfolders more than 8000 deep...When you said deeply nested, you weren't joking! The precise max stack depth depends on the version of REBOL and the platform you run on. But if you have had to tweak code to avoid the max stack depth, maybe it is time to refactor as a non-recursive routine. *** As an example, here is two versions of my own folder search. The first is recursive, and the second depth-first iterative. They don't do *quite* the same as yours (they return a block of all files that matched at least one of the target substrings). But you could easily adapt them -- or post-process the output Sunanda. =================== find-files-recursive: func [ folder [file!] targets [block!] /local matches ][ matches: copy  foreach file read folder [ either dir? join folder file [ append matches find-files-recursive join folder file targets ][ file: to-string file foreach target targets [ if find/any file target [ append matches join folder file break ] ;; if ] ;; for ] ;; either ] ;; for return matches ] print length? res: find-files-recursive %/c/ [".txt" "*log"] =================== find-files-iterative: func [ folder [file!] targets [block!] /local matches folders-to-search current-folder ][ matches: copy  folders-to-search: copy  insert folders-to-search folder while [0 <> length? folders-to-search][ current-folder: folders-to-search/1 remove folders-to-search ;; drop first entry foreach file read current-folder [ either dir? join current-folder file [ insert folders-to-search join current-folder file ][ file: to-string file foreach target targets [ if find/any file target [ append matches join folder file break ] ;; if ] ;; for ] ;; either ] ;; for ] ;;while return matches ] print length? res: find-files-iterative %/c/ [".txt" "*log"]
- Quoted lines have been omitted from some messages.
View the message alone to see the lines that have been omitted