Scan for files through deep directory
[1/4] from: tmoeller::fastmail::fm at: 3-Mar-2006 13:34
Hi,
i am playing around with various versions of how to scan through a deep
directory tree to find .cmd files. They are basically all working and
doing their job, creating a file containing the full path and filename
for the results.
The problem is, that all these versions consume up to 500 MB of Memory
which is too much on my Notebook and even a lot on normal Computers.
Is there way to achieve this with less memory consumption???
Perhaps i am missing something. I tried out some scripts i found,
looking around on the internet, but they all have the same problem.
Maybe some Guru is around to shade some light on me.
Thorsten
--
http://www.fastmail.fm - The way an email service should be
[2/4] from: SunandaDH::aol::com at: 3-Mar-2006 10:02
Thorsten:
> Is there way to achieve this with less memory consumption???
Undoubtable!
But it would help having a sample bad script and some knowledge of what is
going wrong.
Diagnosing blindly, I suspect most such scripts are written to run
recursively, so a very deep folder structure could cause them to need a lot of stack
space.
A non-recursive solution will need far less interim storage.
But 500meg sounds a lot.
I dusted off the equivalent script I once wrote and tried it on my folder
structure, and it needed less than 100K of interim storage. But then may not have
so deep a tree as you.
Can you post an example of the code you are using?
Sunanda
[3/4] from: tmoeller::fastmail::fm at: 5-Mar-2006 19:12
Hi Sunanda,
this is the function i used last. It is a modification of a script i
found.
dir-tree: func [
current-path [file! url!] "directory to explore"
/inner
tree [block!] "useful to avoid stack overflow"
/depth "recursion depth, 1 for current level, -1 for infinite"
depth-arg [integer!]
/local
current-list
sub-tree
item
server
][
if all [not inner not block? tree] [tree: copy []]
depth-arg: either all [depth integer? depth-arg] [depth-arg - 1][-1]
current-list: read current-path
if all [not none? current-list] [
foreach item current-list [
if not find/any to-string item "*log*" [
if any [find/any to-string item ".cmd" find/any to-string item
"/"] [
if find/any to-string item ".cmd" [write/append
%/c/allcmds.txt join current-path/:item newline]
if all [dir? current-path/:item not-equal? depth-arg 0] [
sub-tree: copy []
dir-tree/inner/depth current-path/:item sub-tree
depth-arg
]
]
]
]
]
]
Perhaps you have some advice for me.
Thanks
Thorsten
On Fri, 3 Mar 2006 10:02:14 EST, SunandaDH-aol.com said:
> Thorsten:
> > Is there way to achieve this with less memory consumption???
<<quoted lines omitted: 17>>
> To unsubscribe from the list, just send an email to
> lists at rebol.com with unsubscribe as the subject.
--
http://www.fastmail.fm - IMAP accessible web-mail
[4/4] from: SunandaDH::aol::com at: 5-Mar-2006 20:26
Thorsten:
> Perhaps you have some advice for me.
I compared the runstats with my standard folder search routine. Yours seems
to need twice the working space. But that's still nowhere near close to 500Meg.
If the comment in your code is for real ("useful to avoid stack overflow")
then you must have subfolders more than 8000 deep...When you said deeply nested,
you weren't joking!
The precise max stack depth depends on the version of REBOL and the platform
you run on. But if you have had to tweak code to avoid the max stack depth,
maybe it is time to refactor as a non-recursive routine.
***
As an example, here is two versions of my own folder search. The first is
recursive, and the second depth-first iterative. They don't do *quite* the same
as yours (they return a block of all files that matched at least one of the
target substrings). But you could easily adapt them -- or post-process the
output
Sunanda.
===================
find-files-recursive: func [
folder [file!]
targets [block!]
/local
matches
][
matches: copy []
foreach file read folder [
either dir? join folder file [
append matches find-files-recursive join folder file targets
][
file: to-string file
foreach target targets [
if find/any file target [
append matches join folder file
break
] ;; if
] ;; for
] ;; either
] ;; for
return matches
]
print length? res: find-files-recursive %/c/ [".txt" "*log"]
===================
find-files-iterative: func [
folder [file!]
targets [block!]
/local
matches
folders-to-search
current-folder
][
matches: copy []
folders-to-search: copy []
insert folders-to-search folder
while [0 <> length? folders-to-search][
current-folder: folders-to-search/1
remove folders-to-search ;; drop first entry
foreach file read current-folder [
either dir? join current-folder file [
insert folders-to-search join current-folder file
][
file: to-string file
foreach target targets [
if find/any file target [
append matches join folder file
break
] ;; if
] ;; for
] ;; either
] ;; for
] ;;while
return matches
]
print length? res: find-files-iterative %/c/ [".txt" "*log"]
Notes
- Quoted lines have been omitted from some messages.
View the message alone to see the lines that have been omitted