[REBOL] Re: Depth first search
From: brett:codeconscious at: 7-Dec-2001 23:35
Hi Nate,
I've been looking at your script. I believe that it has a couple of
problems. First the less serious one. It
seems to create invalid urls. Something like http://domain//toplevel/...
This will mean that you will get errors reading these urls.
The more serious error which I belive is causing you the performance
problems is this line from the LinkCrawler function:
append _LINKS LinkCrawler (depth - 1) getlinks url _LINKS
Note that it is the last line in this function. After append is finished it
will return _LINKS as it's result. This in turn is
returned as the result of the function LinkCrawler. It is returned "by
reference". Now LinkCrawler is recursive, so you are actually doing the
following:
append _links _links
And by so doing, creating huge blocks of duplicate urls. That's your
performance problem I believe.
Brett.
