World: r3wp

so far, preallocating too large buffer takes much more time... but 
look at my funcs... I do even better  :-)

result: clear []


which re-uses the same block over and over, unless ladislav knows 
otherwise, the memory for that block isn't released by clear, only 
its internal size is reset to 0  which is why clear is so fast AFAIK.

so it will grow to match the needs of the dataset, optimising itself 
in size within a few searches and then not re-allocating itself too 
often.

there are many important differences:

*make series 0 does not necessarily make a block

*regarding the allocation length - if you can estimate reliably the 
necessary length, then you are better off

with 10 million records, I estimated my dense datasets at about 120000 
records and did a few tests, they wher MUCH slower than using 
result: clear [ ]

but, make block! 1000 is certainly slower than make block! 0 if it 
turns out, that you need only 10 elements, e.g.

(estimated 120000 results)

thing is many times, if not most of the times, you don't need to 
copy the result as a new block, and that also saves A LOT of ram 
in my tests...


overall, about 500MB of RAM where saved by not pre-allocating large 
buffers

memory reuse: yes, if it is possible at all (it is like optimizing 
GC by hand), but that can be used only in special circumstances

for searches its usually ok... since the result is often used only 
to cycle over and create other data.

and in any case, you can copy the result of the search, at which 
point you have a perfect size and as little wasted RAM as possible.

Max - where do I get the dataset from, if I would try to rewrite 
your find-fast into a version using 'parse? :-) Do you generate one?

look in profiling, there is a full script with verbose printing and 
everything you need, just replace the loop in one of the funcs  :-)

you can easily compare your results with the current best ... I'll 
be happy if you can beat the ultimate-find and give the exact same 
feature...

searching on any field of a record and return the whole record.

overall, about 500MB of RAM where saved by not pre-allocating large 
buffers

 <- hmm... I thought the allocation did not necessarily mean usage, 
 only that GC is simpler, or is it different under Unix and Windows?.

I use windows task manager to look at ram use... the peak was 900MB 
and average was 700MB... removing pre-allocation it went down to 
350 with peaks of  ~ 500  IIRC

linux usually has more precise RAM reports AFAIK.

ok, let's say you allocate 2 GB, if you can, does Windows start to 
swap?

because if Windows only reports allocation and not actual use, then 
the task manager doesn't report true usage.

the process manager reports  a few different values, current, swapped, 
peak, and some more obscure ones.

if an application allocates and reserves 2GB I really don't care 
if its only using 10mb of it... my system is clogged and its not 
the OS's fault.

though I did a special of XP install which forces the OS NEVER to 
swap... and XP tends to be MUCH smoother because of it.

(special install of XP)  playing around with some obscure registry 
keys.

though for these tests, no swapping occured.

I recently watched a talk by Poul Henning Kamp, author of Varnish, 
who talked about how many people misunderstand how memory allocation 
works in modern OS'es. Since he's a FreeBSD kernel developer, he 
has some bias, but he made some interesting points in that memory 
allocation is nearly free in various unixes, but most people ignore 
that an only allocate, perhaps just enough or below what they need.

Whether this can be translated directly to REBOL, I don't know.

problem is when you task switch, or run several RAM intensive apps... 
they do kill each other, even on unix.

but that's because the RAM is actually used, correct?

(based on rendering 3D animations which required 4GB of swap file 
, just to load a scene  ;-)

yes... but as long as only one application is running the CPU, you 
can have A LOT of apps in virtual RAM without real system slow down 
(on unix).

I guess I'm wrong with Windows. allocating a 100 MB string takes 
time.

takes even longer under OSX.

while[here: find/skip here key 2][insert tail result second here 
here: at here 2]

rebol0  (untried)   suspect  parse is more efficent

ahh party moved to profileing and it has all been done

For more precise system usage under Windows, please use SysInternals 
tools (now part of MS)

http://technet.microsoft.com/en-us/sysinternals/default.aspx... 
look at RAM Map, Process Manager, etc.

Q. how to use a word as a string value in path?
ie: ["a" 1 "b" 2]
n: "b"
ie/(n)
>> 2

nvm

very strange    a: true      logic? a  => true

but a:[true]  logic? a/1 => false

That's because [true] is a word, not a value. Try this:
     a: reduce [true]  logic? a/1  >> true

oki thanks

What's the advantage of using words in blocks? 
ie: [a "one b "two] vs ie2: ["a" "one" "b" "two"]

seeing you run out global word space very quickly?

dialects and selection

ie2/("a")
>> "one"

so dialects then?

yes, and the code readability maybe - ie2/("a") vs ie2/a

using words as table keys is probably not a good idea.

why? because of word limit? or any other consequences?