World: r3wp

There's a fundamental flaw with this.. let say i find 3000 customers, 
but now i want to pull their email property

foreach customers  customer [ (pseudo) intersect the  customer with 
i1 and email i2)
once is fine.. but 3000 loops is really slow

probably need to store the properties as objects

data: [ "bob" [ email "[bob-:-example-:-com]" lname "Jones"] ...]

then continue to index email, lname etc

OK, take all the noise above and throw it away.. i have the perfect 
solution (how's that for a tease?)

It ties in with my old AtomDB developments, but uses some of the 
methods we went over.. AtomDB used foreach to crawl.. fine for smaller 
datasets but too slow on large data

foreach is alway slower than find/part  so whatever algorithm you 
use, replace it with that in its loop and it will be even better.

It's becoming apparent that these large datasets need to be reduced 
when it comes to FINDing strings



imagine: ["one" "two" "three" "four" "five"... "one hundred million"] 
 

should be

o: ["one" "one hundred million"]
t; ["two" "three"]
f: ["four" "five"]

or even

fo: ["four" "fox" "food"]
fi: ["five" "fifty" "fiver niner"]

pull out the first twh chars and there's your search block

You could make a complete hash of it.

http://sociopathways.files.wordpress.com/2009/12/hash-browns.jpg

Steeve, back to your method with some changes.. 
Some early benchmarks on this new method
3,000,000 objects


subjects:  [ [age 42 fname "Steeve"][ fname "Tweety" age 75] [url 
"http://rebol.com"] ... ]

3,000,000 key / value pairs that index the subjects block

i1: [ "Steeve" 1 "Tweety" 2 "Rebol" 3 ... ] 

then.. 

	theindex: select i1 "Rebol"
	out: pick subjects theindex


I can do 1,000,000 of these in 1.3 seconds or one 769, 230 / s (older 
box with 2gb ram)

then finish the other two triples index (preds vals)

i2: [age [ 1 2] fname [1 2]  url [ 3]]

i3 should have it's values converted to md5s (as suggested) 

i3: [ #{12CE1E73A3EABF1623970A0C53B9CA1F} [3] ]

The problem with the other method was the overall cost of looking 
up properties .. if all you needed was the single property, (ie all 
things that are age = 75) then that was fine, but usually you want 
more info. So if you found 5000 things that match the query, you'd 
need to run the whole process 5000 times to get all their email addresses. 


It just makes more practical sense to keep things as things, along 
with all of their properties.

(I prefer the term "things" over "objects" as triples don't usually 
contain methods, just properties)

Any thoughts on speeding up intersect when comparing large blocks?

You should not use Intersect on blocks

(if you want speed)

my suggestion is to try it on hashes

intersect is a native so the only way you can get faster performance 
is if you know something about the blocks contents. For example, 
if both blocks contain sorted integers *and* one block is smaller 
than the other *and* the bigger block contains the first value of 
the smaller then the following code *may* be faster than intersect:

	a: [2 3 4]
	b: [1 2 3 4 5]
	x: last b
	b: find b first a
	r: copy []
	foreach i a [
		all [i > x break]
		all [find b i append r i]
	]
	b: first b

When using Intersect on blocks, you should take into account, that 
their contents are hashed first. Therefore, using hashes instead 
of blocks, the hashing operation itself may be spared.

but, I did not do any speed comparisons, so it is up to you, Terry

i just tried some comparisons and the time is the same

however, I think I've finally created a rocket db

the time is the same

 - then, it looks, that the Intersect native is not "clever enough", 
 doing an unnecessary operation.

nevertheless, hashing can certainly be spared, so, doing Intersect 
"by hand" can be faster, in my opinion

i have a block with 1 million sub blocks representing objects, and 
they each have a random 'age value  between 1 - 100.. i can pull 
the indexes of all the blocks where age = 42 in 0.7 seconds.. 
that's the slowest query i have.

in that same 1 million objects, i have an object with fname = "Bob" 
.. i can pull that out in 0.0014 secs.

There should be a ratio where the iterative method is faster than 
intersect, but some tests have to be done.
ratio = (lenght? bigest-block) / (length? smallest-block)

In my comparisons with Redis, I was getting 7200 GETS (pulling the 
value of a key) per second from a DB with 100,000 key/value pairs.. 
 With this rebol db, i can do 1,000,000 GETS in 0.64 seconds, from 
a db with 1,000,000 key/value pairs.

Terry's informations are quite inexact. Intersection of hashes is 
*much* faster, than intersection of blocks, exactly as I expected.

I agree:
>> b1: make hash! [1 2 3 4] b2: make hash! [2 3 4 5 6]
>> tm 100000 [intersect b1 b2]
== 0:00:00.313
>> b1: make block! [1 2 3 4] b2: make block! [2 3 4 5 6]
>> tm 100000 [intersect b1 b2]
== 0:00:00.766

My timings were for a script, where intersecting was one part. Whether 
intersecting block or hash made no noticeable difference.

Well, since your timings did not detect, that hash intersects are 
*much* faster than block intersects, this is a proof, that your timings 
do have very little in common with the speed of intersect

one odd thing i find with timing in general is why do i get random 
values each time i run a script? shouldn't it always be the same?

I would be very suspicious if the timings where the same. All machines 
these days are running multiple processes so your script's access 
to the processor will differ each time you run it.


Rebol itslef may well be inconsistent as the garbage collector will 
run at differnet points in you tests.

What do you mean by random values? I can see some differencies, but 
not so random. The function I use for timing is:

tm: func [count [integer!] code [block!] /local t][t: now/time/precise 
loop count code probe now/time/precise - t]

As opposed to that, I propose a more precise http://www.fm.tul.cz/~ladislav/rebol/timblk.r

Vista --> 0.000990825688073395

thanks, Steeve, that looks like 1 millisecond, taking into account 
the given 5% precision

Hi all, I adjusted the  http://www.fm.tul.cz/~ladislav/rebol/timblk.r
file to reflect Steeve's measurement, but would like to obtain more 
results (results for other operating systems). If you have the results 
that are not already in the file, please, let me know.

R2, XP ---> 1.54857142857143E-2

sorry, it's already in the file., And it's XP SP3

9.8876404494382E-4 Vista

XP x64
== 1.54678899082569E-2

thanks, I assume, Gregg, that your value is either for XP x64 SP2 
or SP3?

hmm, Paul's value is quite strange, taking into account, that Steeve 
measured it to be 1 ms

mine is on 32 bit Vista running AMD.

aha, OK, the difference is still just 1.12%, i.e. it is in the tolerance

Sorry, yes, SP2.

Btw, got the same results with R2 and R3

This test shows well why some graphical demos work bad on some systems 
even if the computers are faster than mine  (Vista with a tiny celeron).

hmm, I just checked a Vista Business 64 bit (not updated for quite 
some time), and it used 15.5 milliseconds too...

The same result with R2 and R3
 - of course, this value is a property of the operating system

it could have been that they use different timer libs.