World: r3wp

it's maxim's ultimate-find above ( and im using real world data)

ladislav, there is a script earlier in this discussion which has 
a complete working example.

aha, you are using real-world data. OK, then, you should tell me 
how many matches you see

(and a revision to ultimate-find, just after it)

the example shows the problem very well.

3270 matches

the example creates several sets of data with different organizations 
and it compares all of them amongst each other.


so with that script, you should be able to do all the analysis you 
need.

495 matches against the same 732981 hash takes only .003

above I said "its rehashing its content... which is strange."

that is a guess... it should say:

its *might* be rehashing its content... which *would be* strange.

hmm, what if the hash is optimized for unique elements?

...then you are most probably out of luck trying to use hash! for 
indexing purposes

ah   yes.... ususally a hash will have to skip over  elements which 
return the same hash key.


so if your table has a few thousand similar items, you aren't benifiting 
from the hashing... and its quite possible that looking up a hash 
itself is actually longer when it has to skip over and over (comparing 
data on top of the hash).


though one could argue, that the speeds should be a bit slower than 
using a block, not this slower...  possibly related to the implementation 
itself.

my dilema is indexing triples in a key/value world

generall speaking :) ?

you could have a look at the various dedicated triplestores available 
(even though many of them have a semweb/rdf/... background).

or have a look at cassandra and/or monetdb (w/o knowing anything 
about your intended usage)

yeah, I've looked a a few

rdf is to xml what War and Peace is to Cat in the Hat -- Triples 
are working even with Maxim's code above (just not in hashes for 
more than a query with a single value).. but i crave the speed of 
index? against large datasets.

I WILL NOT STOP TILL I HAVE A FAST AND SIMPLE TRIPLE STORE!  
(sleep is my enemy)

terry, index? is not a procedure within rebol .. its the same as 
length?
its a stored value which is simply looked up when you call index?


nothing will be as fast as index?    its the "getting to" index which 
consumes cycles

Where's the dilema ? you just have to maintain 3 indexes at the same 
time (for triples), there isn't any other choice if you looking for 
speed on readings.

i know .. keys can be integers that are indexes of values in map! 
or hash.

yeah Steeve, im scratching out notes on that now.. it's not quite 
as simple as it sounds

ie: a value might be a large binary ..

1 GB values as keys don't work very well.

I already said to you to compute a checksum to build keys from large 
data, it's built-in in Rebol

yeah, but then you risk collisions

with an md5 checksum ??? don't be silly :-)

you can negate collisions by building two checksums out of different 
properties of you data and merging them.

fair enough.. not running bank accounts with this thing

the other issue is the time it takes to build the checksum vs brute 
force

but it will be 100 or 1000 times faster, then to access the data 
using an index.

your actual trial to make a lookup with foreach or find+loop is insanly 
slow by comparison

Got to decide what is more important:
-- time to build data structure
-- time to update it (add/remove on the fly)
-- time to search it

And build data structures optimized to your priorities. There is 
no one true solution, just the best match for the situation at hand.

*current trial

ok.. here's an example.. 

take this simple rdf triple   "Tweety" "isa" "Canary"
How would create 3 indexes to manage it and 10,000,000 like it?

It's the problem, I think Terry can't decide :-)

Ok, I give it to you...

Tweety
 "age" "75"
Steeve
 "isa" "Rebol"
Steeve
 "age" "unknown"

I have a system working now that's fast enough.. but I'm a speed 
junkie.. there must be a BEST way (not better... BEST)

First i add the triple in the triples store (a simple block)

The, I recovers its index from the block (actually it's the last 
one)

And i use this index as the value for the 3 indexes, and i create 
the keys+value
Tweety
: index
Isa
: index
Canary
: index

that's all...

yeah, but ...

Perhaps the explanations are not clear, but it's pretty clear in 
my head ;-)

>> ie: ["Tweety" "isa" "canary"]
== ["Tweety" "isa" "canary"]
>> index? find ie "Tweety"
== 1

Great.. now what

( only in Rebol does ie: actually mean ie: )

(added In index1) "Tweety"= index
(added In index2) "Isa"= index
(added In index3) "Canary"= index

(added In index1) "Tweety"= index
(added In index2) "Isa"= index
(added In index3) "Canary"= index