World: r3wp

what does it mean "field you want to search on"?

what item of your record you want to match against... basically what 
you meant by searching  subject, predicate or value

yeah, so if i say 1 then that's all subjects?

yep it will match the value against subjects only.

no go then

>> dataset

== [6 27744 92191 1 61175 9905 9 62225 72852 7 31935 71556 4 59248

>> ultimate-find dataset 1 1 zz 1
ultimate find(): 1. -> 0:00:00.031   0 matches found
== []

should have picked up that fourth index (1)

this worked.. 
>> ultimate-find dataset 6 1 zz 1
ultimate find(): 1. -> 0:00:00.219   1 matches found

zz should be 3.

oh.. i thought zz was the length? of dataset

ah.. that works GREAT

>> dataset: [1 "a" "B" 4 "h" "V" 1 "z" "Z" 4 "p" "d" 4 "k" "i" 4 
"y" "o"]
== [1 "a" "B" 4 "h" "V" 1 "z" "Z" 4 "p" "d" 4 "k" "i" 4 "y" "o"]
>> ultimate-find dataset 4 1 3 1
ultimate find(): 1. -> 0:00   4 matches found
== [4 "h" "V" 4 "p" "d" 4 "k" "i" 4 "y" "o"]

very nice

ultimate find(): 1. -> 0:00   1 matches found
== [1 "a" "B"]

:-)

oops missing cmd line...

so if the dataset is key/value just use 2 as the record-length

>> ultimate-find dataset "a" 2 3 1
ultimate find(): 1. -> 0:00   1 matches found
== [1 "a" "B"]

yep

cool

i inserted "maxim" "age" "unknown" and appended "terry" "age" "42" 
into the dataset containing 6 million records.. 

>> ultimate-find dataset "age"  2 3 1
ultimate find(): 1. -> 0:00:00.093   2 matches found
== ["maximn" "age" "unknown" "terry" "age" "42"]

I'll say that's a respectable time... and the leading contestant 
:)

:-)

now if only i was 42 again...

But wait, there's more.... 

convert dataset to hash! and run ultimate-find again!

>> ultimate-find dataset "age"  2 3 100
ultimate find():  -> 0:00   2 matches found
== ["maximn" "age" "unknown" "terry" "age" "42"]

100 iterations not even registering

1000 iterations 0.40

OMG !

exactly

but I'm getting an odd deadlock here on some tests... hum...

I'm getting extremely slow results on dense tests...

interesting... im not too worried as density isn't a big issue with 
triple stores

im off.. good luck with your optimizations

I'm talking like 100 times worse!   the larger the list the worse 
it gets... seems like an exponential issue.

that seems like an anomaly

both dense tests perform pretty much the same, the moment I convert 
it to a hash, it gets reallllly slow.

yeah, i see that too

mind you, that's pretty dense data

the strange thing is i did tests using a record size of 2, which 
wouldn't trigger strange mis aligned key/value issues.  I even removed 
the copy to make sure that wasn't the issue and one test with only 
400000 records took more than 4 minutes to complete vs .297 for the 
feach test!

I'm looking for the 6 integer.. it's still cranking and i can hear 
my system struggling..

must be a loop error

well, the results where the same at the end... pretty weird... maybe 
someone has encountered this before and can explain why this happens....

Max - just a question - wouldn't using parse be faster than find/skip?

my advice would be:

1) to test Parse as Pekr noted (traversing only the respective field)
2) to use a hash to index the respective field

I didn't do any parse test tweaks... but find/skip is very fast so 
far, we can skip over 100 million records within a millisecond.  
not sure parse can beat that

Did you find a solution to the density issue Max?

nope... I'm working on urgent stuff won't have time for a few days 
to put more time on this.

didn't tested since a while in R2, but in R3, parse is faster in 
most of the cases (if you write correctly the rules)

I'm wondering if it has something to do with recreating the hash 
each time a value is found?

Looking at Maxim's ulitmate-find (above, monday 11:32) , does anyone 
have an idea why when dealing with hash! , the more matches it finds, 
the slower it gets?

I think, that it is quite natural. You should probably generate some 
random data having (approximately) similar properties as what you 
intend to process and try some variant approaches to really find 
out, which one is best for the task. Do you know, that it is possible 
to index just a specific record field, i.e. you don't need to make 
a hash containing all the data from the database?