AltME groups: search

I think there's a big gap between the usefulness of a spanish or 
Italian localization and a chinese localization.

If Alias permits one to set up word equivalencies, then it's solving 
a problem I have with Rebmu.

(Which I will again say is not some kind of joke.  It is trying to 
position Rebol as the leader in Code Golf, and I'm tuning it so that 
knowledge transfer with Rebol is as good as it can get.  Random example: 
I did some tweaking so that now first+ isn't FP anymore, it's F+ 
and all the functions which end in "+" are understood as being in 
the family of those which modify their arguments.  A+ FOO BAR is 
understood to add foo to bar and store the result in foo, for example.)

Also I changed AN for AND~ and OO for OR~ into "a~" and "o~", thus 
keeping consistency with the Rebol notation for prefix operations, 
because the prefix ones are the ones that make sense in character 
count minimization.  I'm bending and thinking of what makes those 
look common and came up with "n~" for NOT, even though Rebol doesn't 
have a tilde after prefix NOT (since there's no infix not).  So it's 
a balance of consistency.

Anyway, the various crack-smoking mods I'm doing (such as if-true-mu 
being able to take constants in the code block and evaluate to those 
constants without the brackets) are intended to be minor things... 
something one could quickly unlearn in Rebol programming and say 
"Oh, right, you need a block around that."  Increasingly I'm backing 
off anything which is overly focused on character optimization at 
the price of creating a pattern that is too far off from a reasonable 
Rebol programming practice.

Another issue of using single character symbolic things for operators 
is that because they lack a "case", they don't play well with mushing. 
 [A&b] unmushes to [a: & b]...  [a&b] unmushes to [a&b]... [a&B] 
unmushes to [a& b] .... and [A&B] unmushes to [a&b:]

Consequently, you're better off with a two character A~ for AND~, 
because you'd end up having to throw in spaces on the left and right 
of the & all the time otherwise, which gets you three characters, 
and you'd probably have to throw one on the left half the rest of 
the time.

Is there a "weirdest dialect" award I'm going to win with this?  
:)

It doesn't make sense if trying to retain compatibility with lowercase 
Rebol code.  I want to be able to paste "to-string" in the middle 
of a Rebmu program and have it "just work"

Specifically, there are cases when lower numbered dots can connect 
and cross over the second digit of a larger numbered dot

That forces you to basically build a map of the numbers before you 
start drawing the lines :(

A lot of these problems have gotchas like that where you think "oh, 
easy I'll just do it like this..."

Hopefully you get the idea.  It's really simple.  I'm not trying 
to beat the winners based on libraries or tricks, I'm not trying 
to write clever code, I'm writing REALLY boring straightforward code 
with function definitions and everything which can be adapted easily 
if the problem spec is adapted.  And still within a stone's throw 
of the winners, or even beating them.  In this case I got 218 characters 
compared to the winning "perl" at 222.

Your unrebmu proposal would be a compiled dialect, with the target 
being DO dialect code. This is a good technique for many dialects, 
and can be very fast. That makes it a good idea for a demo.

It's a good question, and regarding that, I noted that many of the 
solutions to Code Golf problems utilize base-64.

Janko wanted some script obfuscator and as I see, Rebmu is already 
less readable than Brainfuck, so I think it's a good candidate.

Spite for Perl is a noble motivation :)

But also a desire to get "fringe"/"creative" people to become aware 
of Rebol.

That's why making an ultra-cool demo is a priority :)

I assume it's because of using a standard that expected the "do not 
compress" to be taken care of at a higher level.

In any case, the single-character operators in Rebmu are a tricky 
bunch for reasons I explained above, but one case that they do pop 
up in is assignments where the left hand is an all-caps target (set-word) 
and the right hand is a string or block or something.  That's when 
you can get them easily.

I think there's a big gap between the usefulness of a spanish or 
Italian localization and a chinese localization.

 - actually not, as Jerry mentioned some time ago, there are exactly 
 the same reasons why it is not useful

@Ladislav: Tonight I was discussing the Firefox/Mozilla thing with 
my roommate.  It was an issue where Mozilla was building a lot of 
functional stuff, but they didn't have quite the ability or agility 
to see and take advantage of opportunities.  Some weird dude came 
along and said "Hey, I'm gonna chop all this stuff off, and insert 
this stuff, and call it 'Firefox', and let's see what the heck happens..."

It would have enough overhead that it should not be there by default 
- LOAD does enough as it is. However, it might be an interesting 
tool for somebody, and a great demonstration. And it can be an add-on 
module, which makes it an even better demonstration.

In my opinion, Rebol is a more significant and interesting technical 
artifact than Mozilla.  Bear in mind I have a lot of context, so 
I'm not saying it's the greatest technology ever, I'm just going 
to say that on many axes it is essentially the best in its class 
that we know of.

:)  I skipped a step.

(and a revision to ultimate-find, just after it)

above I said "its rehashing its content... which is strange."

that is a guess... it should say:

its *might* be rehashing its content... which *would be* strange.

ah   yes.... ususally a hash will have to skip over  elements which 
return the same hash key.


so if your table has a few thousand similar items, you aren't benifiting 
from the hashing... and its quite possible that looking up a hash 
itself is actually longer when it has to skip over and over (comparing 
data on top of the hash).


though one could argue, that the speeds should be a bit slower than 
using a block, not this slower...  possibly related to the implementation 
itself.

my dilema is indexing triples in a key/value world

you could have a look at the various dedicated triplestores available 
(even though many of them have a semweb/rdf/... background).

or have a look at cassandra and/or monetdb (w/o knowing anything 
about your intended usage)

yeah, I've looked a a few

rdf is to xml what War and Peace is to Cat in the Hat -- Triples 
are working even with Maxim's code above (just not in hashes for 
more than a query with a single value).. but i crave the speed of 
index? against large datasets.

I WILL NOT STOP TILL I HAVE A FAST AND SIMPLE TRIPLE STORE!  
(sleep is my enemy)

terry, index? is not a procedure within rebol .. its the same as 
length?
its a stored value which is simply looked up when you call index?


nothing will be as fast as index?    its the "getting to" index which 
consumes cycles

ie: a value might be a large binary ..

I already said to you to compute a checksum to build keys from large 
data, it's built-in in Rebol

your actual trial to make a lookup with foreach or find+loop is insanly 
slow by comparison

I have a system working now that's fast enough.. but I'm a speed 
junkie.. there must be a BEST way (not better... BEST)

First i add the triple in the triples store (a simple block)

Ron Everett presented  a database that did much of what you want 
at DevCon2007. The live discussion is here:

   http://www.rebol.org/aga-display-posts.r?offset=0&post=r3wp500x1919
The video of the presentation may be on qtask.

Should have listened to my mother and became a lawyer.

Terry, I think INTERSECT is fine the way it is, and it's easy to 
wrap if you want.

    fold: func [
        series [series!]
        fn     [any-function!]
        /local value
    ][
        value: pick series 1
        foreach item next series [value: fn value item]
    ]

    intersect-all: func [
        series [block!] "A block of series values to intersect"
    ][
        fold series :intersect
    ]

I knew someone would comback with a function.. might as well have 
been you Gregg.
I like PHP's array_merge()

steeve, REBOL doesn't support path with strings, and furthermore, 
it would only return the first index, if you used it within a paren.


so I'd really like you to give a small snippet of code with your 
idea, I am curious about your method... cause I don't see *how* what 
you say works.

I would use map! (or hash!) as indexes and a key would contain one 
or several triples (inside a block)
verb/"age": [ index indexn ...]

To make it practical for other operations too, the triple store should 
be defined so, that:


1) every triple is stored as a fourtuple, the additional field being 
a unique triple ID
2) create the main index using the triple ID, 

3) create three field indices searchable by by the field contents, 
and "inside" also by the main ID


This way, the triple storage will be moderately fast for triple removal 
as well

For R2, I wrote a function called Stagger for that:
Stagger: func [L I /local][
    forall L [
        L/1: copy/part L I 
        remove/part next L (I - 1)
    ] 
    return L
]

>> Stagger ["2c3cukne98" "femaleend" "E1016 Camlok" "2c3cukne98" 
"isa" "cable" "2c3cukne98" "uid" "8558"] 3

== [["2c3cukne98" "femaleend" "E1016 Camlok"] ["2c3cukne98" "isa" 
"cable"] ["2c3cukne98" "uid" "8558"]]

AFAIK, R2 doesn't have a built in function for it (at least, I never 
ran across it)

It is?  I've only used it for relatively small data, so never had 
a problem..

Besides you don't need to store triples as blocks (they can be flattened) 
in the triple store, then you'll just need to do some math to calculate 
the index of a triple.

The advantage is that you could still use the triple store do some 
slow searchings.

yeah, i thought of that, but once the inference is there, not sure 
there's a need

another advantage of a flat triple store is for simple key/value 
lookups

ie: if you have a html element  <div id="20984"> where the contents 
of the div is  "pick ie 20984"
doesn't get any faster than that

And, this data is tight.. the  250,000 real world triples (manages 
a multi-million dollar electrical equipment rental biz, including 
 1500 contacts, invoicing, and 10,000s of items)  is only 9mb... 
zips to 1.3mb

that's all the data a logic for a medium sized company stored on 
a floppy.

To make it practical for other operations too, the triple store should 
be defined so, that:

1) every triple is stored as a fourtuple, the 
additional field being a unique triple ID...

Why not just use the index of the first part of the triple? 

ie: ["tweety" "isa" "canary" "Sylvester" "isa" "cat"...]


so 1 and 4 are the two triples( i1)  and the next two are i2 and 
i3

i1: [ "Tweety" [1] "Sylvester" [4][
i2: ["isa" [1 4]
i3: ["canary" [ 1] "cat" [4]]

how...

 - this way, you will have a triple ID that does not change even if 
 other triples are removed. That is not true for your index.

If you remove a triple, you'd need to reindex everything, no?

I have thought of a fourth ID for a triple before.. but it represents 
the triple itself. .. rarely needed it though. .there's always a 
way around

...but you are right, that there is always a way around

In my old AtomDB i made in Rebol some time ago, I had an index for 
each value.. ie: the number "42" was stored only once, and used the 
index with the inference.. 

values: [ "42" "November" "Baseball"..]


You could have a million birthdays recorded in "November", with a 
single element

billbdaymonth: 2
frankbdaymonth: 2
frankage: 1
the answer to life the universe and everything: 1

It makes more sense for larger values.. nice when you can use a single 
symbol to represent a 1GB binary

Anyway, thanks everyone for the help.

terry, any benchmark results on a dense search with a few million 
records?

also: how long is generating the index on that data

and how much RAM is your triple index consuming?

Ladislav, no prob with a map!

probably a faster way, but I'll let the gurus beat their chests over 
it :)

Izkata.. here's another option.. not sure how efficient it is.. 

orig: ["one" "two" "three" "four" "five" "six"]
blk: []

foreach [ a b c ] orig [ d: array/initial [1] reduce [a b c] append 
blk d]

Well, if your doing it that way, "append/only blk reduce [a b c]" 
should be faster - no call to 'array


May as well also have "blk: make block! (length? orig) / 3" - don't 
remember how much of a speedup that was in your previous tests, though

Max: Indexing 244,327 triples took 4:20 on a rather snappy windows 
box with 6Gb RAM
DB is consuming 114 Mb Ram

Within that data (real world) i have 3,733 cable triples ("dfuflfi33s", 
"isa" "cable") 
("dfuflfi33s" isa a unique 10 byte ID that i stick on everything)


Using the code posted earlier, i can pull a final block of indexes 
for each of those cables in 0.0012 seconds

pulling a single unique email address (out of 429) took  0.013

so the time seems somewhat fixed around 0.12 seconds regardless of 
the density

There's a fundamental flaw with this.. let say i find 3000 customers, 
but now i want to pull their email property

foreach customers  customer [ (pseudo) intersect the  customer with 
i1 and email i2)
once is fine.. but 3000 loops is really slow

probably need to store the properties as objects

data: [ "bob" [ email "[bob-:-example-:-com]" lname "Jones"] ...]

then continue to index email, lname etc

OK, take all the noise above and throw it away.. i have the perfect 
solution (how's that for a tease?)

You could make a complete hash of it.

intersect is a native so the only way you can get faster performance 
is if you know something about the blocks contents. For example, 
if both blocks contain sorted integers *and* one block is smaller 
than the other *and* the bigger block contains the first value of 
the smaller then the following code *may* be faster than intersect:

	a: [2 3 4]
	b: [1 2 3 4 5]
	x: last b
	b: find b first a
	r: copy []
	foreach i a [
		all [i > x break]
		all [find b i append r i]
	]
	b: first b

however, I think I've finally created a rocket db

i have a block with 1 million sub blocks representing objects, and 
they each have a random 'age value  between 1 - 100.. i can pull 
the indexes of all the blocks where age = 42 in 0.7 seconds.. 
that's the slowest query i have.

There should be a ratio where the iterative method is faster than 
intersect, but some tests have to be done.
ratio = (lenght? bigest-block) / (length? smallest-block)

In my comparisons with Redis, I was getting 7200 GETS (pulling the 
value of a key) per second from a DB with 100,000 key/value pairs.. 
 With this rebol db, i can do 1,000,000 GETS in 0.64 seconds, from 
a db with 1,000,000 key/value pairs.

My timings were for a script, where intersecting was one part. Whether 
intersecting block or hash made no noticeable difference.

Well, since your timings did not detect, that hash intersects are 
*much* faster than block intersects, this is a proof, that your timings 
do have very little in common with the speed of intersect

one odd thing i find with timing in general is why do i get random 
values each time i run a script? shouldn't it always be the same?

As opposed to that, I propose a more precise http://www.fm.tul.cz/~ladislav/rebol/timblk.r

This test shows well why some graphical demos work bad on some systems 
even if the computers are faster than mine  (Vista with a tiny celeron).

hmm, I just checked a Vista Business 64 bit (not updated for quite 
some time), and it used 15.5 milliseconds too...

The same result with R2 and R3
 - of course, this value is a property of the operating system

...its strange that they do not also apply...

 - nothing strange, it is a different kind of counter, not the one 
 used to get the system time (that might mean, that it actually is 
 less precise, e.g.)

What I think too...


but if R3 can get precise timing info, it could arguably add this 
precise elapsed period since last *time* check and add it to the 
result when now/precise is called in a script. 

just an idea... thinking out loud.

precise timing

 - that is exactly the opposite meaning than what I am using. Example:


clocks usually display time in seconds, using second as the smallest 
unit. That does not mean, clocks are imprecise, they may actually 
be more precise, than a counter counting every microsecond, e.g., 
but with a 0.1 % error, meaning, that in one day, such a counter 
may be e.g. one and a half minute slow in one day

hence the use of word "timing"  ;-P


adding timing resolution within the "passage of time" resolution 
would be arguably sufficient for me even if it meant we are ~0.1 
ms off, as long as we get a constant increment of milliseconds (or 
better) to use.


right now the time resolution is sooo poor (on windows) its often 
frustrating.

but, generally, it looks, that Windows may well be the only operating 
system currently, that still has a detectable resolution time of 
operating system clock

Ladislav - It's a 2.5 year old MacBookPro with a 2.4GHz Core 2 Duo 
- I think there are much faster processors around these days.

yes, your values correspond well to each other, the loop body calls 
now/precise and uses a comparison and IF

Yes, interesting, so, even though your processor is a bit slower 
than mine, the NOW/PRECISE evaluation is significantly faster

my mac mini is a super chitty system HW, but its still faster at 
editing video than PCs four times as capable.

What initially surprised me was that there did not appear to be a 
significant difference bewteen OS X (native) and XP(VM) with  time-block[now/precise] 
0,05

so... does an XP running on a OSX VM run faster than the same on 
a PC board?  ;-)

(i.e. not using a VM)

Max: As I understand, there is at least a 10 per cent performance 
penalty for running Windows under a VM rather than directly (using 
Bootcamp).

From a quick browse of the Virtual Box User Manual, a VM uses the 
host systems "time stamp counter" by default. There are a number 
of options to change the "guest clock".


It's section 9.10 in the Virtual Box User Manual if you want to take 
a look.