AltME groups: search

my ultimate goal is storing triples..  [a1 a2 a3 b1 b2 b3...] preferably 
strings if that's fast enough

where i can search against any part of the triple , and return the 
whole thing (triple)

if strings are quick enough and not much overhead, it's possible 
to use key/value   but the key is made up of subject:predicate ie: 
"maxim:age" "unknown"  so that select/part key ":age" would return 
everything including "unknown"

using find/skip and an index for what item to search for in the record... 
we get by FAR the fastest search... if you count the fact that generating 
the keys for find-very-fast often takes as long as the search itself, 
in dense searches.

ultimate-find below gives the results of this new function


preparing sparse data
6000005
sparse tests:
feach(): 5.4.3.2.1. -> 0:00:04.89   4 matches found
----------------
find-fast(): 5.4.3.2.1. -> 0:00:00.719   4 matches found
----------------
find-really-fast(): 5.4.3.2.1. -> 0:00:00.234   4 matches found
----------------
ultimate find(): 5.4.3.2.1. -> 0:00:00.594   4 matches found
----------------

dense match with key/value collisions
6000000
feach(): 5.4.3.2.1. -> 0:00:13.343   221606 matches found
----------------
find-fast(): 5.4.3.2.1. -> 0:00:14.813   221606 matches found
----------------

find-really-fast(): 5.4.3.2.1. -> 0:00:07.718   221606 matches found
----------------
ultimate find(): 5.4.3.2.1. -> 0:00:06.735   221606 matches found
----------------

dense match without key/value collisions
6000000
feach(): 5.4.3.2.1. -> 0:00:13.969   222405 matches found
----------------
find-fast(): 5.4.3.2.1. -> 0:00:09.812   222405 matches found
----------------

find-really-fast(): 5.4.3.2.1. -> 0:00:07.672   222405 matches found
----------------
ultimate find(): 5.4.3.2.1. -> 0:00:06.531   222405 matches found
----------------
done

i inserted "maxim" "age" "unknown" and appended "terry" "age" "42" 
into the dataset containing 6 million records.. 

>> ultimate-find dataset "age"  2 3 1
ultimate find(): 1. -> 0:00:00.093   2 matches found
== ["maximn" "age" "unknown" "terry" "age" "42"]

I'll say that's a respectable time... and the leading contestant 
:)

But wait, there's more.... 

convert dataset to hash! and run ultimate-find again!

the strange thing is i did tests using a record size of 2, which 
wouldn't trigger strange mis aligned key/value issues.  I even removed 
the copy to make sure that wasn't the issue and one test with only 
400000 records took more than 4 minutes to complete vs .297 for the 
feach test!

I'm looking for the 6 integer.. it's still cranking and i can hear 
my system struggling..

well, the results where the same at the end... pretty weird... maybe 
someone has encountered this before and can explain why this happens....

I think, that it is quite natural. You should probably generate some 
random data having (approximately) similar properties as what you 
intend to process and try some variant approaches to really find 
out, which one is best for the task. Do you know, that it is possible 
to index just a specific record field, i.e. you don't need to make 
a hash containing all the data from the database?

Yeah, i've tried some actual data finding 3270 matches out of a hash 
that is 732981 in length.. 

when it's block the search takes .033 s, and same run against has 
is 0.6

but if the matches are just a few, hash is 1000x faster

.033 s, and same run against has is 0.6
 - do you mean 0.6s, ie. roughly 18 times slower?

it's maxim's ultimate-find above ( and im using real world data)

(and a revision to ultimate-find, just after it)

the example creates several sets of data with different organizations 
and it compares all of them amongst each other.


so with that script, you should be able to do all the analysis you 
need.

ah   yes.... ususally a hash will have to skip over  elements which 
return the same hash key.


so if your table has a few thousand similar items, you aren't benifiting 
from the hashing... and its quite possible that looking up a hash 
itself is actually longer when it has to skip over and over (comparing 
data on top of the hash).


though one could argue, that the speeds should be a bit slower than 
using a block, not this slower...  possibly related to the implementation 
itself.

or have a look at cassandra and/or monetdb (w/o knowing anything 
about your intended usage)

rdf is to xml what War and Peace is to Cat in the Hat -- Triples 
are working even with Maxim's code above (just not in hashes for 
more than a query with a single value).. but i crave the speed of 
index? against large datasets.

I WILL NOT STOP TILL I HAVE A FAST AND SIMPLE TRIPLE STORE!  
(sleep is my enemy)

you can negate collisions by building two checksums out of different 
properties of you data and merging them.

Got to decide what is more important:
-- time to build data structure
-- time to update it (add/remove on the fly)
-- time to search it

And build data structures optimized to your priorities. There is 
no one true solution, just the best match for the situation at hand.

ok.. here's an example.. 

take this simple rdf triple   "Tweety" "isa" "Canary"
How would create 3 indexes to manage it and 10,000,000 like it?

And i use this index as the value for the 3 indexes, and i create 
the keys+value
Tweety
: index
Isa
: index
Canary
: index

And my comment...

it's remember me the IDE Plex(obsydian) in the nineties. it used 
widely the concept of triples (tuples) to modelize applications and 
databases.

Should have listened to my mother and became a lawyer.

Terry, I think INTERSECT is fine the way it is, and it's easy to 
wrap if you want.

    fold: func [
        series [series!]
        fn     [any-function!]
        /local value
    ][
        value: pick series 1
        foreach item next series [value: fn value item]
    ]

    intersect-all: func [
        series [block!] "A block of series values to intersect"
    ][
        fold series :intersect
    ]

steeve, REBOL doesn't support path with strings, and furthermore, 
it would only return the first index, if you used it within a paren.


so I'd really like you to give a small snippet of code with your 
idea, I am curious about your method... cause I don't see *how* what 
you say works.

I would use map! (or hash!) as indexes and a key would contain one 
or several triples (inside a block)
verb/"age": [ index indexn ...]

and i1,i2,i3 should be of type hash! or map!

To make it practical for other operations too, the triple store should 
be defined so, that:


1) every triple is stored as a fourtuple, the additional field being 
a unique triple ID
2) create the main index using the triple ID, 

3) create three field indices searchable by by the field contents, 
and "inside" also by the main ID


This way, the triple storage will be moderately fast for triple removal 
as well

And, this data is tight.. the  250,000 real world triples (manages 
a multi-million dollar electrical equipment rental biz, including 
 1500 contacts, invoicing, and 10,000s of items)  is only 9mb... 
zips to 1.3mb

To make it practical for other operations too, the triple store should 
be defined so, that:

1) every triple is stored as a fourtuple, the 
additional field being a unique triple ID...

Why not just use the index of the first part of the triple? 

ie: ["tweety" "isa" "canary" "Sylvester" "isa" "cat"...]


so 1 and 4 are the two triples( i1)  and the next two are i2 and 
i3

i1: [ "Tweety" [1] "Sylvester" [4][
i2: ["isa" [1 4]
i3: ["canary" [ 1] "cat" [4]]

as you can see, it will print "Hello" first, and after 5 secs it 
will print "Hello in a task" .... the question is, if I got the usage 
right :-)

It's been like that for more than a year now. Oh, and if an error 
is triggered in a task and not handled, it will crash R3. I'm not 
sure it is stable to trigger an error and handle it either.

Yup, that works Pekr, and the task ends on its own.

>> a: make task! [wait 5 print "Hello in a task"] do a print "Hello"
Begin Task
Hello
>> Hello in a task
End Task

I wonder where the code is that prints "Begin Task" and "End Task"...

you've got me wondering how we could already setup some thread IPC 
and burst mode control with the current  !task.implementation.... 
hum... worth looking into.

the "begin/end task" printing happens somewhere in r3lib, and in 
native code, afaict

We're not disputing that it is a bug - actually, we think it is debug 
code that is only there temporarily. But reporting it is of no good 
for the moment, and it's not nearly as bad as its other bugs.

i implemented the thread management functions in the host lib. the 
create_thread hostlib function gets a function pointer as argument 
that is to be run in the new thread. and the error is occuring within 
this function, so it's a bit outside my reach

good to know that tasks are this far already, though. should be fairly 
straightforward to get linux to the same level as windows, and from 
there on to continue and make them actually useful :)

So, what exactly is safe and unsafe in using task!

ok another question i know i'm a bother ... but if  us tiny bunch 
of crazy fans we don't ask us silly asks we will never get steps 
further and get rebol writed in history. So the crazy is is why not 
doing rebol in java instead of C or C++ ?


I mean now in days having rebol based on java what are the possible 
gains ?

1) Easier way to handle multiplateform

2) Big big java comunity and so getting strong specialised people 
to enhance rebol VM will be easier.
3) no need to adapt algorithms across plateform.
4) all the curent technologies are adapted to java. 

the bad points are:


1) jre weights alot and rebol being based on it will not be stand 
alone so it will need the jre to be installed to run
2) goodbye AGG 
3) interfacing with java libraries can be problematic.

I think the good points overcome the bad points.

i know rebol wants to be something else ... but that's the main reason 
why rebol isn't not being use more extensively and is so much undergrounded...

Add a few bad points:

4) We would have to throw all of the current code away and start 
over.

5) The semantics of REBOL have very little in common with Java, so 
we would be very limited in what Java libraries we could use.

6) The JVM is much worse at dynamic languages than its competitors, 
such as the CLR.
7) SLOW.
This might tip the tables in the other direction.

Brianh's bad point: 
 

4) hum yes but maybe you will attrack some more hella programers 
ultra specialised that will help us do the translation ...

5) the semantic and datatypes of rebol have few in comon with C/C++ 

6) yes but looking at the jobs offers now in day most of them aorund 
70% are based on java jdbc job offers ... 

7) slow but now we have 4 to 6 cores processors that was a good excuses 
10 years ago but today that's not...

But if you make this change: "doing rebol in java instead of C or 
C++" -> "doing another rebol in java in addition to the current C/C++ 
version"

Then you can get rid of bad points 1, 2 and 4. And add a good point: 
5) R3 for Android.

brianH Rebol on android i say Woooooooooooooooohooooooooooooo that 
smells good 

and iphone too since apple doesn't seems to be willing to support 
flash but have no problems with java

4) "ultra specialised that will help us do the translation" Manual 
translation. The code would not be comparable, and the entire implementation 
strategy would need to change because of the differences between 
C and Java.

5) REBOL doesn't have classes. And accessing C libraries from REBOL 
is awkward too.

ok lets be frantic and compare rebol with tcl/tk, perl, python,  
ruby. All those scripts langage have in common to use GNU bases main 
portable libs so the community making them grow and known is bigger 
than our community 1000 times ...

It's the replacement idea that is bad. We can make a vaguely REBOL-like 
language for Java that is good enough to interoperate with the C-based 
R3, but throwing away the R3 codebase like you said and starting 
over would kill the language and make us wait another 5-10 years. 
The "instead of" part is a BAD IDEA.

BrianH R3 is stuck in alpha since 3 years and it will take 3 more 
years to be officially released ... and mean while we keep a design 
based on  90's year design  i mean it implicates too a lot of difficulties 
that can't be overcomed easyly

Perl, Python and Ruby are compiled, and so is TCL now. They are not 
interpreted.

R3 is not stuck in alpha. What you are doing is suggesting throwing 
away a codebase and starting over. Perl 6 did that and they are still 
in alpha after 10 years.

BrianH yes that can be a side project ... but it have to be seriously 
done not another abandoned clone project that convince no one and 
get no support...

BrainH yes they are compiled but using the Glib the GTK+ libs etc 
GNU libs and tools so people feels more confortable to contribute

BrianH i heard alot java sux etc ... but truth is it's the most wanted 
language now in day ... and you can't tell it will be better or not 
for rebol we didn't tryed ... and if java is not your prefered ok 
how about mono (the GPL ed  C# package) but mono have less visibility 
than java.

I have no problem with there being a semi-compatible R3 clone based 
on Java, and if I buy an Android phone I will start such a clone. 
But it will not replace R3 because that code is useful for many people 
*now*, and doesn't have any of the significant downsides of the JVM 
and its ilk, which is part of what makes it useful. And any clone 
that is based on (L)GPL can't share code with R3, so anyone who wants 
to help the community can't contribute it it.

and in that hundred of fanatics you have only 10 to 20 guys participating 
to the REBOL project actively ... personally i don't feel motivated 
to help apporting my know how to a project that will feed someone 
else without retribution i think as american you understand my point 
buissness is buissness ... I don't work for free for a project that 
will bring money to someone else.. that's plain and simple ... And 
it will be different if it was organised as a foundation people give 
the money they want and then the monney is splitted to remunerate 
the contributions according to their importance ...

I have no problem with doing what it takes to get REBOL interoperating 
with Java and .NET (particularly Silverlight). It's just a matter 
of resources. It's the "instead of" idea that is bad.

brianH wrong in the worst case i can retain my ideas and improvements 
for ruby for myself and then make my own monney on selling my work 
... wich is out of consideration in rebol ...

all i can do in rebol is selling the  programs i do using rebol... 
but what credibility i have imagine my clients how they work they 
give me specs i propose them to do it in rebol they say no because 
if you get any problem we the client will not be able to get easyly 
a substitute to you  to retake your work and get it enhanced ...

Since you haven't been participating as much, you don't know how 
far along we are, so you don't understand how incredibly dumb it 
would be to scrap the code base and start over. We are only working 
on one VM. If someone else wants to start another project, fine, 
but they should have a good case for allocating the resources, and 
not do anything to preclude others from helping them.

BrianH hum and no people asked for my help neither ... supposing 
i could bring some help which is not the case i think...

I have on many occasions asked for contributions and help, and many 
have answered. If you feel that you can't help, you will be missed 
but we will make do.

i know but my way to help is different but not uninterresting ... 
and franckly i'm maybe one  of the few more interrested in what can 
be done with rebol than what can be done to improve rebol ... what 
is the meaning of adding things if we already don't use what exists 
in rebol ?

if i can if i have access to the whole source code, but i will try 
...  most of the time i need to see the big picture to get ideas 
on how to proceed ... a mono C# clone intent have been tryed 4 years 
ago it was spirit then sudently the author disapeared and that's 
it ...

Because Java sucks. This is why Scala and Clojure exist too.

(lisp :) ahahaha and :( say no to car cdr car and all the parenthésis 
that goes with them)

The only reason C# and the .NET platform has any traction at all 
is because C# is better than Java at the exact same things that Java 
is used for. And then the .NET runtime is better than the Java runtime 
for not only what people try to use the Java runtime for, but what 
the Java runtime is designed for. It's the libraries that fall short.

but .net runtime only exists in windows plateform and i'm not sure 
that its all windows plateform non X86 based...

Scala is a functional-oop programming language, so it can do everything 
Java can do and a lot more, plus strong concurrency support that 
Java drastically lacks. Its syntax has nothing in common with Java.

mono the opened C# doesn't have a complete compatibility with official 
.net thing  and it's slower than the real thing ...

.NET runtime (or semantic equivalent) exists in Silverlight, Mono, 
the iPhone (through MonoTouch and Unity), Second Life, Linux (through 
Intel's port), Windows phones.

Scala is not like PHP (and stop swearing: words like PHP are not 
fit for polite company). Sacal is more like OCaml, Erlang, F#, etc. 
Loosely though.

Silverlight works on Windows, Mac, Windows Phone 7, Meego, and Linux 
(as Moonlight). You are underestimating the Mono project: They have 
a lot of funding and support.

Basing some REBOL-related tools on Silverlight and/or the DLR would 
be a plus. Keep in mind that we don't have to make the whole of REBOL 
run within the JVM or DLR - REBOL is great for making development 
tools, which can be used to generate Silverlight applications, just 
like they generate Flash applications. Or R3 could be used as a library 
by Java, or as a supplemental developemtn tool, or a code generator, 
whatever. Remember, most of REBOL is a set of awesome libraries and 
dialects. REBOL without most of DO or AGG is still useful.

ok so we have a rebol base in a language that means alot of work 
to have it on windows / linux/ macOSX  and some mobile thingies

but then we have extension/ modules what ever things based on moonlight 
and C# easyly portable but then obviously the part that would be 
a better fit to moonlight would be the VID / draw part ... and can 
that really coexists with the rest of rebol ?

It's only a lot of work if you add it up. Noone will need all of 
this. The small part that they need will be a smaller bit of work. 
And others will need different things, and will do their own work. 
As long as people contribute back to the community and don't say 
things like "I don't work for free for a project that will bring 
money to someone else", without realizing that you will befefit from 
those other people's contributions, then we will all benefit.

Moonlight has its own network layer and video stream playing abilities 
already. That is not where REBOL will help.

Silverlight and Moonlight are good. If you want to make a REBOL browser 
plugin, they are the thing to beat. Flash is not as good, and not 
really competition if it weren't for the installed base.

yeah but most of people use flash ... mostly because of the tools 
and what you can do with the tools rather than of what the result 
perf will be and what the language structure is

BrianH don't forget that mostly flash is used by people who don't 
know what an algorithms is and which i call the "click nation"

could then moonlight and mono be interfaced with other side libraries 
like opengl  can i make a part of my interface as an opengl display 
etc??

brianh right ... to the problem can rebol display formated dynamic 
text using Draw/agg dialect it only took me a steeve and 3 yeart 
to apport the solution ... and lot of discutions ...

and when you look at the solution that's so evident that you are 
amazed no one thought about that before ... mainly area-tc and viva-rebol 
are splitted in 3 you have the color text engine based on color-code.r 
carl script but instead of generating html it generates draw code 
in a block then it's rendered to a vid face using it's draw block 
 for the ihm the feel block is used on the same vid face ...

and when i see carl saying i pause R3 and patch R2  i'm like stunned 
... anyway ...

and having a proper resume of what is done is hard ...

Oh, then this conversation is more off-topic in this group than I 
thought. Well, here's what you missed:

- The R3 GUI is being actively developed by commercial developers, 
and it is much faster than the approach you just mentioned.

- We have had a couple releases in the last week, with more to come. 
In other words, we're not stuck.

- The GUI code has been moved to the host, and is thus open source.

- Everyone involved is hard at work, communicating, and busy. This 
includes Carl.
- All of your criticisms of the project are outdated.

but looking things changing perspective someitmes open new ways ... 
and i saw last carl's blog about having problems R2.7 with unbunt 
10 for example and that's the kind of thing that I don't want  R3 
and that can be avoided if you don't have to port 10 times the code 
...

and i clearly foresee the actual i port 10 times the code  of rebol 
isn't a suitable way for futur...

I should clarify something, ANYTHING uploaded to Qtask is still there, 
and available.  In other words, Qtask loses nothing.

So anything that was uploaded like Videos can be found simply by 
searching the Files.

Janko, i'll gather the videos from 2005 and make them available to 
you. if you have the time and patience to upload them to youtube, 
that would be great. i don't know when i'll ever get around to do 
that. :)

Gabriele: great ... I can upload them to youtube no problem ! 


Reichart: I see there are videos I didn't previously see yet there. 
I dl-ding and looking them. Thanks. 

I don't know exactly being in DevCon project means, but I am taking 
rebol and it's community serriously and wouldn't mind participating 
*with what's in my powers* at next devcon (and am interested in eventually 
seeing all videos from past devcons)

Maybe we should have a central place for all  the devcon videos ... 
I was just looking at Yahoo's developer movie collection  ( JS and 
the YUI )

Alright 2010 is almost over now. Why don't we set something up virtually 
and "just do it"?  I just had a Teamviewer http://www.teamviewer.com/download/index.aspx?os=windows
session with some people from Singapore and Taiwan from California 
and it worked flawlessly. You guys sound like you are all doing some 
very interesting things.

it would be neet to setup, I think that setting up an official contest 
to ask for a date & time amongst all rebolers as well as proposed 
demos and topics would be a good first start.


I don't thing this should be planned for 2010 though, the holidays 
are here and I know that I'm not available until january, already, 
and I presume this is a common issue for all of us now.

I know nick has all the infrastructure setup for multi-user chat 
rooms and IIRC that is what was used for the last "virtual" devcon.

Very nice. Had you tested glass and liquid with the web plugin from 
several years ago? Just curious.