World: r3wp

R3 won't utilise more than 1 core either way :)

Of course :)

Ooo, tasks! But error handling is still broken in tasks, so this 
test won't work :(

Ah, here's the jump :)

From 1.0G to 1.9G, after 14m cpu time

But it's still running

Mine just jumped up a hundred megs, still running :)

It failed a lot earlier for me (some time in the last half hour), 
never getting over 1.2GB, and now there isn't enough memory to figure 
out what the number it failed at was. Looks like you run out of memory 
before you run into the word limit, and that the symbol table isn't 
cleaned up by the recycler. Good to know.

these experiments remind me how painful was it to figure out why 
are we getting those funky out of memory messages, saying something 
about PermGen in java

The internal symbol table is there to make symbols work at all.


I don't think, I understand this fully. To me, it's like needing 
an internal string table to make strings work. Or an internal integer 
table to make integers work. Why not just have contexts, and don't 
put words in any context, if the word is just data? And if a word 
go from being data to hold more meaning (have a value attached), 
then put it in a context.

If you take a book written in finnish, you see a lot of words, but 
they have no meaning to you. When you close the book, the finnish 
words shouldn't take up any space in your brain.

Geomol, I think you are confusing design with implementation. The 
implementation can be improved; however, it's a compromise between 
the complexity of it and how common your "millions of words" scenario 
is in practice.

Makes sense.

Putting words in a context

 is binding. That's very different from the symbol table, which you 
 could say "binds" symbol strings to integer IDs

If you can't read Finnish, it means the Finnish symbols are not bound 
to values in contexts in your head. Closing the book and erasing 
the symbol table is equivalent to quitting the REBOL proces, which 
does release the memory

I'm sure some Finnish words are also words in Danish, so forgetting 
the symbol table wouldn't help your situation :-)

I'm sure because my name is one of them...

I prefer Finnish Vodka, and Finnish music :-)

John, you are missing some things others know and find obvious. For 
example, do you know the answer to the following question?

    What is the ratio between

        stringn: func [n] [head insert/dup copy "" #"a" n]


        word: to word! stringn 1 t1: time-block [equal? word word] 0,05

        word: to word! stringn 1000 t2: time-block [equal? word word] 0,05
        t2 / t1

compared to


        string: stringn 1 t1: time-block [equal? string string] 0,05

        string: stringn 1000 t2: time-block [equal? string string] 0,05
        t2 / t1
?

time-block? u mean delta-time?

(your naming is more intuitive though... it's also 'time in bash...)

Thanks, Ladislav. Good examples!


The thing, I missed, was that REBOL has this extra internal data 
structure to hold symbols (words), I though, just the contexts was 
used for that. So comparing words (that are not bound to any context) 
are much faster than comparing strings in REBOL. I see different 
possibilities, depending on implementation. If a word is changed 
to the result from a hash calculation, then two different words might 
give the same result, right? It's unlikely, but it could happen. 
That's why map datastructures are combined with lists, when two different 
hash calculations give same result. The other possibility is, that 
words are changed to pointers pointing to their entry in the map 
(hash table). Do you know, what of the two, REBOL implement? In other 
words, could two different words be equal in REBOL?


About the strings, then it's possible to do kind of the same with 
hashing. Lua does that. If you have two different but identical strings 
(same string content) in Lua, they share the same memory area. Hashing 
is involved, and I guess, comparing string would be equal fast as 
comparing words, if REBOL did the same. (unless words are exchanged 
with the result from the hash calculation, but then two words might 
be equal)

After a little testing, it seems, words are changed to a hash result, 
and REBOL gives an error, if two different words give same result:

>> s: "abcdefghijklmn"

>> forever [if equal? a: to word! random s b: to word! random s [print 
[a b]]]
** Internal Error: No more global variable space

The error comes really fast.

Petr, Nightwish? :-)

Above test is in R2.

R3 keeps going and can't be stopped with <Esc>.

<Esc> + <Ctrl>-c seem to stop it.

Global variable space should be the special object holding global 
values in R2. That's not the symbol table, either

My guess is that it must do more useless binding than R3

You're right. to word! put it in system/words, I need to do to block!, 
I guess.

Maybe to-lit-word works

In R2:

s: "abcdefghijklmn"

forever [if equal? a: to block! random s b: to block! random s [print 
[a b]]]

rebol-console: line 2:  5658 Floating point exception./rebol --noviewtop

A crash. Interesting! :)

Indeed

to lit-words! also put it in system/words in R2.

That's probably one of the things optimised away in R3

It's such a shame it's stalled

I think, Carl will return soon and he will open source R3.

He must have a hard time doing that other job, but probably he will 
experience the bright side of the open collaboration too.

Im expecting him to realize that he can actually help the world effectively 
if he can unleash the power us, the long time fans of rebol.

Don't hold your breath

John, to keep your mind on the edge of the cliff, did you know about 
the global series table? :-)

Not sure, what's that?

Quite similar to the global symbols table, the existence of a global 
series table can be deduced, from the R3 extensions interface

A series value does not include an index. Only a reference does. 
So that's one indirection, from a value slot (with an index) to a 
series value

But there must be another indirection, from the series value to the 
actual storage location of its item slots, because the item storage 
can't be counted on to stay at the same memory address over garbage 
collection and callbacks or multitasking, but it must be deduced 
that the series value can be counted on to stay at the same memory 
address

So there must be a table of series descriptors apart from the actual 
series content

Maybe it doesn't really need to be a global table, but it seems very 
convenient for the garbage collector

No, haven't heard of that.

It wouldn't need to be a global table, it could be a heap space dedicated 
to series descriptors (a typed array in memory). Same purpose, more 
likely implementation. Precise collectors are often implemented with 
type-specific heap spaces, since it reduces fragmentation when you 
don't have a copying or compacting collector - REBOL's collector 
doesn't move values, supposedly. REBOL could get away with heap spaces 
for block data, string/binary data, and descriptors (maybe both object 
and series in the same heap space); there might be more, but adding 
more can make the memory management more complicated.

I'm not sure how that differs from my description

can't be counted on to stay at the same memory address over garbage 
collection

 - as far as I know this has never been demonstrated. The performance 
 characteristics of REBOL's garbage collection are more consistent 
 with mark-and-sweep, and there hasn't been any indication that it 
 compacts or copies. Series reallocation on expansion does copy though. 
 Plus, there is a bunch of indication that R3 isn't yet task-safe 
 and that callbacks can choke on shared data too. Aside from that, 
 your argument's good, and your conclusion more so.

Well, I wouldn't count on it, but indeed, what the extensions documentation 
specifies is that series can move when they are changed. It's all 
very vague, but if the collector doesn't compact as you say, it would 
be restricted to expanding series beyond the internally allocated 
memory region