AltME groups: search

but its not a big deal... unset words are created just by loading 
any word in a block.  it just means, I've seen this symbol before 
and it now has its reserved number in the word list/hash table. any 
future reference will reuse the same word id (number).

Yeah, they will be reused. But the way, REBOL do it, if you have 
an application, that do a lot of block parsing for example, with 
new words coming in all the time, then that global context will just 
grow and grow. In reality, it will probably come to an end, as there 
are a finite number of words in a human language, if that's what 
being parsed.


If words were not bound by just being loaded, but only when evaluated 
(or compiled, if that's the case), then parsing blocks would not 
produce any unset! words in the global context. But a consequence 
of this is, that blocks of code being sent to a function (like LOOP), 
will be able to see the words local to that function, unless the 
block is first bound outside the function, like

	count: 1
	loop 10 bind [print count] 'count
, which will then print the number 1 10 times.

So to not exhaust the global context (if that's a problem), we should 
parse like this:


parse to block! "a few words and then some more" to block! "'a 'few 
'words to end"

john, you are in error when you say: "exhaust the global context" 
 the number of words in the global context is irrelevant to exhausting 
the number of usable words in rebol.

the reason is that binding is not what reserves words in master word 
table.  its anything that creates a new word value, bound or not.

here is an example, using your own to block! example:

>> repeat i 100000 [to-block join "random-word" i]
>> probe length? first system/words
== 2616


pump up the number to 500000 (in 2.7.8) and it crashes.   IIRC this 
was as low as 32k in older versions ! 


with each increment of 100000 you will see the rebol process gobble 
up a few MBs more as it assigns new word-ids to those words.

This is an indirection, but different from binding. To abolish that 
table would mean to keep strings everywhere internally instead of 
simple numbers. If you want a language to work that way, you should 
use shell scripting. It's very slow

I think, there is a third alternative. When we deal with strings, 
a data structure is made, and we just have a pointer to that.

var: "a string"
var: none


When noone is using the string anymore, memory can be completely 
cleaned for it. If I do the same with a word:

var: 'word
var: none


I don't see, why memory can't be cleaned just the same. Is it a design 
flaw, the way it is?

which suggest using some kind of reference counter for the words, 
but what would decrement such a reference counter?

REBOL use garbage collection, but in the case of counters, the same 
that would decrement a string counter. If a word stop pointing to 
it, decrement. If it's in a block, and the block is destroyed, decrement 
the block content.

R3 doesn't have anything like R2's system/user. For all we know symbols 
could be garbage collected. In 32biit R3 though, afaik you will not 
reach the total number of possible words until you have already hit 
the limits of the memory address space first. Does someone have a 
computer with enough RAM to test this? I only have 4 GB.

for x 0 to-integer #7fffffffffffffff 1 [to-word ajoin ["a" x]]


Then watch the memory usage of the process, and tell us what happens 
and which error is triggered when it fails.

>> to-integer #7fffffffffffffff
== 9223372036854775807

That will be many more words than could be held in memory in a 32bit 
address space.

R3 has system/contexts/user , which seem to work like R2's system/words. 
Try

? system/contexts/user


before and after making some random words, e.g. in a block like [some 
random words]

No, it really doesn't. All loaded words are added to system/words. 
Only words that are referenced directly in user scripts, or added 
explicitly with INTERN, are added to system/contexts/user. I had 
to add a bit of really careful code to make sure that system/contexts/user 
doesn't get words added to it until absolutely necessary, because 
it is an isolated context from the runtime library system/contexts/lib.

Symbols in R3 are stored in an internal symbols table, a btree(?) 
or some other unknown data structure that you can't reference externally.

They aren't added to any context until you add them explicitly. The 
R3 interpreter would not presume to know what a word would mean to 
you until you tell it what it means, by binding the word to a context 
or by just using the word as data.

>> repeat i to-integer #7fffffffffffffff [to word! ajoin ["a" i]]
** Internal error: not enough memory
** Where: to repeat
** Near: to word! ajoin ["a" i]

This code might be a better test: repeat i to-integer #7fffffffffffffff 
[if zero? i // 1'000'000 [recycle] to-hex i]

It should have less memory usage overall and if words are recycled 
then it won't run out. I'll run it now.

TO-HEX generates an issue!, which is a word type in R3. Yes, you 
can even bind them.

I figure that not creating the temporary string, and running recycle 
every once in a while, might make the memory problems go away. So 
far the R3 process is staying at exactly the same memory, less than 
a gig. I also tossed an assignment in there so I can know which number 
it fails on.

Mine just jumped up a hundred megs, still running :)

It failed a lot earlier for me (some time in the last half hour), 
never getting over 1.2GB, and now there isn't enough memory to figure 
out what the number it failed at was. Looks like you run out of memory 
before you run into the word limit, and that the symbol table isn't 
cleaned up by the recycler. Good to know.

The internal symbol table is there to make symbols work at all.


I don't think, I understand this fully. To me, it's like needing 
an internal string table to make strings work. Or an internal integer 
table to make integers work. Why not just have contexts, and don't 
put words in any context, if the word is just data? And if a word 
go from being data to hold more meaning (have a value attached), 
then put it in a context.

If you take a book written in finnish, you see a lot of words, but 
they have no meaning to you. When you close the book, the finnish 
words shouldn't take up any space in your brain.

Geomol, I think you are confusing design with implementation. The 
implementation can be improved; however, it's a compromise between 
the complexity of it and how common your "millions of words" scenario 
is in practice.

Putting words in a context

 is binding. That's very different from the symbol table, which you 
 could say "binds" symbol strings to integer IDs

John, you are missing some things others know and find obvious. For 
example, do you know the answer to the following question?

    What is the ratio between

        stringn: func [n] [head insert/dup copy "" #"a" n]


        word: to word! stringn 1 t1: time-block [equal? word word] 0,05

        word: to word! stringn 1000 t2: time-block [equal? word word] 0,05
        t2 / t1

compared to


        string: stringn 1 t1: time-block [equal? string string] 0,05

        string: stringn 1000 t2: time-block [equal? string string] 0,05
        t2 / t1
?

Thanks, Ladislav. Good examples!


The thing, I missed, was that REBOL has this extra internal data 
structure to hold symbols (words), I though, just the contexts was 
used for that. So comparing words (that are not bound to any context) 
are much faster than comparing strings in REBOL. I see different 
possibilities, depending on implementation. If a word is changed 
to the result from a hash calculation, then two different words might 
give the same result, right? It's unlikely, but it could happen. 
That's why map datastructures are combined with lists, when two different 
hash calculations give same result. The other possibility is, that 
words are changed to pointers pointing to their entry in the map 
(hash table). Do you know, what of the two, REBOL implement? In other 
words, could two different words be equal in REBOL?


About the strings, then it's possible to do kind of the same with 
hashing. Lua does that. If you have two different but identical strings 
(same string content) in Lua, they share the same memory area. Hashing 
is involved, and I guess, comparing string would be equal fast as 
comparing words, if REBOL did the same. (unless words are exchanged 
with the result from the hash calculation, but then two words might 
be equal)

After a little testing, it seems, words are changed to a hash result, 
and REBOL gives an error, if two different words give same result:

>> s: "abcdefghijklmn"

>> forever [if equal? a: to word! random s b: to word! random s [print 
[a b]]]
** Internal Error: No more global variable space

The error comes really fast.

In R2:

s: "abcdefghijklmn"

forever [if equal? a: to block! random s b: to block! random s [print 
[a b]]]

rebol-console: line 2:  5658 Floating point exception./rebol --noviewtop

A crash. Interesting! :)

It's such a shame it's stalled

I think, Carl will return soon and he will open source R3.

He must have a hard time doing that other job, but probably he will 
experience the bright side of the open collaboration too.

Im expecting him to realize that he can actually help the world effectively 
if he can unleash the power us, the long time fans of rebol.

Quite similar to the global symbols table, the existence of a global 
series table can be deduced, from the R3 extensions interface

A series value does not include an index. Only a reference does. 
So that's one indirection, from a value slot (with an index) to a 
series value

So there must be a table of series descriptors apart from the actual 
series content

Maybe it doesn't really need to be a global table, but it seems very 
convenient for the garbage collector

It wouldn't need to be a global table, it could be a heap space dedicated 
to series descriptors (a typed array in memory). Same purpose, more 
likely implementation. Precise collectors are often implemented with 
type-specific heap spaces, since it reduces fragmentation when you 
don't have a copying or compacting collector - REBOL's collector 
doesn't move values, supposedly. REBOL could get away with heap spaces 
for block data, string/binary data, and descriptors (maybe both object 
and series in the same heap space); there might be more, but adding 
more can make the memory management more complicated.

can't be counted on to stay at the same memory address over garbage 
collection

 - as far as I know this has never been demonstrated. The performance 
 characteristics of REBOL's garbage collection are more consistent 
 with mark-and-sweep, and there hasn't been any indication that it 
 compacts or copies. Series reallocation on expansion does copy though. 
 Plus, there is a bunch of indication that R3 isn't yet task-safe 
 and that callbacks can choke on shared data too. Aside from that, 
 your argument's good, and your conclusion more so.

could two different words be equal in REBOL?
 - in my opinion, that would be a bug

If you allocate strings only in multiples of 128 bits and aligned 
the same as blocks, fragmentation wouldn't be a problem if they are 
allocated in the same space. It would be a waste of resources for 
strings though, as even a single-char string would take 16 bytes 
of space.

Some memory management systems segregate the values by the size, 
with objects of similar sizes allocated from the same space, to cut 
down on fragmentation. Large enough objects are sometimes allocated 
on their own, in a seperate chunk of memory allocated from the OS. 
I haven't seen any indications that REBOL's memory management does 
this though.

About logic, I guess, it's related to a wish to use PICK with logic 
argument. It works like this for SKIP:

>> skip [a b] true
== [a b]
>> skip [a b] false
== [b]

and you can use PICK instead of EITHER sometimes like:

value: pick [1 2] b > 1

instead of

value: either b > 1 [1] [2]

But I don't know if it is a bug or not.

To be able to type a word of datatype word!, we need lit-word!. Else 
the word would be evaluated to it's value. So when you write
	length? 'word
, it's the same as
	length? first [word]
To get what you want:
	length? first ['word]
or
	length? quote 'word

Funny!

These different words are all part of the typeset any-word!. In R3, 
you can see, what datatypes are part of a typeset with:
	? any-word!

It's interesting, issue! is also an any-word! in R3. It's not in 
R2. But issues have different syntactic rules than words:

>> #1a
== 1a
>> '1a
** Syntax Error: Invalid word-lit -- '1a

it is because of lit-words cannot start with a number but refinements 
and issues can.

Yes, but what benefit do you have from refinements starting with 
a number?

actually nothing :)

it is useless because you can create a function like below, but you 
cannot use it:

>> f: func [/1a] []
>> f/1a
** Syntax Error: Invalid integer -- 1a

Geomol, refinements are not just for translating to paths. You can 
use a subset of the possible refinements for that purpose, but rest 
can be used for whatever reason you like in other dialects. REBOL 
is not just the DO dialect, remember. (This is the official answer; 
I am just writing it out again, for the 4th time this year.)

Jerry, the /local refinement is just another function option. If 
you provide that option when you call that function, you can provide 
initial values for the "local variables". In the case of sys/load-module, 
the security and control flow of that function depends on the local 
variables being initialized with the none value, so we want to avoid 
the /local option being used when the function is called. When a 
function refinement is not used its value is none, so if you want 
to ensure that it is none and trigger an error if it isn't, ASSERT/type 
[option none!] is the most efficient way to do this in R3.

After Carl asked me that question, Jerry, and got that answer, he 
wrote a blog where he suggested making /local special in functions. 
While most people are in favor of that (not half of the gurus), this 
proposed change to /local hasn't been done yet and we don't know 
if it will ever be done.

After Carl asked me that question, Jerry, and got that answer, he 
wrote a blog where he suggested making /local special in functions. 
While most people are in favor of that (not half of the gurus)

 - then I must have a problem with reading comprehension. I rechecked 
 the discussion of this twice and came to a completely different conclusion

I'm looking at what kind of functions, R3 has:
>> ? any-function!

ANY-FUNCTION! is a typeset of value: make typeset! [native! action! 
rebcode! command! op! closure! function!]


And then I come to think, if it's a problem having both native!, 
action! and function!. Should all 3 be seen as function! ? Having 
them all and now with the addition of closure, what if a native act 
like a closure? Should it then be a native-closure! ? Same with action! 
.


And what if it's possible one day to make operators, should we then 
call the current operators for nop! (or native-op!)? I guess, these 
types originated, because they're dealt with differently internally, 
but should the users have to deal with the differences?

I think, that the answers to your questions are simple.


Closures and functions (while being nonnative) are so different, 
that it is more convenient to have them as different datatypes.

what if a native act like a closure?
 - can you name any native that does not?


As you mention, the current datatypes are likely the most convenient 
for the implementation.

I agree on distinguish between closures and functions, but maybe 
those whould be the only two, and then skip native and action. I 
remember being a bit confused about especially action! types, when 
I started on REBOL.

It confused me, because they are called differently, so as a programmer, 
I think "ok, this is two different things, so I have to use time 
to learn the differences between e.g. functions and natives, or natives 
and actions, and how to use them correctly". And in fact, as a programmer 
I can very well ignore the differences, because the 3 types works 
exactly the same in my programs.

:)

I badly written native might be slower than a well written function.

I think, it was Bertrand Meyer, who wrote, that a datatype is recognized 
by the methods, you can use on it. In REBOL terms, what you can do 
with a datatype specify it. If you can do the same with a native! 
and a function!, they're the same datatype.

OK, then, how do you get the body of a native?

That is up to the implementor/designer to deside. :-) Maybe it should 
be a number, like actions returns in R2, or none as R3 do.

We can still try to get the body of a native and action, like we 
can with functions. So we can use them exactly the same way.

There is not "body of a native" you get. Both in R2 as well as in 
R3 you get just a "replacement value"

Aha, so you are saying, that it does not matter whether you get a 
#[none!] or a block?

I see it as a huge difference.

It is actually the same difference as a difference between causing 
an error or obtaining a block.

(a "caused error" can be considered a "special value" as well)

I wonder what happens in there. I would assume it would be slower 
to divide, if you have to make a check afterwards that it can be 
converted to integer, or is there something else fancy going on?

But I guess, it's convenient, it works this way, as we think this 
way as humans. 10 / 2 is an integer, 10 / 3 is a real.

So it returns a real.

Is it crazy to make the rule, that DIVIDE always returns a decimal! 
? POWER always returns a decimal!, even if the inputs are integers 
and the output could be an integer. It would make DIVIDE faster.

Hm, I may take that back. It may be compiler optimization, but always 
giving a real as output seem to be slower in my test. Or it could 
be, that giving the result involve slower cpu circuity, when it's 
a real compared to an integer.

But:

>> a: 3x1 / 3
== 1x0.3333333

>> a/1
== 1.0

so you are saying, that it does not matter whether you get a #[none!] 
or a block?


If natives and actions were called functions (datatype function!), 
then getting their bodies could also just return an empty block instead 
of none (if that's a problem from a language technical view). But 
it can't be a big problem, as so many functions in REBOL return a 
value different from none, if success, else none. Like PICK, SELECT, 
...

...it can't be a big problem...

 - this is not about any "big problem". This is "only" a problem of 
 a definition. You seem to be saying, that, formally, you can get 
 the "body" of a native. Of course, you can define the BODY-OF function 
 however you want. But, that is a "super-formal" approach, since, 
 in fact, you are unable to get the body of a native no matter how 
 much you pretend "it is not a big problem". In the same "super-formal" 
 way you can evaluate BODY-OF 1 and say, that it (formally) is possible 
 too, "yielding" a "triggered error". What I am trying to note is, 
 that a part of the definition should be when we can we reasonably 
 "can get the body" and when not.

Geomol: "Why bother?"

The reason that this kind of question is asked is because a lot of 
people seem to forget that REBOL is any more than its built-in dialects. 
I have to answer to remind people that the point of these syntax 
types is for users to use when creating their own dialects. Given 
that, a feature being useless for the built-in dialects is not a 
suffcient reason to restrict it. In order to justify restrictions 
the feature has to be actively harmful, as opposed to just passively 
unused, because it could be useful to user-level dialects.

Geomol, about native-op!, native-closure! and native-action!, these 
won't be necessary. Functions of the native! type don't need to bind 
their bodies to contexts, so they don't need the feature that makes 
closures closures. Ops and actions both redirect to the actual implementing 
function, which in the case of actions is an internal native associated 
with the type of the first argument, and in the case of op is a (at 
the moment only native) function provided at op creation time. If 
you wanted to allow the creation of user-defined datatypes with function!, 
closure! or command! implementations of the type's actions, the regular 
action! functions would still redirect to these implementations. 
Similarly, the regular op! type could be extended to support redirecting 
to other function types without needing the creation of a new function 
type.

Why was REFLECT introduced?

 - REFLECT was my idea, as a way to get metadata about a type without 
 overloading the ordinal functions. This was a security measure so 
 that you could make the ordinal functions available to untrusted 
 code safely, and not make REFLECT and the *-OF functions available, 
 which would make untrusted code potentially more useful.

Couldn't SELECT be used?
>> select context [body: 1] 'body
== 1
>> reflect context [body: 1] 'body
== [
    body: 1
]

Functions of the native! type don't need to bind their bodies to 
contexts

 - in that sense, natives behave both as functions and as closures 
 at the same time. That is not as a big of an exception, as it may 
 look, e.g.:

    f: func [][]

behaves as a closure, similarly as

   g: func [x y][add :x :y]


, simply because in such simple cases it is not discernible whether 
the given "function" is a function or a closure

I'm not sure, and maybe it's a gray area. Are you familar with contracting 
between caller and function as described by Bertrand Meyer?


If you promise to call *r* with *pre* satisfied then I, in return, 
promise to deliver a final state in which *post* is satisfied.

*r* is a routine in his words.

the blog discussion attacks the use of /local in a myriad of ways, 
but few actually talk about  making /local or what they would accept 
or not if it were a "reserved"/uncallable refinement.

Geomol, you can use any other refinement, why use /local and allow 
a huge gap in security wrt intent of a local varable.  

I'd actually replace the /local handling of the function dialect 
and allow you to specify default values right in the function block.

f: func [a b /local i: 2 blk: [ ] ] [...]


The idea of /local is that its a protected, internal, set of words... 
most people don't even realize that it is just another normal refinement.

How are arguments used? You can set them using set-word! or just 
use the word. If you just use an argument, your "inject code" could 
be a problem, not if you use it as a set-word!. But normal arguments 
you just use, so your problem is there already, or what?

Take REPLACE as an example of a function with locals. See how locals 
are used by looking at the source. I don't think, the thing about 
/local is an existing problem.

All I'm saying, John, is that currently we have to manually "close 
the doors", there are no ways within the language to make words private 
from the outside without doing the work ourself.


and most people don't realize that there is no such thing as a local 
word in rebol functions.  we only have arguments, which looks like 
a weird missing feature.

I'm a bit in doubt, because as you say, people might see this as 
a missing feature, even if it's not a problem. But wouldn't it be 
able to produce a function creator like FUNC, that makes a context 
with the locals, and then the function itself within the context 
and with the rest of the arguments?

Can you give an example of a function, where it's a problem, locals 
are not really local?

Maybe a function, where the local var is set, if some condition is 
fulfilled, and then the local is returned in the end, being NONE, 
if it wan't set. Is that an example of such a problem function?

Here is an example of a function hi-jacking.


it is something that can commonly be seen in larger apps, where some 
locals are only used conditionally.  in this case, the original function 
is hijacked and we could really do nasty things here.


--------------------------------------------------------------------
rebol []

f: func [a [string! block!] /local str][

	; uncomment to make the function safe
	; str: none

	if block? a [
		str: mold a
	]
	print any [str a]
]

evil-func: func [a ][
	; do something evil here
	print "EVIL!" 
	head insert a " >:-P >>>  "
]

f/local "Print occurs as if nothing is wrong" :evil-func

ask "!"

Good example! :)
Houston, we have a problem!

using this trick we can get internal function values even if we cannot 
change function bodies anymore.


my point is not that this can be done with any arguments.  its just 
that the way we all use /locals, we don't include argument protection.

so I think its just a good thing to allow this functionality by default 
in the language, since we all use /locals as if it where already 
there.

IIRC, its just a stub which creates a 'FUNC spec.

Maxim, in your example, the user has access to the function f, can 
call it direcly. Let's say, we couldn't give the /local refinement, 
then I could just write:

	f evil-func "Print occurs as if nothing is wrong"


Why is it a problem, we can give the /local refinement? Isn't it 
false security, if it is changed, so we can't call with /local ?

Ah, a scheme engine leftover :-)

i wish we would live in a world where u shouldn't be concerned about 
this kind of security issues :D

this concept is sooo lovely, that u can even build such a fundamental 
feature as local variable via the already existing specs dialect, 
my heart hurts to see we can't keep this virgin beauty and have to 
sacrifice it on the altar of security...

The surprise being that there aren't really any local variables, 
just arguments which might be optional. There was a proposal to make 
there be non-argument local variables in R3, and I wouldn't be opposed 
to this, but "least surprise" isn't a good enough argument for why 
in such an inherently surprising language as REBOL. I also wouldn't 
be opposed to having FUNCT add an ASSERT/type [local none!] to the 
beginning of the code block of every function it generates.

I can't figure out, if the /local behaviour is a real problem, or 
just one in our minds.

It has become a problem, now that people know it can be abused :-)

But who can abuse it? Just ourselves or outside hackers in a web 
application?

However, if someone gets a sandbox, every function made available 
can now be probed for code injection through this hole

For better local variables, you can use USE. But it's a bit clunky.

   f: func [a b][use [c] [c: a + b c]]   ;; 'c is local to the function, 
   and not easily/obviously affected by anything outside