AltME groups: search

R2.   


since we compile just about all the rules from other datasets and 
simplified user-data, the R3 advantage is much less significant (because 
we can simulate all the R3 improvements by using R2 idoms, though 
its sometimes tricky).


Using R3, it probably would be a few percent faster since some of 
the rules we have would be simpler and those tricks would be managed 
natively by parse rather than by *more* parse rules.

The problem with R3 right now is that it isn't yet compiled in 64-bits 
we still have the 1.6GB RAM limit for a process which is the biggest 
issue right now.   I have blown that limit a few times already, so 
it makes things a bit more complex and it doesn't allow me to fully 
optimize speed by using more pre-generated tables and unfolded state 
rules.

Our datasets are huge and we optimise for performance by unfolding 
and indexing a lot of stuff into rules... for example instead of 
parsing by a list of words, I parse by a hierarchical tree of characters. 
 its much faster since the speed is linear to the length of the word 
instead of to the number of items in the table. i.e.  the typical 
 O*n   vs.   O*O*n  type of scenario .  just switching to parse already 
was 10 times faster than using  hash! tables and using find on them.... 


In the end, we had a 100 time speed improvement from before parse 
to compiled parse datasets.  this means going from 30 minutes to 
less than 20 seconds....but this comes at a huge cost in RAM... a 
400MB Overhead to be precise.

(btw that 1.6GB limit used to be a real problem when I was doing 
3D stuff...  3D animation apps are memory hogs, and in some cases, 
we could only work 15 minutes before high-end apps would crash.

which is a problem when a 3D scene takes 30 minutes to save to disk 
over the network  ;-)

You can use FORM as well.

And having alternatives should not be something to complain about. 
:)

O*O*n
  == a typo  :-)

I guess I really meant  something like O(n*n) 


Its the kind of dramatic  linear vs logarithmic scaling difference 
when we unfold our datasets into parse.


but its not exactly that kind of scaling, since the average topology 
of the sort tree will have a lot of impact on the end-result.  for 
example in my system, when I try to index more than the first 5 characters, 
the speed gain is so insignificant that the ratio is quickly skewed, 
when compared to the difference which the first 3 letters give.  


Its 100% related to the actual dataset I use.  in some, going past 
2 is already almost useless, in others I will have to go beyond 5 
for sure.  in some other datasets we unfold them using hand-picked 
algorythms per branch of data, and others its a pure, brute force 
huge RAM gobler.

it also looks in the current-dir... but that path will depend of 
how you launched rebol.


use WHAT-DIR just before you try to load your dll  to know where 
the current-dir is at that time and put your dll there.


you can also add a path in the user or system path environment and 
place the dll there.

I'll continue here for now, as /library is now a free part of Core, 
and DLL.SO is not web-public.

If datatypes equals words, like word! = 'word!, then maybe the refinement 
in type?/word isn't needed? But what are the consequences? The next 
two examples would return the same:

>> find [integer! 42] integer!
== [42]
>> find [integer! 42] 'integer!
== [integer! 42]


I came to think of this, because I find myself writing things like 
the following all the time now:

	either find [block! paren!] type?/word value [ ...
and
	switch type?/word value [ ...


If datatypes equals words, only type? without the refinement would 
be needed.

I know, I today can write things like


 either find [#[datatype! block!] #[datatype! paren!]] type? value 
 [ ...


but I don't do that, because it has too much syntax for my taste, 
and therefore isn't very readable.

refinement! is member of the any-word! typeset together with word!, 
set-word!, get-word! and lit-word!. My thoughts above lead to asking 
if also none! and logic! should be part of any-word! with datatype! 
too? Examples from R2:

>> /ref = 'ref
== true
>> find [/ref]Ê'ref
== none		; this is strange to me

Maybe all the next should succeed?

>> find [true] true
== none
>> find [none] none
== none
>> find [integer!] integer!
== none

none! and logic! are simply not word types, so it makes no sense 
to have them in the any-word! typeset. none/true/false being words 
conveniently pre-bound to values of the corresponding datatypes does 
not change that.

Note that we also have a literal syntax for none! and logic! values 
now, which makes all your finds succeed even without reducing:

>> find [#[true]] true
== [true]

Etc.

Integers are not decimals, but they're both numbers, and we can check 
like:

>> 1 = 1.0
== true

Refinement are not words, but they're both any-words.


Why not let datatypes (and none and logic) be any-words just the 
same? If the benefit from doing so is big, then it could be a good 
idea.

There is a big difference between having datatypes be word values, 
versus having them fall under the any-word pseudotype. The latter 
seems OK, but not the former. If I understand you, it would cause 
things like [datatype? integer!] to fail, because it would return 
word!. That is, we lose them as an identifiable datatype. I use them 
for reflective tools and dialects. While the change wouldn't make 
that impossible, I like them being first class citizens as they are 
today.

No, let me clarify. I want integer! to represent a datatype, like 
1 represents an integer. So datatype? integer! should return true, 
and word? integer! should return false, just like decimal? 1 returns 
false.


I simple suggest equal? to return true, when comparing a datatype 
with a word of the same spelling. Like this is true:

>> equal? 1 1.0
== true

Technical speaking, it's an expanded coercion for the equal operator, 
=, (and so also for the equal? function).

MAP is an associative (Key <-> Value) data "storage". In R2 a correspoding 
way would be to use the hash! datatype, however, if you want to discern 
keys from values you need to use a separate Keys hash! and a separate 
Values block, otherwise you end up having Keys and Values intermixed. 
Your way of using the /skip refinement and a block is slower, however 
it searches only in Keys as well due to the /skip 2 use. When not 
used, it would search in Values.

I know the theory:/ To have separate hashes for key and values would 
be even more complicated. I would be fine if the select/skip would 
not return a block which is simply stupid... or correct me if there 
is any reason for that. It's sad we cannot have map! in R2.

To have separate hashes for key and values would be even more complicated

 - that is wrong, there is no need to have two hashes, moreover, it 
 is not complicated, I wrote the corresponding software, and it is 
 easy

If I could move time back a few years and I could vote, I would like 
Carl to enhance R2 a little bit instead of starting R3 which he probably 
never finish.

And I will not ask.. I was asking so many times without any response 
that I gave up long time ago.

Or ... if the language makes you do anything, like e.g. C, and what 
it does, it does well, then it doesn't matter, if it's closed source 
or not.

Geomol's right. R2 can be extended via DLLs, and R3 via extensions. 
But that might not solve all usage cases or needed fixes in Core 
...

I don't care, if my C compiler is closed source or not, because it 
just works. I also shouldn't care, if my COBOL compiler and interpreter 
is closed source or not, but I actually do, because the company behind 
can't figure out to make graphical tables (called GRIDs) the correct 
way, so my COBOL programs doesn't work as intended, and I have to 
create work-arounds. Years ago, I didn't care, if REBOL was closed 
source or open, but later I did, because I couldn't finish projects, 
I made with it.


Any future language, I would use, I don't care if it's closed or 
open, if it delivers, what it promices. If it doesn't, it's another 
case.

A good language is also easy to expand and integrate with other technologies. 
For some projects, it may be a good idea to have the language as 
a dynamic linked library.

We all do mistakes even after years and years of practices

We all do mistakes even after years and years of practices

But that doesn't exclude, that if I use some time and think about 
some problem, I can figure out, if it's doable in some language the 
'right' way.

Geomol, I already know that you made some technical choices in World 
that I would not have done because I think (maybe I'm wrong) I know 
better ways to do faster VM.

So, to my mind,you already failed in the task to deliver a promising 
clone.

Just to say that your 'needs' , expectations and technical skills 
are probably not the best in each room.
;-)

Lad +1


The REOL Syntax project is very important in this regard, and documenting 
the differences between the clones. Being the fastest VM my not be 
John's goal, if it means other tradeoffs.

bool is in C since C99, and it's, in fact, *drumroll* a typedef/#define 
for _Bool

The C standard is (as always) very vague and doesn't specify a particular 
storage size. I think it just defines that _Bool must be able to 
hold 0 and 1.

So when encountering a "bool", one really has to ask what kind of 
bool that is. More often than not, it won't be a C99 _Bool and will 
actually be int-sized.

And I think Pekr really has a different case, because he's trying 
to use a C++ lib. C++ really has a bool, and I think it's char-sized 
(but I don't know whether that's impl-specific in C++ or defined 
in the C++ standard).

I guess that other sizes are atypical and not exactly respecting 
the standard

when it is int it could be 64bit or 32bit.. depend on the compiler 
and the OS. 

Pekr: So may be the C library returns a 64bit integer: #{0000000001D30000} 
= #{00000000 - 01D30000}, the left part is 0 (=false) the second 
part is just a number from stack.. 
When you get in from R2 it become #{01D30000} = 30605312

Or it's a 16 bit return, and the machine, it's from, is little endian, 
where World is big endian. Then it's the last 4 zeros in the result, 
that is the bool.

A slightly better test would be:

#include <stdio.h>

int main(int argc, char *argv[]) {printf("%ld\n", sizeof(_Bool)); 
return 0;}


And then compile that in C99 mode (i.e. -std=c99 for GCC; but C89/90 
compilers will bark on the unknown _Bool keyword anyway).


Better, because there exist(ed) a few stdbool.h versions prior to 
the final C99 standard which used e.g. unsigned integers for bools. 
Should be gone/fixed by now, but one never knows :)

Under OS X using gcc, I get bool length 1 both with Cyphre and Andreas 
versions, and also with or without -std=c99 option.

We do not need to be too limited, currently the source is R3 specific, 
but I do intend to put in also R2, and the inclusion of other alternatives 
may be a worthy enterprise as well.

that is called IMPLICIT-BLOCK and is there

ah.. hehe my browser had an old version of the rules... did a page 
refresh and there it was   ;-)

Ladislav, I see what you did there ;-)
Tuples:
Each number is 3 digit max
Leading and trailing zeros are optional 
.0. == 0.0.0

actually, the rule should load the digits and fail if the integer 
it represents is higher than 255.

yes but it's hard to do it without code evaluation.
Same remarks apply to integer! and decimal! datatypes.

We are aiming for a purely syntactic description. So semantic checks 
at a later stage are out of the picture for now (no action blocks 
in general, and here: no loading of digits in particular).

(Same for integer and decimal overflows, as Steeve already remarked.)

tuple-syntax: [0 3 digit #"." 1 3 digit #"." 0 7 [1 3 digit #"."] 
and termination]

I know there is still the need to check the overflow but I think 
it's slightly better than using some or any digits

tuple-syntax: [0 3 digit #"." 1 3 digit #"." 0 7 [1 3 digit #"."] 
0 3 digit and termination]

to get rid of the 3-digit enforcement:

tuple-syntax: [any digit #"." some digit #"." 0 7 [some digit #"."] 
any digit and termination]

But but... it's not only the leading and trailing zeros that can 
be removed.
It takes me time......

or/and the first one

tuple-syntax: [[some digit 2 9 [#"." any digit] | #"." some digit 
1 8 [#"." any digit] ] and termination]

Andreas and Ladislav, already off ?

I think it was mainly meant as a way to make / based ops   things 
like //  and  ///  .  I don't see why it should be removed.  it can 
only contain  "/"  characters.  otherwise its a refinement.

about the more-less-word rule:
>> and << are allowed

hmm, just tested, and load "=>" does not work in R2 either

how about the + and - words? Is there a ticket for the refinement 
syntax?

I also forgot to add the special cases for + and -

Well it's more complex than that.
+' and -' are invalid

last attempt for tonight

word-syntax: [
	[#"." | #"+" not #"'" | #"-" not #"'" | not #"'" and c-word] 
	[
		c-word any [c-word | digit]
		| termination
	]
]

Too obfuscated maybe
Feel free to go on anyone

When it comes to things like word syntax, the errors raised for bad 
syntax and the particular cases where they are raised need to be 
part of the rules.

last but not least


word-char: complement union charset "()[]^"{}/;<>,\#$%:@^@" whitespace
word-ext: [
	word-char any [word-char | digit]
	[and [#"<" | #">"] | termination]
]
word-syntax: [
	not sign not #"." not #"'" word-ext 
	| #"." [and [#"<" | #">"] | termination]
	| #"." word-ext
	| sign termination
	| sign not #"'" word-ext
	| more-less-word
]

grumpffff, still missing the cases with prefix [+.] and [-.]

word-syntax: [
	not sign not #"." not #"'" word-ext 
	| #"." [and [#"<" | #">"] | termination]
	| #"." word-ext
	| sign termination
	| sign not #"'" opt #"." word-ext      ; <--- there
	| more-less-word
]

Remaining problems with [termination] in word-syntax.
When a word is stuck with a less-word or a tag (R2 and R3)
a<
 == [a <] valid
a<=
 == [a <=] valid
a<>
 == [a <>] valid
a<<
 == [a <<] valid
a<tag>
 == [a <tag>] valid

a>=
 invalid
a>
 invalid
a>>
 invalid

IMO, t's enough to check if the following char is #"<" only

word-syntax: [
	slash-word termination
	| more-less-word termination
	| opt sign [#"." | not #"'"] not digit any word-char
		[termination | and #"<"]
]

A112 (not released) is reported to have other improvements related 
to get-words and set-words

A question for Brian: do you think the case:


load "a<a>" ; == [a <a>] shall be mentioned in CC? (and, eventually, 
where?, a new ticket or an old one?)

Steve are you aware of the R3 parse AND operator?  maybe that is 
throwing you off?

the AND is a look ahead.  it doesn't advance the input, so whatever 
is matched by    [ AND termination ] only tries to find a delimiter.

Al least it's a problem with [termination] used in tuple-syntax and 
decimal-syntax.
Don't say i'm wrong here again :-)

ah, yes... you are struggling with the AND here   :-)   the termination 
is not part of the tuple... but it does mark it  :-)

the key here really is that AND  ;-)

That way, word-syntax must have one character length à least
word-syntax: [
	[
		slash-word
		| more-less-word

  | and word-char opt sign [#"." | not #"'"] not digit any word-char
	]
	termination 
]

There were some good reasons for the complexity of the rules in http://issue.cc/r3/1302
and the error handling was necessary too, since the actual word syntax 
is only recognized after it eliminates the error conditions.

I've been meaning to adapt those rules to R2 though. There should 
be more bugs there though, and unlike the bugs in the R3 syntax we 
can't fix them in R2.

for R3, I think the following trick is enough

word-syntax: [
	...
	[and tag-syntax | termination] 
]

There really weren't that many, and most of them were fixed in alpha 
97. Some of them are deliberate tradeoffs, such as http://issue.cc/r3/1317

One interesting thing is that R2 and R3 have binary parsers, not 
text parsers. The difference doesn't matter that much in R2 but in 
R3 it matters a lot.

try this in R2
[a<] and [a<  ]

Similarly the above

    load "+<tag>"

and

    load "-<tag>"

look like an inconsistency in syntax.

We could produce several documents

(Btw I don't think it's a practical idea to continue further mixing 
R2 and R3 syntax)
- R3 pure expected syntax (without glitch, inconsistency)
- R2 pure expected syntax (without glitch, inconsistency)
- R3 with glichs
- R2 with glichs

When people wanted to refer to the < word in R2, and they can't use 
the lit-word syntax for arrow words in R3 and pre-a97 R3, one way 
is to store that word in a block and use FIRST to get the value. 
However, in R2 that resulted in a value that LOAD choked on. The 
<] tradeoff was made really early on in the R3 project to solve that 
issue. The alternative would be to make MOLD mold [<] as [< ], or 
more specifically to make < mold as "< ", with an extra space every 
time.

in R3 and pre-a97 R3 -> in R2 and pre-a97 R3

The way MOLD is written, the values are molded by code that doesn't 
know it's in a block. You could have the ] handling code check against 
a charset of iffy characters and then optionally insert an extra 
space if found, but that doesn't deal with user-written code where 
[>] works and [<] doesn't. The usage of ] as the first character 
in a tag is so rare that it's not a bad tradeoff to make.

When I was trying to replicate the R3 word syntax, it was partly 
to document R3, partly to serve as the basis of a more flexible TRANSCODE 
that would allow people to handle more sloppy syntax without removing 
the valuable errors from the regular TRANSCODE, but mostly it served 
to generate new CC tickets for syntax bugs that we weren't aware 
of because the syntax wasn't well enough documented, and they hadn't 
come up in practice yet.

I'm a little more concerned with R3 URL syntax though, since in that 
case there are real bugs that have already affected people in real 
cases, and because hypothetically a lot of the bugs are fixable in 
mezzanine code.

And as the email! datatype can be used for many a purpose within 
dialects, it does not necessarily have to match RFC822 (or rather 
5322) exactly.

Corrected version, works with R2 and R3:

escape-uri: [#"%" 2 hex-digit]
email-char: complement union charset {%@:} termination-char
email-esc: [email-char | escape-uri]
email-syntax: [
	[
		#":" any [email-esc | #":" ] #"@" any [email-esc | #":" ]
		| not #"<" some email-esc #"@" any email-esc
	]
	termination
]

That's a good start! I'm really curious about whether ulrs and emails 
deal with chars over 127, especially in R3. As far as I know, the 
URI standards don't support them directly, but various internationalization 
extensions add recodings for these non-ASCII characters. It would 
be good to know exactly which chars supported in the data model, 
so we can hack the code that supports that data to match.

When last I checked, R3 considers all chars over 127 to be word-chars. 
It is considered to be non of REBOL's business whether a printer 
or display would show the character, so that even includes the additional 
Unicode space and control characters beyond ASCII. R3 has a binary 
parser, you see.

Do you know if the REBOL syntax parser (LOAD and TRANSCODE) handles 
the unescaping and puts the decoded data into the url! structure, 
or if that is handled by the DECODE-URL mezzanine code? I'm hoping 
it's handled by the mezzanine, because it's broken in both R2 and 
R3 and mezzanine changes are the only kind we can make at the moment.

AFAICT  it's part of the datatype... since a space will go back and 
forth when you go to/from URL! and other types like string

(in R2 at least):
>> to-url "gogo://a.com/space here"
== gogo://a.com/space here
>> to-string gogo://a.com/space here
== "gogo://a.com/space here"

The escape decoding gets done too early. The decoding should not 
be done after until the URI structure has been parsed. If you do 
the escape decoding too early, characters that are escaped so that 
they won't be treated as syntax characters (like /) are treated as 
syntax characters erroneously. This is a bad problem for schemes 
like HTTP or FTP that can use usernames and passwords, because the 
passwords in particular either get corrupted or have inappropriately 
restricted character sets. IDN encoding should be put off until the 
last minute too, once we add support for Unicode to the url handlers 
of HTTP, plus any others that should support that standard.

yep... and I've lost hours trying to get some ftp code to work because 
it had strange urls (with passwds)... which the interpreter would 
break all the time. 

At some point you are mystified by what is the actual URL being sent 
to the server.


once you see what is going on, you can get it to work, but realizing 
that you didn't actually send the url you expect, can take quite 
a long time to realize and properly fix once you've got a whole app 
expecting/playing with urls.

I've been hoping to fix that. I can load a hot-patch into R2, and 
include a patch in a host kit build in R3 or replace functions from 
%rebol.r if necessary.

Ok I try to resume our concern.

The url! and email! syntax is more permissive than a valid URI. It's 
not a problem nor a design flaw.

The escape decoding should not be done at all when decoded as a part 
of an url! or email!. Right, but it will not be corrected until Carl 
does it.

DECODE-URL can be rewritten (used by schemes). The parser is too 
strict and can't deal with complex forms.

Lot of inconsistencies with file! datatype between R2 and R3.
Escaping notation = huge mess

Worse than being a huge mess, R2 and R3 have different messes. R2 
MOLD fails to encode the % character properly. R3 chokes on the ^ 
character in unquoted mode, and allows both ^ and % escaping in quoted 
mode, and MOLDs the ^ character without encoding it (a problem because 
it chokes on that character). Overall the R2 MOLD problem is worse 
than all of the R3 problems put together because % is a more common 
character in filenames than ^, but both need fixing. I wish it just 
did one escaping method for files, % escaping, or did only % escaping 
for unquoted files and only ^ escaping for quoted files. % escaping 
doesn't support Unicode characters over 255, but no characters like 
that need to be escaped anyways - they can be written directly.

I guess that I just want the escaping behavior Steeve described for 
R2, but with the MOLD of %%25 fix from R3, along with % by itself 
being interpreted as and molding as %"".