World: r4wp

I am for sure no expert regarding unicode, but as red is a compiler 
and open source, why not not add flags that the user has to choose 
which unicode/string support he wants; either flexibility, but of 
cost of speed or no unicode support, then he  has to do the hard 
work by himself

One hypothetical advantage you have with Red is that you can make 
the logical behavior fairly high-level and have the compiler/optimizer 
get rid of that at runtime. REBOL, being interpreted, is effectively 
a lower-level language requiring hand optimization, the kind of hand 
optimization that you'd want to prohibit in Red because it would 
interfere with the machine optimization. This means that, for strings 
at least, it would make sense to have the logical model have a lot 
of the same constraints as that of R3 (because those constraints 
were inherent in the design of Unicode), but make the compiler aware 
of the model so it can translate things to a much lower level. If 
you break the logical model though, you remove the power the compiler 
has to optimize things.

sqlab, it would make sense to have the user choose the underlying 
model if you are doing Red on bare metal and implementing everything 
yourself, or running on a system with no Unicode support at all. 
If you are running a Red program on an existing system with Unicode 
support, the choice of which model is best has already been made 
for you. In those cases choosing the best underlying model would 
best be made by the Red porter, not the end developer.

but that means, that Red has to support all unicode models on all 
the systems, it can be compiled for.

That's not as hard as it sounds. There are only 3 API models in wide 
use: UTF-16, UTF-8, and no Unicode support at all. A given port of 
Red would only have to support one of those on a given platform.

Red user code would only need to support the codepoint-series model; 
Red would translate that into the system's preferred underlying model. 
More encodings would need to be supported for conversion during I/O, 
of course, but not for API or internal use.

So far, my short-list of encodings to support are UTF-8 and UTF-16LE. 
UTF-32 might be needed at some point in the future, but for now, 
I'm not aware of any system that uses it?


The Unicode standard by itself is not the problem (having just one 
encoding would have helped, though). The issue lies in different 
OSes supporting different encodings, so it makes the choice for an 
internal x-platform encoding hard. It's a matter of Red internal 
trade-offs, so I need to study the possible internal resources usage 
for each one and decide which one is the more appropriate. So far, 
I was inclined to support both UTF-8 and UTF-16LE fully, but I'm 
not sure yet that's the best choice. To avoid surprizing users with 
inconsistent string operation performances, I thought to give users 
explicit control over string format, if they need such control (by 
default, Red would handle all automatically internally). For example, 
on Windows::

    s: "hello"		;-- UTF-8 literal string

    print s		;-- string converted to UCS2 for printing through win32 
    API
    write %file s	;-- string converted back to UTF-8

    set-modes s 'encoding 'UTF-16 ;-- user deciding on format
or
    s/encoding: 'UTF-16

    print length? s	;-- Length? then runs in O(1), no surprize.



Supporting ANSI as internal encoding seems useless, being able to 
just export/import it should suffice.

BTW, Brian, IIRC, OS X relies on UTF-8 internally not UTF-16.

Thanks, I don't know much about OSX's Unicode support.

set-modes s 'encoding 'UTF-16
should rather be:
    set-modes s [encoding: UTF-16]

Be sure to not forget the difference between UTF-16 (variable-length 
encoding of all of Unicode) and UCS2 (fixed-length encoding of a 
subset of Unicode). Windows, Java and .NET support UTF-16 (barring 
the occasional buggy code that assumes fixed-length encoding). R3's 
current underlying implementation is UCS2, with its character set 
limitations, but its logical model is codepoint-series.

IIRC Python 3 uses UCS4 internally for its Unicode strings, with 
all of the overhead that implies. UCS4 and UTF-32 are the same thing, 
both fixed-length.

If you support different internal string encodings on a given platform, 
be sure to not give logical access to the underlying binary data 
to Red code. The get/set-modes model is good for that kind of thing. 
If the end developer knows that the string will be grabbed from something 
that provides UTF-8 and passed along to something that takes UTF-8, 
they might be better off choosing UTF-8 as an underlying encoding. 
However, that should just be a mode - their interaction with the 
string should follow the codepoint model. If the end developer will 
be working directly with encoded data, they should be working with 
binary! values.

Btw, in this code above:
    s/encoding: 'UTF-16
    print length? s	;-- Length? then runs in O(1), no surprize.


Length is not O(1) for UTF-16, it's O(n). Length is only O(1) for 
the fixed-length encodings.

Since Python 3.3, things have changed: http://www.python.org/dev/peps/pep-0393/

Brian: right, my claim is valid for BMP characters only.

Ah, but length is even O(n) for BMP characters in a UTF-16 string, 
because figuring out that there are only BMP characters in there 
is an O(n) operation. To be O(1) you'd have to mark some flag in 
the string when you add the characters in there in the first place.

Ok, if you really want to be nitpicking, replace UTF-16 with UCS-2. 
;-)

If you are ensuring that only BMP characters are in there then you 
have UCS2, not UTF-16 :)

Python 3.3 seems to finally be following the R3 model, good for them. 
Even better for them because it's actually implemented.

Don't worry, I'm only nitpicking to make things better. There's a 
lot of buggy code out there that assumes UTF-16 is UCS2, so we're 
better off making that distinction right away :)

Well, then I'm sure you'll be glad to write string unit tests for 
Red in order to ensure things are done in the proper way. ;-)

Doc, pardon me because I don't know what the intended datatype model 
is for Red. Something like the REBOL datatype/action model could 
be used to implement the different underlying string encodings that 
you want in-memory support for. Each supported encoding would have 
its own set of action handlers, which would all have the same external 
interface. Swapping the encoding would be as simple as swapping the 
handler set. Resolving the difference at compile time could be similar 
to generic type instantiation, or C++ template generation.

The datatype code that's committed so far seems to go that way

Brian: implementing an abstraction layer over string encodings is 
a trivial task. The intended datatype model is very similar to REBOL's 
one, even more since recentl,y as I moved to an hybrid dynamic/static 
type system. I'll commit the new code in a few days, so you'll see 
how close to REBOL it can be.


I hope that this hybrid model will help us get the best of both worlds.

*recently,

Got my first real Red program working: 
    Red [] print 1
outputs:
    1


It doesn't look like much, but it validates the compiler + runtime 
from end to end, and at this point, it's really cool! FYI, the native 
PRINT here triggers a FORM (action) on the passed argument. No REDUCE 
yet (not implemented).

Congratulations Doc. That is so cool. Really!

Thanks James!

Wow!!! awesome .. from print "hello worl" to print "<html>...</html>" 
to make webapps is not that long way, I was just afraid you would 
never get to RED, Awesome!

Yeah! Way to go!

Congratulations Doc! This is a great day.

A milestone

( kilometre in France )

+1 for Red :)

Congratulations, doc. :-)

This gives hope

Cool! Another 30€ for the good news :)

Wooow.. Wooow... Did I hear the crying of a baby... I think this 
is the birthday of Red. The past 1.5 years are just pregnancy.

:-) Can't wait to show Red to the Chinese people.

From now on, the child is going to grow day by day :-)

I can tell, it's gonna be a super star :-)

Congrats, Doc!

Endo: thank you!

Thanks to all for your support!

Pekr: right, from now on, you can expect daily progress on Red layer. 
I will push the new code soon, I still need to complete it a bit 
and clean it up.


Jerry: the baby looks nice, we'll just have to keep it away from 
junk food and it will grow up well. ;-)

Many congratulations Nenad.

Thanks Peter!

Félicitations Doc - I will also submit an additional  donation to 
help the baby to grow further ... 50€  was sent to you Doc

Doc, I announce this good news in Chinese http://weibo.com/rebol/
 :-)

Thanks Gerard! 

Jerry: Cool! I'll make a blog entry too.

Pekr: "I wonder what the strategy for Red is going to be in regards 
to GC ..."


Currently Red is using a per-OS thread mark-sweep-compacting GC. 
I intend to make it concurrent (maybe parallel) and incremental when 
(and if) possible. The collector is not yet fully implemented. It 
is also possible to easily make it generational if we need it. When 
we'll have big Red apps, we'll profile them and determine the best 
tuning options for the GC.


There's no silver bullet for memory management & GC, you need to 
adapt it to the language specific needs (and they may vary depending 
on the type of application: client-side, server-side, realtime, ...).