World: r4wp

Red user code would only need to support the codepoint-series model; 
Red would translate that into the system's preferred underlying model. 
More encodings would need to be supported for conversion during I/O, 
of course, but not for API or internal use.

So far, my short-list of encodings to support are UTF-8 and UTF-16LE. 
UTF-32 might be needed at some point in the future, but for now, 
I'm not aware of any system that uses it?


The Unicode standard by itself is not the problem (having just one 
encoding would have helped, though). The issue lies in different 
OSes supporting different encodings, so it makes the choice for an 
internal x-platform encoding hard. It's a matter of Red internal 
trade-offs, so I need to study the possible internal resources usage 
for each one and decide which one is the more appropriate. So far, 
I was inclined to support both UTF-8 and UTF-16LE fully, but I'm 
not sure yet that's the best choice. To avoid surprizing users with 
inconsistent string operation performances, I thought to give users 
explicit control over string format, if they need such control (by 
default, Red would handle all automatically internally). For example, 
on Windows::

    s: "hello"		;-- UTF-8 literal string

    print s		;-- string converted to UCS2 for printing through win32 
    API
    write %file s	;-- string converted back to UTF-8

    set-modes s 'encoding 'UTF-16 ;-- user deciding on format
or
    s/encoding: 'UTF-16

    print length? s	;-- Length? then runs in O(1), no surprize.



Supporting ANSI as internal encoding seems useless, being able to 
just export/import it should suffice.

BTW, Brian, IIRC, OS X relies on UTF-8 internally not UTF-16.

Thanks, I don't know much about OSX's Unicode support.

set-modes s 'encoding 'UTF-16
should rather be:
    set-modes s [encoding: UTF-16]

Be sure to not forget the difference between UTF-16 (variable-length 
encoding of all of Unicode) and UCS2 (fixed-length encoding of a 
subset of Unicode). Windows, Java and .NET support UTF-16 (barring 
the occasional buggy code that assumes fixed-length encoding). R3's 
current underlying implementation is UCS2, with its character set 
limitations, but its logical model is codepoint-series.

IIRC Python 3 uses UCS4 internally for its Unicode strings, with 
all of the overhead that implies. UCS4 and UTF-32 are the same thing, 
both fixed-length.

If you support different internal string encodings on a given platform, 
be sure to not give logical access to the underlying binary data 
to Red code. The get/set-modes model is good for that kind of thing. 
If the end developer knows that the string will be grabbed from something 
that provides UTF-8 and passed along to something that takes UTF-8, 
they might be better off choosing UTF-8 as an underlying encoding. 
However, that should just be a mode - their interaction with the 
string should follow the codepoint model. If the end developer will 
be working directly with encoded data, they should be working with 
binary! values.

Btw, in this code above:
    s/encoding: 'UTF-16
    print length? s	;-- Length? then runs in O(1), no surprize.


Length is not O(1) for UTF-16, it's O(n). Length is only O(1) for 
the fixed-length encodings.

Since Python 3.3, things have changed: http://www.python.org/dev/peps/pep-0393/

Brian: right, my claim is valid for BMP characters only.

Ah, but length is even O(n) for BMP characters in a UTF-16 string, 
because figuring out that there are only BMP characters in there 
is an O(n) operation. To be O(1) you'd have to mark some flag in 
the string when you add the characters in there in the first place.

Ok, if you really want to be nitpicking, replace UTF-16 with UCS-2. 
;-)

If you are ensuring that only BMP characters are in there then you 
have UCS2, not UTF-16 :)

Python 3.3 seems to finally be following the R3 model, good for them. 
Even better for them because it's actually implemented.

Don't worry, I'm only nitpicking to make things better. There's a 
lot of buggy code out there that assumes UTF-16 is UCS2, so we're 
better off making that distinction right away :)

Well, then I'm sure you'll be glad to write string unit tests for 
Red in order to ensure things are done in the proper way. ;-)

Doc, pardon me because I don't know what the intended datatype model 
is for Red. Something like the REBOL datatype/action model could 
be used to implement the different underlying string encodings that 
you want in-memory support for. Each supported encoding would have 
its own set of action handlers, which would all have the same external 
interface. Swapping the encoding would be as simple as swapping the 
handler set. Resolving the difference at compile time could be similar 
to generic type instantiation, or C++ template generation.

The datatype code that's committed so far seems to go that way

Brian: implementing an abstraction layer over string encodings is 
a trivial task. The intended datatype model is very similar to REBOL's 
one, even more since recentl,y as I moved to an hybrid dynamic/static 
type system. I'll commit the new code in a few days, so you'll see 
how close to REBOL it can be.


I hope that this hybrid model will help us get the best of both worlds.

*recently,

Got my first real Red program working: 
    Red [] print 1
outputs:
    1


It doesn't look like much, but it validates the compiler + runtime 
from end to end, and at this point, it's really cool! FYI, the native 
PRINT here triggers a FORM (action) on the passed argument. No REDUCE 
yet (not implemented).

Congratulations Doc. That is so cool. Really!

Thanks James!

Wow!!! awesome .. from print "hello worl" to print "<html>...</html>" 
to make webapps is not that long way, I was just afraid you would 
never get to RED, Awesome!

Yeah! Way to go!

Congratulations Doc! This is a great day.

A milestone

( kilometre in France )

+1 for Red :)

Congratulations, doc. :-)

This gives hope

Cool! Another 30€ for the good news :)

Wooow.. Wooow... Did I hear the crying of a baby... I think this 
is the birthday of Red. The past 1.5 years are just pregnancy.

:-) Can't wait to show Red to the Chinese people.

From now on, the child is going to grow day by day :-)

I can tell, it's gonna be a super star :-)

Congrats, Doc!

Endo: thank you!

Thanks to all for your support!

Pekr: right, from now on, you can expect daily progress on Red layer. 
I will push the new code soon, I still need to complete it a bit 
and clean it up.


Jerry: the baby looks nice, we'll just have to keep it away from 
junk food and it will grow up well. ;-)

Many congratulations Nenad.

Thanks Peter!

Félicitations Doc - I will also submit an additional  donation to 
help the baby to grow further ... 50€  was sent to you Doc

Doc, I announce this good news in Chinese http://weibo.com/rebol/
 :-)

Thanks Gerard! 

Jerry: Cool! I'll make a blog entry too.

Pekr: "I wonder what the strategy for Red is going to be in regards 
to GC ..."


Currently Red is using a per-OS thread mark-sweep-compacting GC. 
I intend to make it concurrent (maybe parallel) and incremental when 
(and if) possible. The collector is not yet fully implemented. It 
is also possible to easily make it generational if we need it. When 
we'll have big Red apps, we'll profile them and determine the best 
tuning options for the GC.


There's no silver bullet for memory management & GC, you need to 
adapt it to the language specific needs (and they may vary depending 
on the type of application: client-side, server-side, realtime, ...).

Will it be possible to swap it out, depending on needs?

I plan to use an abstraction API for memory management, so that we 
could in future, implement different ones.

Brian: I was wrong for OS X, it uses UTF-16 internally according 
to http://en.wikipedia.org/wiki/UTF-16

I am giving a speech in SDCC conference tomorrow. http://sdcc.csdn.net/speaker.html
My topic is Dart. I hope it will be Red next year when Red is mature 
enough. :-)

In a year, we should have passed v1.0 since a while. ;-)