World: r3wp

It would also be very useful if the Amiga patches were incorporated 
into the host kit

They probably use UTF-8

That might also require some conversion, but at least then the conversion 
would be there to use. R3 uses UCS for strings internally for speed 
and code simplicity, though strangely enough words are stored in 
UTF-8 internally, since you don't have to access and change words 
on a character basis.

R3 uses both, single-byte by default and multi-byte if needed.

Windows uses UTF-16 for its APIs, not UCS-2, so by using UCS-2 R3 
is limited to the BMP codepoints.

Good to hear that the UCS-1 to UCS-2 autoexpansion is still there. 
We don't have the UCS-4 expansion supported yet though.

Is UCS-2 sufficient for Chinese?

I don't know, but know that I need wchar at this point and don't 
have it:

https://github.com/rebolsource/r3-hostkit/blob/f331c6a46947e6e5afedc90f3d375bcd3f7ad8a1/src/agg/agg_truetype_text.cpp#L696

Yes, I think so.

(to Kaj)

I must dig deeper.. the solution is visible even with the Carl's 
version as I can see the text using draw... it's like solving a puzzle...:)

The BMP should cover most stuff, but there is a whole supplemental 
plane dedicated to more obscure asian ideographic scripts, and some 
of those might come up in Chinese language eventually.

It's still an ongoing issue, as many asian nations don't like each 
other very much, so commonalities in their character sets are often 
controversial.

I've found it.. rich-text is doing the conversion here:

https://github.com/rebolsource/r3-hostkit/blob/f331c6a46947e6e5afedc90f3d375bcd3f7ad8a1/src/os/win32/host-graphics.c#L714

The question is, how to use RL_GET_STRING if I already have REBCHR 
instead of REBSER.

Also isn't it REBOL's week spot if we need so much text manipulations? 
At least for international languages?

For example the CMD_TEXT_TEXT command is called on each mouse move 
event (redraw).

It is not a REBOL-secific weak spot. All programming languages have 
to deal with Unicode issues in some way or another, and every means 
of dealing with it has its good and bad points. Any cross-platform 
language will run into a little difficulty if it has to interact 
with the platform-specific Unicode APIs, because different platforms 
deal with the problems in different ways.

REBOL-specific

Mistake - extensive ansi text will be affected as AGG is using widechar, 
each ansi string which we want to display in view must be converted 
to wchar on each redraw!

It could be fixed if we could change the ansi string to unicode and 
store it for later use.. I'm just a C newbie, but I don't think it's 
how it works now.

Does AGG have separate 8bit and 16bit rendering APIs or is it always 
one or the other?

I think it has only 16bit.

But I'm not so far to be sure.

If it is UCS-2 or UTF-16, then all that would need to be done is 
to convert UCS-1 model R3 strings to UCS-2 mode somewhere before 
rendering. (He says glibly, having not analyzed the AGG sources or 
APIs.)

I'm still guessing this only applies to AGG on Windows, using UTF-16. 
On other platforms, AGG uses FreeType, and I guess that would accept 
UTF-8

Submited it to CC http://curecode.org/rebol3/ticket.rsp?id=1814so 
Carl could answer eventualy.

I think the file you're looking at is not really part of AGG, but 
written by Cyphre as a bridge between AGG and Windows

I know. But it does not solve the issue.

The issue is we have to wait for the Amiga patches, which should 
constitute the bridge to FreeType systems

Also I'm not sure REBOL is using UTF-8 internally, I think it has 
only ANSI or UCS2

No, as Brian says, it's using fixed width vectors internally. You 
get UTF-8 only from conversions

which means that FreeType will be affected as well.

Yes, we knew since the Unicode epic that we would need conversions

But for each redraw? The conversion is fine, if is only one.

That can be cached

The text conversion is actually a minor matter. Code points also 
need to be converted to pixel glyphs, and heavy caching is always 
used there to arrive at usable performance

Btw... it's the reason why I'm hacking the code... I would like to 
enable bitmap fonts. AGG supports that, but I don't think that Cyphre 
has enough time to work on it alone:)

Unfortunately instead of experimenting with agg itself, I must solve 
issues which are already solved but not public yet.. but at least 
I'm learning.

Rebol strings are stored either as ANSI (one byte) or wide char (double 
byte). Of course the rich-text module is currently doing the conversion 
for every rendered ANSI string in realtime. Any sophisticated rich-text 
caching is not yet implemented. (note: this has nothing to do with 
font glyph cache which works well) But even though the cache of large 
text block is missing the performance is still very usable for normal 
GUI cases so the priority to spend time on the line-cache is not 
too high at the moment.

I understand what Oldes wants. He want to be able to change already 
allocated Rebol string! from ANSI to WIDE-CHAR 'mode' so next time 
the conversion is not needed. Tha could be useful but has to be decided/implemented 
by Carl in the hostkit api.

Also it would be interesting to see how big is the difference between 
on-the-fly conversion performance to see if it really makes sense 
to permanently take two times more memory for such Rebol string or 
just do the temporary on-the fly conversion as it  works now.

I can imagine, if there is a lot of the text, that scrolling such 
a text, might be influenced in performance, or not?

It's possible to do such a test... just to compare the performance 
with ansi strings only and with strings which contains non ansi chars 
as well (so are stored as wide char internally). I can try it once 
I will have some time again.

that would be nice ...

But I would rather wait for the RMA's host-kit release before the 
test.

For those who are impatient I put the RMA Host-Kit on this URL: http://www.rm-asset.com/code/downloads/files/rma-host-kit.zip

It's not yet on the official RMA download page but Robert will be 
updating the page content soon.

Published: http://www.rm-asset.com/code/downloads/

That "soon" was 3s

LOL thats really quick :)

R3 final will be out soon.