World: r4wp

Red/System c-strings are UTF-8 compatible.

I mean my ongoing request for getting UTF-8 in routines

As I said above, that part is not implemented, it will be once the 
work on Red's I/O will begin.

Yes, you said you might speed it up if I have a big need for it, 
so I'm expressing my need

But someone could contribute string! <=> UTF-8 conversion routines 
in the meantime.

I'm afraid my short-term schedule is already full, I've kept postponing 
very important parts for months, I need to get them implemented now 
(shared libs, error!, typeset!, object!).

I think that those UTF-8 conversion routines would take at least 
two days of work to get implemented and debugged.    I'll see once 
I get shared libs done if I can afford them before working on the 
other urgent tasks.

It's just that I thought Unicode was already there. Most of my future 
work is blocked by it

Unicode support is there, but we don't have all the external encoders/decoders 
yet.

It's nice that you can get data in, but it's useless if you can't 
get it out

We have hardwired the stdout/stdin Unicode support because we don't 
yet have the I/O infrastructure for Red (ports and devices).

True, but I do have the infrastructure. Just can't really use it 
this way

Well, as I said, someone could contribute those UTF-8 conversion 
routines.

I think Peter already did some work on it as have Rudolf (meijeru).

I wanted to publish an article with examples on the birth of the 
interpreter and the porting of the bindings to Red, but if a simple 
READ WRITE example doesn't work, I think it's too early to talk about 
it

I will have a look at it once shared libs are done, that's the best 
I can do for now.

OK, but I can't imagine what I would do with shared libraries if 
I can't get string data out

Shared libraries are on the critical path to Android support for 
Red and I've delayed it way too much already.

Same thing: what would you do on Android if you can get data in, 
but not out?

It's worse than having no Unicode, then you can at least get out 
what you put in

For Android, java uses UTF-16, so the conversion from string! is 
(almost) trivial.

But it's not there yet, is it?

No, I will implement it when I'll need it, and I have a lot of other 
stuff to code for Android support before that.

Is it wise that Red won't work on other platforms before it works 
on Android?

The features currently implementd in Red are working.

Sure, but it's pretty useless like this

That's why it is called an alpha. ;-)

I was hoping a little more could be done, but I'll have to postpone 
a lot of work

I told you I will have a look at it once the shared libs will be 
done, just wait a few days more. If it's critical to you, you might 
want to contribute the required conversion routines?

I could, but I know very little of Unicode, so there would be a lot 
of overhead in getting up to speed

I have no idea how long it will take you to finish the shared libraries. 
It has been a backburner project for a long time

Not very long, I just kept it postponed since almost a year now, 
and it's getting on my way for Android support since a while, so 
I've scheduled it since a few weeks to get it done just after the 
interpreter is finished (Exit/Return support).

OK, that's fine. It sounded like data out support was of undetermined 
priority

The only "data out" support we need for now for building Red is the 
stdout support, and we have it since a while.

Red I/O full support is next on my list after the above mentioned 
tasks will be completed.

BTW, if you stick to Latin-1, you shouldn't have the need for any 
conversion?

Also, there might be a cheap way to achieve the conversion in the 
meantime using wsprintf() or similar function.

Hmm, it might not be enough, so you might want to have a look and 
maybe wrap libiconv:
http://www.gnu.org/software/libiconv/

For once, the API looks good and simple enough (4 functions to wrap).

From a routine, if str is a red-string! pointer, this is the dispatch 
code you would need to use:

s: GET_BUFFER(str)
switch GET_UNIT(str) [
	Latin-1 [...conversion code...]
	UCS-2 [...conversion code...]
	UCS-4 [...conversion code...]
]

Beginning of internal string buffer is given by:

	string/rs-head str	(returns a byte-ptr!)

Should be GET_UNIT(s) above, sorry for the typo/

Another typo: should be Latin1.

Anyway, you don't need any conversion for Latin1, so you just have 
to do it for the other two formats.

Sticking to Latin1 is not much use these days. Many data such as 
web sites is in Unicode. It would be fine if it worked like R2, as 
a transparent passthrough, but Red eats your Unicode and won't give 
it back from its internal format

How does stdout support deal with that? Is there no conversion to 
the platform format there?

I'd be happy to look at a UCS-2 to UTF-8 conversion function but 
I don't have the time to do it at the moment.

I'm pretty sure that would be enough for Kaj's immediate needs.

Yes

I see there are specialised platform specific print functions only 
for printing the internal format. They look like a base for the general 
purpose conversions, though