## [REBOL] Re: UTF-8 revisited

### From: jan:skibinski:sympatico:ca at: 26-Nov-2002 14:51

Hello RebOldes, Thank you for you response and suggestions. I already have had some optimization done, and discovered some of the things you just mentioned: some by trial and error, some by a common sense. So the "for" loops have been replaced, subroutines removed, etc. But I would not even think about other things you mention here, such as using to integer! instead of to-integer, etc. So thank you so much for those. --------------------------------------------------- Anyway, here is a status: I think my latest unpublished version is quite acceptable in speed. But I am too tired now to do the final cleanup, and comparison with what you just posted today. This must wait till I get few hours of sleep. But I'll post here the encode function for your revision. I was using your little sentence of Latin-1 characters chars: =EC=9A=E8=F8=9E=FD=E1=ED=E9 looping 1000 times or so. And while your version was doing it in about 0.20 s, my original non-optimized version was dragging its feet at about 13 s, or so. That was clearly unacceptable. To cut the long story short, my latest version is working it in about 0.60 s in a general case and in about 0.35 using specialization to Latin-1 but still keeping the same framework as for other more general cases. Further specialization would obviously degenerate to utf8-encode. I am attaching a skeleton of one function only, so you will see what I am bragging about here. There are few subtle points to be made : + If you can do all your arithmetic on chars, then you can be as fast as in utf8-encode. First of all, the time consuming to-integer is not required, because in this case the operation / behaves as the integer division. That means that a / b gives you a character, which can be directly stored in an output string - no convertion is required. However, you also have to remember that the multiplication works modulo 256 too. This is why am fiddling with (unfinished for the case k=4) function 'f at the beginning of 'encode. Adding 0 and assuming correct order of multiplication is important if one expects values to be greater than 256. + To assure that after the division of two integers I still get the integer, I use little trick of 'and-ing with negative numbers as in "x and -64 / 64". This seem to be faster than to-integer. I have not tried yor suggested to integer! yet. The cascade below goes down from the fastest/cheapest case to some more elaborate cases: nothing much to do for ascii characters, a bit more for ansi (latin 1), and quite a bit of work with to-char convertion for the most generic case. Notice that, compared to the original version, I simplified the entire scheme as well. Best wishes, Jan encode: func [ { Encode string of k-wide characters into UTF-8 string, where k: 1, 2 or 4. Case k = 1 could have been isolated for much improved speed. (integer -> string -> string) } k [integer!] ucs [string!] /local x m result [string!] ][ result: make ucs 0 f: switch :k [ 1 [func[u][u/1]] 2 [func[ u ][ either u/1 > 0 [0 + u/2 + (256 * u/1)][u/2] ]] 4 [func[u][u/4 + (256 * u/3) + (65536 * u/2) + (16777216 * u/1)]] ] while [not tail? ucs][ x: f ucs result: tail result either x < 128 [ insert result x ][ either x < 256 [ insert result x and 63 or 128 insert result x / 64 or 192 ][ m: 1 while [x > 127 ][ insert result to-char (x and 63 or 128) x: x and -64 / 64 m: m + 1 ] insert result to-char (x or udata/3/:m) ] ] ucs: skip ucs k ] head result ] Oh, you will need this too: udata/3 == [0 192 224 240 248 252] I hope I did not miss anything here.