Documention for: str-enc-utils.r
Created by: peterwood
 on: 23-May-2009
Last updated by: peterwood on: 4-Jul-2009
Format: text/editable
Downloaded on: 30-Apr-2025

[h1 String Encoding Utilities
[contents
[numbering-on

[h2 Introduction

[p This is a small set of utilities to help deal with different 8-bit string encoding schemes. It was initially developed to meet the requirements of the REBOL.org system so does not cover all possible character encodings.

[h2 The str-enc-utils Object

[p The script contains a single object, surprisingly named str-enc-utils. It provides a number of functions related to text encoding, including conversions to and from utf-8.

[p Converting utf-8 to another 8-bit encoding system is inevitably a ""lossy"" conversion as the other encoding systems cannot represent all possible utf-8 characters. If a utf-8 character does not have an equivalent in the target encoding scheme, it is substitued with a replacement character. The default replacement character is the question mark. An alternative replacement character can easily be used:

[asis

  str-enc-utils/replacement-char: #"!"
asis]

[h2 The str-enc-utils Functions

[h3 bom?

[p This function checks to see if a string starts with a Unicode Byte Order Mark.

[p Input: any String
[p Output: One of ""utf-32be"", ""utf-32le"", ""utf-16be"", ""utf-16le"", ""utf-8"", or #[none].

[h3 encoding?

[p This function guesses the encoding of a string. IF the string starts with a Unicode BOM, it will return the encoding method infered by the BOM. After that it is very limited as it only considers the main Western encoding systems. Its method is explained in the appendix - Some Thoughts About Guessing How A String Is Encoded.

[p Input: any string
[p Output: One of ""us-ascii"", ""utf-8"", ""iso-8859-1"", ""macintosh"", ""windows-1252"", ""utf-32be"", ""utf-32le"", ""utf-16be"" or""utf-16le"".

[h3 iso-8859-1-to-html
[p This function converts an ISO-8859-1 encoded string to pure ASCII with characters 128 and above converted to html escape sequences. It has one refinement to also escape <, > and &. A second refinement that leaves HTML tags untouched.
[p Input: an iso-8859-1 encoded string
[p Output: an hmtl ""escaped"" string
[p Refinement: /esc-lt-gt-amp - escapes <, > and &
[p Refinement: /keep-tags - leaves HTML tags alone

[h3 iso-8859-to-utf-8
[p A base function that is used in converting iso-8859 series encoded strings. By default, it converts iso-8859-1 encoded strings to utf-8.
[p Input: an iso-8859 series encoded string
[p Output: a utf-8 encoded string

[h3 iso-8859-1-to-utf-8
[p Input: an iso-8859-1 encoded string
[p Output: a utf-8 encoded string

[h3 iso-8859-2-to-utf-8
[p Input: an iso-8859-2 encoded string
[p Output: a utf-8 encoded string

[h3 iso-8859-9-to-utf-8
[p Input: an iso-8859-9 encoded string
[p Output: a utf-8 encoded string

[h3 iso-8859-15-to-utf-8
[p Input: an iso-8859-15 encoded string
[p Output: a utf-8 encoded string

[h3 macroman-to-utf-8
[p Input: a MacRoman encoded string
[p Output: a utf-8 encoded String

[h3 mail-encoding?
[p This function searches a mail string for the first ""Content-type"" header and extracts the ""charset"" if present.
[p Input: a string containing the ""raw source"" of a mail message
[p Output: a sting containing the first ""charset"" found in the mail or #[none]

[h3 strip-bom
[p Strips any Byte Order Mark from the start of a string.
[p Input: any string - note the string is modified in place
[p Output: the input string with any BOM removed

[h3 utf-8-to-iso-8859
[p A base function that is used in converting utf-8 to iso-8859 series encoded strings. By default, it converts utf-8 encoded strings to iso-8859-1.
[p Input: a utf-8 series encoded string
[p Output: an iso-8859 series encoded string

[h3 utf-8-to-iso-8859-1
[p Input: a utf-8 series encoded string
[p Output: an iso-8859-1 series encoded string

[h3 utf-8-to-iso-8859-15
[p Input: a utf-8 series encoded string
[p Output: an iso-8859-1 series encoded string

[h3 utf-8-to-macroman
[p Input: a utf-8 series encoded string
[p Output: a MacRoman series encoded string

[h3 utf-8-to-win-1252
[p Input: a utf-8 series encoded string
[p Output: a Windows codepage 1252 series encoded string

[h3 win-1252-to-utf-8
[p Input: a Windows codepage 1252 series encoded string
[p Output: a utf-8 series encoded string 

[h2 Appendix - Some Thoughts About Guessing How A String Is Encoded
  
[h3 Caveats and Assumptions
    
    [list/style/list-style-type:decimal
      [li
        [p
        	The function only tries to distinguish between the following encodings. It is blissfully unaware of other character encodings.
        [list
          [li ASCII
          [li UTF-8
          [li ISO-8859-1
          [li Windows Codepage 1252
          [li MacRoman
        list]
      [li 
        [p 
        	The following line endings give a hint as to the operating system on which the string was created:
        [list
          [li Line Feed - 'nix and Mac OSX
          [li Carriage Return - Mac OS 1 to Mac OS 9 
          [li Carriage Return followed by Line Feed - Windows 
        list]
       
      [li 
        [p 
         	The default character encodings on the different operating systems are:
         
        [list
          [li 'nix - UTF-8 
          [li Mac OS X - UTF-8 
          [li Mac OS 1 - 9 - MacRoman 
          [li Windows - Codepage 1252 
        list]
       
      [li 
        [p 
        	The differences between ISO-8859-1, Windows Codepage 1252 and MacRoman can be seen in the following table:
         
        [table
          [row 
            [cell Decimal 
            [cell Hexadecimal 
            [cell ISO-8859-1 
            [cell Windows 1252 
            [cell MacRoman 
           
          [row 
            [cell 127 
            [cell 7F 
            [cell Notused 
            [cell DEL 
            [cell DEL 
           
          [row 
            [cell 128 
            [cell 80 
            [cell Notused 
            [cell € 
            [cell Ä 
           
          [row 
            [cell 129 
            [cell 81 
            [cell Notused 
            [cell Notused 
            [cell Å 
           
          [row 
            [cell 130 
            [cell 82 
            [cell Notused 
            [cell ‚ 
            [cell Ç 
           
          [row 
            [cell 131 
            [cell 83 
            [cell Notused 
            [cell ƒ 
            [cell É 
           
          [row 
            [cell 132 
            [cell 84 
            [cell Notused 
            [cell „ 
            [cell Ñ 
           
          [row 
            [cell 133 
            [cell 85 
            [cell Notused 
            [cell … 
            [cell Ö 
           
          [row 
            [cell 134 
            [cell 86 
            [cell Notused 
            [cell †; 
            [cell Ü 
           
          [row 
            [cell 135 
            [cell 87 
            [cell Notused 
            [cell ‡ 
            [cell á 
           
          [row 
            [cell 136 
            [cell 88 
            [cell Notused 
            [cell ˆ 
            [cell à 
           
          [row 
            [cell 137 
            [cell 89 
            [cell Notused 
            [cell ‰ 
            [cell â 
           
          [row 
            [cell 138 
            [cell 8A 
            [cell Notused 
            [cell Š 
            [cell ä 
           
          [row 
            [cell 139 
            [cell 8B 
            [cell Notused 
            [cell ‹ 
            [cell ã 
           
          [row 
            [cell 140 
            [cell 8C 
            [cell Notused 
            [cell Π
            [cell å 
           
          [row 
            [cell 141 
            [cell 8D 
            [cell Notused 
            [cell Notused 
            [cell ç 
           
          [row 
            [cell 142 
            [cell 8E 
            [cell Notused 
            [cell Ž 
            [cell é 
           
          [row 
            [cell 143 
            [cell 8F 
            [cell Notused 
            [cell Notused 
            [cell è 
           
          [row 
            [cell 144 
            [cell 90 
            [cell Notused 
            [cell Notused 
            [cell ê 
           
          [row 
            [cell 145 
            [cell 91 
            [cell Notused 
            [cell ‘ 
            [cell ë 
           
          [row 
            [cell 146 
            [cell 92 
            [cell Notused 
            [cell ’ 
            [cell í 
           
          [row 
            [cell 147 
            [cell 93 
            [cell Notused 
            [cell “ 
            [cell ì 
           
          [row 
            [cell 148 
            [cell 94 
            [cell Notused 
            [cell ” 
            [cell î 
           
          [row 
            [cell 149 
            [cell 95 
            [cell Notused 
            [cell • 
            [cell ï 
           
          [row 
            [cell 150 
            [cell 96 
            [cell Notused 
            [cell – 
            [cell ñ 
           
          [row 
            [cell 151 
            [cell 97 
            [cell Notused 
            [cell — 
            [cell ó 
           
          [row 
            [cell 152 
            [cell 98 
            [cell Notused
            [cell ˜ 
            [cell ò 
           
          [row 
            [cell 153 
            [cell 99 
            [cell Notused
            [cell ™ 
            [cell ô 
           
          [row 
            [cell 154 
            [cell 9A 
            [cell Notused 
            [cell š 
            [cell ö 
           
          [row 
            [cell 155 
            [cell 9B 
            [cell Notused 
            [cell › 
            [cell õ 
           
          [row 
            [cell 156 
            [cell 9C 
            [cell Notused 
            [cell œ 
            [cell ú 
           
          [row 
            [cell 157 
            [cell 9D 
            [cell Notused 
            [cell Notused 
            [cell ù 
           
          [row 
            [cell 158 
            [cell 9E 
            [cell Notused 
            [cell ž 
            [cell û 
           
          [row 
            [cell 159 
            [cell 9F 
            [cell Notused 
            [cell Ÿ 
            [cell ü 
           
          [row 
            [cell 160 
            [cell A0 
            [cell &nbsp 
            [cell   
            [cell † 
           
          [row 
            [cell 161 
            [cell A1 
            [cell &iexcl 
            [cell ¡ 
            [cell ° 
           
          [row 
            [cell 162 
            [cell A2 
            [cell &iexcl 
            [cell ¡ 
            [cell ¢ 
           
          [row 
            [cell 163 
            [cell A3 
            [cell &pound 
            [cell £ 
            [cell £ 
           
          [row 
            [cell 164 
            [cell A4 
            [cell &curren 
            [cell ¤ 
            [cell § 
           
          [row 
            [cell 165 
            [cell A5 
            [cell &yen 
            [cell ¥ 
            [cell • 
           
          [row 
            [cell 166 
            [cell A6 
            [cell &brvbar 
            [cell ¦ 
            [cell ¶ 
           
          [row 
            [cell 167 
            [cell A7 
            [cell &sect 
            [cell § 
            [cell ß 
           
          [row 
            [cell 168 
            [cell A8 
            [cell &uml 
            [cell ¨ 
            [cell ® 
           
          [row 
            [cell 169 
            [cell A9 
            [cell &copy 
            [cell © 
            [cell © 
           
          [row 
            [cell 170 
            [cell AA 
            [cell &ordf 
            [cell ª 
            [cell ™ 
           
          [row 
            [cell 171 
            [cell AB 
            [cell &laquo 
            [cell « 
            [cell ´ 
           
          [row 
            [cell 172 
            [cell AC 
            [cell &not 
            [cell ¬ 
            [cell ¨ 
           
          [row 
            [cell 173 
            [cell AD 
            [cell &shy 
            [cell ­ 
            [cell ≠ 
           
          [row 
            [cell 174 
            [cell AE 
            [cell &reg 
            [cell ® 
            [cell Æ 
           
          [row 
            [cell 175 
            [cell AF 
            [cell &macr 
            [cell ¯ 
            [cell Ø 
           
          [row 
            [cell 176 
            [cell B0 
            [cell &deg 
            [cell ° 
            [cell ∞ 
           
          [row 
            [cell 177 
            [cell B1 
            [cell &plusmn 
            [cell ± 
            [cell ± 
           
          [row 
            [cell 178 
            [cell B2 
            [cell &sup2 
            [cell ² 
            [cell ≤ 
           
          [row 
            [cell 179 
            [cell B3 
            [cell &sup3 
            [cell ³ 
            [cell ≥ 
           
          [row 
            [cell 180 
            [cell B4 
            [cell &acute 
            [cell ´ 
            [cell ¥ 
           
          [row 
            [cell 181 
            [cell B5 
            [cell &micro 
            [cell µ 
            [cell µ 
           
          [row 
            [cell 182 
            [cell B6 
            [cell &para 
            [cell ¶ 
            [cell ∂ 
           
          [row 
            [cell 183 
            [cell B7 
            [cell &middot 
            [cell · 
            [cell ∑ 
           
          [row 
            [cell 184 
            [cell B8 
            [cell &cedil 
            [cell ¸ 
            [cell ∏ 
           
          [row 
            [cell 185 
            [cell B9 
            [cell &sup1 
            [cell ¹ 
            [cell π 
           
          [row 
            [cell 186 
            [cell BA 
            [cell &ordm 
            [cell º 
            [cell ∫ 
           
          [row 
            [cell 187 
            [cell BB 
            [cell &raquo 
            [cell » 
            [cell ª 
           
          [row 
            [cell 188 
            [cell BC 
            [cell &frac14 
            [cell ¼ 
            [cell º 
           
          [row 
            [cell 189 
            [cell BD 
            [cell &frac12 
            [cell ½ 
            [cell Ω 
           
          [row 
            [cell 190 
            [cell BE 
            [cell &frac34 
            [cell ¾ 
            [cell æ 
           
          [row 
            [cell 191 
            [cell BF 
            [cell &iquest 
            [cell ¿ 
            [cell ø 
           
          [row 
            [cell 192 
            [cell C0 
            [cell &Agrave 
            [cell À 
            [cell ¿ 
           
          [row 
            [cell 193 
            [cell C1 
            [cell &Aacute 
            [cell Á 
            [cell ¡ 
           
          [row 
            [cell 194 
            [cell C2 
            [cell &Acirc 
            [cell  
            [cell ¬ 
           
          [row 
            [cell 195 
            [cell C3 
            [cell &Atilde 
            [cell à 
            [cell √ 
           
          [row 
            [cell 196 
            [cell C4 
            [cell &Auml 
            [cell Ä 
            [cell ƒ 
           
          [row 
            [cell 197 
            [cell C5 
            [cell &Aring 
            [cell Å 
            [cell ≈ 
           
          [row 
            [cell 198 
            [cell C6 
            [cell &AElig 
            [cell Æ 
            [cell ∆ 
           
          [row 
            [cell 199 
            [cell C7 
            [cell &Ccedil 
            [cell Ç 
            [cell « 
           
          [row 
            [cell 200 
            [cell C8 
            [cell &Egrave 
            [cell È 
            [cell » 
           
          [row 
            [cell 201 
            [cell C9 
            [cell &Eacute 
            [cell É 
            [cell … 
           
          [row 
            [cell 202 
            [cell CA 
            [cell &Ecirc 
            [cell Ê 
            [cell   
           
          [row 
            [cell 203 
            [cell CB 
            [cell &Euml 
            [cell Ë 
            [cell À 
           
          [row 
            [cell 204 
            [cell CC 
            [cell &Igrave 
            [cell Ì 
            [cell à 
           
          [row 
            [cell 205 
            [cell CD 
            [cell &Iacute 
            [cell Í 
            [cell Õ 
           
          [row 
            [cell 206 
            [cell CE 
            [cell &Icirc 
            [cell Î 
            [cell Π
           
          [row 
            [cell 207 
            [cell CF 
            [cell &Iuml 
            [cell Ï 
            [cell œ 
           
          [row 
            [cell 208 
            [cell D0 
            [cell &ETH 
            [cell Ð 
            [cell – 
           
          [row 
            [cell 209 
            [cell D1 
            [cell &Ntilde 
            [cell Ñ 
            [cell — 
           
          [row 
            [cell 210 
            [cell D2 
            [cell &Ograve 
            [cell Ò 
            [cell “ 
           
          [row 
            [cell 211 
            [cell D3 
            [cell &Oacute 
            [cell Ó 
            [cell ” 
           
          [row 
            [cell 212 
            [cell D4 
            [cell &Ocirc 
            [cell Ô 
            [cell ‘ 
           
          [row 
            [cell 213 
            [cell D5 
            [cell &Otilde 
            [cell Õ 
            [cell ’ 
           
          [row 
            [cell 214 
            [cell D6 
            [cell &Ouml 
            [cell Ö 
            [cell ÷ 
           
          [row 
            [cell 215 
            [cell D7 
            [cell &times 
            [cell × 
            [cell ◊ 
           
          [row 
            [cell 216 
            [cell D8 
            [cell &Oslash 
            [cell Ø 
            [cell ÿ 
           
          [row 
            [cell 217 
            [cell D9 
            [cell &Ugrave 
            [cell Ù 
            [cell Ÿ 
           
          [row 
            [cell 218 
            [cell DA 
            [cell &Uacute 
            [cell Ú 
            [cell ⁄ 
           
          [row 
            [cell 219 
            [cell DB 
            [cell &Ucirc 
            [cell Û 
            [cell € 
           
          [row 
            [cell 220 
            [cell DC 
            [cell &Uuml 
            [cell Ü 
            [cell ‹ 
           
          [row 
            [cell 221 
            [cell DD 
            [cell &Yacute 
            [cell Ý 
            [cell › 
           
          [row 
            [cell 222 
            [cell DE 
            [cell &THORN 
            [cell Þ 
            [cell fi 
           
          [row 
            [cell 223 
            [cell DF 
            [cell &szlig 
            [cell ß 
            [cell fl 
           
          [row 
            [cell 224 
            [cell E0 
            [cell &agrave 
            [cell à 
            [cell ‡ 
           
          [row 
            [cell 225 
            [cell E1 
            [cell &aacute 
            [cell á 
            [cell · 
           
          [row 
            [cell 226 
            [cell E2 
            [cell &acirc 
            [cell â 
            [cell ‚ 
           
          [row 
            [cell 227 
            [cell E3 
            [cell &atilde 
            [cell ã 
            [cell „ 
           
          [row 
            [cell 228 
            [cell E4 
            [cell &auml 
            [cell ä 
            [cell ‰ 
           
          [row 
            [cell 229 
            [cell E5 
            [cell &aring 
            [cell å 
            [cell  
           
          [row 
            [cell 230 
            [cell E6 
            [cell &aelig 
            [cell æ 
            [cell Ê 
           
          [row 
            [cell 231 
            [cell E7 
            [cell &aelig 
            [cell æ 
            [cell Á 
           
          [row 
            [cell 232 
            [cell E8 
            [cell &egrave 
            [cell è 
            [cell Ë 
           
          [row 
            [cell 233 
            [cell E9 
            [cell &eacute 
            [cell é 
            [cell È 
           
          [row 
            [cell 234 
            [cell EA 
            [cell &eacute 
            [cell é 
            [cell Í 
           
          [row 
            [cell 235 
            [cell EB 
            [cell &euml 
            [cell ë 
            [cell Î 
           
          [row 
            [cell 236 
            [cell EC 
            [cell &igrave 
            [cell ì 
            [cell Ï 
           
          [row 
            [cell 237 
            [cell ED 
            [cell &iacute 
            [cell í 
            [cell Ì 
           
          [row 
            [cell 238 
            [cell EE 
            [cell &icirc 
            [cell î 
            [cell Ó 
           
          [row 
            [cell 239 
            [cell EF 
            [cell &iuml 
            [cell ï 
            [cell Ô 
           
          [row 
            [cell 240 
            [cell F0 
            [cell &eth 
            [cell ð 
            [cell  
           
          [row 
            [cell 241 
            [cell F1 
            [cell &ntilde 
            [cell ñ 
            [cell Ò 
           
          [row 
            [cell 242 
            [cell F2 
            [cell &ograve 
            [cell ò 
            [cell Ú 
           
          [row 
            [cell 243 
            [cell F3 
            [cell &oacute 
            [cell ó 
            [cell Û 
           
          [row 
            [cell 244 
            [cell F4 
            [cell &oacute 
            [cell ó 
            [cell Ù 
           
          [row 
            [cell 245 
            [cell F5 
            [cell &otilde 
            [cell õ 
            [cell ı 
           
          [row 
            [cell 246 
            [cell F6 
            [cell &ouml 
            [cell ö 
            [cell ˆ 
           
          [row 
            [cell 247 
            [cell F7 
            [cell &divide 
            [cell ÷ 
            [cell ˜ 
           
          [row 
            [cell 248 
            [cell F8 
            [cell &oslash 
            [cell ø 
            [cell ¯ 
           
          [row 
            [cell 249 
            [cell F9 
            [cell &ugrave 
            [cell ù 
            [cell ˘ 
           
          [row 
            [cell 250 
            [cell FA 
            [cell &uacute 
            [cell ú 
            [cell ˙ 
           
          [row 
            [cell 251 
            [cell FB 
            [cell &ucirc 
            [cell û 
            [cell ˚ 
           
          [row 
            [cell 252 
            [cell FC 
            [cell &uuml 
            [cell ü 
            [cell ¸ 
           
          [row 
            [cell 253 
            [cell FD 
            [cell &yacute 
            [cell ý 
            [cell ˝ 
           
          [row 
            [cell 254 
            [cell FE 
            [cell &thorn 
            [cell þ 
            [cell ˛ 
           
          [row 
            [cell 255 
            [cell FF 
            [cell &yuml 
            [cell ÿ 
            [cell ˇ
        table]
     list]
        
[h3 Rules of Thumb
    
    [p Applied in the following order: 
    
    [list/style/list-style-type:decimal
      [li 
        [p 
          If the string starts with a BOM, the encoding infered by the BOM will be returned.
      
     
      [li 
        [p 
          If the string contains only characters in the range 0x00 - 0x7F, it is an ASCII string.
         
       
      [li 
        [p 
          If the string contains more UTF-8 multi-byte characters than it does invalid utf-8 characters and invalid multi-byte sequences, it is a UTF-8 string.
         
       
      [li 
        [p 
          If the string contains characters in the range 0xA0 - 0xFF but none in the range 0x80 - 0x9F, it is an ISO-8859-1 string.
         
       
      [li 
        [p 
          If the string contains any of 0x81, 0x8D, 0x8F, 0x90 or 0x9D, it is a MacRoman string. .
         
       
      [li 
        [p 
          If the string contains carriage returns but no line feeds, it is a MacRoman string.
         
       
      [li 
        [p 
          It is a Windows 1252 Codepage string.
    list]