AltME groups: search

Red: I'm still working on both the compiler and the minimal runtime 
required to run simple Red programs. I have only the very basic datatypes 
working for now, no objects (so no ports) yet. I not yet at the point 
where I can give an accurate ETA for the first alpha, but I hope 
to be able to provide that ETA in a week.


Red string! datatype will support Unicode (UTF-8 and UTF-16 encoding 
internally). I haven't implemented Unicode yet, so if some of you 
are willing to provide efficient code for supporting Unicode, that 
would greatly speedup Red progress. 

The following functions would be needed (coded in Red/System):

- UTF-8 <=> UTF-16 LE conversion routines

- (by order of importance) length?, compare (two strings), compare-case, 
pick, poke, at, find, find-case
- optinally: uppercase, lowercase, sort


All the above functions should be coded both for UTF-8 and UTF-16 
LE.

In case, you wonder why Red needs both UTF formats, well, it's simple, 
 Windows and UNIX worlds use different encodings, so we need to support 
both. Red will use by default UTF-8 for string values, but on Windows 
platform, it will convert the string to UTF-16 on first call to an 
OS API, and will keep that encoding later on (and avoid the overhead 
of converting it each time). 


We might want to make the UTF-16 related code platform-depend and 
not include it for other platforms, but I think that some text processing 
algorithms might benefit from a fixed-size encoding, so for now, 
I'm for including both encoding for all targets.


It will be also possible for users to check and change the encoding 
of a Red string! value at runtime.

Keep in mind that even UTF-16 is not a fixed-size encoding. Each 
codepoint either takes 2 or 4 bytes.

UTF-32 (aka UCS4) is a fixed-size encoding. It's rarely used though.

So far, my short-list of encodings to support are UTF-8 and UTF-16LE. 
UTF-32 might be needed at some point in the future, but for now, 
I'm not aware of any system that uses it?


The Unicode standard by itself is not the problem (having just one 
encoding would have helped, though). The issue lies in different 
OSes supporting different encodings, so it makes the choice for an 
internal x-platform encoding hard. It's a matter of Red internal 
trade-offs, so I need to study the possible internal resources usage 
for each one and decide which one is the more appropriate. So far, 
I was inclined to support both UTF-8 and UTF-16LE fully, but I'm 
not sure yet that's the best choice. To avoid surprizing users with 
inconsistent string operation performances, I thought to give users 
explicit control over string format, if they need such control (by 
default, Red would handle all automatically internally). For example, 
on Windows::

    s: "hello"		;-- UTF-8 literal string

    print s		;-- string converted to UCS2 for printing through win32 
    API
    write %file s	;-- string converted back to UTF-8

    set-modes s 'encoding 'UTF-16 ;-- user deciding on format
or
    s/encoding: 'UTF-16

    print length? s	;-- Length? then runs in O(1), no surprize.



Supporting ANSI as internal encoding seems useless, being able to 
just export/import it should suffice.

BTW, Brian, IIRC, OS X relies on UTF-8 internally not UTF-16.

set-modes s 'encoding 'UTF-16
should rather be:
    set-modes s [encoding: UTF-16]

Be sure to not forget the difference between UTF-16 (variable-length 
encoding of all of Unicode) and UCS2 (fixed-length encoding of a 
subset of Unicode). Windows, Java and .NET support UTF-16 (barring 
the occasional buggy code that assumes fixed-length encoding). R3's 
current underlying implementation is UCS2, with its character set 
limitations, but its logical model is codepoint-series.

If you support different internal string encodings on a given platform, 
be sure to not give logical access to the underlying binary data 
to Red code. The get/set-modes model is good for that kind of thing. 
If the end developer knows that the string will be grabbed from something 
that provides UTF-8 and passed along to something that takes UTF-8, 
they might be better off choosing UTF-8 as an underlying encoding. 
However, that should just be a mode - their interaction with the 
string should follow the codepoint model. If the end developer will 
be working directly with encoded data, they should be working with 
binary! values.

Btw, in this code above:
    s/encoding: 'UTF-16
    print length? s	;-- Length? then runs in O(1), no surprize.


Length is not O(1) for UTF-16, it's O(n). Length is only O(1) for 
the fixed-length encodings.

Doc, pardon me because I don't know what the intended datatype model 
is for Red. Something like the REBOL datatype/action model could 
be used to implement the different underlying string encodings that 
you want in-memory support for. Each supported encoding would have 
its own set of action handlers, which would all have the same external 
interface. Swapping the encoding would be as simple as swapping the 
handler set. Resolving the difference at compile time could be similar 
to generic type instantiation, or C++ template generation.

It reads it as a stream of bytes. As UTF-8 doesn't use null bytes 
in its encoding (except for codepoint 0), it can be fully loaded 
as string! or binary! in R2 (but you'll see garbage for non-ASCII 
characters).

The above string doesn't work as-is in Red though, you should pass 
the codepoints escaped instead of the UTF-8 encoding.

Pekr: try to set the "encoding" field to UTF-8 in the saving panel 
(Save as...).

You should have zipped the hello.red script to preserve its original 
encoding.

You need to change the encoding selector when saving with Notepad 
to UTF-8.

Note that if you specify the length, it applies to the length of 
the script after the header and an optional newline after it (cr, 
crlf or lf). Same goes for the checksum. Both apply to binary data, 
meaning the source in UTF-8 encoding and with newlines in the style 
that they are specified in the file.

The most space efficient encoding I can come up with would be something 
like ~a-1 for 'a and ~A-2 for 'A. That would be cheap to evaluate 
for strict-equal? but expensive for equal?

A faster encoding would be to reserve a part of the integer identifier 
for the alias number, for example one byte. That would reduce the 
number of different symbols to 2^24 and the maximum number of aliases 
for one symbol to 256. That would only allow a word up to 8 characters 
to have all its aliases, but it would be cheap to evaluate for both 
strict-equal? and equal?

A possible cause of the hello app not working on Android anymore 
is a mismatch between the output encoding of Red and the expected 
one from the Java wrapper. I will check that when I'll start working 
fully on the Android port.

We haven't yet implemented UTF-8 encoding functions in the standard 
library. It will be done during the I/O implementation (unless you 
have a strong need for it, then I'll have a look at it).

UTF-8 encoding would be very welcome. My I/O frameworks are of little 
use without it

I'm only pushing in so far as I have been inquiring about encoding 
support for several weeks, and the state of things has been downplayed, 
so I want it to be noted

For example, there's an internal single byte encoding that's marked 
"Latin1", but I now know there is no way to get Latin-1 data in or 
out, so I wonder if this encoding will ever be used for more than 
7-bit ASCII

As far as I understood the plan, the internal "Latin 1" encoding 
is used to store Unicode codepoints 0-255. So this should be used 
already.

1) "I found out that not only does Red not support Unicode, it doesn't 
support Latin-1, not even on Windows" Red *does* "support" Unicode, 
Latin-1 "support" was not claimed in Red, except for the console 
script. I've put quotes around support word, because you're mixing 
up internal representation and I/O encoding formats.

5) "Yes, I think it's very dangerous to claim that Red has Unicode 
and Latin-1 support". Red *has* Unicode support, string! and word! 
value support Unicode, input Red scripts are Unicode, PRINT outputs 
Unicode characters. Latin-1 is used as an *internal* encoding format, 
I don't remember ever claiming that "Red supports Latin-1 for I/O" 
except for the console script (which is wrong, I agree). OTOH, I 
do remember thinking about supporting it at the beginning for printing, 
then I found it cumbersome to support in addition to Unicode mode 
and dropped it during the implementation.

Wrt the Unicode plan (my blog entry link above), I would like to 
highlight only one sentence: "Conversion for input and output strings 
will be done on-the-fly between one of the internal representation 
and UTF-8/UTF-16." This is what have been implemented for Red input 
scripts (except from the console), and for outputting text on screen 
with the currently hardwired PRINT output support because the I/O 
sub-system has not been yet implemented in Red. The PRINT backend 
will be rewritten once we get ports/devices support. Also, the "on-the-fly" 
part (no intermediary buffer) should have hinted you that I could 
not implement encoders/decoders before I/O sub-system is done. This 
also means that the current encoding/decoding logic you've implemented 
these last days probably won't be useful for Red's I/O.

Kaj, it seems to me that you were confused by a few things:
- console script banner wrong statement (my fault)

- internal "Latin-1" naming (like in Python's internals) which might 
be misleading (there's no other closer naming in Unicode for one 
byte representation AFAIK, though some people call it "UCS-1", maybe 
we should adopt that too).

- "Unicode support" seems to imply to you that *all* possible Unicode 
encodings have to be supported (with encoders/decoders). It doesn't, 
having just one encoding supporting the full Unicode range (like 
UCS-4) is enough for claiming "Unicode support".

Doc: what do you mean when you say Textpad doesn't support  Unicode 
? I use Textpad 5.4.2 and see options to set UTF-8 as default text 
encoding and others options for BOM and so on.

I implemented UTF-8 output support for Red. I ended up writing optimised 
versions based more on the Red print backend. I integrated them in 
my I/O routines and made heavy performance optimisations. Thanks 
to Peter for leading the way. There are the following Red/System 
encoders embedded in %common.red:

http://red.esperconsultancy.nl/Red-common/dir?ci=tip


to-UTF8: encodes a Red string into UTF-8 Red/System c-string! format.

to-local-file: encodes a Red string into Latin-1 Red/System c-string! 
format on Windows, and into UTF-8 on other systems. This yields a 
string suitable for the local file name APIs. Latin-1 can be output 
as long as it was input into Red via UTF-8. Non-Latin-1 code points 
cannot be encoded in Latin-1 and yield a NULL for the entire result.


These encoders make use of the Latin1-to-UTF8, UCS2-to-UTF8 and UCS4-to-UTF8 
encoding functions. An example of their use in the Red READ and WRITE 
functions is in %input-output.red

I used the new encoding functions in all my Red bindings: those for 
the C library, input/output via files and cURL, 0MQ, SQLite and GTK+. 
In as many places as possible, data marshalled to the external libraries 
now supports UTF-8. File names on Windows support Latin-1. Files 
and URLs are always read and written as UTF-8, including on Windows. 
Red does not support loading Latin-1 strings.

I've updated the binary downloads. The red console interpreters and 
all the Red examples include the above encoding support now, and 
all the latest Red features:

http://red.esperconsultancy.nl/Red-test/dir?ci=tip


For example, the Red/GTK-text-editor now supports writing UTF-8 files 
with UTF-8 or Latin-1 names.


I've added an MSDOS\Red\red-core.exe for Windows 2000, because the 
GTK+ libraries in red.exe require Windows XP+.

I can't test the encoding on Mac, so I would be interested to hear 
if it works there, especially UTF-8 file names

Arnold: I believe that Rebol/View uses Windows Codepages under Windows, 
MacRoman on OS X and ISO-8859-1 on  Linux. Sadly this means it only 
really supports true ASCII characterrs cross platform unless you 
manage encoding your self.

I wrote a run length encoding function, may be useful for someone 
else too:

rle: func ["Run length encode" b /local v r i j] [
	v: 1 r: copy []
	j: next i: b
	unless empty? b [
		until [
			either all [not tail? j equal? first i first j] [
				v: v + 1 j: next j
			] [
				append r reduce [v first i] v: 1 i: ++ j
			]
			tail? i
		]
	]
	r
]

RLE might help for binary data too, including that reduced encoding 
I mentioned for strings above.

Note, however, that the 

%00" percent-encoding (NUL) may require special handling and should 
be rejected if the application is not expecting to receive raw data 
within a component." -RFC 3986

>> write http://www.rebol.com/index.html[ HEAD ]
make object! [
    name: none
    size: none
    date: none
    type: 'file
    response-line: "HTTP/1.1 200 OK"
    response-parsed: none
    headers: make object! [
        Content-Length: "7407"
        Transfer-Encoding: none
        Last-Modified: "Sat, 15 Dec 2012 07:02:21 GMT"
        Date: "Wed, 09 Jan 2013 09:24:53 GMT"
        Server: "Apache"
        Accept-Ranges: "bytes"
        Content-Type: "text/html"
        Via: "1.1 BC5-ACLD"
        Connection: "close"
    ]
]

Moving from the 'random topic back to prot-send.r, I found a somewhat 
serious bug with attachments and base-64 encoding.  I sent the exact 
same attachment using webmail and R3.


at position 32677 in the email sent by webmail (the headers are slighly 
different sizes and the base-64 line breaks are different between 
the two clients) we have the following data:


eMHgCm4jUznXtDnpVKaErkAc107QbjVC6siyHYBu96hxLUjnXDKeWqPOTzVmWxuUY7kJ+lRGF16x

tn6VmO4ncYpZX34HYU0qw7EU3afSgdyUKCvy1s6DpEN3doL93jtyD9089Caq6bZO0g3Llj0Wu8t9

OhsNDu7uUK9wYiq/7OeOK1p03Mwq1eXRbnnUVuZJcKCecD3rctbVbYjdy5/Si1ks7WTLyAerYzUH


at position 32521 in the email sent by R3, we have the following 
data:


vaHNtZYHTmoHtcDpXRSQDk1UeMHgCm4jUznXtDnpVKaErkAc107QbjVC6siyHYBu96hxLUjn
XDKeWq

doL93jtyD9089Caq6bZO0g3Llj0Wu8t9OhsNDu7uUK9wYiq/7OeOK1p03Mwq1eXRbnnUVuZJ


I copied the three lines of data around where the problem occurs. 
 On the short line in the R3 data, the following sequence is missing:


POTzVmWxuUY7kJ+lRGF16xtn6VmO4ncYpZX34HYU0qw7EU3afSgdyUKCvy1s6DpEN3


You can imagine the kind of trouble that causes with binary data. 
;-)

Well. Don't bother. I have found another solution that just remove 
the encoding problem at the start.

I want a way to generate mp3 voice mail .. this does the mp3 encoding.

uses lame as well for the mp3 encoding.

...Shared files and stored with encoding... So let the newsgroups 
now misuse ALTME for anything ;-) Good for business though...

another issue is how text encoding is done. I just had a conversation 
in Danish with Geomol. While Danish letters look right on a mac it 
will look different in Windows and vice versa.

test-address-import: func [

 {Returns true if pass, false if discrepency and none if failed with 
 error.}
	limit [integer!] "Number of TO address to generate."
	/quiet "Does not display error."
	/local to-list eml sep msg obj
] [
	to-list: copy {} sep: ""
	repeat i limit [
		eml: to-email join "test-" [i "@test.com"]
		repend to-list [sep {"'} eml {'" <} eml {>}]
		if empty? sep [sep: {,^/	}]
	]

	msg: replace copy {Date: Thu, 12 Feb 2004 11:41:49 +0100
From: test <[test-:-test-:-com]>
MIME-Version: 1.0
To: TO_LIST
Subject: [REBOL] test message - edited copy of real message
Content-type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 8bit
Sender: [rebol-bounce-:-rebol-:-com]
Reply-to: [testing-:-testing-:-com]
Status:   


Test message.
} {TO_LIST} to-list

	either error? set/any 'result try [
		obj: import-email msg
		limit = length? obj/to
	][if not quiet [print mold disarm result] none][result]

]

Anton, the problem is actually that rebol does not handle percent 
encoding correctly in URLs. (i hope we can fix this in r3)

if you have more char enties than just hex encoding

I've added encoding: 'cp1252 into header... it's up to rebol.org 
now to use such an info and convert such a script into utf8 before 
displaying it in html

The core of the library system is old enough that it was written 
without considering character encoding at all.

At the moment, I'd be worried about standarising the Library on utf-8 
as the effect of multibyte characters would have during script and 
mail processing is not understood. It could well be that the system 
handles multibyte characters without a hitch but nobody knows yet.


I have started to write some scripts to try to help move to a consistent 
character encoding of the Library data but, due to time constraints, 
I have been very slow.

sunanda, you can force the character encoding in the html page header... 
I've used that before and it worked for me.

I had the same kind of issues on another system.  nowadays, the default 
encoding has become UTF-8 for many/most html handlers, so if its 
not specified, many new browers and tools will incorrectly break 
up the character data.

I think the root of the problem is that when the Library system was 
first written, no account was taken of character encoding. As a result, 
not only is the data encoded as it was when originally submitted 
but the method of encoding is not even known.


Whatever charset is specified in the http header is not going to 
be correct for all scripts and messages. Using charset=utf8 seems 
to cause the least problems. Though for example, it will cause many 
ISO-8859-1 "high bit" characters to be incorrectly displayed.

Sunanda, you're right about that ascii-math.r file. When I clicked 
the [Download script] link, the browser (konqueror) downloaded and 
directly opened it with the editor (SciTE). SciTE thought it was 
8-bit ascii, and showed the characters incorrectly. All I had to 
do was change the file encoding from 8-bit to utf-8 and the characters 
appeared correctly. I guess the editor had no way of determining 
the encoding, and incorrectly guessed 8-bit ascii.

The view-script.r html source for the page correctly advertises the 
encoding as utf-8, so the browser shows it correctly.

Thanks Gabriele -- that's a clear explanation, and has helped me 
work out what is going on.


Anton and Gabriele -- I have tried changing the charset we emit on 
the download to say UTF-8. But that makes little difference. As both 
of you note, once the file has been saved then (without a MAC-type 
resource fork) there is no obvious indication of the encoding. And 
several editors I have tried get it wrong -- thus "revealing" the 
extra ASCII chars.


Not sure what the solution is other than to de-UTF-8 files on download.

Which editors?

I think most editors these days allow manually changing the encoding, 
so developers who notice strange characters can just change it themselves.

Maybe it would be helpful to add a rebol.org library script header 
advertising the encoding (when it is known, and when not).

I don't recommend 'de-UTF-8'ing files on download - that's just going 
to confuse things more, especially when the file is view-script.r'd 
as utf-8 just beforehand.

It seems the responsibility lies with the clients to interpret encodings 
properly. As we move to a unicode world, software assuming 8-bit 
encodings are some ASCII encoding should drop off. But until the 
transition is complete, there's not much we can do about client software 
guessing wrong like that, except stating the encoding in the script 
header, in the web page that provides the download link, and by helping 
confused newbies.

Are rebol.org uploaders asked to declare the encoding used?

Ok, so there are some editors which don't support unicode, don't 
guess encoding correctly, or can change encoding only with difficulty.

How about this suggestion; if a rebol.org script is known to be UTF-8, 
then an additional link should appear:

[Download as ASCII] download-a-script?script-name=ascii-math.r&encoded-as=8-bit-ascii
which transcodes a UTF-8 file to ASCII.
Just have to get a conversion function in place for this to work.

Ok, so things seem to be proceeding well. The rebol.org Library's 
support for utf-8 was actually stronger than thought, and what're 
being added are functions to help deal with legacy client apps which 
misidentify the file encoding.

Using Peter's code (thanks again!), I've made two changes to the 
download-a-script link:


1. if we find UTF-8 chars in a script, we download it with the HTTP 
content type charset=utf-8


But that probably makes no practical difference.  A downloaded script 
will be saved by the browser, and then opened by a text editor. The 
text editor is unlikey to be passed the charset setting. So:


2. Scripts with UTF-8 encoding are downloaded with a few lines of 
comment at their top. The comment explains the possible problem.


Thanks to all for the comments and help with getting things this 
far.

I'd suggest recognizing 3D as the beginning of a name could be an 
encoding error

Is 'sha1 encoding available in free REBOL cores ?

This new 'read implementation avoids the SQL encoding issues encountered 
in previous versions. The side effect it that it will break compatibility 
with previous sources using 'read.

hi doc, 
is there any trick or encoding to prevent 
inserting errors due to strings containing 
special characters like ' or /  etc?

With Apache 2.x (normal CGI), we have: make object! [ 
	server-software: "Apache/2.0.54 (Fedora)" 
	server-name: "localhost" 
	gateway-interface: "CGI/1.1" 
	server-protocol: "HTTP/1.1" 
	server-port: "80" 
	request-method: "GET" 
	path-info: "/sample01.rhtml" 
	path-translated: "/var/www/html/sample01.rhtml" 
	script-name: "/cgi-bin/magic.cgi" 
	query-string: "" 
	remote-host: none 
	remote-addr: "127.0.0.1" 
	auth-type: none 
	remote-user: none 
	remote-ident: none 
	Content-Type: none 
	content-length: none 
	other-headers: [
		"HTTP_HOST" "localhost" 

  "HTTP_USER_AGENT" {Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.10) 
  Gecko/20050720 Fedora/1.0.6-1.1.fc4 Firefox/1.0.6} 

  "HTTP_ACCEPT" {text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5} 
		"HTTP_ACCEPT_LANGUAGE" "en-us,en;q=0.5" 
		"HTTP_ACCEPT_ENCODING" "gzip,deflate" 
		"HTTP_ACCEPT_CHARSET" "ISO-8859-1,utf-8;q=0.7,*;q=0.7" 
		"HTTP_KEEP_ALIVE" "300" 
		"HTTP_CONNECTION" "keep-alive" 
		"HTTP_COOKIE" "PHPSESSID=7f84fd7766f23e1462fed550ecbbfda4"
	] 
]

Received: from web26108.mail.ukl.yahoo.com ([217.12.10.232])

        by mail.prosygma-asp.com (Merak 7.5.2) with SMTP id 1TI26716
        for <[me-:-you-:-com]>; Tue, 26 Apr 2005 04:58:31 +0200

Received: (qmail 67900 invoked by uid 60001); 26 Apr 2005 02:58:30 
-0000

Message-ID: <[20050426025830-:-67898-:-qmail-:-web26108-:-mail-:-ukl-:-yahoo-:-com]>

Received: from [81.248.68.164] by web26108.mail.ukl.yahoo.com via 
HTTP; Tue, 26 Apr 2005 04:58:29 CEST
Date: Tue, 26 Apr 2005 04:58:29 +0200 (CEST)
From: Rodrigue <[you-:-me-:-com]>
Subject: Re: Erreur lors de l'enregistrement d'une news
To: Admin Rebol <[me-:-you-:-com]>
In-Reply-To: 6667
MIME-Version: 1.0

Content-Type: multipart/alternative; boundary="0-370637848-1114484309=:67141"
Content-Transfer-Encoding: 8bit
X-Antivirus: avast! (VPS 0517-0, 25/04/2005), Inbound message
X-Antivirus-Status: Clean

--0-370637848-1114484309=:67141
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: 8bit

Salut Fabrice,
 
Peux tu me redire sous quel format je dois envoyer les photos

Rodrigue

Admin Rebol <[me-:-you-:-com]> wrote:
Bonjour Rodrigue,

>Salut Fabrice,

>contrairement à ce que je t'expliquais hier soir, les news ne s'enregistrent 
pas.
>voilà le message d'erreur : 

Peux-tu m'envoyer ce que tu veux mettre en ligne par @ ?

Je vérifierais directement avec tes données car il n'y a pas de problème 
de mon côté pour ajouter les news.

Merci.

-- 
Fabrice

		
---------------------------------

 Découvrez le nouveau Yahoo! Mail : 250 Mo d'espace de stockage pour 
 vos mails !
Créez votre Yahoo! Mail
--0-370637848-1114484309=:67141
Content-Type: text/html; charset=iso-8859-1
Content-Transfer-Encoding: 8bit

<DIV>Salut Fabrice,</DIV>
<DIV>&nbsp;</DIV>

<DIV>Peux tu me redire sous quel format je dois envoyer les photos<BR></DIV>
<DIV>Rodrigue</DIV>

<DIV><BR><B><I>Admin Rebol &lt;[me-:-you-:-com]&gt;</I></B> wrote:</DIV>

<BLOCKQUOTE class=replbq style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; 
BORDER-LEFT: #1010ff 2px solid">Bonjour Rodrigue,<BR><BR>&gt;Salut 
Fabrice,<BR>&gt;contrairement à ce que je t'expliquais hier soir, 
les news ne s'enregistrent pas.<BR>&gt;voilà le message d'erreur 
: <BR><BR>Peux-tu m'envoyer ce que tu veux mettre en ligne par @ 
?<BR>Je vérifierais directement avec tes données car il n'y a pas 
de problème de mon côté pour ajouter les news.<BR><BR>Merci.<BR><BR>-- 
<BR>Fabrice<BR></BLOCKQUOTE><p>
		<hr size=1> 

Découvrez le nouveau Yahoo! Mail : <font color="red">250 Mo d'espace</font> 
de stockage pour vos mails !<br><a href="http://fr.rd.yahoo.com/mail/taglines/*http://us.rd.yahoo.com/evt=25917/*http://us.rd.yahoo.com/mail_fr/mail_campaigns/splash/taglines_250/default/*http://fr.promotions.yahoo.com/mail/creer28.html">Créez 
votre Yahoo! Mail</a>


--0-370637848-1114484309=:67141--

If this list doesn't fulfill all your needs, here's the additionnal 
features planned for the 1.0 release :

    * RSP: REBOL Server Pages support.
    * General CGI support (run any CGI script).

    * Chunk-encoding transferts support (streamed data transferts).
    * Standard compression methods support: gzip, deflate, bzip2.

    * Byte-ranges request support (ability to request files in parts 
    and resume broken downloads).

    * mod-rewrite module for powerful request URL transformations (without 
    the regexp complexity!).

    * mod-map-url module for direct URL to REBOL functions or objects 
    mapping.
    * SSL support.
    * Advanced GUI client for local and remote administration.

Chris -- I don't get that problem,

But you did make me look closer, and my earlier statement was wrong.
I'm using

http://www.rebol.org/cgi-bin/cgiwrap/rebol/download-a-script.r?script-name=xml-object.r
Which is similar to xml-parse, but not identical.
Example of usage:
probe: first reduce xml-to-object parse-xml
     {<?xml version="1.0" encoding="ISO-8859-1"?>
      <xxx>11</xxx>
     }

John, I've downloaded it from your website - thank you!

One more question from an unexperienced REBOL-user:

What is the most commen way to enhance a block I've got with xml2rebxml, 
source is
<?xml version="1.0" encoding="iso-8859-1"?>
<chapter id="ch_testxml" name="Test XML">
  <title>A chapter with some xml tests</title>
  <sect1 id="sct_about" name="About my Tests">
    <title>What kind of tests I will do</title>
    <body>
      <para>Some simple paragraph.</para>
    </body>
  </sect1>
</chapter>

After read in the file with
my-doc: xml2rebxml read %test.xml

I'd like to insert a second sect1-element in the block my-doc, whats 
the best way - just to avoid some stupid mistakes.

RebXML: I did some testing with rebxml, the documents I used can 
be found here:
 http://www.simplix.de/rebol/resources/xml/xmltests.zip

There is also a simple script that reads the XML docs in and writes 
them back.

Some problems I found:
- empty attributes, I have fixed this in the zip

- entities in content: all should be escaped, because they can be 
found there, otherwise a " gets "
- comments after last element missed
- comments before first element - missing line feed
- missing PIs in output


Another question: encoding - it seems that all output files will 
be written in iso-8859-1 ?

About encoding in RebXML, rebxml2xml let you produce utf-8 by specifying 
the /utf-8 refinement:
rebxml2xml/utf-8 <some rebxml data>

the only difference I can see between POST and SOAP is that the encoding 
method ... SOAP uses text/xml;

The web and soap/http are in a sense REST applications (REST is just 
WS over HTTP) though both use a limited subset of REST arguments 
and have to work around as such.


The web is limited (by the HTML spec) to the verbs 'get and 'post, 
and post content types of 'application/x-www-form-urlencoded (default) 
or 'multipart/form-data (used to upload files).  For the most part, 
the web is RESTful as we usually only want to 'get resources.  However, 
other operations typically violate REST principles, as all other 
resource actions (create, update, delete) have to squeeze through 
the get/post/url-encode/multipart pipe.  The Rebol HTTP protocol 
as standard is designed to mimic this and requires patching to move 
beyond get/post/url-endode (even for multipart)


SOAP as I understand it, when using HTTP only uses 'post - the post 
content contains the actual request.  Where it varies from the web 
(and Rebol HTTP) is the need for the 'text/xml content type.


REST itself is limited only by HTTP.  It uses theoretically limitless 
HTTP (v1.1) verbs (though most common patterns use 'get, 'put, 'post 
and 'delete).  It uses any encoding.  It uses HTTP headers as parameters 
(eg. the 'Accept header specifies the desired return type).  Therefore, 
any HTTP protocol designed for REST will accomodate SOAP requests.

As I said the problems we have last time were not related to our 
broadcasting or encoding format. It was due to our physical instruments 
(camaras, mics, and camera man).

A question about nenads original pgsql (0.90),


I have a database using UNICODE as the encoding, and try to connect 
via pgsql.
Characters get encoded as, e.g.  \010 ( #"^/" ) etc. 

I tried to change the client encoding within postgresql to winxxxx, 
or latinxxx, but it made no difference.

any ideas?

Im' using the non async version with these changes with no problems 
and use UNICODE encoding as well

Something else that works: 

insert  db "set client_encoding to 'win'"


now pgsql understands what rebol sends it, doesn't work for "€", 
though.

Thanks Oldes - it worked!

The minor thing now is that I can not use UTF8 encoding for databases 
then I am getting in case of language specific characters:


>> insert db {INSERT INTO my_first_table VALUES (1, 'kaíçni', 12.34);}

** User Error: ERROR:  invalid byte sequence for encoding "UTF8": 
0xede76e

** Near: insert db {INSERT INTO my_first_table VALUES (1, 'kaíçni', 
12.34);}
>>


But well - it seems that I can still use Win1257 encoding or try 
to set client encoding - that I did not yet tried/learn.

or let the sql server know, what encoding you are using (that it's 
not UTF8)

if you set the encoding on the server side, you don't need to convert 
your strings in Rebol but let the server to do that job

It seems for me too the best aproach too.

One another case where to use encoding funcions could be PDF maker 
- I had trouble to get pdf output into special characters of my language.

Hi All,  
Have any Rebolers dealt with UniCode files?


Here is my situation.  I work on an IBM AS400 that can "port" files 
over to the PC.  Notebook opens it up just fine, but Rebol doesn't 
see it the same way.  If I Cut & Paste the contents of the file into 
an empty notebook and save it, Rebol can see it just fine.  Upon 
further study, I noticed at the bottom of the SAVE AS window that 
Encoding was set to UNICODE for the AS400 file, while the cut & paste 
one was set to ANSI.  


Does Rebol want ANSI text files only, or can it read UNICODE files 
too?

When you try to save a document under Notebook, the encoding choices 
are UTF-8, UNICODE, ANSI among others.  UNICODE may be the same as 
UTF-16 because it does look like every single character is saved 
as two bytes.  


The code (rejoin extract read InFile 2) does eliminate the double 
characters but I noticed that the entire file is still double spaced 
-- as if the newline is coded twice and not removed from the rejoin. 
 But that extra newline may be an annoyance than anything else.

Hi PhilB -- The formatted text report is generated on the AS400 into 
the work spool area.  I then can use the INavigator software on the 
PC to connect to it and drag and drop it on the PC, where I can look 
at it via Word or Notebook.  I'm not sure where the encoding to UniCode 
is happening -- I suspect the INavigator software, but then, it may 
not be an issue since Rebol can convert it to readable text, even 
with the extra newline between each line, I'm sure that annoyance 
can be overcome too.

At first, I thought it just be some stray bytes comming from the 
AS400, but I was able to re-create a file using Notebook and get 
same results.
Any of you should be able to test this out by:
1.  Open Notebook
2.  Type in some text
3.  Save the file with Encoding to UNICODE

To give you an example.. this.. 
<?xml version="1.0" encoding="utf-8"?>

<mx:Application xmlns:mx="http://www.adobe.com/2006/mxml"layout="absolute">

<mx:Panel title="My Application" width="200" height="300" x="0" y="0">

<mx:Label text="Welcome to Flex!" mouseDownEffect="WipeRight" height="45"/>


</mx:Panel>
	<mx:PopUpButton x="483" y="20" label="PopUpButton"/>
	<mx:Accordion x="441" y="50" width="200" height="200">
		<mx:Canvas label="Accordion Pane 1" width="100%" height="100%">
		</mx:Canvas>
		<mx:Canvas label="asdf" width="100%" height="100%">
		</mx:Canvas>
		<mx:Canvas label="asdf" width="100%" height="100%">
		</mx:Canvas>
		<mx:Canvas label="adsf" width="100%" height="100%">
		</mx:Canvas>
	</mx:Accordion>
	<mx:CheckBox x="441" y="258" label="Checkbox"/>
	<mx:DateChooser x="238.5" y="31"/>

</mx:Application>

What would be the best way to implement switch like function in rebcode 
(with many cases - encoding/decoding table)

Each type of COM object may support different string encodings, I 
think. We'll have to look up what type of encoding Skype supports.

REBOL postscript dialect version 0.3.0: http://home.tiscali.dk/john.niclasen/postscript/postscript.r

New thing is, that you don't have to make all those blocks as in 
the prior version. Also alignment of text is possible and output 
is ISOLatin1 encoding, both thanks to Henrik. I should write some 
documentation soon!

So this is very simple encoding. ASCII85 encoding is a bit more difficult, 
but will take up less space.

It's not a short paste...
[HTTPd] ================== NEW REQUEST ==================

                                                         [HTTPd] Request Line=>GET /testapp/ HTTP/1.1

                                                                                                     [HTTPd] Request Headers=>
Host: localhost:8080

User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.3) Gecko/20070310 
Iceweasel/2.0.0.3 (Debian-2.0.0.3-1)

Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive


[HTTPd] Phase url-to-filename done ( mod-alias )

                                                [HTTPd] Phase url-to-filename done ( mod-rsp )

                                                                                              [HTTPd] Phase url-to-filename done ( mod-fastcgi )
   [HTTPd] Phase url-to-filename done ( mod-static )

                                                    [HTTPd] Phase access-check done ( mod-action )

                                                                                                  [HTTPd] Phase set-mime-type done ( mod-action )
    [HTTPd] Phase make-response done ( mod-rsp )

                                                [HTTPd] Response=>

                                                                  HTTP/1.1 302 Moved Temporarily
Server: Cheyenne/0.9.11
Connection: close
Location: /testapp/login.rsp


[HTTPd] Phase logging done ( mod-static )

                                         [HTTPd] Phase clean-up done ( mod-rsp )

                                                                                [HTTPd] Connection closed

[HTTPd] ================== NEW REQUEST ==================        
                                        /

                                                         [HTTPd] Request Line=>GET /testapp/login.rsp HTTP/1.1

                                                                                                              [HTTPd] Request Headers=>
Host: localhost:8080

User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.3) Gecko/20070310 
Iceweasel/2.0.0.3 (Debian-2.0.0.3-1)

Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive


[HTTPd] Phase url-to-filename done ( mod-alias )

                                                [HTTPd] Phase url-to-filename done ( mod-rsp )

                                                                                              [HTTPd] Phase url-to-filename done ( mod-fastcgi )
   [HTTPd] Phase url-to-filename done ( mod-static )

                                                    [HTTPd] Phase access-check done ( mod-action )

                                                                                                  [HTTPd] Phase set-mime-type done ( mod-action )
    [HTTPd] Phase make-response done ( mod-rsp )
[HTTPd] Response=>
                  HTTP/1.1 200 OK
Server: Cheyenne/0.9.11
Content-Length: 482
Content-Type: text/html
Connection: Keep-Alive

Set-Cookie: RSPSID=EISPOMAZTPDFKVIWJAFONZDE; expires=Sat, 02 Jun 
2007 11:54:30 GMT; path=/testapp; HttpOnly
Pragma: no-cache
Cache-Control: no-cache, must-revalidate
Expires: Thu, 01 Dec 1994 16:00:00 GMT


[HTTPd] Phase logging done ( mod-static )

                                         [HTTPd] Phase clean-up done ( mod-rsp )

                                                                                [HTTPd] Phase task-done done ( mod-rsp )

[HTTPd] ================== NEW REQUEST ==================        
                                                       \

                                                         [HTTPd] Request Line=>POST /testapp/login.rsp HTTP/1.1

                                                                                                               [HTTPd] Request Headers=>
Host: localhost:8080

User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.3) Gecko/20070310 
Iceweasel/2.0.0.3 (Debian-2.0.0.3-1)

Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://localhost:8080/testapp/login.rsp
Content-Type: application/x-www-form-urlencoded
Content-Length: 23


[HTTPd] Phase url-to-filename done ( mod-alias )

                                                [HTTPd] Phase url-to-filename done ( mod-rsp )

                                                                                              [HTTPd] Phase url-to-filename done ( mod-fastcgi )
   [HTTPd] Phase url-to-filename done ( mod-static )

                                                    [HTTPd] Posted data=>login=test&pass=letmein

                                                                                                [HTTPd] Phase access-check done ( mod-action )
 [HTTPd] Phase set-mime-type done ( mod-action )

                                                [HTTPd] Phase make-response done ( mod-rsp )
[HTTPd] Response=>
                  HTTP/1.1 301 Moved Permanently
Server: Cheyenne/0.9.11
Connection: close
Location: /testapp/

Set-Cookie: RSPSID=YDADUIONKJPHLFBWEDZDFCXN; expires=Sat, 02 Jun 
2007 11:54:37 GMT; path=/testapp; HttpOnly


[HTTPd] Phase logging done ( mod-static )

                                         [HTTPd] Phase clean-up done ( mod-rsp )

                                                                                [HTTPd] Phase task-done done ( mod-rsp )

                                                                                                                        [HTTPd] Connection closed
    [HTTPd] ================== NEW REQUEST ==================

                                                             [HTTPd] Request Line=>GET /testapp/ HTTP/1.1

                                                                                                         [HTTPd] Request Headers=>
Host: localhost:8080

User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.3) Gecko/20070310 
Iceweasel/2.0.0.3 (Debian-2.0.0.3-1)

Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://localhost:8080/testapp/login.rsp


[HTTPd] Phase url-to-filename done ( mod-alias )

                                                [HTTPd] Phase url-to-filename done ( mod-rsp )

                                                                                              [HTTPd] Phase url-to-filename done ( mod-fastcgi )
   [HTTPd] Phase url-to-filename done ( mod-static )

                                                    [HTTPd] Phase access-check done ( mod-action )

                                                                                                  [HTTPd] Phase set-mime-type done ( mod-action )
    [HTTPd] Phase make-response done ( mod-rsp )

                                                [HTTPd] Response=>

                                                                  HTTP/1.1 302 Moved Temporarily
Server: Cheyenne/0.9.11
Connection: close
Location: /testapp/login.rsp


[HTTPd] Phase logging done ( mod-static )

                                         [HTTPd] Phase clean-up done ( mod-rsp )

                                                                                [HTTPd] Connection closed

                                                                                                         [HTTPd] ================== NEW REQUEST ==================

                     [HTTPd] Request Line=>GET /testapp/login.rsp HTTP/1.1

                                                                          [HTTPd] Request Headers=>
Host: localhost:8080

User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.3) Gecko/20070310 
Iceweasel/2.0.0.3 (Debian-2.0.0.3-1)

Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://localhost:8080/testapp/login.rsp


[HTTPd] Phase url-to-filename done ( mod-alias )

                                                [HTTPd] Phase url-to-filename done ( mod-rsp )

                                                                                              [HTTPd] Phase url-to-filename done ( mod-fastcgi )
   [HTTPd] Phase url-to-filename done ( mod-static )

                                                    [HTTPd] Phase access-check done ( mod-action )

                                                                                                  [HTTPd] Phase set-mime-type done ( mod-action )
    [HTTPd] Phase make-response done ( mod-rsp )

                                                [HTTPd] Response=>

                                                                  HTTP/1.1 200 OK
Server: Cheyenne/0.9.11
Content-Length: 482
Content-Type: text/html
Connection: Keep-Alive

Set-Cookie: RSPSID=RTJSUKAVYBNOLCJCJBSTNUHP; expires=Sat, 02 Jun 
2007 11:54:37 GMT; path=/testapp; HttpOnly
Pragma: no-cache
Cache-Control: no-cache, must-revalidate
Expires: Thu, 01 Dec 1994 16:00:00 GMT


[HTTPd] Phase logging done ( mod-static )

                                         [HTTPd] Phase clean-up done ( mod-rsp )

                                                                                [HTTPd] Phase task-done done ( mod-rsp )

My working login.rsp script ...

<%
		in-user: select request/content 'login
		in-pass: select request/content 'pass
		encoding-salt: to-binary "My encryption string"


  print [ <p/> "Login is: " in-user " and pass is " in-pass <p/> ]
				
		encode-pass: func [
			pass [string!]
			salt [binary!]
		] [
			checksum/secure append to binary! pass salt
		]

		
		if all [ in-user in-pass ][
			print <pre>


   qry: rejoin [ {select staffname, sid, fullname from staff where staffname 
   = '} in-user {' and pwd = '} form encode-pass in-pass encoding-salt 
   {'} ]

			probe qry
			print </p>
			
			sql: do-sql 'remr qry 
			print [ "Query result: " sql ]
			print </pre>	
			if found? sql/1 [
				response/redirect "/testapp/"
			]
		]
%>

Output from a Ice Weasel  http://dev:8080       - dev.rsp redirects 
to show.rsp...
Back

Timestamp: 2-Jun-2007/19:37:48-4:00

Request parameters :

    * HTTP Method: GET
    * HTTP Port: 8080
    * Client IP address: 192.168.1.102

Request headers :

    * Host : "dev"

    * User-Agent : {Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.3) 
    Gecko/20070310 Iceweasel/2.0.0.3 (Debian-2.0.0.3-1)}

    * Accept : {text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5}
    * Accept-Language : "en-us,en;q=0.5"
    * Accept-Encoding : "gzip,deflate"
    * Accept-Charset : "ISO-8859-1,utf-8;q=0.7,*;q=0.7"
    * Keep-Alive : "300"
    * Connection : "keep-alive"

Request variables :

    * No variable passed

Session :

    * No session

Another approach could be to install in your module a callback for 
'filter-output phase (not used in any builtin modules, yet) and test 
the return code. This way, you wouldn't need to patch mod-static 
and if you give it  order: 'first, it should be able to work even 
when I'll add callbacks to that phase. The purpose of this phase 
is to allow last minute changes on the response content, like encoding 
(think about JSON, for example), compressing (gzip, deflate) or encrypting 
it. These kinds of handlers would be installed as 'last in the phase 
callbacks order.

you need to encode # with percent encoding, and you need to make 
sure rebol is nod decoding that for you silently (the url! datatype 
does this unfortunately).

No tested, but I don't why this would be a problem for Cheyenne, 
as long as it's a valid HTTP request. There's only two upload encodings 
that Cheyenne currently doesn't support: multipart/mixed and chunked. 
Chunked encoding is only supported in Cheyenne's responses.

i need to find some time and look at what i need to do (eg encoding) 
to put them there.

YouTube I know you just upload.  No need to worry about encoding 
there