pop problems
[1/26] from: gchiu::compkarori::co::nz at: 12-Oct-2002 21:09
Anyone had any problems accessing some messages from a
pop3 server?
I just was not able to access one particular message ...
rebol just hung. Yet, I had no problems with retrieving
the same message using OE.
The message did not seem malformed as far as I could see,
but presumably there is something about the message that
rebol can't with.
--
Graham Chiu
[2/26] from: rebol:optushome:au at: 12-Oct-2002 20:18
Hi Graham,
Do you have something like norton anti-virus running on the same machine? It
might have intercepted the email (if you have anti-virus email monitoring
on), detected a virus and was waiting for a reply to a popup dialog. But
then if it didn't behave that way for OE, it probably isn't the case.
Cheers,
Allen K
[3/26] from: gchiu:compkarori at: 12-Oct-2002 23:48
On Sat, 12 Oct 2002 21:09:44 +1300
"Graham Chiu" <[gchiu--compkarori--co--nz]> wrote:
>I just was not able to access one particular message ...
>rebol just hung. Yet, I had no problems with retrieving
>the same message using OE.
>
I started up Ethereal to see what was wrong. I was able
to retrieve messages smaller and larger than this one, but
not this message using the pick method.
Ethereal shows that the complete message arrived, but
Rebol was hung, possibly attempting to retrieve more data
than was available.
--
Graham Chiu
[4/26] from: gchiu:compkarori at: 13-Oct-2002 9:06
On Sat, 12 Oct 2002 20:18:11 +1000
"Allen Kamp" <[rebol--optushome--com--au]> wrote:
>Do you have something like norton anti-virus running on
>the same machine? It
Hi Allen,
No I don't have NAV running. I tried retrieving this
message on 3 different PCs with the same result. This
suggests that the problem is with Rebol.
As an aside, I notice that in the Rebol POP code, there is
this function:
read-message: func [
"Read a message from the POP server"
port
n [integer!]
/local buf line
][
insert port/sub-port reform [
"RETR" system/words/pick
port/locals/msg-nums port/state/index + n]
net-utils/confirm port/sub-port
write-check
read-til-dot port buf: make string! 1024
; guess at size
]
The buffer is created each time a message is read. Since
the message size is already known within locals/sizes,
presumably this could be sped up a little by allocating
the exact amount of memory instead of 1024 each time.
read-til-dot: func [port buf][
while [(line: system/words/pick
port/sub-port 1) <> "."] [
insert tail buf line
insert tail buf newline
]
buf
]
This function reads from the POP port until it retrieves
only a ".". I was thinking of returning the buffer if the
expected number of chars are retrieved as an additional
check.
I'm gonna have to wait till I hit another message I can't
download to test this out. I had another one like this
last week, and I just deleted it from the pop server so I
could download all the other messages. For reference,
this troublesome message was 16674 octets long.
--
Graham Chiu
[5/26] from: gchiu:compkarori at: 13-Oct-2002 11:21
On Sun, 13 Oct 2002 09:06:23 +1300
"Graham Chiu" <[gchiu--compkarori--co--nz]> wrote:
>This function reads from the POP port until it retrieves
>only a ".". I was thinking of returning the buffer if
>the expected number of chars are retrieved as an
>additional check.
>
Hmm. That doesn't work. Although RFC1939 says
In order to simplify parsing, all POP3 servers are
required to use a certain format for scan listings.
A scan listing consists of the message-number of the
message, followed by a single space and the exact size
of the message in octets.
the size calculation depends upon how the pop server
internally represents the end-of-line.
--
Graham Chiu
[6/26] from: gchiu:compkarori at: 17-Oct-2002 8:33
>I'm gonna have to wait till I hit another message I can't
>download to test this out. I had another one like this
>last week, and I just deleted it from the pop server so I
>could download all the other messages. For reference,
>this troublesome message was 16674 octets long.
Well, I'm stuck again. I can't retrieve message 3 from my
pop server. Here's the session:
>> rebol/version
== 1.0.2.3.1
>> mailbox: open pop://userid:[password--pop--isp]
>> probe mailbox/locals
make object! [
total-size: 1046586
sizes: [1 2357
2 7735
3 14992
4 2475
5 5430
6 4813
7 6487
8 2445
9 835
10 21810
11 2974
12 5832
13 2518
(escape)
>> eml: pick mailbox 2
reading messageno 2 size: 7736
== {Return-Path:
<sentto-93429-30506-1034776780-gchiu=[compkarori--co--nz--returns--groups--yahoo--com]>
Received: from n35.grp.scd.yahoo.c...
>> eml: pick mailbox 5
reading messageno 5 size: 5431
== {Return-Path:
<sentto-93429-30507-1034777182-gchiu=[compkarori--co--nz--returns--groups--yahoo--com]>
Received: from n18.grp.scd.yahoo.c...
>> eml: pick mailbox 3
reading messageno 3 size: 14993
Got none .. aborting
== {Return-Path:
<sentto-93429-30505-1034776706-gchiu=[compkarori--co--nz--returns--groups--yahoo--com]>
Received: from n33.grp.scd.yahoo.c...
>> eml: pick mailbox 2
reading messageno 2 size: 7736
** User Error: Server error: tcp connection failed
** Near: eml: pick mailbox 2
>>
I tracked the problem down to
read-til-dot: func [port buf][
while [(line: system/words/pick port/sub-port 1) <>
.
] [
if none? line [ print "Got none .. aborting" break
]
insert tail buf line
insert tail buf newline
]
buf
]
where /pick port 1 returns none ... and keeps returning
none in an endless loop.
I put the above trap in the function, but then get a tcp
error, as can be seen when I then do
pick mailbox 2
Does anyone knowledgeable in matters Rebol tcp have a work
round for me?
--
Graham Chiu
[7/26] from: gscottjones:mchsi at: 16-Oct-2002 15:57
Hi, Graham,
From: "Graham Chiu"
message: http://www.escribe.com/internet/rebol/m26582.html
...
> I tracked the problem down to
> read-til-dot: func [port buf][
<<quoted lines omitted: 14>>
> Does anyone knowledgeable in matters Rebol tcp have a work
> round for me?
Getting none on the port suggests an error that is not necessarily
originating from REBOL. (Maybe you weren't implying it is a REBOL problem.
:-) The POP protocol says that for multiline messages, that the message is
sent and then a single line with a period is sent. A none return suggests a
pop server error or dropped packets/congestion. At any rate, if you are
checking for this state in order to debug the whole situation, then I think
your way will work, but the function needs to return a value in order to
avoid an error. Try something like:
func [port buf][
while [(line: system/words/pick port/sub-port 1) <> "."] [
if none? line [
print "Got none .. aborting"
break/return buf
]
insert tail buf line
insert tail buf newline
]
buf
]
where the break returns the buffer named buf. This should avoid an error.
If it is some sort of network congestion problem, then I will guess that you
may likely see other shortened messages due to a premature abortion on the
port read (meaning there was momentarily nothing to be read).
Experimentation will readily reveal if this is a problem.
Good luck, and hope my mini-fix avoids the error.
--Scott Jones
[8/26] from: gscottjones:mchsi at: 16-Oct-2002 16:37
From: "G. Scott Jones"
...
> Try something like:
> func [port buf][
<<quoted lines omitted: 8>>
> buf
> ]
Ooops. I forgot that the break would break out of the while not the
function. Obviously my addition should be of no benefit. I guess that is
an additional problem with not being able to test on the error condition
before sending. Sorry.
--Scott Jones
[9/26] from: brett:codeconscious at: 17-Oct-2002 8:26
Hi Graham,
> >> rebol/version
> == 1.0.2.3.1
Why this oldie?
> if none? line [ print "Got none .. aborting" break
None of the REBOLs I've checked into have this line. Are you using an
unmodified pop scheme? If so maybe it was a bug that got squashed a while
ago.
Regards,
Brett.
----- Original Message -----
From: "Graham Chiu" <[gchiu--compkarori--co--nz]>
To: <[rebol-list--rebol--com]>
Sent: Thursday, October 17, 2002 5:33 AM
Subject: [REBOL] Re: pop problems
> >
> >I'm gonna have to wait till I hit another message I can't
<<quoted lines omitted: 27>>
> reading messageno 2 size: 7736
> == {Return-Path:
<sentto-93429-30506-1034776780-gchiu=[compkarori--co--nz--returns--groups--yahoo--c]
om>
> Received: from n35.grp.scd.yahoo.c...
> >> eml: pick mailbox 5
> reading messageno 5 size: 5431
> == {Return-Path:
>
<sentto-93429-30507-1034777182-gchiu=[compkarori--co--nz--returns--groups--yahoo--c]
om>
> Received: from n18.grp.scd.yahoo.c...
> >> eml: pick mailbox 3
> reading messageno 3 size: 14993
> Got none .. aborting
> == {Return-Path:
>
<sentto-93429-30505-1034776706-gchiu=[compkarori--co--nz--returns--groups--yahoo--c]
om>
[10/26] from: gchiu:compkarori at: 17-Oct-2002 11:45
>If it is some sort of network congestion problem, then I
>will guess that you
<<quoted lines omitted: 3>>
>read).
>Experimentation will readily reveal if this is a problem.
Hi Scott,
I don't think the problem is a network congestion problem
as I can download messages both larger/smaller. I can
reproduce the problem exactly with the same message, so
the error is specific to a particular message.
Since Outlook Express is able to download these
troublesome messages, I have concluded that the problem is
with Rebol.
Since my debugging line actually allows me to retrieve the
entire message, what I would like to do is return without
the tcp error ie. reset whatever flag is flagging an
error.
I note that if I return from the read-til-dot function
after just retrieving one line from any pop read, I get a
similar error. So could rebol be flagging an error in both
cases because I have not retrieved all the data??
Any ideas???
--
Graham Chiu
[11/26] from: greggirwin:mindspring at: 16-Oct-2002 17:01
Hi Brett,
<<
> >> rebol/version
> == 1.0.2.3.1
Why this oldie?
>>
Probably a Link version.
--Gregg
[12/26] from: brett:codeconscious at: 17-Oct-2002 9:05
Oh yeah... eheh...
Second foolish comment was about the line - I realise now that it is
Graham's self defence :^)
Still my question about the unmodified pop is valid, though I'm guessing
what the answer will be.
Graham - is it possible to get a copy of your problem message to a different
mailbox, without affecting your original? On the off chance that you can
test if it is a single message thing vs a position in a mailbox thing.
Regards,
Brett.
[13/26] from: gchiu:compkarori at: 17-Oct-2002 12:20
>> >> rebol/version
>> == 1.0.2.3.1
>
>Why this oldie?
Hi Brett,
That's the version that comes up from IOS. It's easy
enough for me to break out a console from IOS, but the
problems I'm experiencing are on the new betas as well.
>> if none? line [ print "Got none .. aborting"
>>break
<<quoted lines omitted: 3>>
>squashed a while
>ago.
That's a debugging line I added to the pop scheme to try
and see what's going on.
--
Graham Chiu
[14/26] from: gchiu:compkarori at: 17-Oct-2002 14:37
>Graham - is it possible to get a copy of your problem
>message to a different
>mailbox, without affecting your original? On the off
>chance that you can
>test if it is a single message thing vs a position in a
>mailbox thing.
Hi Brett,
Unfortunately as soon as you transport it, new headers get
added changing the original message.
I don't think it is a positional thing as this problem has
occurred in different positions in the pop queue.
--
Graham Chiu
[15/26] from: brett:codeconscious at: 17-Oct-2002 12:49
Hi Graham,
I can't decompress the file - "Out of memory" error - which is odd in
itself.
Perhaps you base 64 version (write mold Enbase/base .... 64)?
Regards,
Brett.
[16/26] from: brett:codeconscious at: 17-Oct-2002 12:53
Better yet, perhaps you could publish the message in IOS so that I can see
it (private folder perhaps).
[17/26] from: gchiu:compkarori at: 18-Oct-2002 9:39
An update on my pop problems.
I've got another message that I can't download.
Interestingly it is again from the same person ( on a
yahoogroups mailing list ). So, there is something about
her messages that exposes a problem with the pop scheme.
More debugging to do ...
--
Graham Chiu
[18/26] from: gscottjones:mchsi at: 17-Oct-2002 16:09
From: "Graham Chiu"
...
> Interestingly it is again from the same person ( on a
> yahoogroups mailing list ). So, there is something about
> her messages that exposes a problem with the pop scheme.
>
> More debugging to do ...
Hi, Graham,
Isn't it fun to debug a difficult to reproduce problem?
;-)
I was trying to create errors last night and this morning, but was only
successful
once, so it is hard to reproduce the problem while trying
different things.
It may be totally obvious to you, so apologies in advance, but some of the
things I was trying, in order to see where problems may lies, is to have the
function print every line, like the following:
read-til-dot: func [port buf][
while [(line: system/words/pick port/sub-port 1) <> "."] [
print line
insert tail buf line
insert tail buf newline
]
buf
]
Similarly, I telnet-ed to my mail server on port 110 then used something
like the following to get to a message 1. It also prints everything up to
the single period on a line.
USER yourname
PASS yourpass
LIST
RETR 1
If you find the problem, I would certainly be interested in hearing what it
is. Good luck.
--Scott Jones
[19/26] from: gchiu:compkarori at: 18-Oct-2002 12:06
>read-til-dot: func [port buf][
> while [(line: system/words/pick port/sub-port 1) <>
<<quoted lines omitted: 5>>
> buf
>]
Hi Scott,
Yes, I was trying that out as well. Nothing obviously
abnormal about the mail.
I've put them into my ios/developer/users/graham/wip
folder if you want to have a peek.
>Similarly, I telnet-ed to my mail server on port 110 then
>used something
>like the following to get to a message 1. It also prints
>everything up to
>the single period on a line.
Unfortunately I don't have telnet access to my pop server.
But I might just try and bypass the pop scheme next time
this happens, and just open a tcp port to 110, and do the
above. Gives me another thing to try :)
--
Graham Chiu
[20/26] from: brett:codeconscious at: 18-Oct-2002 14:56
I've asked Graham to check out my theory. In the meantime this I what I
reckon - the POP scheme can't handle a line length greater than 4093. Here's
a little view program for those interested to test the idea:
Regards,
Brett.
REBOL [
Author: "Brett Handley"
Date: 18-Oct-2002
Purpose: "To test email lengths."
]
;
; It appears we are hitting a 4096 byte limit - like HTTP post problem.
;
; 4094 for a line length is not ok, 4093 is.
send-message: does [
if not all [
not error? try [address: first to-block msg-email/text]
email? :address
not error? try [message-length: first to-block msg-len/text]
integer? :message-length
] [request/ok "Invalid data." return]
address: to-email msg-email/text
header: make system/standard/email [
Subject: {An email REBOL can't read.}
]
message: head insert/dup make string! 10000 "X" message-length
send/header address message header
]
view layout [
across
label "Message length [integer!]:" tab
msg-len: field "4094" return
label "Email address [email!]:" tab
msg-email: field form system/user/email return
button "Send message" [send-message]
]
[21/26] from: gchiu:compkarori at: 18-Oct-2002 20:47
On Fri, 18 Oct 2002 14:56:25 +1000
"Brett Handley" <[brett--codeconscious--com]> wrote:
>I've asked Graham to check out my theory. In the meantime
>this I what I
>reckon - the POP scheme can't handle a line length
>greater than 4093. Here's
>a little view program for those interested to test the
>idea:
Interestingly although this does not reproduce the problem
for me, it seems Brett does have the same problem in that
read-til-dot gets stuck in a loop returning none.
The length of the string may be OS dependent. He is using
NT 4, and I'm using XP.
--
Graham Chiu
[22/26] from: gscottjones:mchsi at: 18-Oct-2002 9:26
Hi, Graham and Brett,
> On Fri, 18 Oct 2002 14:56:25 +1000
> "Brett Handley" wrote:
<<quoted lines omitted: 3>>
> >a little view program for those interested to test the
> >idea:
From: "Graham Chiu"
> Interestingly although this does not reproduce the problem
> for me, it seems Brett does have the same problem in that
> read-til-dot gets stuck in a loop returning none.
>
> The length of the string may be OS dependent. He is using
> NT 4, and I'm using XP.
Brett's mutli-x (4094 to be exact :-) -rated email did trip up REBOL on
Win98se for me.
When I looked at the offending email in your (Graham's) link folder, I am
wondering if Brett not be on the right trail, in that the offending email
uses HTML with no line breaks.
When I experimented, I discovered that it appears to be REBOL's line buffer
cannot "absorb" enough characters, at least for NT and Win98. Given the
inherent line-oriented nature of POP servers, it is understandable "why" RT
used open/lines on the port. But "inconsiderate" email client's that don't
offer line breaks seem to break REBOL with its current line buffer.
So, I rewrote the pop scheme to switch from line mode to character mode once
it begins to read messages. After some fiddling (forgot that newline is not
necessarily equivalent to "^M^/" ... whoops there goes an hour of time:-), I
got the function to work, so I reintegrated it as the central "read-til-dot"
function, and so far it seems to work. It is less than elegant, because I
needed to maintain the carriage-return and line-feeds for the port
processing logic, but wanted to return the buffer back to the equivalent
REBOL newline (hence the replace/all call before the return).
If indeed the base problem turns out to be the line buffer, it would be more
elegant to improve the native functionality and leave the scheme alone
(which was neat in its original form). But my function replacement seems to
be more robust for now.
Graham, if you decide to try it and it "fixes" the hanging port problem, I
would like to report this to feedback, of course.
Hope this helps. And thanks to Brett for detecting at least one problem that
breaks the pop scheme!!
--Scott Jones
read-til-dot: func [port buf][
set-modes port [lines: false]
line: system/words/copy ""
while [line <> ".^M^/"][
line: system/words/copy ""
while [not find line "^M^/"][
a: system/words/pick port/sub-port 1
if integer? a [
append line to-char a
]
]
insert tail buf line
]
set-modes port [lines: true]
replace/all buf "^M^/" newline
buf
]
[23/26] from: gchiu:compkarori at: 19-Oct-2002 7:59
>Graham, if you decide to try it and it "fixes" the
>hanging port problem, I
>would like to report this to feedback, of course.
>
Hi Scott,
I think Brett has already reported this to feedback. I'll
give your function a go the next time Rebol hangs on an
email, but I fear that it will be much slower reading a
character at a time vs a line at a time.
Since the problem is not just with pop, hopefully RT will
address this problem real soon :)
--
Graham Chiu
[24/26] from: gscottjones:mchsi at: 18-Oct-2002 15:18
From: "Graham Chiu"
> I think Brett has already reported this to feedback. I'll
> give your function a go the next time Rebol hangs on an
> email, but I fear that it will be much slower reading a
> character at a time vs a line at a time.
Hi, Graham,
I sent 4 messages to myself: two of 16000 bytes in a single line and two of
8000 bytes in a line. Unchanged pop protocol, of course, choked. Doing a
pop read (broadband connection) with my revision over three runs:
0:00:12.42
0:00:07.85
0:00:07.8
Then I sent the same lengths of messages, but this time in a more line
friendly 76 characters per line. First, I ran a pop read with unchanged
protocol:
0:00:04.88
0:00:01.32
0:00:01.37
Then with my changed protocol:
0:00:03.96
0:00:03.96
0:00:03.79
I made a few revisions to my version, including the dumping of the final
period, and I changed the "append" to and "insert tail", which doubled its
speed for the 76 char line length trial (it made very little difference in
the "long line" versions).
Newest version, including the proper binding, for others who may wish to
experiment:
system/schemes/pop/handler/read-til-dot: func [port buf] bind [
set-modes port [lines: false]
line: system/words/copy ""
while [line <> ".^M^/"][
insert tail buf line
line: system/words/copy ""
while [not find line "^M^/"][
a: system/words/pick port/sub-port 1
if integer? a [
insert tail line to-char a
]
]
]
set-modes port [lines: true]
replace/all buf "^M^/" newline
buf
] in system/schemes/pop/handler 'self
I certainly agree that "something" needs to be fixed, and I think we would
all prefer the "/lines" buffer to be more robust in the face of less than
courteous tcp programs. I think the robustness of the patch is a reasonable
trade-off until a proper repair is made. Just trying to be helpful. :-)
> Since the problem is not just with pop, hopefully RT will
> address this problem real soon :)
Agreed!
--Scott Jones
[25/26] from: gchiu:compkarori at: 19-Oct-2002 10:39
Hi Scott,
>I certainly agree that "something" needs to be fixed, and
>I think we would
<<quoted lines omitted: 4>>
>trade-off until a proper repair is made. Just trying to
>be helpful. :-)
Thanks for the Sterling efforts ( sad pun here! ).
In my WIP folder, I hacked Jeff's changes to the pop
scheme to allocate the correct amount of memory for the
buffer rather than guessing at 1024 bytes.
Would you try that, and apply your patch as well, to see
if it speeds it up a little more?
--
Graham Chiu
[26/26] from: brett:codeconscious at: 19-Oct-2002 11:01
> > I think Brett has already reported this to feedback. I'll
Yes I've sent feedback about two problems: this POP problem and
READ/DIRECT/LINES limit of 4094...
> Newest version, including the proper binding, for others who may wish to
> experiment:
Great work, I tried your patch and it allowed me to read the "problem"
email.
Regards,
Brett.
Notes
- Quoted lines have been omitted from some messages.
View the message alone to see the lines that have been omitted