Mailing List Archive: 49091 messages
  • Home
  • Script library
  • AltME Archive
  • Mailing list
  • Articles Index
  • Site search
 

pop problems

 [1/26] from: gchiu::compkarori::co::nz at: 12-Oct-2002 21:09


Anyone had any problems accessing some messages from a pop3 server? I just was not able to access one particular message ... rebol just hung. Yet, I had no problems with retrieving the same message using OE. The message did not seem malformed as far as I could see, but presumably there is something about the message that rebol can't with. -- Graham Chiu

 [2/26] from: rebol:optushome:au at: 12-Oct-2002 20:18


Hi Graham, Do you have something like norton anti-virus running on the same machine? It might have intercepted the email (if you have anti-virus email monitoring on), detected a virus and was waiting for a reply to a popup dialog. But then if it didn't behave that way for OE, it probably isn't the case. Cheers, Allen K

 [3/26] from: gchiu:compkarori at: 12-Oct-2002 23:48


On Sat, 12 Oct 2002 21:09:44 +1300 "Graham Chiu" <[gchiu--compkarori--co--nz]> wrote:
>I just was not able to access one particular message ... >rebol just hung. Yet, I had no problems with retrieving >the same message using OE. >
I started up Ethereal to see what was wrong. I was able to retrieve messages smaller and larger than this one, but not this message using the pick method. Ethereal shows that the complete message arrived, but Rebol was hung, possibly attempting to retrieve more data than was available. -- Graham Chiu

 [4/26] from: gchiu:compkarori at: 13-Oct-2002 9:06


On Sat, 12 Oct 2002 20:18:11 +1000 "Allen Kamp" <[rebol--optushome--com--au]> wrote:
>Do you have something like norton anti-virus running on >the same machine? It
Hi Allen, No I don't have NAV running. I tried retrieving this message on 3 different PCs with the same result. This suggests that the problem is with Rebol. As an aside, I notice that in the Rebol POP code, there is this function: read-message: func [ "Read a message from the POP server" port n [integer!] /local buf line ][ insert port/sub-port reform [ "RETR" system/words/pick port/locals/msg-nums port/state/index + n] net-utils/confirm port/sub-port write-check read-til-dot port buf: make string! 1024 ; guess at size ] The buffer is created each time a message is read. Since the message size is already known within locals/sizes, presumably this could be sped up a little by allocating the exact amount of memory instead of 1024 each time. read-til-dot: func [port buf][ while [(line: system/words/pick port/sub-port 1) <> "."] [ insert tail buf line insert tail buf newline ] buf ] This function reads from the POP port until it retrieves only a ".". I was thinking of returning the buffer if the expected number of chars are retrieved as an additional check. I'm gonna have to wait till I hit another message I can't download to test this out. I had another one like this last week, and I just deleted it from the pop server so I could download all the other messages. For reference, this troublesome message was 16674 octets long. -- Graham Chiu

 [5/26] from: gchiu:compkarori at: 13-Oct-2002 11:21


On Sun, 13 Oct 2002 09:06:23 +1300 "Graham Chiu" <[gchiu--compkarori--co--nz]> wrote:
>This function reads from the POP port until it retrieves >only a ".". I was thinking of returning the buffer if >the expected number of chars are retrieved as an >additional check. >
Hmm. That doesn't work. Although RFC1939 says In order to simplify parsing, all POP3 servers are required to use a certain format for scan listings. A scan listing consists of the message-number of the message, followed by a single space and the exact size of the message in octets. the size calculation depends upon how the pop server internally represents the end-of-line. -- Graham Chiu

 [6/26] from: gchiu:compkarori at: 17-Oct-2002 8:33


>I'm gonna have to wait till I hit another message I can't >download to test this out. I had another one like this >last week, and I just deleted it from the pop server so I >could download all the other messages. For reference, >this troublesome message was 16674 octets long.
Well, I'm stuck again. I can't retrieve message 3 from my pop server. Here's the session:
>> rebol/version
== 1.0.2.3.1
>> mailbox: open pop://userid:[password--pop--isp] >> probe mailbox/locals
make object! [ total-size: 1046586 sizes: [1 2357 2 7735 3 14992 4 2475 5 5430 6 4813 7 6487 8 2445 9 835 10 21810 11 2974 12 5832 13 2518 (escape)
>> eml: pick mailbox 2
reading messageno 2 size: 7736 == {Return-Path: <sentto-93429-30506-1034776780-gchiu=[compkarori--co--nz--returns--groups--yahoo--com]> Received: from n35.grp.scd.yahoo.c...
>> eml: pick mailbox 5
reading messageno 5 size: 5431 == {Return-Path: <sentto-93429-30507-1034777182-gchiu=[compkarori--co--nz--returns--groups--yahoo--com]> Received: from n18.grp.scd.yahoo.c...
>> eml: pick mailbox 3
reading messageno 3 size: 14993 Got none .. aborting == {Return-Path: <sentto-93429-30505-1034776706-gchiu=[compkarori--co--nz--returns--groups--yahoo--com]> Received: from n33.grp.scd.yahoo.c...
>> eml: pick mailbox 2
reading messageno 2 size: 7736 ** User Error: Server error: tcp connection failed ** Near: eml: pick mailbox 2
>>
I tracked the problem down to read-til-dot: func [port buf][ while [(line: system/words/pick port/sub-port 1) <> . ] [ if none? line [ print "Got none .. aborting" break ] insert tail buf line insert tail buf newline ] buf ] where /pick port 1 returns none ... and keeps returning none in an endless loop. I put the above trap in the function, but then get a tcp error, as can be seen when I then do pick mailbox 2 Does anyone knowledgeable in matters Rebol tcp have a work round for me? -- Graham Chiu

 [7/26] from: gscottjones:mchsi at: 16-Oct-2002 15:57


Hi, Graham, From: "Graham Chiu" message: http://www.escribe.com/internet/rebol/m26582.html ...
> I tracked the problem down to > read-til-dot: func [port buf][
<<quoted lines omitted: 14>>
> Does anyone knowledgeable in matters Rebol tcp have a work > round for me?
Getting none on the port suggests an error that is not necessarily originating from REBOL. (Maybe you weren't implying it is a REBOL problem. :-) The POP protocol says that for multiline messages, that the message is sent and then a single line with a period is sent. A none return suggests a pop server error or dropped packets/congestion. At any rate, if you are checking for this state in order to debug the whole situation, then I think your way will work, but the function needs to return a value in order to avoid an error. Try something like: func [port buf][ while [(line: system/words/pick port/sub-port 1) <> "."] [ if none? line [ print "Got none .. aborting" break/return buf ] insert tail buf line insert tail buf newline ] buf ] where the break returns the buffer named buf. This should avoid an error. If it is some sort of network congestion problem, then I will guess that you may likely see other shortened messages due to a premature abortion on the port read (meaning there was momentarily nothing to be read). Experimentation will readily reveal if this is a problem. Good luck, and hope my mini-fix avoids the error. --Scott Jones

 [8/26] from: gscottjones:mchsi at: 16-Oct-2002 16:37


From: "G. Scott Jones" ...
> Try something like: > func [port buf][
<<quoted lines omitted: 8>>
> buf > ]
Ooops. I forgot that the break would break out of the while not the function. Obviously my addition should be of no benefit. I guess that is an additional problem with not being able to test on the error condition before sending. Sorry. --Scott Jones

 [9/26] from: brett:codeconscious at: 17-Oct-2002 8:26


Hi Graham,
> >> rebol/version > == 1.0.2.3.1
Why this oldie?
> if none? line [ print "Got none .. aborting" break
None of the REBOLs I've checked into have this line. Are you using an unmodified pop scheme? If so maybe it was a bug that got squashed a while ago. Regards, Brett. ----- Original Message ----- From: "Graham Chiu" <[gchiu--compkarori--co--nz]> To: <[rebol-list--rebol--com]> Sent: Thursday, October 17, 2002 5:33 AM Subject: [REBOL] Re: pop problems
> > > >I'm gonna have to wait till I hit another message I can't
<<quoted lines omitted: 27>>
> reading messageno 2 size: 7736 > == {Return-Path:
<sentto-93429-30506-1034776780-gchiu=[compkarori--co--nz--returns--groups--yahoo--c] om>
> Received: from n35.grp.scd.yahoo.c... > >> eml: pick mailbox 5 > reading messageno 5 size: 5431 > == {Return-Path: >
<sentto-93429-30507-1034777182-gchiu=[compkarori--co--nz--returns--groups--yahoo--c] om>
> Received: from n18.grp.scd.yahoo.c... > >> eml: pick mailbox 3 > reading messageno 3 size: 14993 > Got none .. aborting > == {Return-Path: >
<sentto-93429-30505-1034776706-gchiu=[compkarori--co--nz--returns--groups--yahoo--c] om>

 [10/26] from: gchiu:compkarori at: 17-Oct-2002 11:45


>If it is some sort of network congestion problem, then I >will guess that you
<<quoted lines omitted: 3>>
>read). >Experimentation will readily reveal if this is a problem.
Hi Scott, I don't think the problem is a network congestion problem as I can download messages both larger/smaller. I can reproduce the problem exactly with the same message, so the error is specific to a particular message. Since Outlook Express is able to download these troublesome messages, I have concluded that the problem is with Rebol. Since my debugging line actually allows me to retrieve the entire message, what I would like to do is return without the tcp error ie. reset whatever flag is flagging an error. I note that if I return from the read-til-dot function after just retrieving one line from any pop read, I get a similar error. So could rebol be flagging an error in both cases because I have not retrieved all the data?? Any ideas??? -- Graham Chiu

 [11/26] from: greggirwin:mindspring at: 16-Oct-2002 17:01


Hi Brett, <<
> >> rebol/version > == 1.0.2.3.1
Why this oldie?
>>
Probably a Link version. --Gregg

 [12/26] from: brett:codeconscious at: 17-Oct-2002 9:05


Oh yeah... eheh... Second foolish comment was about the line - I realise now that it is Graham's self defence :^) Still my question about the unmodified pop is valid, though I'm guessing what the answer will be. Graham - is it possible to get a copy of your problem message to a different mailbox, without affecting your original? On the off chance that you can test if it is a single message thing vs a position in a mailbox thing. Regards, Brett.

 [13/26] from: gchiu:compkarori at: 17-Oct-2002 12:20


>> >> rebol/version >> == 1.0.2.3.1 > >Why this oldie?
Hi Brett, That's the version that comes up from IOS. It's easy enough for me to break out a console from IOS, but the problems I'm experiencing are on the new betas as well.
>> if none? line [ print "Got none .. aborting" >>break
<<quoted lines omitted: 3>>
>squashed a while >ago.
That's a debugging line I added to the pop scheme to try and see what's going on. -- Graham Chiu

 [14/26] from: gchiu:compkarori at: 17-Oct-2002 14:37


>Graham - is it possible to get a copy of your problem >message to a different >mailbox, without affecting your original? On the off >chance that you can >test if it is a single message thing vs a position in a >mailbox thing.
Hi Brett, Unfortunately as soon as you transport it, new headers get added changing the original message. I don't think it is a positional thing as this problem has occurred in different positions in the pop queue. -- Graham Chiu

 [15/26] from: brett:codeconscious at: 17-Oct-2002 12:49


Hi Graham, I can't decompress the file - "Out of memory" error - which is odd in itself. Perhaps you base 64 version (write mold Enbase/base .... 64)? Regards, Brett.

 [16/26] from: brett:codeconscious at: 17-Oct-2002 12:53


Better yet, perhaps you could publish the message in IOS so that I can see it (private folder perhaps).

 [17/26] from: gchiu:compkarori at: 18-Oct-2002 9:39


An update on my pop problems. I've got another message that I can't download. Interestingly it is again from the same person ( on a yahoogroups mailing list ). So, there is something about her messages that exposes a problem with the pop scheme. More debugging to do ... -- Graham Chiu

 [18/26] from: gscottjones:mchsi at: 17-Oct-2002 16:09


From: "Graham Chiu" ...
> Interestingly it is again from the same person ( on a > yahoogroups mailing list ). So, there is something about > her messages that exposes a problem with the pop scheme. > > More debugging to do ...
Hi, Graham, Isn't it fun to debug a difficult to reproduce problem? ;-) I was trying to create errors last night and this morning, but was only successful once, so it is hard to reproduce the problem while trying different things. It may be totally obvious to you, so apologies in advance, but some of the things I was trying, in order to see where problems may lies, is to have the function print every line, like the following: read-til-dot: func [port buf][ while [(line: system/words/pick port/sub-port 1) <> "."] [ print line insert tail buf line insert tail buf newline ] buf ] Similarly, I telnet-ed to my mail server on port 110 then used something like the following to get to a message 1. It also prints everything up to the single period on a line. USER yourname PASS yourpass LIST RETR 1 If you find the problem, I would certainly be interested in hearing what it is. Good luck. --Scott Jones

 [19/26] from: gchiu:compkarori at: 18-Oct-2002 12:06


>read-til-dot: func [port buf][ > while [(line: system/words/pick port/sub-port 1) <>
<<quoted lines omitted: 5>>
> buf >]
Hi Scott, Yes, I was trying that out as well. Nothing obviously abnormal about the mail. I've put them into my ios/developer/users/graham/wip folder if you want to have a peek.
>Similarly, I telnet-ed to my mail server on port 110 then >used something >like the following to get to a message 1. It also prints >everything up to >the single period on a line.
Unfortunately I don't have telnet access to my pop server. But I might just try and bypass the pop scheme next time this happens, and just open a tcp port to 110, and do the above. Gives me another thing to try :) -- Graham Chiu

 [20/26] from: brett:codeconscious at: 18-Oct-2002 14:56


I've asked Graham to check out my theory. In the meantime this I what I reckon - the POP scheme can't handle a line length greater than 4093. Here's a little view program for those interested to test the idea: Regards, Brett. REBOL [ Author: "Brett Handley" Date: 18-Oct-2002 Purpose: "To test email lengths." ] ; ; It appears we are hitting a 4096 byte limit - like HTTP post problem. ; ; 4094 for a line length is not ok, 4093 is. send-message: does [ if not all [ not error? try [address: first to-block msg-email/text] email? :address not error? try [message-length: first to-block msg-len/text] integer? :message-length ] [request/ok "Invalid data." return] address: to-email msg-email/text header: make system/standard/email [ Subject: {An email REBOL can't read.} ] message: head insert/dup make string! 10000 "X" message-length send/header address message header ] view layout [ across label "Message length [integer!]:" tab msg-len: field "4094" return label "Email address [email!]:" tab msg-email: field form system/user/email return button "Send message" [send-message] ]

 [21/26] from: gchiu:compkarori at: 18-Oct-2002 20:47


On Fri, 18 Oct 2002 14:56:25 +1000 "Brett Handley" <[brett--codeconscious--com]> wrote:
>I've asked Graham to check out my theory. In the meantime >this I what I >reckon - the POP scheme can't handle a line length >greater than 4093. Here's >a little view program for those interested to test the >idea:
Interestingly although this does not reproduce the problem for me, it seems Brett does have the same problem in that read-til-dot gets stuck in a loop returning none. The length of the string may be OS dependent. He is using NT 4, and I'm using XP. -- Graham Chiu

 [22/26] from: gscottjones:mchsi at: 18-Oct-2002 9:26


Hi, Graham and Brett,
> On Fri, 18 Oct 2002 14:56:25 +1000 > "Brett Handley" wrote:
<<quoted lines omitted: 3>>
> >a little view program for those interested to test the > >idea:
From: "Graham Chiu"
> Interestingly although this does not reproduce the problem > for me, it seems Brett does have the same problem in that > read-til-dot gets stuck in a loop returning none. > > The length of the string may be OS dependent. He is using > NT 4, and I'm using XP.
Brett's mutli-x (4094 to be exact :-) -rated email did trip up REBOL on Win98se for me. When I looked at the offending email in your (Graham's) link folder, I am wondering if Brett not be on the right trail, in that the offending email uses HTML with no line breaks. When I experimented, I discovered that it appears to be REBOL's line buffer cannot "absorb" enough characters, at least for NT and Win98. Given the inherent line-oriented nature of POP servers, it is understandable "why" RT used open/lines on the port. But "inconsiderate" email client's that don't offer line breaks seem to break REBOL with its current line buffer. So, I rewrote the pop scheme to switch from line mode to character mode once it begins to read messages. After some fiddling (forgot that newline is not necessarily equivalent to "^M^/" ... whoops there goes an hour of time:-), I got the function to work, so I reintegrated it as the central "read-til-dot" function, and so far it seems to work. It is less than elegant, because I needed to maintain the carriage-return and line-feeds for the port processing logic, but wanted to return the buffer back to the equivalent REBOL newline (hence the replace/all call before the return). If indeed the base problem turns out to be the line buffer, it would be more elegant to improve the native functionality and leave the scheme alone (which was neat in its original form). But my function replacement seems to be more robust for now. Graham, if you decide to try it and it "fixes" the hanging port problem, I would like to report this to feedback, of course. Hope this helps. And thanks to Brett for detecting at least one problem that breaks the pop scheme!! --Scott Jones read-til-dot: func [port buf][ set-modes port [lines: false] line: system/words/copy "" while [line <> ".^M^/"][ line: system/words/copy "" while [not find line "^M^/"][ a: system/words/pick port/sub-port 1 if integer? a [ append line to-char a ] ] insert tail buf line ] set-modes port [lines: true] replace/all buf "^M^/" newline buf ]

 [23/26] from: gchiu:compkarori at: 19-Oct-2002 7:59


>Graham, if you decide to try it and it "fixes" the >hanging port problem, I >would like to report this to feedback, of course. >
Hi Scott, I think Brett has already reported this to feedback. I'll give your function a go the next time Rebol hangs on an email, but I fear that it will be much slower reading a character at a time vs a line at a time. Since the problem is not just with pop, hopefully RT will address this problem real soon :) -- Graham Chiu

 [24/26] from: gscottjones:mchsi at: 18-Oct-2002 15:18


From: "Graham Chiu"
> I think Brett has already reported this to feedback. I'll > give your function a go the next time Rebol hangs on an > email, but I fear that it will be much slower reading a > character at a time vs a line at a time.
Hi, Graham, I sent 4 messages to myself: two of 16000 bytes in a single line and two of 8000 bytes in a line. Unchanged pop protocol, of course, choked. Doing a pop read (broadband connection) with my revision over three runs: 0:00:12.42 0:00:07.85 0:00:07.8 Then I sent the same lengths of messages, but this time in a more line friendly 76 characters per line. First, I ran a pop read with unchanged protocol: 0:00:04.88 0:00:01.32 0:00:01.37 Then with my changed protocol: 0:00:03.96 0:00:03.96 0:00:03.79 I made a few revisions to my version, including the dumping of the final period, and I changed the "append" to and "insert tail", which doubled its speed for the 76 char line length trial (it made very little difference in the "long line" versions). Newest version, including the proper binding, for others who may wish to experiment: system/schemes/pop/handler/read-til-dot: func [port buf] bind [ set-modes port [lines: false] line: system/words/copy "" while [line <> ".^M^/"][ insert tail buf line line: system/words/copy "" while [not find line "^M^/"][ a: system/words/pick port/sub-port 1 if integer? a [ insert tail line to-char a ] ] ] set-modes port [lines: true] replace/all buf "^M^/" newline buf ] in system/schemes/pop/handler 'self I certainly agree that "something" needs to be fixed, and I think we would all prefer the "/lines" buffer to be more robust in the face of less than courteous tcp programs. I think the robustness of the patch is a reasonable trade-off until a proper repair is made. Just trying to be helpful. :-)
> Since the problem is not just with pop, hopefully RT will > address this problem real soon :)
Agreed! --Scott Jones

 [25/26] from: gchiu:compkarori at: 19-Oct-2002 10:39


Hi Scott,
>I certainly agree that "something" needs to be fixed, and >I think we would
<<quoted lines omitted: 4>>
>trade-off until a proper repair is made. Just trying to >be helpful. :-)
Thanks for the Sterling efforts ( sad pun here! ). In my WIP folder, I hacked Jeff's changes to the pop scheme to allocate the correct amount of memory for the buffer rather than guessing at 1024 bytes. Would you try that, and apply your patch as well, to see if it speeds it up a little more? -- Graham Chiu

 [26/26] from: brett:codeconscious at: 19-Oct-2002 11:01


> > I think Brett has already reported this to feedback. I'll
Yes I've sent feedback about two problems: this POP problem and READ/DIRECT/LINES limit of 4094...
> Newest version, including the proper binding, for others who may wish to > experiment:
Great work, I tried your patch and it allowed me to read the "problem" email. Regards, Brett.

Notes
  • Quoted lines have been omitted from some messages.
    View the message alone to see the lines that have been omitted