Mailing List Archive: 49091 messages
  • Home
  • Script library
  • AltME Archive
  • Mailing list
  • Articles Index
  • Site search
 

Reading empty directories via FTP error

 [1/69] from: carl:cybercraft at: 24-Dec-2003 22:45


When I attempt to get a directory listing via FTP and the directory is empty I get this error...
>> read dir
** Access Error: Port none not open ** Where: parse-dir-list ** Near: read dir I would've expected it to return an empty block, or perhaps none. No problems when there's files in the directory. Any way round this, other than by examining the error returned? -- Carl Read

 [2/69] from: brett:codeconscious at: 9-Nov-2003 11:29


That's odd. I get an empty block returned when I create an empty directory on my website and then read it. Maybe you should trace the connection and see what conversation the server is having with REBOL. Regards, Brett.

 [3/69] from: carl:cybercraft at: 24-Dec-2003 22:45


On 09-Nov-03, Brett Handley wrote:
> That's odd. I get an empty block returned when I create an empty > directory on my website and then read it. > Maybe you should trace the connection and see what conversation the > server is having with REBOL.
Except I don't know how to do that. :) I get the error using different OSs, different versions of REBOL and from two different ftp servers. If it's a bug, I'm sure someone must have hit it before me. Anyway, I've found I get an empty block returned if I use a port instead of an URL, so that provides a way around it. Would've been easier using an URL though... Carl.
>> When I attempt to get a directory listing via FTP and the >> directory is
<<quoted lines omitted: 7>>
>> problems when there's files in the directory. Any way round this, >> other than by examining the error returned?
-- Carl Read

 [4/69] from: g:santilli:tiscalinet:it at: 9-Nov-2003 9:44


Hi Carl, On Sunday, November 9, 2003, 5:37:38 AM, you wrote: CR> Anyway, I've found I get an empty block returned if I use a port CR> instead of an URL, so that provides a way around it. Would've been CR> easier using an URL though... When using the URL, did you try both with and without the trailing slash? (My guess is that you get the error without the trailing slash.) Regards, Gabriele. -- Gabriele Santilli <[g--santilli--tiscalinet--it]> -- REBOL Programmer Amiga Group Italia sez. L'Aquila --- SOON: http://www.rebol.it/

 [5/69] from: carl:cybercraft at: 24-Dec-2003 22:45


On 09-Nov-03, Gabriele Santilli wrote:
> Hi Carl, > On Sunday, November 9, 2003, 5:37:38 AM, you wrote:
<<quoted lines omitted: 4>>
> slash? (My guess is that you get the error without the trailing > slash.)
Yep - tried both. And I was wrong about using a port... Grrr. Here's my current test script... rebol: [] x: err: try [read [ scheme: 'ftp host: "..." user: "..." pass: "..." path: %/d/ ]] probe x and this is the error I get if there's nothing in the d directory...
>> probe disarm err
make object! [ code: 501 type: 'access id: 'not-open arg1: "Port^@" arg2: none arg3: none near: [read [ scheme: 'ftp host: "..." user: "..." pass: "..." path: %/d/ ]] where: 'parse-dir-list ] Put a file in the d directory and it works fine! I'm picking there's something wrong with parse-dir-list, which you can look at using source, but it's code is way beyond me. -- Carl Read

 [6/69] from: gscottjones:mchsi at: 9-Nov-2003 7:37


Carl Read wrote:
> Brett Handley wrote: > > That's odd. I get an empty block returned when I create an empty > > directory on my website and then read it. > > Maybe you should trace the connection and see what conversation the > > server is having with REBOL. > Except I don't know how to do that. :)
trace/net true --Scott Jones

 [7/69] from: carl:cybercraft at: 24-Dec-2003 22:45


On 10-Nov-03, G. Scott Jones wrote:
> Carl Read wrote: >> Brett Handley wrote:
<<quoted lines omitted: 4>>
>> Except I don't know how to do that. :) > trace/net true
Ah - thanks. Didn't expect it to be so simple. So, here's the end of the trace output when it reads a directory with some files in it... Net-log: [ "SYST" "*"] Net-log: "215 UNIX Type: L8" Net-log: [ ["PORT" port/locals/active-check] "200"] Net-log: "200 PORT command successful." Net-log: [ ["CWD" either empty? port/path ["./"] [join "./" port/path]] 250 ] Net-log: {250 "/test" is new cwd.} Net-log: [ ["TYPE A"] ["200"]] Net-log: "200 Type okay." Net-log: [ ["LIST"] ["150" "125"]] Net-log: {150 Opening ASCII mode data connection for /bin/ls.} Net-log: [ none "226"] Net-log: "226 Listing completed." And here's the output when reading an empty directory... Net-log: [ "SYST" "*"] Net-log: "215 UNIX Type: L8" Net-log: [ ["PORT" port/locals/active-check] "200"] Net-log: "200 PORT command successful." Net-log: [ ["CWD" either empty? port/path ["./"] [join "./" port/path]] 250 ] Net-log: {250 "/d" is new cwd.} Net-log: [ ["TYPE A"] ["200"]] Net-log: "200 Type okay." Net-log: [ ["LIST"] ["150" "125"]] Net-log: {150 Opening ASCII mode data connection for /bin/ls.} Net-log: [ none "226"] Net-log: "226 Listing completed." Net-log: [ ["PORT" port/locals/active-check] "200"] Net-log: "200 PORT command successful." ** Access Error: Port none not open ** Where: parse-dir-list As far as I can tell, the extra... Net-log: [ ["PORT" port/locals/active-check] "200"] Net-log: "200 PORT command successful." on the end there just before the error is the only difference the trace shows. And I'm still none the wiser... -- Carl Read

 [8/69] from: brett:codeconscious at: 10-Nov-2003 16:20


Hi Carl, Try one more quick test: apply http://www.codeconscious.com/rebsite/rebol-library/patches.r or just run the FTP part of it and see if that changes anything. BTW, what version of REBOL are you having the problem with? Regards, Brett. Carl wrote:

 [9/69] from: antonr:iinet:au at: 10-Nov-2003 18:16


Ok, to dig more into the guts of it, here's what to do: Make a new rebol script file, paste this into it: -------------------- rebol [ notes: { write clipboard:// mold system/schemes/ftp/handler } ] system/schemes/ftp/handler: -------------- Now use the notes to paste the ftp-handler code in at the bottom. Now when you execute the script, you are setting the ftp handler. So that means you can put in print statements wherever you like to bug-test it. It's not so hard, you can use divide and conquer to isolate the approximate bug position. I advise you to put print statements that are distinguished easily from the trace/net output, eg: print "CR> at point 1" Anton.

 [10/69] from: carl:cybercraft at: 24-Dec-2003 22:45


On 10-Nov-03, Brett Handley wrote:
> Hi Carl, > Try one more quick test: > apply http://www.codeconscious.com/rebsite/rebol-library/patches.r > or just run the FTP part of it and see if that changes anything.
Made no difference I'm afraid. ):
> BTW, what version of REBOL are you having the problem with?
On Amiga: View 1.2.1 and Core 2.5.0. On Windows95se: View 1.2.1, View 1.25 and View 1.210. And I get the same results on two different FTP sites - paradise.net.nz and nebularis.com. Here's another console session where I make directories and examine them. ftp-site's the FTP server's URL...
>> read join ftp-site %dir/
== [%file.txt]
>> make-dir join ftp-site %dir/test-dir/
== ftp:// ... etc. ... /dir/test-dir/
>> read join ftp-site %dir/
== [%file.txt %test-dir/]
>> probe info? join ftp-site %dir/
make object! [ size: 2 date: none type: 'directory ]
>> probe info? join ftp-site %dir/test-dir/
** Access Error: Port none not open ** Where: parse-dir-list ** Near: info? join ftp-site %dir/test-dir/ Grrrr! -- Carl Read

 [11/69] from: antonr:iinet:au at: 11-Nov-2003 1:54


Carl, Don't let it stop you doing what you want to do. I sense you are frustrated by this... Any ftp reading/writing (in fact, all your errors) should be caught by your program and a window popped open to show the error. Very easy and quick to do. Your program does not need to stop running. (I am assuming that you are making a gui-based program). I will appreciate it if you find out the reason for this unexpected error though. Have a look what parse-dir-list is doing. Anton.

 [12/69] from: gscottjones:mchsi at: 10-Nov-2003 23:41


Carl Read wrote:
> Grrrr!
Grrrr is right! You were correct in that the error occurs in the parse-dir-list function in the ftp protocol. The code closes the "port/sub-port" and then tries to use that port/sub-port specification. Whoops! The corrected function is as follows (as usual watch for line wraps): parse-dir-list: func [] [ all [not passive not proxy listen-port port/sub-port: first listen-port] if all [not passive proxy listen-port] [ net-utils/accept-proxy port/sub-port ] temp: 'list file-list: make string! 2000 while [line: system/words/pick port/sub-port 1] [append file-list join line "^/"] net-utils/confirm/multiline port/locals/cmd-port transfer-check if empty? file-list [ data-connect port/locals/dir-cache: system/words/copy [] insert port/locals/cmd-port "NLST" if (first system/words/pick port/locals/cmd-port 1) <> #"5" [ while [line: system/words/pick port/sub-port 1] [ append file-list join line "^/" ] temp: 'nlist net-utils/confirm/multiline port/locals/cmd-port transfer-check ] system/words/close port/sub-port port/locals/dir-cache: parse-files file-list temp foreach temp [%./ %../] [ if loc: find port/locals/dir-cache temp [loop 2 [system/words/remove loc]] ] port/state/tail: length? port/locals/dir-cache ] My bindology skills are rusty at best, so perhaps a bindologist would be so kind as to create a patch for this. The way I do it, when I forget bindology (which is usually within two days of learning it), is to print the network scheme to disk, patch the scheme and then read it back in. One way to do this is to: 1) at a console prompt: type "echo %/path-to-file/ftp-scheme.txt" type "probe system/schemes/ftp" type "quit" 2) open the ftp-scheme.txt file and change the opening to read: system/schemes/ftp: make object! [ scheme: 'FTP host: none ... 3) replace the parse-dir-list function 4) clean the ">> " from the end of the file and save. 5) this file may be then pasted into a console or "do" the file (adding a rebol header) I look forward to seeing the "bind" version of the patch (I don't have time to refigure out the bind command again). Hope that helps! --Scott Jones

 [13/69] from: brett:codeconscious at: 11-Nov-2003 11:11


Hi Scott, Well done on isolating the error. I'm just wondering though if your corrected function has the right logic. The original function closes the port outside the IF block for "IF empty? file-list" but you have moved it inside along with the lines starting port/locals/dir-cache: So I guess I'm wondering if there needs to be another close for the case when the IF fails and also whether those extra lines for the directory cache should have been moved? I don't know one way or the other - just looking at what has changed. Regards, Brett.

 [14/69] from: brett:codeconscious at: 11-Nov-2003 12:11


I think I found where the missing bracket should be and solved my other query about the logic change. Based on this I've made a patch based on Scott's solution. It needs to be run before the FTP scheme is first used in a REBOL session. I've put it in http://www.codeconscious.com/rebsite/rebol-library/patches.r Can you confirm it works for you Carl? Regards, Brett.

 [15/69] from: carl:cybercraft at: 24-Dec-2003 22:46


On 11-Nov-03, Brett Handley wrote:
> I think I found where the missing bracket should be and solved my > other query about the logic change. Based on this I've made a patch
<<quoted lines omitted: 3>>
> http://www.codeconscious.com/rebsite/rebol-library/patches.r > Can you confirm it works for you Carl?
I'm afraid it doesn't, but Scott's solution does (after I put the missing bracket in.) Except... Except I'd got in touch with Nebularis, (one of my hosts), explaining the problem to them and asking if they got the same error using REBOL - and they did - and they fixed it: "I got the ftp listings working via REBOL. My assumption that REBOL expects certain response on an empty directory was correct. This just required a simple modification to the FTP server." Which is fine as far as it goes, except if I've implimented Scott's solution I can no longer access the Nebularis server... ** Access Error: Network timeout ** Where: confirm That may be just due to the logic error you think is in Scott's code, though. So, could you provide a copy of parse-dir-list the way you think it should be modified and I'll try it to see how it behaves. And thanks to you and Scott for your replies. It's nice to finally be getting somewhere with this. -- Carl Read

 [16/69] from: brett:codeconscious at: 11-Nov-2003 20:01


> > Can you confirm it works for you Carl? > > I'm afraid it doesn't, but Scott's solution does (after I put the > missing bracket in.) Except...
Hmmm.
> though. So, could you provide a copy of parse-dir-list the way you > think it should be modified and I'll try it to see how it behaves.
No problem. I figure the inner If block is the last value in the outer If block - the missing bracket "closes" the outer If immediately after the inner If case. That appears to cause less logic change in my eyes - but I'm only guessing. Code included below in compressed form.
> And thanks to you and Scott for your replies. It's nice to finally be > getting somewhere with this.
With pleasure - I'm glad you're making progress. Regards, Brett. decompress #{ 789CA553CB4EC330103C275FB10455C02175A9C4A542F4071017B8454532CEA6 35756CCB7628F97BD6490BE94B482045511CCFCECECCDA963B8F79295DAEA40F 33A81A2DA0584091265C2928B40960B9F7F203A1FB76E6B38588459D5BE3E80F BD986FDEBA151148E7C310B0481359C111D91151D733D118F22648E51917026D C87BDC5E933421CE80B59DC1552448934A2ADC1AA8F91AC10727F5F202A693C9 244D362BDA8642498D33F02D75ACD9C6B8D2332BC57A9F1B6E7B1DDC5AD4257C 13C3BB911A220564AF2C23013F4A85D1E4BA6675A3681D211DA53282C7DDBAEC 9983E3DA57E872B142B1EE52210FA19D0FBAC4D6250F3C274E8D82AC2543AE38 27C1A9FEC08730B6A5A9115A6A8FBBA11C0AC89E1E9F5FB208AAE0BA9FD39934 0E2B6F6FE0FE012EB3BBAC53F8A7447FCD3462A285DD68753FDBE4FF49475A7A F62353C6E3E1B93A93B5ED2E49D4ED07EAA34C3A7AC621A1BA1514A33183D178 CC7ACB943391C53B41B64F726FCB943116A650EC2974581BBA2854B2E8F5F762 030FC802976A060AF532ACE6A7A9D345FA05D04E50FEDD030000 }

 [17/69] from: atruter:labyrinth:au at: 11-Nov-2003 21:10


Hi Carl, I've just hit the same problem (1.2.10.3.1 on Windows2000), so it's not just you ;)
>> read ftp://userid:[password--www--site--com--au]/public_html/aa/
connecting to: www.site.com.au ** Access Error: Port none not open ** Where: parse-dir-list ** Near: read ftp://userid:[password--www--site--com--au]/public_html/aa/ You can "avoid" the error by wrapping it in an attempt block
>> attempt [read ftp://userid:[password--www--site--com--au]/public_html/aa/]
connecting to: www.site.com.au == none which then lets you handle empty dirs as follows: attempt [blk: read dest] if not none? blk [ foreach file blk [ ... but I am finding the whole REBOL ftp thing a bit flaky (empty dirs or not) with various port errors cropping up at weird places (almost like once a port error occurs then all downstream operations are impacted). Need to look at this a bit more before giving up on the whole excercise (trying to do a simple FTP sync script in REBOL that actually works). Regards, Ashley

 [18/69] from: gscottjones:mchsi at: 11-Nov-2003 11:47


Hi, Brett and Carl, et al, Regarding the missing bracket: I was not working at my own computer, so I was trying to manually pretty up my patch and evidently deleted a bracket. Sorry for the confusion, but I'm glad that my own error was evident. I was able to reproduce Carl's original error on an ftp site, and my suggested patch worked correctly. When I looked at Brett's patches.r file, the "new" patch does not yet show up in that file. Perhaps that is why Carl was not able to access the Nebularis server using Brett's patch file. Brett's correction of my missing bracket appears to be correct. Thanks. --Scott Jones

 [19/69] from: antonr:iinet:au at: 11-Nov-2003 23:53


Rebol ftp might not be at fault. The handler may in fact be following the ftp spec, but various servers are "interpreting" it poorly. I would need to check that though. That could be because of cached connections. If a connection errors out, when it's reused later it could be in an invalid state. Just guessing. Look here for "cache-size" : http://www.codeconscious.com/rebol/rebol-net.html Anton.

 [20/69] from: antonr:iinet:au at: 11-Nov-2003 23:53


Scott, I do not see where, after the sub-port has been closed that it is attempted to be used. Can you point it out, please? It looks to me like data-connect would reconnect the sub-port before it is used again. Anton.

 [21/69] from: greggirwin:mindspring at: 11-Nov-2003 8:50


Hi Ashley, AT> but I am finding the whole REBOL ftp thing a bit flaky (empty dirs or not) AT> with various port errors cropping up at weird places (almost like once a AT> port error occurs then all downstream operations are impacted). Need to AT> look at this a bit more before giving up on the whole excercise (trying to AT> do a simple FTP sync script in REBOL that actually works). I think the FTP protocol is part of the problem, and different implementations may be another part. I seem to recall Reichart saying that getting the FTP stuff robust in FTPGadget was their biggest issue, but not because of REBOL. -- Gregg

 [22/69] from: maximo:meteorstudios at: 11-Nov-2003 11:20


Hi, sorry for butting in late, but as mentioned in another mail, I have not followed the list for a few days... hope I am not repeating past information.
> -----Original Message----- > From: Gregg Irwin [mailto:[greggirwin--mindspring--com]]
<<quoted lines omitted: 15>>
> that getting the FTP stuff robust in FTPGadget was their biggest > issue, but not because of REBOL.
I've basically had the same kind of server-centric ugly reactions. Doing file sync engine is easy in theory, but when each ftp server replies at his own whim, its hard to make all features symmetric on any server... the RFC for ftp is filled with nasties about different implementations and its apparent by reading it that too many people "contributed" to it over such a long period of time. for example on some servers, creating directories does not map to the same paths as creating files ... because the web root and site root do not map the same with files and directories... what a shit load of fun. BUT if you find the magical url base path, you eventually find something that sort of works, but not always, again a head splitter. finally, opening the ftp port with open() and running another function (I don't recall which one, Gregg will ;-), worked well for my issues, because open forces the path in a way that you are sure the path you are opening is based on your site root and not the web root. If you're really just doing stuff for your site/workgroup... iron out the issues with your server and then you'll be happy, cause once you understand its quirks, ftp **UPLOADING** becomes easy (cause D/L is always a snap). -MAx --- You can either be part of the problem or part of the solution, but in the end, being part of the problem is much more fun.

 [23/69] from: gscottjones:mchsi at: 11-Nov-2003 16:45


Hi, Anton,
> Scott, I do not see where, after the sub-port > has been closed that it is attempted to be used. > Can you point it out, please? ...
In the while loop in the second to the last line in the snippet below: ... while [line: system/words/pick port/sub-port 1] [append file-list join line "^/"] system/words/close port/sub-port net-utils/confirm/multiline port/locals/cmd-port transfer-check if empty? file-list [ data-connect port/locals/dir-cache: system/words/copy [] insert port/locals/cmd-port "NLST" if (first system/words/pick port/locals/cmd-port 1) <> #"5" [ while [line: system/words/pick port/sub-port 1 ] [append file-list join line "^/"] ...
> It looks to me like data-connect would reconnect > the sub-port before it is used again.
You make a good point. I had not investigated that issue carefully. I intended to start over today on the issue, but I discovered that I was not able to recreate the original error. Curious. However, looking at the code, I think you raise a very good point. If I can figure out why the error no longer occurs today, I could relook at the problem. Strange. Thank you for noticing. Now if you could only figure out why the error on my test site no longer occurs.... :-) --Scott Jones

 [24/69] from: atruter:labyrinth:au at: 12-Nov-2003 10:52


OK, it can't get much simpler than this. A fresh rebol session with the following console commands:
>> site: ftp://userid:[password--www--site--com--au]/public_html/
== ftp://userid:[password--www--site--com--au]/public_html/
>> write site/test/a "a"
connecting to: www.site.com.au
>> write site/test/b "b"
** User Error: Server error: FTP 226-File successfully transferred ** Near: write site/test/b "b"
>> read site/test
connecting to: www.site.com.au ** User Error: Server error: tcp 550 I can only retrieve regular files ** Near: read site/test It makes no difference whether the test directory is empty to start with or not, or what order the above commands are given in. The first [write] command works, subsequent ones fail. I've also tried: system/schemes/ftp/passive: true and / or cache-size: 0 with no effect. Any ideas? Regards, Ashley

 [25/69] from: brett:codeconscious at: 12-Nov-2003 11:08


Hi Scott, No, I deleted the patch from patches.r as soon as Carl said it didn't work - I should have delivered the patch in email in the first place until it was confirmed. Regards, Brett.

 [26/69] from: brett:codeconscious at: 12-Nov-2003 11:13


Max wrote: If you're really just doing stuff for your site/workgroup... iron out the issues with your server and then you'll be happy, cause once you understand its quirks, ftp **UPLOADING** becomes easy (cause D/L is always a snap). I agree, my view based website upload scripts are just click, enter password and Ok and work very reliably for my needs - just based on the straight URL form of FTP read and write. Regards, Brett.

 [27/69] from: brett:codeconscious at: 12-Nov-2003 11:14


Hi Ashley, For your errors, the 226 error looks like something my patch file fixes. But as you have not mentioned trying it, I'll repeat the URL again: http://www.codeconscious.com/rebsite/rebol-library/patches.r Regards, Brett.

 [28/69] from: atruter:labyrinth:au at: 12-Nov-2003 12:09


> Hi Ashley, > > For your errors, the 226 error looks like something my patch file fixes. > But > as you have not mentioned trying it, I'll repeat the URL again: > http://www.codeconscious.com/rebsite/rebol-library/patches.r
No luck ;(
>> write site/test/a2 "a"
URL Parse: userid password www.site.com.au none public_html/test/ a2 Net-log: ["Opening" "tcp" "for" "FTP"] connecting to: www.site.com.au Net-log: [ none ["220" "230"]] Net-log: {220---------- Welcome to Pure-FTPd 1.0.14 ----------} Net-log: "220-You are user number 1 of 50 allowed." Net-log: {220-Local time is now 11:53 and the load is 1.11. Server port: 21.} Net-log: "220-This is a private system - No anonymous login" Net-log: {220 You will be disconnected after 30 minutes of inactivity.} Net-log: [ ["USER" port/user] "331"] Net-log: "331 User userid OK. Password required" Net-log: [ ["PASS" port/pass] "230"] Net-log: "230-Your bandwidth usage is restricted" Net-log: "230-User userid has group access to: 2000 " Net-log: "230 OK. Current directory is /" Net-log: [ "SYST" "*"] Net-log: "215 UNIX Type: L8" Net-log: [ ["PORT" port/locals/active-check] "200"] Net-log: "200 PORT command successful" Net-log: [ ["CWD" either empty? port/path ["./"] [either slash = port/path/1 [port/path ] [join "./" port/path]]] "25*"] Net-log: "250 OK. Current directory is /public_html/test" Net-log: [ ["TYPE I"] ["200"]] Net-log: "200 TYPE is now 8-bit binary" Net-log: [ ["STOR" port/target] ["150" "125"]] Net-log: "150 Connecting to port 1263" Net-log: ["low level write of " 1 "bytes"] Net-log: [ none "226"] Net-log: "226-66555.9 Mbytes free disk space"
>> write site/test/b2 "b"
URL Parse: userid password www.site.com.au none public_html/test/ b2 ** User Error: Server error: FTP 226-File successfully transferred ** Near: write site/test/b2 "b" I'm wondering if it isn't just easier to [under Windows] do a "Map Network Drive" and then work on the remote file systems directly. Anyone had any luck with this? Regards, Ashley

 [29/69] from: brett:codeconscious at: 12-Nov-2003 14:01


Hi Ashley
> No luck ;(
Bugger.
> Net-log: [ > none "226"]
<<quoted lines omitted: 3>>
> ** User Error: Server error: FTP 226-File successfully transferred > ** Near: write site/test/b2 "b"
This snippet from your log looks to me like the Net-utils/confirm function has not been called with the multiline refinement. The only place I can find this happening is in the Close function of the FTP protocol - but I'm not sure it will help. Use a new session and paste this into your console, then retry your tests please. code: second get in system/schemes/ftp/handler 'close if equal? code/5/3 'net-utils/confirm [ head insert tail code/5/3 'multiline ] I'm not convinced that is the problem yet, but I have no other leads to follow right now.
> I'm wondering if it isn't just easier to [under Windows] do a "Map Network > Drive" and then work on the remote file systems directly. Anyone had any > luck with this?
I've no experience with that. Regards, Brett.

 [30/69] from: atruter:labyrinth:au at: 12-Nov-2003 15:01


Hi Brett,
> code: second get in system/schemes/ftp/handler 'close > if equal? code/5/3 'net-utils/confirm [ > head insert tail code/5/3 'multiline > ]
That did the trick! Thanks for saving my sanity ;) Regards, Ashley

 [31/69] from: brett:codeconscious at: 12-Nov-2003 15:11


Ah good. I'll add it to my patches.r and send it to feedback. Regards, Brett.

 [32/69] from: carl:cybercraft at: 24-Dec-2003 22:46


On 11-Nov-03, Brett Handley wrote:
>>> Can you confirm it works for you Carl? >>
<<quoted lines omitted: 8>>
> logic change in my eyes - but I'm only guessing. Code included below > in compressed form.
No go I'm afraid. I get the same results as with Scott's code, in that it works on the one site but not on the fixed Nebularis site. Anyway, I'll examine both your versions and the original and see if I can come up with something that works. Will get back to you on this. Going by the other posts to this thread, it looks like the problem is too loose FTP specs, rather than REBOL. -- Carl Read

 [33/69] from: g:santilli:tiscalinet:it at: 12-Nov-2003 10:20


Hi Ashley, On Wednesday, November 12, 2003, 2:09:56 AM, you wrote: AT> Net-log: "226-66555.9 Mbytes free disk space"
>>> write site/test/b2 "b"
AT> URL Parse: userid password www.site.com.au none public_html/test/ b2 AT> ** User Error: Server error: FTP 226-File successfully transferred AT> ** Near: write site/test/b2 "b" My guess is that the server is sending TWO lines of answer, i.e.: 226-66555.9 Mbytes free disk space 226-File successfully transferred but REBOL only expects and picks one. On the subsequent operation, it picks the second line, but of course this confuses it because it is not the answer it is expecting. I don't think this has an easy solution, because you have no way to know how many lines the server is sending as answer; you can patch the protocol for this particular server only by waiting for two lines of answer instead of one, but then it won't work on other servers. Regards, Gabriele. -- Gabriele Santilli <[g--santilli--tiscalinet--it]> -- REBOL Programmer Amiga Group Italia sez. L'Aquila --- SOON: http://www.rebol.it/

 [34/69] from: brett:codeconscious at: 12-Nov-2003 21:48


Having got success with Ashley's problem, I've had another stab at Carl's and I think I may have it - but I can't duplicate the original problem so I'm running blind here. First I think Scott was right about the port. Second I think Anton was right about data-connect. I think that data-connect creates a listen-port but the code to create the sub-port is missing for the case where NLST is issued. That's why the port none error will occur. If I'm right, then I don't see how the NLST code ever worked :-/ Carl and Ashley could you incorporate the code below and test it please: Regards, Brett. decompress #{ 789CA5554D8FD330103D27BF62085A150E69BA455C0A62A5959040425CD85B14 24D79936A68E6DD9CE96FE7BC64E7749FA214AB787AAB667DECCBC793335CC3A CC6B6173299C5FC0AA531CCA0ACA344D9894502AEDC130E7C42342FC6DF5EF1D 046354B9D1966EE8AB70DD329E084158E78706559A88151C811D0185A049A2D0 E79D17D2158C73343EEFED4641D284303DB66601930090262B21715F41CB3608 CE5BA1D6AF603E9BCDD264DBD0339452285C80DB51C4B6D86A5BBBC208BE1963 C32DE5C18C4155C3332AFCD24241F087EC67915544CE08864BEDF030C7BF9570 AD8895B6683B49E780124DA5E62CBCB6751FD95BA6DC0A6DCE1BE49B34D24645 FADDDD2093D098A4669EE584AA90FB70FE00F9159FE8788F6B2AEDFE0BDCCE73 A51F7362EC5D7C786830AAC10BAD6018101C7A3752C0B2F3411BD16D44C234A2 705D1394EE8851C93C5AA0809EEE3591FC1C223ABB4677B286250275C06A6305 D983D3F075428AA93B2305A79B1A8487062D4EAFAF9D3C47AA8C2A1BEAD3EF0C C2C74F3051B80D03D2CB33F997DA83CD096D1C580C8E0B8AAA902EAB414A17A7 73D9B89C0BF832E97CA6869E10CEB5ED18CE4458489CD11C1CCC2BD76647EB29 30A51C3E2D9FC341CABE7FFBF190F574BEE93B7466EA0F3D6FDF068E5F67EFB3 9EDEFFDB1CC1E392ED4166FBEDA5FAF5755232473D7CE94289120B4D3FC3B489 FF05216D37483E644A0B565B24AB7882F2665AC0CD745AF4222496092CCC0255 7D127BEF26B53630877254ACC55693BEC9A58AE9ED27CCD39C179E09B900896A ED9BBBD3D0695AA57F000CD92034C5060000 }

 [35/69] from: brett:codeconscious at: 12-Nov-2003 21:56


Hi Gabriele,
> My guess is that the server is sending TWO lines of answer, i.e.: > > 226-66555.9 Mbytes free disk space > 226-File successfully transferred
Spot on.
> but REBOL only expects and picks one. On the subsequent operation, > it picks the second line, but of course this confuses it because > it is not the answer it is expecting.
Yes.
> I don't think this has an easy solution, because you have no way > to know how many lines the server is sending as answer; you can > patch the protocol for this particular server only by waiting for > two lines of answer instead of one, but then it won't work on > other servers.
No, it turns out to be easy. You may have missed my email with the solution - I didn't get it sent back to me. But you can see it on the REBOL website list page here: http://www.rebol.net/list/list-msgs/32665.html Mulitple lines are indicated by the "-" just after the 3 digit response code sent back by the server - its like a continuation flag ie "there are more lines coming". REBOL has the code built-in to handle multiple lines (net-utils/confirm/mulitple) - it just wasn't turned on. Regards, Brett.

 [36/69] from: g:santilli:tiscalinet:it at: 12-Nov-2003 12:21


Hi Brett, On Wednesday, November 12, 2003, 11:56:19 AM, you wrote: BH> You may have missed my email with the solution - I didn't get it sent back BH> to me. But you can see it on the REBOL website list page here: Yup, saw it after sending my mail. I didn't recall about /multiline at the moment. BH> Mulitple lines are indicated by the "-" just after the 3 digit response code BH> sent back by the server - its like a continuation flag ie "there are more BH> lines coming". REBOL has the code built-in to handle multiple lines BH> (net-utils/confirm/mulitple) - it just wasn't turned on. I wonder, does that mean that the server is actually sending three lines at least? I see the - in the second line too. (I'd have to check the RFCs on FTP etc, at least before talking about them... ;) Regards, Gabriele. -- Gabriele Santilli <[g--santilli--tiscalinet--it]> -- REBOL Programmer Amiga Group Italia sez. L'Aquila --- SOON: http://www.rebol.it/

 [37/69] from: brett:codeconscious at: 12-Nov-2003 22:34


> I wonder, does that mean that the server is actually sending three > lines at least? I see the - in the second line too. (I'd have to > check the RFCs on FTP etc, at least before talking about them...
I would say so - what a chatty server! :-) This from RFC 959: Thus the format for multi-line replies is that the first line will begin with the exact required reply code, followed immediately by a Hyphen, "-" (also known as Minus), followed by text. The last line will begin with the same code, followed immediately by Space <SP>, optionally some text, and the Telnet end-of-line code. So, my comment was a little misleading the "-" begins multiple lines the " " is on the last line. What is in between is a mystery. Regards, Brett

 [38/69] from: carl:cybercraft at: 24-Dec-2003 22:46


On 12-Nov-03, Brett Handley wrote:
> Having got success with Ashley's problem, I've had another stab at > Carl's and I think I may have it - but I can't duplicate the
<<quoted lines omitted: 7>>
> Carl and Ashley could you incorporate the code below and test it > please:
Well, it works okay with directories that are empty or contain files on the fixed Nebularis site, and is okay on the other site for directories that contain files, but I get nothing back if the directory is empty - have to press Escape to get the Console back. This is what trace outputs... Net-log: [ "SYST" "*"] Net-log: "215 UNIX Type: L8" Net-log: [ ["PORT" port/locals/active-check] "200"] Net-log: "200 PORT command successful." Net-log: [ ["CWD" either empty? port/path ["./"] [join "./" port/path]] 250 ] Net-log: {250 "/dir/test-dir" is new cwd.} Net-log: [ ["TYPE A"] ["200"]] Net-log: "200 Type okay." Net-log: [ ["LIST"] ["150" "125"]] Net-log: {150 Opening ASCII mode data connection for /bin/ls.} Net-log: [ none "226"] Net-log: "226 Listing completed." Net-log: [ ["PORT" port/locals/active-check] "200"] Net-log: "200 PORT command successful." Nothing more comes after that, but I'm not given the console back either - as I said, I need to press Escape. I've still not had time to play with parse-dir-list myself. Will get there though. Carl.
> Regards, > Brett.
<<quoted lines omitted: 19>>
> ED9BBBD3D0695AA57F000CD92034C5060000 > }
-- Carl Read

 [39/69] from: atruter:labyrinth:au at: 13-Nov-2003 13:13


Hi Brett,
> Carl and Ashley could you incorporate the code below and test it please:
No luck with the "empty dir" problem. ;(
>> URL Parse: userid password www.site.com.au none public_html/test3/ none
Net-log: ["Opening" "tcp" "for" "FTP"] connecting to: www.site.com.au Net-log: [ none ["220" "230"]] Net-log: {220---------- Welcome to Pure-FTPd 1.0.14 ----------} Net-log: "220-You are user number 2 of 50 allowed." Net-log: {220-Local time is now 13:08 and the load is 1.29. Server port: 21.} Net-log: "220-This is a private system - No anonymous login" Net-log: {220 You will be disconnected after 30 minutes of inactivity.} Net-log: [ ["USER" port/user] "331"] Net-log: "331 User userid OK. Password required" Net-log: [ ["PASS" port/pass] "230"] Net-log: "230-Your bandwidth usage is restricted" Net-log: "230-User userid has group access to: 2000 " Net-log: "230 OK. Current directory is /" Net-log: [ "SYST" "*"] Net-log: "215 UNIX Type: L8" Net-log: [ ["PORT" port/locals/active-check] "200"] Net-log: "200 PORT command successful" Net-log: [ ["CWD" either empty? port/path ["./"] [either slash = port/path/1 [port/path] [join "./" port/path]]] "25*"] Net-log: "250 OK. Current directory is /public_html/test3" Net-log: [ ["TYPE A"] ["200"]] Net-log: "200 TYPE is now ASCII" Net-log: [ ["LIST"] ["150" "125"]] Net-log: "150 Connecting to port 1099" Net-log: [ none "226"] Net-log: "226-Options: -l " Net-log: "226 0 matches total" Net-log: [ ["PORT" port/locals/active-check] "200"] Net-log: "200 PORT command successful" ** Access Error: Port none not open ** Where: parse-dir-list ** Near: read join site %test3/
>>
Regards, Ashley

 [40/69] from: brett:codeconscious at: 13-Nov-2003 14:51


> > Carl and Ashley could you incorporate the code below and test it please: > > No luck with the "empty dir" problem. ;(
Carl and Ashley - thanks for testing my attempt at a patch. I think that parse-dir-list function now needs a ridiculous amount of debugging statements added to identify the exact error point and a port guru added to interpret them! :-) Regards, Brett

 [41/69] from: carl:cybercraft at: 24-Dec-2003 22:47


On 13-Nov-03, Carl Read wrote:
> I've still not had time to play with parse-dir-list myself. Will get > there though.
Okay - have now. I found the difference between the empty directories on the two sites was that Nebularis was returning something so file-list wasn't an empty string, unlike the other site. Changing the original parse-dir-list from... ... system/words/close port/sub-port net-utils/confirm port/locals/cmd-port transfer-check if empty? file-list [ ... to... ... if not empty? file-list [system/words/close port/sub-port] net-utils/confirm port/locals/cmd-port transfer-check if empty? file-list [ ... was enough to get it working on both sites. However, that's just my sites and in cases where the second if wasn't acted on system/words/close port/sub-port would never happen, so I changed the second if to an either to ensure it did. ie, from this... if (first system/words/pick port/locals/cmd-port 1) <> #"5" [ while [line: system/words/pick port/sub-port 1 ] [append file-list join line "^/"] temp: 'nlist system/words/close port/sub-port net-utils/confirm/multiline port/locals/cmd-port transfer-check ] to this... either (first system/words/pick port/locals/cmd-port 1) <> #"5" [ while [line: system/words/pick port/sub-port 1 ] [append file-list join line "^/"] temp: 'nlist system/words/close port/sub-port net-utils/confirm/multiline port/locals/cmd-port transfer-check ][ system/words/close port/sub-port ] I tried having "system/words/close port/sub-port" only at the end of and outside the two if blocks but this didn't work. I see this is how your code worked Brett, so I assume that was the problem with it. It's very hard to see why it shouldn't work that way, other than the port needs to be closed before net-utils/confirm/multiline is acted on. I've no idea what's really happening here, so whether that's true or not I don't know. Anyway, here's what works for me... decompress #{ 789C9D544D4FE33010BDF7570C5921E090BA20EDA542F007567BD9BD454132CE 949A3AB6654F97EDBFDFB1D384A401C1E243555B6FDE7BF3152F43C4B2D1A134 3AD21A367BABA0AAA15A001F690C54D6117819A3FE8390FF07F7F700098EB6F4 2EF00BFF88B87FCC37E6D021D218506732BD99F3CDB87AE5742C52B9276DA290 4AA1A7B2434FD432B8E3276CFD1A2E1259BE6FB4C16356ADDC21440ADA3E9DC1 CD6AB5CA80972D23A032DAE21AE2813DB4E2C585260AAFD56EAA03D7EC4C7A8F B679258667A72DA478281E4431A49932643374B81F61AB8982322EE254A20B7F 4D5A39CB956C3B90714AA6B7B6E9EC5090366E30946A8B6AD70BCF4587623692 64C99416150D8F63EA34034A32DD492D94F3079E882146DB887DCF4F5D153F7F FCFA5D0C50D4B4C50097DD40BC53E1538EEB2BB8BD836FC5F762E4FEFFDB3509 FD6CEFFA731C253BCC527F3EEAE1043C6BA568F786EF49F3B34DCDEEABAF79A8 47CBF14EA77D5EFF5494382A4D4ABF5B21179081F901AAF3A580F3E5528C7794 878E59D3CE7371DF1439061BE73CDC9C6C41C0D6F1578043EA99D948925090D4 660D06ED136DEFDF1658D48BC53F76FE885BC7040000 } Does that work for you, Ashley? And Brett? And any others who've been testing this? -- Carl Read

 [42/69] from: atruter:labyrinth:au at: 13-Nov-2003 16:36


Hi Brett,
> Carl and Ashley - thanks for testing my attempt at a patch. I think that > parse-dir-list function now needs a ridiculous amount of debugging > statements added to identify the exact error point and a port guru added > to interpret them! :-)
Sounds like a job for . . . RT perhaps? Anyway, I'm not that fussed about it now as the multiline fix was the show-stopper for me (I can at least work around the empty dir problem). Hi Carl,
> Does that work for you, Ashley?
No, but see above. Regards, Ashley

 [43/69] from: brett:codeconscious at: 13-Nov-2003 16:40


Hi Carl,
> Okay - have now. I found the difference between the empty directories > on the two sites was that Nebularis was returning something so > file-list wasn't an empty string, unlike the other site. Changing > the original parse-dir-list from...
So, the Nebularis admin has changed his FTP server such that you get back a string, which when parsed by REBOL will give you an empty file list, and therefore sidesteps the NLST code. Your other server is not returning anything so, in this the original REBOL code would attempt to get a directory listing a second time by issuing a NLST command and waiting for the response.
> was enough to get it working on both sites. However, that's just my > sites and in cases where the second if wasn't acted on > "system/words/close port/sub-port" would never happen, so I changed > the second if to an either to ensure > it did. ie, from this...
I'm glad you got something working, however, I think your code might be working for the wrong reasons but I'm not sure of that.
> I tried having "system/words/close port/sub-port" only at the end of > and outside the two if blocks but this didn't work. I see this is > how your code worked Brett, so I assume that was the problem with it.
Yes - the first attempt
> It's very hard to see why it shouldn't work that way, other than the > port needs to be closed before net-utils/confirm/multiline is acted
Yes, I think that is right. When the sub-port (the data connection) is closed, the server may send an informative message via the command port about how much was transferred - which in turn is handled by the transfer-check. I am becoming more sure that the code for NLST in the PARSE-DIR-LIST function is a temporary and very unsightly, putting it mildy, hack job to fix some obscure error in the past and which has finally fallen over. It doesn't even support network tracing like the rest of the code. Sheesh. In any case, the code is hard to follow when trying to mentally prove it. It needs more attention. Regards, Brett.

 [44/69] from: brett:codeconscious at: 13-Nov-2003 16:46


> > Does that work for you, Ashley? > > No, but see above.
Hmm.. Thinking more on this, given the whole NLST handling (if empty? file-list [....]) is a second bite of the cherry at getting a directory list, and if you can assume that your FTP server will always respond to the LIST command in a way that REBOL can parse, you could probably cut the whole NLST handling from parse-dir-list and happily ignore most of this thread. :-) Regards, Brett.

 [45/69] from: carl:cybercraft at: 24-Dec-2003 22:47


On 13-Nov-03, Ashley Trüter wrote:
> Hi Brett, >> Carl and Ashley - thanks for testing my attempt at a patch. I think
<<quoted lines omitted: 7>>
>> Does that work for you, Ashley? > No, but see above.
Oh well. Incidentally, I just added prints and probes to parse-dir-list to get an idea of what was happening. Probing file-list was the biggest help. -- Carl Read

 [46/69] from: carl:cybercraft at: 24-Dec-2003 22:47


On 13-Nov-03, Brett Handley wrote:
>>> Does that work for you, Ashley? >>
<<quoted lines omitted: 6>>
> you could probably cut the whole NLST handling from parse-dir-list > and happily ignore most of this thread. :-)
Quite possibly, since I've just tried it and it works for me. ;) Try that Ashley - cut the "if empty? file-list [....]" code right out of the original parse-dir-list and see what happens... Has anyone sent RT feedback about this? As it does seem as though parse-dir-list could attempt to close the same port twice, right? Which at least shouldn't happen, never mind the rest of the code, right? -- Carl Read

 [47/69] from: rotenca:telvia:it at: 13-Nov-2003 11:47


Hi, One of many problems of the ftp handler is that it does create directory outside the root: when you do make-dir ftp://www.bbb.ccc/main/sub is created the dir: sub/ and not the dir: main/sub/ One patch can be this: mkdir-check: [ ["MKD" head system/words/remove back tail join port/path port/target] 25 ] --- Ciao Romano

 [48/69] from: brett:codeconscious at: 13-Nov-2003 23:19


Hi
> > Thinking more on this, given the whole NLST handling (if empty? > > file-list [....]) is a second bite of the cherry at getting a
<<quoted lines omitted: 3>>
> > and happily ignore most of this thread. :-) > Quite possibly, since I've just tried it and it works for me. ;)
Ahah. So I was on the right track. Not to be beaten by this - I've hammered away at it and eventually built my own dodgy FTP server to replicate the problem and get a different perspective on things. I think I see light at the end of the tunnel!! - or it could be an oncoming heavy vehicle.... Playing with my dodgy server I've found that my last fix - was soooo close, but I screwed up by closing the listen-port - I should have left it open. Here then is some code that makes my dodgy server happy. While I was at it, I added a net-log call to show the hidden NLST communication. Please tell me that's the end of the tunnel :-) Regards, Brett. decompress #{ 789CA554D16ED330147D4EBEE21034010F69BA4D7BE910D3AA4D8084CA038897 A848AEE3B4A68E1DD9EE4AFE7ED74981A64D51115155C5F63DE7DE7B6E8E6B66 9D480B6953259D9FA0DC688E7C8E3C8E9852C8B5F1A89973F249A07DB7E66783 102B745A1B4B3BF497B9CDA25D1181B4CEEF07CCE3489638223B226A73465AF8 74E3A57219E35CD43EEDE27A49E28838BDA8EA095E0582382AA512BB062AB616 70DE4ABD7C81ABF1781C47DB151D2357528B095C4319AB6C6B6CE1B25AF2759F 1B975407AB6BA10BFC66C50F2335021EC9F72CA1EC3D16AE8C138725FE69841B 4DA25459B551B40E246DA8329C85D3AAE8127BCBB42B854DF94AF0752B1AB5E8 9BBBBD3A824205F32C254E2D38A589F6B9C2183923FC419BDCD40D0D35EEE91B DE9459224F669FBE7C4D90DCEBC6AF48362C361E0C37E3319CB05238086B8D4D E6B8C5F4E1032EAFD3D9E76F29697B1D18A5A6283FDC53471D824ABCEEBE8C13 FA1F222FDFE0ED3BBC4C6E126A9A08A25BA4070FA6BD5A307D7CFF7176149576 E0FBA20037858037F02B81A022762A4AA3914248DAB75834D832E949870E581A 1B6478A22382EE10A0CDD29A8A646ABFCFD17085E73E6D83432619745C6794DD E4FF62BC10343FC57C92F53C0306E6133D1F8EE571F630204C8BFE3763B659CF 3167F4EB72D0DDED109D61D7E8FF0D1BC4A6DF0943D6ED451BAA767BB58742E9 FA32565054BB427E31CA70311A65DD4468784416C64B4D0F72EF60CA981A57C8 7BBD5A51199A3641E67BD539CFBCC83C936A0225F4D2AFEE86A9639AF2339E27 F2C122060000 }

 [49/69] from: brett:codeconscious at: 13-Nov-2003 23:41


Hi Romano, That's interesting. I may have encountered it in the past and forgot about it as my routines are probably ensuring each level is created before going on to the next. I tried briefly to replicate the problem but couldn't - I got a different error! Regards, Brett.

 [50/69] from: rotenca:telvia:it at: 13-Nov-2003 14:14


Hi, another problem of the ftp handler is error handling (see http://www.rebol.net/list/list-msgs/32024.html) one patch i tested a little is this: old: repeat x length? connections [ conn: system/words/pick connections x if all [conn conn/host = temp conn/user = port/user not error? catch [get-cur-dir conn]] [ patched: repeat x length? connections [ conn: system/words/pick connections x if all [ conn conn/host = temp conn/user = port/user ;romano not error? catch [throw-on-error [get-cur-dir conn]] ] this kind of patch can be used also in other points of the handler code, but i did not tested it (some erros must happen other errors must not), in this specific case no error must be throw back to the calling code. --- Ciao Romano

 [51/69] from: rotenca:telvia:it at: 13-Nov-2003 14:37


Hi, i forgot to say that this patch:
> patched: > repeat x length? connections [
<<quoted lines omitted: 3>>
> not error? catch [throw-on-error [get-cur-dir conn]] > ]
should fix some error problems (not all) of the kind: port none not open --- Ciao Romano

 [52/69] from: rotenca:telvia:it at: 13-Nov-2003 14:45


Hi Brett,
> I tried briefly to replicate the problem but couldn't - I got a different > error!
What error? My error manifests itself only under some conditions (i am not able to say exactly what conditions). I think that they depend from cached ftp port, but i am not sure. With a freshly opened shell and new ftp port it should not happen. I think that the initial Ashley error with empty dirs was due to a not existing dir + cached ftp port. --- Ciao Romano

 [53/69] from: carl:cybercraft at: 24-Dec-2003 22:47


On 14-Nov-03, Brett Handley wrote:
> Hi >> > Thinking more on this, given the whole NLST handling (if empty?
<<quoted lines omitted: 16>>
> NLST communication. > Please tell me that's the end of the tunnel :-)
Well - I came out the other side without a scratch, so, daylight for me! No problems at all on either server. Good work, Brett! But what say Ashley? -- Carl Read

 [54/69] from: brett:codeconscious at: 14-Nov-2003 10:10


Hi Romano,
> another problem of the ftp handler is error handling (see > http://www.rebol.net/list/list-msgs/32024.html)
I see your point about the error handling code in the FTP protocol - it does not look robust to me. As I understand it, Catch does not trap errors at all, yet the FTP protocol code has "error? catch" in multiple places through the code. As users we can wrap our FTP stuff in a Try or Attempt to protect ourselves, but still the FTP code may fall apart unreasonably early.
> > I tried briefly to replicate the problem but couldn't - I got a
different
> > error! > > What error?
I think the FTP server was complaining about the directory not existing - I haven't traced the problem, but I did find that at some point I couldn't then do previously working commands.
> My error manifests itself only under some conditions (i am not able to say > exactly what conditions). I think that they depend from cached ftp port,
but i
> am not sure. With a freshly opened shell and new ftp port it should not > happen.
Ok.
> I think that the initial Ashley error with empty dirs was due to a not > existing dir + cached ftp port.
Ashley's 226 error was solved (see patch in REBOL library). Ashley also had the same error as Carl's (NLST) - which hopefully now is solved. Ashley's third problem was a general one about having had an FTP error, FTP could become unusable for the rest of the session - which I think you have identified in your error handling comments. Regards, Brett.

 [55/69] from: maximo:meteorstudios at: 13-Nov-2003 18:40


> Ashley's > third problem was a general one about having had an FTP > error, FTP could > become unusable for the rest of the session - which I think you have > identified in your error handling comments.
I've had this problem EVERY time I play around with the ftp port spec. whenever the ftp goes bad, you have to restart rebol. has this been solved in your various patchwork? -MAx

 [56/69] from: brett:codeconscious at: 14-Nov-2003 10:48


Romano identified a cause - I've found four occurrences of it in the protocol and I think I'll have a new patch to test shortly. Hoping to count you in on that :-) Regards, Brett.

 [57/69] from: atruter:labyrinth:au at: 14-Nov-2003 10:59


Brett,
> Please tell me that's the end of the tunnel :-)
No luck ;( Carl,
> Try that Ashley - cut the "if empty? file-list [....]" code right out > of the original parse-dir-list and see what happens...
Also no luck Romano,
> repeat x length? connections [ > conn: system/words/pick connections x
<<quoted lines omitted: 4>>
> ] should fix some error problems (not all) of the kind: > port none not open
Those are the symptons alright, but where / how exactly should this patch be applied? Regards, Ashley

 [58/69] from: rotenca:telvia:it at: 14-Nov-2003 2:34


Hi Brett,
> > another problem of the ftp handler is error handling (see > > http://www.rebol.net/list/list-msgs/32024.html) > > I see your point about the error handling code in the FTP protocol - it does > not look robust to me.
In the connections case is robust. In the other cases, tests must be done to see if it is needed (i do not know) and if it is robust.
> As I understand it, Catch does not trap errors at all, yet the FTP protocol > code has "error? catch" in multiple places through the code. As users we > can wrap our FTP stuff in a Try or Attempt to protect ourselves, but still > the FTP code may fall apart unreasonably early.
Catch trap errors, but only throwed errors. What errors are throwed in a protocol? All errors handled by the net-error helper. Here it is the source: net-error: func [info][throw make error! info] try this: error? catch [throw net-error ""] ;==true error? catch [throw-on-error [2 / 0]] ; == true But if an inner catch prevents the throw and fires the error, it will not reach our catch statement and we'll not able to catch the error. A function with the [catch] attribute catches all throwed errors and then return it without throwing it again. error? catch [do func [[catch]][throw-on-error [2 / 0]]] ** Math Error: Attempt to divide by zero ** Where: throw-on-error ** Near: [throw-on-error [2 / 0]] Every function which handles ports has an hidden [catch] attribute, so when the protocol handler throws the error, they can catch it and show, in the near error field, the user line. In the case of connections handling, we must detect every error, also errors already catched, else we'll use an invalid port.
> > > I tried briefly to replicate the problem but couldn't - I got a > different
<<quoted lines omitted: 4>>
> haven't traced the problem, but I did find that at some point I couldn't > then do previously working commands.
The same happened to me about previously working commands. But it is not easy to replicate.
> > I think that the initial Ashley error with empty dirs was due to a not > > existing dir + cached ftp port. > > Ashley's 226 error was solved (see patch in REBOL library). Ashley also had
The multiline error. Ok for the patch. But it seems to me that there are other points of code where the multiline refinement is not used. What say the RFC about multiline: every ftp command can be multiline?
> the same error as Carl's (NLST) - which hopefully now is solved.
I have many doubts about this. The doubts are: 1) why to use NLIST? some servers do not implement LIST? 2) If LIST is not implemented, the server should return an error, not an empty list of files. 3) The NLIST command is not sent in the same mode as LIST. The directory name is not sent, like is sent in LIST. So i think that can exists other errors about NLIST code. You patched the connection problem but I think it can be not the only one.
> Ashley's > third problem was a general one about having had an FTP error, FTP could > become unusable for the rest of the session - which I think you have > identified in your error handling comments.
This was the Ashley error I also had. --- Ciao Romano

 [59/69] from: brett:codeconscious at: 14-Nov-2003 13:52


> > As I understand it, Catch does not trap errors at all, yet the FTP
protocol
> > code has "error? catch" in multiple places through the code. As users
we
> > can wrap our FTP stuff in a Try or Attempt to protect ourselves, but
still
> > the FTP code may fall apart unreasonably early. > > Catch trap errors, but only throwed errors.
I think that Catch catches Throws only.
> What errors are throwed in a protocol? > > All errors handled by the net-error helper. Here it is the source: > > net-error: func [info][throw make error! info]
Ah this is interesting. View 1.2.10 does not have the throw. I've only realised this by re-reading your email after I created ftp-error-handling patch (see other email on this thread). So without the throw I thought to myself "this ERROR? CATCH thing is useless - I should replace the CATCH with a TRY". This is what the patch file does.
> Every function which handles ports has an hidden [catch] attribute, so
when That's interesting.
> In the case of connections handling, we must detect every error, also
errors
> already catched, else we'll use an invalid port.
Hmm.
> The same happened to me about previously working commands. But it is not
easy
> to replicate.
I finally found a way to replicate a problem - at least for myself - see other email.
> But it seems to me that there are other points of code where the multiline > refinement is not used. > > What say the RFC about multiline: every ftp command can be multiline?
I'm not sure - but testing for multiline on every FTP command probably wouldn't hurt.
> I have many doubts about this. The doubts are: > > 1) why to use NLIST? some servers do not implement LIST? > > 2) If LIST is not implemented, the server should return an error, not an
empty
> list of files. > > 3) The NLIST command is not sent in the same mode as LIST. The directory
name
> is not sent, like is sent in LIST.
Excellent points. REBOL parses the directory list returned by LIST - what if the Parse itself fails because the parse rules are not appropriate for the listing? This might have been a good reason to use NLST which will only return filenames not attributes. So maybe the test for an empty file-list should have been a test to see that the parse of the directory listing actually worked? In this way if LIST returns nothing and parse of nothing is ok, then NLST does not happen, it will be only when LIST returns something we cannot decipher that NLST will be used.
> So i think that can exists other errors about NLIST code.
Quite possibly.
> You patched the connection problem but I think it can be not the only one.
Ouch :-) It took me ages to find that one. But that connection problem was because the code for it was completely missing. Regards, Brett.

 [60/69] from: brett:codeconscious at: 14-Nov-2003 14:43


Hi Romano,
> But if an inner catch prevents the throw and fires the error, it will not > reach our catch statement and we'll not able to catch the error.
I have to think more about this. As I wrote before the throw does not seem to be in net-error for the latest REBOLs. So this issue may become one of backward compatibility maybe. Regards, Brett.

 [61/69] from: rotenca:telvia:it at: 14-Nov-2003 12:40


Hi Brett,
> > > As I understand it, Catch does not trap errors at all, yet the FTP > protocol
<<quoted lines omitted: 6>>
> > Catch trap errors, but only throwed errors. > I think that Catch catches Throws only.
Yes, thowed errors also. See in get-cur-dir: net-error reform ["Server error:" port/scheme cd] the error is thrown, so if you use try, the error by-pass your try:
>> error? try [net-error ""]
** Throw Error: ** User Error: ** Near: throw make error! info These are two solutions:
>> error? try [catch[net-error ""]]
== true
>> error? catch[throw-on-error [net-error ""]]
== true
> So without the throw I thought to myself "this ERROR? CATCH thing is > useless - I should replace the CATCH with a TRY". This is what the patch > file does.
This is a thing to test. In the specific connections case, under 1.2.10 the catch is unuseful. But in other cases must be checked. The problem is that can exists a thow-on-error somewhere, or some function attributes which change the error behaviour when it is cupled with return. Carl S. says something like "let the error throw itself" about 1.2.10 net-error, i do not understand what exactly he means. About the change in 1.2.10, i am not sure that all the existing protocols (third party also) are not affected by this change.
> > In the case of connections handling, we must detect every error, also > errors > > already catched, else we'll use an invalid port. > > Hmm.
Some port error for cached item are due to this.
> I'm not sure - but testing for multiline on every FTP command probably > wouldn't hurt.
I agree.
> > I have many doubts about this. The doubts are: > >
<<quoted lines omitted: 8>>
> > is not sent, like is sent in LIST. > Excellent points.
I have another point: why it tests only the "5", NLIST is like LIST, LIST waits a 2 class answer, the same should be for NLIST.
> REBOL parses the directory list returned by LIST - what if the Parse itself > fails because the parse rules are not appropriate for the listing?
<<quoted lines omitted: 5>>
> will be only when LIST returns something we cannot decipher that NLST will > be used.
The problem in your answer is that the parsing of files is done only after NLIST. Here the check is done only on the raw lines picked from the port: file-list: make string! 2000 while [line: system/words/pick port/sub-port 1] [append file-list join line "^/"] close-the-port answer- "ok"-to-the-server if empty? file-list [ make-nlist ] parse-the-list-of-files But your point opens another bug area of ftp: parse rules. They can fail also only on 1 item, not all (i had this experience some time ago and the same says Carl Read on a recent mail). The solution here is to correct the bugs of parse rules. :-) --- Ciao Romano

 [62/69] from: rotenca:telvia:it at: 14-Nov-2003 13:08


Correction:
> I have another point: why it tests only the "5", NLIST is like LIST, LIST > waits a 2 class answer, the same should be for NLIST.
class 1 Now i'll check the rfc for this. --- Ciao Romano

 [63/69] from: brett:codeconscious at: 15-Nov-2003 0:43


> See in get-cur-dir: > > net-error reform ["Server error:" port/scheme cd]
By the way, get-cur-dir is an anomaly too - it is missing a net-log of the stimulus and of the server's actual response.
> the error is thrown, so if you use try, the error by-pass your try: > > >> error? try [net-error ""] > ** Throw Error: ** User Error: > ** Near: throw make error! info
I might cry now. Authors of robust apps should take note of this last point. In fact it deserves an article all by itself.
> These are two solutions: > > >> error? try [catch[net-error ""]] > == true > >> error? catch[throw-on-error [net-error ""]] > == true
Oh good - I'll stop crying. I like the first one because I don't need to know what is being evaluated in the innermost levels.
> > So without the throw I thought to myself "this ERROR? CATCH thing is > > useless - I should replace the CATCH with a TRY". This is what the patch > > file does. > > This is a thing to test. In the specific connections case, under 1.2.10
the
> catch is unuseful. But in other cases must be checked. The problem is that
can
> exists a thow-on-error somewhere, or some function attributes which change
the
> error behaviour when it is cupled with return.
But doesn't your: error? try [catch[ something ]] solve this generally?
> Carl S. says something like "let the error throw itself" about 1.2.10 > net-error, i do not understand what exactly he means.
I assume it means that: make error! "whatever" will automatically bubble up to whatever is interested in catching it - without requiring a throw. By the way, where does he say this?
> About the change in 1.2.10, i am not sure that all the existing protocols > (third party also) are not affected by this change.
Yeah I wondered about that too. As you have demonstrated a Throw of an error! might bypass a Try, but I think everyone will probably see it if it does. It may cause grief however. If there are extra Catch lying left around after Throw has been removed - it probably will not hurt.
> I have another point: why it tests only the "5", NLIST is like LIST, LIST > waits a 2 class answer, the same should be for NLIST.
I noticed that too - I deferred thinking about it while concentrating on the other things. I agree with you, I think they should be consistent. Testing for success here is probably better than trying to test for failure but missing some cases (like 4xx errors).
> > REBOL parses the directory list returned by LIST - what if the Parse
itself
> > fails because the parse rules are not appropriate for the listing?
...
> The problem in your answer is that the parsing of files is done only after > NLIST.
But NLST was broken (and not pretty) - so I consider logic relating to it suspect as well. I haven't, checked but it should be simple enough to parse the files directly after LIST and again after any NLST that occurs.
> But your point opens another bug area of ftp: parse rules. > > They can fail also only on 1 item, not all (i had this experience some
time
> ago and the same says Carl Read on a recent mail). > > The solution here is to correct the bugs of parse rules. :-)
That is a good step. But maybe there will be a server response to LIST in the future that the parse rules will not be ready for because none of us has seen it yet. A really good step would be to raise an error when it fails (if we are not going to retry again with NLST). I mean, code that does not give you any information one thing - getting the wrong information worries me greatly. Regards, Brett

 [64/69] from: atruter:labyrinth:au at: 15-Nov-2003 22:58


Hi Brett,
> Here you go then (watch line wrap) - try these together
Good news - all issues with the first site have been corrected, bad news is the second site (which had a subset of the problems) now "hangs" on the empty dir read. Trace output follows:
>> Script: {Patch for REBOL FTP protocol 226 response handling.} >> (13-Nov-2003)
No changes to open. Fixed occurrence of FTP 226 handling bug in close.
>> Script: "Patch for REBOL FTP protocol NSLT handling." (14-Nov-2003)
Fixed occurrence of FTP NSLT handling bug in open.
>> Script: "Patch for REBOL FTP protocol error handling." (14-Nov-2003)
Insert missing NET-LOG for PWD in open. Fixed occurrence of FTP error handling bug in open. Fixed occurrence of FTP error handling bug in open. Fixed occurrence of FTP error handling bug in open. Fixed occurrence of FTP error handling bug in query.
>> >> >> >> == ftp://userid:[password--www--site--com]/public_html/ >> URL Parse: userid password www.site.com none public_html/test/ none
Net-log: ["Opening" "tcp" "for" "FTP"] connecting to: www.site.com Net-log: [ none ["220" "230"]] Net-log: {220 ProFTPD 1.2.9 Server (ProFTPD) [froggy.catalog.com]} Net-log: [ ["USER" port/user] "331"] Net-log: "331 Password required for userid." Net-log: [ ["PASS" port/pass] "230"] Net-log: "230 User userid logged in." Net-log: [ "SYST" "*"] Net-log: "215 UNIX Type: L8" Net-log: ["PWD" "25*"] Net-log: [ ["PORT" port/locals/active-check] "200"] Net-log: "200 PORT command successful" Net-log: [ ["CWD" either empty? port/path ["./"] [either slash = port/path/1 [port/path] [join "./" port/path]]] "25*"] Net-log: "250 CWD command successful." Net-log: [ ["TYPE A"] ["200"]] Net-log: "200 Type set to A" Net-log: [ ["LIST"] ["150" "125"]] Net-log: {150 Opening ASCII mode data connection for file list} Net-log: [ none "226"] Net-log: "226 Transfer complete." == [%a %b]
>> URL Parse: userid password www.site.com none public_html/test2/ none
Net-log: ["PWD" "25*"] Net-log: ["Opening" "tcp" "for" "FTP"] connecting to: www.site.com Net-log: [ none ["220" "230"]] Net-log: {220 ProFTPD 1.2.9 Server (ProFTPD) [froggy.catalog.com]} Net-log: [ ["USER" port/user] "331"] Net-log: "331 Password required for userid." Net-log: [ ["PASS" port/pass] "230"] Net-log: "230 User userid logged in." Net-log: [ "SYST" "*"] Net-log: "215 UNIX Type: L8" Net-log: ["PWD" "25*"] Net-log: [ ["PORT" port/locals/active-check] "200"] Net-log: "200 PORT command successful" Net-log: [ ["CWD" either empty? port/path ["./"] [either slash = port/path/1 [port/path] [join "./" port/path]]] "25*"] Net-log: "250 CWD command successful." Net-log: [ ["TYPE A"] ["200"]] Net-log: "200 Type set to A" Net-log: [ ["LIST"] ["150" "125"]] Net-log: {150 Opening ASCII mode data connection for file list} Net-log: [ none "226"] Net-log: "226 Transfer complete." Net-log: [ ["PORT" port/locals/active-check] "200"] Net-log: "200 PORT command successful" Net-log: ["NLST" "Want anything but a 500 series error"] Regards, Ashley

 [65/69] from: rotenca:telvia:it at: 15-Nov-2003 16:06


Ashley, Brett, Carl and all there are many problems in FTP code. Almost every routine has something wrong. I do not think that patching can reach a valid result. So I'm rewriting the protocol trying to put a little of order in the caos :-). I should like to test my beta code on other servers, can you test it (or correct it)? But I do not think that here is a valid solution for LIST/NLIST (see my other message). --- Ciao Romano

 [66/69] from: moliad:aei:ca at: 15-Nov-2003 12:58


>=== Original Message > >I should like to test my beta code on other servers, can you test it (or >correct it)?
I'll be glad to test it
>But I do not think that here is a valid solution for LIST/NLIST (see my > other >message).
but then, how do all ftp softwares fix this. This is going back a while, but isn't there a ftp command which returns a list of commands supported by the server? I may be grosely in error. -MAx

 [67/69] from: rotenca:telvia:it at: 16-Nov-2003 21:29


Hi Max,
> but then, how do all ftp softwares fix this.
The error i know in parsing files with LIST can be fixed with a slight change in the parse rules of the ftp handler. I think that commercial products make fine tuning of the rules they use to parse list output. It is only the task to try some different ftp servers for different OS ;-) Also the Rebol FTP code try to use 2 different set of parse rules with the same list output, if the first fails, it tries the second.
> This is going back a while, but isn't there a ftp command which returns a
list of commands supported by the server? I may be grosely in error. Yes it should be: HELP. --- Ciao Romano

 [68/69] from: brett:codeconscious at: 17-Nov-2003 11:17


> wrong. I do not think that patching can reach a valid result. So I'm
rewriting
> the protocol trying to put a little of order in the caos :-). > > I should like to test my beta code on other servers, can you test it (or > correct it)?
Ambitious! Sure, I'm happy to run it against codeconscious.com and my dodgy ftp test server thingy, but I don't go through a proxy so I can't try that. Regards, Brett.

 [69/69] from: brett:codeconscious at: 17-Nov-2003 11:40


Hi Ashley,
> Good news - all issues with the first site have been corrected, bad news > is the second site (which had a subset of the problems) now "hangs" on the > "empty dir" read. Trace output follows:
...
> Net-log: ["NLST" "Want anything but a 500 series error"]
Thank you for that. This is one of the parts of the protocol that is missing useful tracing information, ie like what the server response was. Two possibilities at least for this, not mutually exclusive, my error handling change has changed the logic unfavourably for this second server, the NLST handling still has issues :-) Will think on this a bit. Regards, Brett.

Notes
  • Quoted lines have been omitted from some messages.
    View the message alone to see the lines that have been omitted