Mailing List Archive: 49091 messages
  • Home
  • Script library
  • AltME Archive
  • Mailing list
  • Articles Index
  • Site search
 

Speed and Memory Management in REBOL

 [1/8] from: princepawn::lycos::com at: 5-Sep-2000 12:52


I was glancing at one of the most popular scripts at the REBOL archive and developed some questions: 1- Is it possible to incrementally write to a file in REBOL? 2- Is it possible to append to a file in REBOL 3- This script seems in-efficient and memory wasteful: it reads and entire file into a temporary array and then converts the array. In Perl, I could do each conversion as I read in each line and write each line out after each conversion. REBOL [ Title: "Comma-Seperated-Values to REBOL converter" Date: 16-Jun-1999 Author: "Bohdan Lechnowsky" Email: [bo--rebol--com] File: %convert-csv.r Purpose: { Convert CSV files to REBOL blocks } Category: [file util db text 2] ] csv: read/lines to-file ask "Filename to convert from CSV: " block: make block! 100 headings: parse/all first csv "," foreach line next csv [append block parse/all line ","] save to-file ask "Filename to save to: " head insert/only block headings

 [2/8] from: doug:vos:eds at: 5-Sep-2000 16:21


have you tried the refinements for write? like: write/append/lines (to write one line at a time).

 [3/8] from: ryanc:iesco-dms at: 5-Sep-2000 13:47


[princepawn--lycos--com] wrote:
> I was glancing at one of the most popular scripts at the REBOL archive and developed some questions: > > 1- Is it possible to incrementally write to a file in REBOL?
autos: open/new %autos.txt insert autos "Ford" insert tail autos " Chevy" close autos
> 2- Is it possible to append to a file in REBOL
write/append %test.txt "test"
> 3- This script seems in-efficient and memory wasteful: it reads and entire file into a temporary array and then converts the array. In Perl, I could do each conversion as I read in each line and write each line out after each conversion. > REBOL [
<<quoted lines omitted: 15>>
> Get your FREE Email and Voicemail at Lycos Communications at > http://comm.lycos.com
Go get em. --Ryan * Ryan Cole * Programmer Analyst www.iesco-dms.com 707-468-5400 ;Fortuneately escape will cancel this endless loop... Forever [ buy microsoft version: version + 1 ]

 [4/8] from: agem:crosswinds at: 6-Sep-2000 0:23


--- [princepawn--lycos--com] wrote on 5-Sep-2000/12:52:36-7:00
> I was glancing at one of the most popular scripts at the REBOL archive and deve > > loped some questions: > > 1- Is it possible to incrementally write to a file in REBOL?
very similar to series. 'skip , 'change ..
> 2- Is it possible to append to a file in REBOL
somewhere in [help open] ?
> 3- This script seems in-efficient and memory wasteful: it reads and entire file > into a temporary array and then converts the array. In Perl, I could do each > conversion as I read in each line and write each line out after each conversion. >
o yes, i remember :) today's systems will not really read <32K , i expect. all other is buffering & simulation :) a single os-call to get it all is usually faster. simpler to, luckily :) memory? some 10 KB compared to any ram outside museum - where is my microscope? ;-) volker

 [5/8] from: bo:rebol at: 5-Sep-2000 16:37


--__REBOL--0.6.13.4.2--5-Sep-2000/16:37:39-6:00--3052 Content-type: text/plain; princepawn, Attached you will find an example that should allow CSV conversion of a file of any length. Shocked and honored that simple old %convert-csv.r is one of the most popular scripts! Have fun! -Bo On 5-Sep-2000/12:52:36-7:00, [princepawn--lycos--com] wrote:
>I was glancing at one of the most popular scripts at the REBOL archive and developed some questions: >1- Is it possible to incrementally write to a file in REBOL?
<<quoted lines omitted: 18>>
>Get your FREE Email and Voicemail at Lycos Communications at >http://comm.lycos.com
-- Bohdan "Bo" Lechnowsky REBOL Adventure Guide REBOL Technologies 707-467-8000 (http://www.rebol.com) Download the REBOL Messaging Language The Official Source for REBOL Books (http://www.REBOLpress.com) --__REBOL--0.6.13.4.2--5-Sep-2000/16:37:39-6:00--3052 Content-Type: application/octet-stream; name="new-csv.r" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="new-csv.r" UkVCT0wgWwoJVGl0bGU6CSJDb21tYS1TZXBhcmF0ZWQtVmFsdWVzIHRvIFJFQk9MIENvbn ZlcnRlciIKCURhdGU6CTUtU2VwLTIwMDAKCUF1dGhvcjoJIkJvaGRhbiBMZWNobm93c2t5 IgoJRW1haWw6CWJvQHJlYm9sLmNvbQoJRmlsZToJJW5ldy1jc3YucgoJUHVycG9zZTogew oJCUNvbnZlcnQgQ1NWIGZpbGVzLCBldmVuIHZlcnkgbGFyZ2Ugb25lcywgdG8gUkVCT0wg YmxvY2tzCgl9CglOb3RlczogewoJCUVhY2ggcmVjb3JkIGdldHMgb3V0cHV0IGFzIGl0cy Bvd24gYmxvY2sgaW4gdGhlIG91dHB1dCBmaWxlLgoJCVRoZSBjb2x1bW4gaGVhZGluZ3Mg YXJlIHRoZSBmaXJzdCBibG9jayBpbiB0aGUgZmlsZS4KCX0KCUNhdGVnb3J5OiBbZmlsZS B1dGlsIGRiIHRleHQgMl0KXQoKY3N2OiBvcGVuL2xpbmVzL2RpcmVjdCB0by1maWxlIGFz ayAiRmlsZW5hbWUgdG8gY29udmVydCBmcm9tIENTVjogJSIKY3N2LW91dDogdG8tZmlsZS Bhc2sgIkZpbGVuYW1lIHRvIHNhdmUgY29udmVydGVkIGZpbGUgYXM6ICUiCgp3cml0ZS9s aW5lcyBjc3Ytb3V0IG1vbGQgcGFyc2UvYWxsIGZpcnN0IGNzdiAiLCIgO291dHB1dHMgaG VhZGluZ3MgdG8gbmV3IGZpbGUKCmluZGV4OiAwCndoaWxlIFtmb3VuZD8gbGluZTogcGlj ayBjc3YgMV1bCglwcmluIFsiQ29udmVydGluZyByZWNvcmQiIGluZGV4OiBpbmRleCArID EgIl5NIl0KCXdyaXRlL2xpbmVzL2FwcGVuZCBjc3Ytb3V0IG1vbGQgcGFyc2UvYWxsIGxp bmUgIiwiCl0KY2xvc2UgY3N2CnByaW50ICJeL0RvbmUhIgo--__REBOL--0.6.13.4.2--5-Sep-2000/16:37:39-6:00--3052--

 [6/8] from: jelinem1:nationwide at: 6-Sep-2000 9:40


--0__=85256952004AC78E8f9e8a93df938690918c85256952004AC78E Content-type: text/plain; charset=us-ascii I always thought it was more "efficient" to read in and write a block of data at a time (or the whole file) as opposed to a line of some arbitrary length. I guess it depends on the OS treatment of buffering data from the device when reading in line-by-line. IMO If you are worried about memory efficiency, program in C. Interpreted languages are inefficient by design, however much I prefer them (esp REBOL). [bo--rebol--com] on 09/05/2000 05:37:39 PM From: [bo--rebol--com] on 09/05/2000 05:37 PM Please respond to [list--rebol--com] To: [list--rebol--com] cc: Subject: [REBOL] Speed and Memory Management in REBOL Re: princepawn, Attached you will find an example that should allow CSV conversion of a file of any length. Shocked and honored that simple old %convert-csv.r is one of the most popular scripts! Have fun! -Bo On 5-Sep-2000/12:52:36-7:00, [princepawn--lycos--com] wrote:
>I was glancing at one of the most popular scripts at the REBOL archive and
developed some questions:
>1- Is it possible to incrementally write to a file in REBOL? >2- Is it possible to append to a file in REBOL >3- This script seems in-efficient and memory wasteful: it reads and entire
file into a temporary array and then converts the array. In Perl, I could do each conversion as I read in each line and write each line out after each conversion.
>REBOL [ > Title: "Comma-Seperated-Values to REBOL converter"
<<quoted lines omitted: 14>>
>Get your FREE Email and Voicemail at Lycos Communications at >http://comm.lycos.com
-- Bohdan "Bo" Lechnowsky REBOL Adventure Guide REBOL Technologies 707-467-8000 (http://www.rebol.com) Download the REBOL Messaging Language The Official Source for REBOL Books (http://www.REBOLpress.com) (See attached file: new-csv.r) --0__=85256952004AC78E8f9e8a93df938690918c85256952004AC78E Content-type: application/octet-stream; name="new-csv.r" Content-Disposition: attachment; filename="new-csv.r" Content-transfer-encoding: base64 UkVCT0wgWwoJVGl0bGU6CSJDb21tYS1TZXBhcmF0ZWQtVmFsdWVzIHRvIFJFQk9MIENvbnZlcnRl ciIKCURhdGU6CTUtU2VwLTIwMDAKCUF1dGhvcjoJIkJvaGRhbiBMZWNobm93c2t5IgoJRW1haWw6 CWJvQHJlYm9sLmNvbQoJRmlsZToJJW5ldy1jc3YucgoJUHVycG9zZTogewoJCUNvbnZlcnQgQ1NW IGZpbGVzLCBldmVuIHZlcnkgbGFyZ2Ugb25lcywgdG8gUkVCT0wgYmxvY2tzCgl9CglOb3Rlczog ewoJCUVhY2ggcmVjb3JkIGdldHMgb3V0cHV0IGFzIGl0cyBvd24gYmxvY2sgaW4gdGhlIG91dHB1 dCBmaWxlLgoJCVRoZSBjb2x1bW4gaGVhZGluZ3MgYXJlIHRoZSBmaXJzdCBibG9jayBpbiB0aGUg ZmlsZS4KCX0KCUNhdGVnb3J5OiBbZmlsZSB1dGlsIGRiIHRleHQgMl0KXQoKY3N2OiBvcGVuL2xp bmVzL2RpcmVjdCB0by1maWxlIGFzayAiRmlsZW5hbWUgdG8gY29udmVydCBmcm9tIENTVjogJSIK Y3N2LW91dDogdG8tZmlsZSBhc2sgIkZpbGVuYW1lIHRvIHNhdmUgY29udmVydGVkIGZpbGUgYXM6 ICUiCgp3cml0ZS9saW5lcyBjc3Ytb3V0IG1vbGQgcGFyc2UvYWxsIGZpcnN0IGNzdiAiLCIgO291 dHB1dHMgaGVhZGluZ3MgdG8gbmV3IGZpbGUKCmluZGV4OiAwCndoaWxlIFtmb3VuZD8gbGluZTog cGljayBjc3YgMV1bCglwcmluIFsiQ29udmVydGluZyByZWNvcmQiIGluZGV4OiBpbmRleCArIDEg Il5NIl0KCXdyaXRlL2xpbmVzL2FwcGVuZCBjc3Ytb3V0IG1vbGQgcGFyc2UvYWxsIGxpbmUgIiwi Cl0KY2xvc2UgY3N2CnByaW50ICJeL0RvbmUhIgo --0__=85256952004AC78E8f9e8a93df938690918c85256952004AC78E--

 [7/8] from: joel:neely:fedex at: 10-Sep-2000 22:49


Hi, Bo... Not to be a complainer ;-) but the venerable new-csv.r also shows what appears to be a bug in parse (or at least a discrepancy between documented and actual behavior...) Consider the "csv" file below, created by a Well-Known Spreadsheet from a Large Redmond Company: 1,1/1/00,"""Moo!"" the cow uttered." 2,1/2/00,"He said ""That's no lady! That's my podiatrist!""" 3,1/3/00,"This ""bad"" data brought to you by MICROS~1." 4,1/4/00,"I'm now crying ""Uncle!""" Notice that the third field of each line is enclosed in quotation marks and that embedded quotation marks are rendered as *pairs* of quotation marks. This can be demonstrated by looking at another copy of the same file, but saved in ".prn" format -- plain printable text. 1 1/1/00 "Moo!" the cow uttered. 2 1/2/00 He said "That's no lady! That's my podiatrist!" 3 1/3/00 This "bad" data brought to you by MICROS~1. 4 1/4/00 I'm now crying "Uncle!" The problem is that parse/all some-string simple-string is described as breaking some-string into a block of sub-strings delimited by occurrences of simple-string (and as ignoring any special significance of characters in some-string -- such as whitespace -- based on the presence of the /all refinement. However, notice that the output of new-csv.r is the file: ["1" "1/1/00" "" "Moo!" " the cow uttered."] ["2" "1/2/00" "He said " "That's no lady! That's my podiatrist!" ""] ["3" "1/3/00" "This " "bad" " data brought to you by MICROS~1."] ["4" "1/4/00" "I'm now crying " "Uncle!" ""] The erroneous number of fields per record result indicates that parse/all is still special-casing the quotation marks, rather than just treating them as plain data characters. As a result of this glitch in the behavior of parse/all new-csv.r didn't produce the expected result of ["1" "1/1/00" {"Moo!" the cow uttered.}] ["2" "1/2/00" {He said "That's no lady! That's my podiatrist!"}] ["3" "1/3/00" {This "bad" data brought to you by MICROS~1.}] ["4" "1/4/00" {I'm now crying "Uncle!"}] (or its equivalent) with three fields per record, the most natural interpretation of the original three-column spreadsheet. -jn- [bo--rebol--com] wrote:

 [8/8] from: g:santilli:tiscalinet:it at: 11-Sep-2000 19:12


Hello [joel--neely--fedex--com]! On 11-Set-00, you wrote: j> Consider the "csv" file below, created by a Well-Known j> Spreadsheet from a Large Redmond Company: Here's a parser for files generated by that monster. :-) HTH, Gabriele. -- Gabriele Santilli <[giesse--writeme--com]> - Amigan - REBOL programmer Amiga Group Italia sez. L'Aquila -- http://www.amyresource.it/AGI/ REBOL [ Title: "CSV parser" Author: "Gabriele Santilli" ] CSV-parser: make object! [ line-rule: [field any [separator field]] field: [[quoted-string | string] (insert tail fields any [f-val copy ""])] string: [copy f-val any str-char] quoted-string: [{"} copy f-val any qstr-char {"} (replace/all f-val {""} {"})] str-char: none qstr-char: [{""} | separator | str-char] fields: [] f-val: none separator: #";" set 'parse-csv-line func [ "Parses a CSV line (returns a block of strings)" line [string!] /with sep [char!] "The separator between fields" ] [ clear fields separator: any [sep #";"] str-char: complement charset join {"} separator parse/all line line-rule copy fields ] ]

Notes
  • Quoted lines have been omitted from some messages.
    View the message alone to see the lines that have been omitted