Speed and Memory Management in REBOL
[1/8] from: princepawn::lycos::com at: 5-Sep-2000 12:52
I was glancing at one of the most popular scripts at the REBOL archive and developed
some questions:
1- Is it possible to incrementally write to a file in REBOL?
2- Is it possible to append to a file in REBOL
3- This script seems in-efficient and memory wasteful: it reads and entire file into
a temporary array and then converts the array. In Perl, I could do each conversion as
I read in each line and write each line out after each conversion.
REBOL [
Title: "Comma-Seperated-Values to REBOL converter"
Date: 16-Jun-1999
Author: "Bohdan Lechnowsky"
Email: [bo--rebol--com]
File: %convert-csv.r
Purpose: {
Convert CSV files to REBOL blocks
}
Category: [file util db text 2]
]
csv: read/lines to-file ask "Filename to convert from CSV: "
block: make block! 100
headings: parse/all first csv ","
foreach line next csv [append block parse/all line ","]
save to-file ask "Filename to save to: " head insert/only block headings
[2/8] from: doug:vos:eds at: 5-Sep-2000 16:21
have you tried the refinements for write?
like: write/append/lines (to write one line at a time).
[3/8] from: ryanc:iesco-dms at: 5-Sep-2000 13:47
[princepawn--lycos--com] wrote:
> I was glancing at one of the most popular scripts at the REBOL archive and developed
some questions:
>
> 1- Is it possible to incrementally write to a file in REBOL?
autos: open/new %autos.txt
insert autos "Ford"
insert tail autos " Chevy"
close autos
> 2- Is it possible to append to a file in REBOL
write/append %test.txt "test"
> 3- This script seems in-efficient and memory wasteful: it reads and entire file into
a temporary array and then converts the array. In Perl, I could do each conversion as
I read in each line and write each line out after each conversion.
> REBOL [
<<quoted lines omitted: 15>>
> Get your FREE Email and Voicemail at Lycos Communications at
> http://comm.lycos.com
Go get em.
--Ryan
* Ryan Cole *
Programmer Analyst
www.iesco-dms.com
707-468-5400
;Fortuneately escape will cancel this endless loop...
Forever [ buy microsoft version: version + 1 ]
[4/8] from: agem:crosswinds at: 6-Sep-2000 0:23
--- [princepawn--lycos--com] wrote on 5-Sep-2000/12:52:36-7:00
> I was glancing at one of the most popular scripts at the REBOL archive and deve
>
> loped some questions:
>
> 1- Is it possible to incrementally write to a file in REBOL?
very similar to series. 'skip , 'change ..
> 2- Is it possible to append to a file in REBOL
somewhere in [help open] ?
> 3- This script seems in-efficient and memory wasteful: it reads and entire file
> into a temporary array and then converts the array. In Perl, I could do each
> conversion as I read in each line and write each line out after each conversion.
>
o yes, i remember :) today's systems will not really read
<32K , i expect. all other is buffering & simulation :)
a single os-call to get it all is usually faster.
simpler to, luckily :)
memory? some 10 KB compared to any ram outside museum - where is my microscope? ;-)
volker
[5/8] from: bo:rebol at: 5-Sep-2000 16:37
--__REBOL--0.6.13.4.2--5-Sep-2000/16:37:39-6:00--3052
Content-type: text/plain;
princepawn,
Attached you will find an example that should allow CSV conversion of a
file of any length.
Shocked and honored that simple old %convert-csv.r is one of the most
popular scripts!
Have fun!
-Bo
On 5-Sep-2000/12:52:36-7:00, [princepawn--lycos--com] wrote:
>I was glancing at one of the most popular scripts at the REBOL archive and developed
some questions:
>1- Is it possible to incrementally write to a file in REBOL?
<<quoted lines omitted: 18>>
>Get your FREE Email and Voicemail at Lycos Communications at
>http://comm.lycos.com
--
Bohdan "Bo" Lechnowsky
REBOL Adventure Guide
REBOL Technologies 707-467-8000 (http://www.rebol.com)
Download the REBOL Messaging Language
The Official Source for REBOL Books (http://www.REBOLpress.com)
--__REBOL--0.6.13.4.2--5-Sep-2000/16:37:39-6:00--3052
Content-Type: application/octet-stream; name="new-csv.r"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="new-csv.r"
UkVCT0wgWwoJVGl0bGU6CSJDb21tYS1TZXBhcmF0ZWQtVmFsdWVzIHRvIFJFQk9MIENvbn
ZlcnRlciIKCURhdGU6CTUtU2VwLTIwMDAKCUF1dGhvcjoJIkJvaGRhbiBMZWNobm93c2t5
IgoJRW1haWw6CWJvQHJlYm9sLmNvbQoJRmlsZToJJW5ldy1jc3YucgoJUHVycG9zZTogew
oJCUNvbnZlcnQgQ1NWIGZpbGVzLCBldmVuIHZlcnkgbGFyZ2Ugb25lcywgdG8gUkVCT0wg
YmxvY2tzCgl9CglOb3RlczogewoJCUVhY2ggcmVjb3JkIGdldHMgb3V0cHV0IGFzIGl0cy
Bvd24gYmxvY2sgaW4gdGhlIG91dHB1dCBmaWxlLgoJCVRoZSBjb2x1bW4gaGVhZGluZ3Mg
YXJlIHRoZSBmaXJzdCBibG9jayBpbiB0aGUgZmlsZS4KCX0KCUNhdGVnb3J5OiBbZmlsZS
B1dGlsIGRiIHRleHQgMl0KXQoKY3N2OiBvcGVuL2xpbmVzL2RpcmVjdCB0by1maWxlIGFz
ayAiRmlsZW5hbWUgdG8gY29udmVydCBmcm9tIENTVjogJSIKY3N2LW91dDogdG8tZmlsZS
Bhc2sgIkZpbGVuYW1lIHRvIHNhdmUgY29udmVydGVkIGZpbGUgYXM6ICUiCgp3cml0ZS9s
aW5lcyBjc3Ytb3V0IG1vbGQgcGFyc2UvYWxsIGZpcnN0IGNzdiAiLCIgO291dHB1dHMgaG
VhZGluZ3MgdG8gbmV3IGZpbGUKCmluZGV4OiAwCndoaWxlIFtmb3VuZD8gbGluZTogcGlj
ayBjc3YgMV1bCglwcmluIFsiQ29udmVydGluZyByZWNvcmQiIGluZGV4OiBpbmRleCArID
EgIl5NIl0KCXdyaXRlL2xpbmVzL2FwcGVuZCBjc3Ytb3V0IG1vbGQgcGFyc2UvYWxsIGxp
bmUgIiwiCl0KY2xvc2UgY3N2CnByaW50ICJeL0RvbmUhIgo--__REBOL--0.6.13.4.2--5-Sep-2000/16:37:39-6:00--3052--
[6/8] from: jelinem1:nationwide at: 6-Sep-2000 9:40
--0__=85256952004AC78E8f9e8a93df938690918c85256952004AC78E
Content-type: text/plain; charset=us-ascii
I always thought it was more "efficient" to read in and write a block of
data at a time (or the whole file) as opposed to a line of some arbitrary
length. I guess it depends on the OS treatment of buffering data from the
device when reading in line-by-line.
IMO If you are worried about memory efficiency, program in C. Interpreted
languages are inefficient by design, however much I prefer them (esp
REBOL).
[bo--rebol--com] on 09/05/2000 05:37:39 PM
From: [bo--rebol--com] on 09/05/2000 05:37 PM
Please respond to [list--rebol--com]
To: [list--rebol--com]
cc:
Subject: [REBOL] Speed and Memory Management in REBOL Re:
princepawn,
Attached you will find an example that should allow CSV conversion of a
file of any length.
Shocked and honored that simple old %convert-csv.r is one of the most
popular scripts!
Have fun!
-Bo
On 5-Sep-2000/12:52:36-7:00, [princepawn--lycos--com] wrote:
>I was glancing at one of the most popular scripts at the REBOL archive and
developed some questions:
>1- Is it possible to incrementally write to a file in REBOL?
>2- Is it possible to append to a file in REBOL
>3- This script seems in-efficient and memory wasteful: it reads and entire
file into a temporary array and then converts the array. In Perl, I could
do each conversion as I read in each line and write each line out after
each conversion.
>REBOL [
> Title: "Comma-Seperated-Values to REBOL converter"
<<quoted lines omitted: 14>>
>Get your FREE Email and Voicemail at Lycos Communications at
>http://comm.lycos.com
--
Bohdan "Bo" Lechnowsky
REBOL Adventure Guide
REBOL Technologies 707-467-8000 (http://www.rebol.com)
Download the REBOL Messaging Language
The Official Source for REBOL Books (http://www.REBOLpress.com)
(See attached file: new-csv.r)
--0__=85256952004AC78E8f9e8a93df938690918c85256952004AC78E
Content-type: application/octet-stream;
name="new-csv.r"
Content-Disposition: attachment; filename="new-csv.r"
Content-transfer-encoding: base64
UkVCT0wgWwoJVGl0bGU6CSJDb21tYS1TZXBhcmF0ZWQtVmFsdWVzIHRvIFJFQk9MIENvbnZlcnRl
ciIKCURhdGU6CTUtU2VwLTIwMDAKCUF1dGhvcjoJIkJvaGRhbiBMZWNobm93c2t5IgoJRW1haWw6
CWJvQHJlYm9sLmNvbQoJRmlsZToJJW5ldy1jc3YucgoJUHVycG9zZTogewoJCUNvbnZlcnQgQ1NW
IGZpbGVzLCBldmVuIHZlcnkgbGFyZ2Ugb25lcywgdG8gUkVCT0wgYmxvY2tzCgl9CglOb3Rlczog
ewoJCUVhY2ggcmVjb3JkIGdldHMgb3V0cHV0IGFzIGl0cyBvd24gYmxvY2sgaW4gdGhlIG91dHB1
dCBmaWxlLgoJCVRoZSBjb2x1bW4gaGVhZGluZ3MgYXJlIHRoZSBmaXJzdCBibG9jayBpbiB0aGUg
ZmlsZS4KCX0KCUNhdGVnb3J5OiBbZmlsZSB1dGlsIGRiIHRleHQgMl0KXQoKY3N2OiBvcGVuL2xp
bmVzL2RpcmVjdCB0by1maWxlIGFzayAiRmlsZW5hbWUgdG8gY29udmVydCBmcm9tIENTVjogJSIK
Y3N2LW91dDogdG8tZmlsZSBhc2sgIkZpbGVuYW1lIHRvIHNhdmUgY29udmVydGVkIGZpbGUgYXM6
ICUiCgp3cml0ZS9saW5lcyBjc3Ytb3V0IG1vbGQgcGFyc2UvYWxsIGZpcnN0IGNzdiAiLCIgO291
dHB1dHMgaGVhZGluZ3MgdG8gbmV3IGZpbGUKCmluZGV4OiAwCndoaWxlIFtmb3VuZD8gbGluZTog
cGljayBjc3YgMV1bCglwcmluIFsiQ29udmVydGluZyByZWNvcmQiIGluZGV4OiBpbmRleCArIDEg
Il5NIl0KCXdyaXRlL2xpbmVzL2FwcGVuZCBjc3Ytb3V0IG1vbGQgcGFyc2UvYWxsIGxpbmUgIiwi
Cl0KY2xvc2UgY3N2CnByaW50ICJeL0RvbmUhIgo
--0__=85256952004AC78E8f9e8a93df938690918c85256952004AC78E--
[7/8] from: joel:neely:fedex at: 10-Sep-2000 22:49
Hi, Bo...
Not to be a complainer ;-) but the venerable new-csv.r also
shows what appears to be a bug in parse (or at least a discrepancy
between documented and actual behavior...)
Consider the "csv" file below, created by a Well-Known Spreadsheet
from a Large Redmond Company:
1,1/1/00,"""Moo!"" the cow uttered."
2,1/2/00,"He said ""That's no lady! That's my podiatrist!"""
3,1/3/00,"This ""bad"" data brought to you by MICROS~1."
4,1/4/00,"I'm now crying ""Uncle!"""
Notice that the third field of each line is enclosed in quotation
marks and that embedded quotation marks are rendered as *pairs* of
quotation marks. This can be demonstrated by looking at another
copy of the same file, but saved in ".prn" format -- plain printable
text.
1 1/1/00 "Moo!" the cow uttered.
2 1/2/00 He said "That's no lady! That's my podiatrist!"
3 1/3/00 This "bad" data brought to you by MICROS~1.
4 1/4/00 I'm now crying "Uncle!"
The problem is that
parse/all some-string simple-string
is described as breaking some-string into a block of sub-strings
delimited by occurrences of simple-string (and as ignoring any
special
significance of characters in some-string -- such as
whitespace -- based on the presence of the /all refinement.
However, notice that the output of new-csv.r is the file:
["1" "1/1/00" "" "Moo!" " the cow uttered."]
["2" "1/2/00" "He said " "That's no lady! That's my podiatrist!" ""]
["3" "1/3/00" "This " "bad" " data brought to you by MICROS~1."]
["4" "1/4/00" "I'm now crying " "Uncle!" ""]
The erroneous number of fields per record result indicates that
parse/all is still special-casing the quotation marks, rather than
just treating them as plain data characters. As a result of this
glitch in the behavior of parse/all new-csv.r didn't produce the
expected result of
["1" "1/1/00" {"Moo!" the cow uttered.}]
["2" "1/2/00" {He said "That's no lady! That's my podiatrist!"}]
["3" "1/3/00" {This "bad" data brought to you by MICROS~1.}]
["4" "1/4/00" {I'm now crying "Uncle!"}]
(or its equivalent) with three fields per record, the most natural
interpretation of the original three-column spreadsheet.
-jn-
[bo--rebol--com] wrote:
[8/8] from: g:santilli:tiscalinet:it at: 11-Sep-2000 19:12
Hello [joel--neely--fedex--com]!
On 11-Set-00, you wrote:
j> Consider the "csv" file below, created by a Well-Known
j> Spreadsheet from a Large Redmond Company:
Here's a parser for files generated by that monster. :-)
HTH,
Gabriele.
--
Gabriele Santilli <[giesse--writeme--com]> - Amigan - REBOL programmer
Amiga Group Italia sez. L'Aquila -- http://www.amyresource.it/AGI/
REBOL [
Title: "CSV parser"
Author: "Gabriele Santilli"
]
CSV-parser: make object! [
line-rule: [field any [separator field]]
field: [[quoted-string | string] (insert tail fields any [f-val copy ""])]
string: [copy f-val any str-char]
quoted-string: [{"} copy f-val any qstr-char {"} (replace/all f-val {""} {"})]
str-char: none
qstr-char: [{""} | separator | str-char]
fields: []
f-val: none
separator: #";"
set 'parse-csv-line func [
"Parses a CSV line (returns a block of strings)"
line [string!]
/with sep [char!] "The separator between fields"
] [
clear fields
separator: any [sep #";"]
str-char: complement charset join {"} separator
parse/all line line-rule
copy fields
]
]
Notes
- Quoted lines have been omitted from some messages.
View the message alone to see the lines that have been omitted