[REBOL] .log to .csv conversion

From: ryan:christiansen:intellisol at: 15-Mar-2001 9:51

Below I am sharing a script that will parse CLF Apache web.log files and
place the data into Excel-friendly .csv files.

I do have a question, however: the script takes a LONG time to run because
of all of the DNS lookups. Is there any way to speed this up?

Thanks.

Ryan C. Christiansen
Web Developer
Intellisol International

REBOL []

log-file: read/lines ftp://username:[password--domain--dom]/logs/web.log

first-line: parse log-file/1 none
first-line/4: remove first-line/4 {[}
first-line/5: remove first-line/5 {]}
checksum-string: rejoin [first-line/4 " " first-line/5]
checksum-date: make date! checksum-string

csv-file-name: make file! (rejoin [checksum-date ".csv"])
write csv-file-name {User IP Address, User Domain Address, Date Hit, Time
Hit, File Hit, Bytes Transferred, Referring Page, Browser Type}
write/append csv-file-name (newline newline)

foreach log-line log-file [

    current-line: parse log-line none

    current-line/4: remove current-line/4 {[}
    current-line/5: remove current-line/5 {]}
    date-string: rejoin [current-line/4 " " current-line/5]
    hit-date: make date! date-string

    either not-equal? hit-date checksum-date [
        csv-file-name: make file! (rejoin [hit-date ".csv"])
        write csv-file-name {User IP Address, User Domain Address, Date
Hit, Time Hit, File Hit, Bytes Transferred, Referring Page, Browser Type}
        write/append csv-file-name (newline newline)

        current-line: parse log-line none
        IP-address: make tuple! current-line/1
        domain-address: read join dns:// IP-address
        current-line/4: remove current-line/4 {[}
        current-line/5: remove current-line/5 {]}
        date-string: rejoin [current-line/4 " " current-line/5]
        hit-date: make date! date-string
        parse date-string [thru ":" copy text to end (hit-time: make time!
text)]
        hit-file: current-line/6
        hit-bytes: current-line/8
        referring-page: make url! current-line/9
        browser-type: current-line/10

        write/append csv-file-name (rejoin [IP-address "," domain-address
,
 hit-date "," hit-time "," hit-file "," hit-bytes "," referring-page ","
browser-type newline])

        checksum-date: hit-date
    ][
        current-line: parse log-line none
        IP-address: make tuple! current-line/1
        domain-address: read join dns:// IP-address
        current-line/4: remove current-line/4 {[}
        current-line/5: remove current-line/5 {]}
        date-string: rejoin [current-line/4 " " current-line/5]
        hit-date: make date! date-string
        parse date-string [thru ":" copy text to end (hit-time: make time!
text)]
        hit-file: current-line/6
        hit-bytes: current-line/8
        referring-page: make url! current-line/9
        browser-type: current-line/10

        write/append csv-file-name (rejoin [IP-address "," domain-address
,
 hit-date "," hit-time "," hit-file "," hit-bytes "," referring-page ","
browser-type newline])

        print (rejoin [IP-address "," domain-address "," hit-date ","
hit-time "," hit-file "," hit-bytes "," referring-page "," browser-type
newline])

        next log-file
    ]
]

1 · [2] · 3 · 4 · 5 · 6 · See whole thread