Mailing List Archive: 49091 messages
  • Home
  • Script library
  • AltME Archive
  • Mailing list
  • Articles Index
  • Site search
 

Multi-process

 [1/14] from: louisaturk:coxinet at: 1-Nov-2002 10:08


Rebols, In a certain script I must manually insert data then save the file. The file is huge and takes over two minutes to save, while I just sit there looking at the screen. How can I save to a separate process so I can immediately get back to data entry (the manual data entry itself takes over two minutes, so there is no worry about saving before the previous save is completed). Is Rugby a solution? or does Rugby not work on a stand alone computer? What are my options? Louis

 [2/14] from: greggirwin:mindspring at: 1-Nov-2002 12:07


Hi Louis, << In a certain script I must manually insert data then save the file. The file is huge and takes over two minutes to save, while I just sit there looking at the screen. >> Is there any processing going on while it's saving, or does it take that long just to write raw data? How large is the file? << How can I save to a separate process so I can immediately get back to data entry (the manual data entry itself takes over two minutes, so there is no worry about saving before the previous save is completed). Is Rugby a solution? or does Rugby not work on a stand alone computer? >> You could use Rugby, or a simple TCP client/server system. --Gregg

 [3/14] from: gchiu:compkarori at: 2-Nov-2002 8:34


>Is Rugby a solution? or does Rugby not work on a stand >alone computer?
Rugby could be used. -- Graham Chiu

 [4/14] from: carl:cybercraft at: 2-Nov-2002 8:52


On 02-Nov-02, Louis A. Turk wrote:
> Rebols, > In a certain script I must manually insert data then save the file.
<<quoted lines omitted: 7>>
> computer? > What are my options?
Rugby is possibly a solution, (not used it so not sure), but a seperate process running a little server just for the job would certainly be an efficient way to do it. See the TCP section of Network Protocols in the Core Guide - simple servers are quiet easy to code. An alternative but probably not very efficient way would be to open a new window and use feel to do your saving. ie, something like this... view layout [ area "You can type stuff in here while the counter's counting." button "Start Count" [ count: 0 win: view/new/offset layout [ across text "Saving: " counter: text 100 rate 10 feel [engage: [ counter/text: to-string count show counter if 101 = count: count + 1 [unview/only win] ]] ] 20x250 ] ] (I didn't attempt a tiny server as I have to read the book to remind me how to code one:) -- Carl Read

 [5/14] from: louisaturk:coxinet at: 1-Nov-2002 14:04


Gregg,, At 12:07 PM 11/1/2002 -0700, you wrote:
>Hi Louis, > ><< In a certain script I must manually insert data then save the file. The >file is huge and takes over two minutes to save, while I just sit there >looking at the screen. >> > >Is there any processing going on while it's saving, or does it take that >long just to write raw data?
Well, I didn't think about it when I made the post, but there is some processing going on. Perhaps you can see a better way that will speed things up. Here is what is happening: start: now/time append buffer/text "Saving..." show buffer if exists? %i-8.bak [delete %i-8.bak append buffer/text "deleted %i-8.bak...renaming %i-8.txt to %i-8.bak..." show buffer] rename %i-8.txt %i-8.bak append buffer/text "saving memory to %i-8.txt..." show buffer foreach line lines [ write/append %i-8.txt join line newline ] append buffer/text "backing up to %/c/i-8.txt ... " write %/c/i-8.txt read %i-8.txt ;print "backing up to %/f/i-8.txt." ;write %/e/i-8.txt read %i-8.txt finish: now/time backup-2-disk-time: finish - start log-string: join now/date join " " join backup-2-disk-time " (hours:minutes:seconds) were spent backing up data to disk." append buffer/text rejoin [log-string newline] show buffer write/append %word-study-log.txt join log-string newline append buffer/text rejoin ["DONE: backing up to disk took (in hr:min:second): " backup-2-disk-time] show buffer
>How large is the file?
8 MB
><< How can I save to a separate process so I can immediately get back to >data >entry (the manual data entry itself takes over two minutes, so there is no >worry about saving before the previous save is completed). > >Is Rugby a solution? or does Rugby not work on a stand alone computer? >> > >You could use Rugby, or a simple TCP client/server system.
How? Many thanks, Louis

 [6/14] from: al:bri:xtra at: 2-Nov-2002 9:13


Louis wrote:
> In a certain script I must manually insert data then save the file. The
file is huge and takes over two minutes to save, while I just sit there looking at the screen. Have you considered defering the saving of the large file? Use the main file as a source of data, then the user's changes or new entries are saved to a separate file. Then much later, the small file of new entries are appended to the main file. Also, consider using the latest beta versions of Rebol. 'save-ing is much quicker with these versions. I hope that helps! Andrew Martin ICQ: 26227169 http://valley.150m.com/

 [7/14] from: greggirwin:mindspring at: 1-Nov-2002 16:39


Hi Louis, The first thing that jumped out at me was this: foreach line lines [ write/append %i-8.txt join line newline ] When you do a write/append (someone correct me if I'm wrong), REBOL is going to open the file, seek to the end, write the data, and close the file for every line you write. This is very inefficient, and likely the cause of the slowdown. Depending on how long your lines are, it could be doing that a couple hundred thousand times for an 8 meg file. Have you tried just using write/lines? E.g.
>> b: [{1 lksjdflksdjf} {2 lksjdflksdjfl} {3 lskjdflksdj}]
== ["1 lksjdflksdjf" "2 lksjdflksdjfl" "3 lskjdflksdj"]
>> write/lines %test.txt b
results in this: 1 lksjdflksdjf 2 lksjdflksdjfl 3 lskjdflksdj --Gregg

 [8/14] from: al:bri:xtra at: 2-Nov-2002 13:41


Louis wrote:
> foreach line lines [ > write/append %i-8.txt join line newline > ]
Ouch! That's a lot of effort to put Rebol into! Instead, try: I8: read %i-8.txt foreach Line Lines [insert tail Line newline] write %i-8.txt join I8 Lines That way the file is read only once, and written to once. This method should be a lot quicker. I hope that helps Andrew Martin ICQ: 26227169 http://valley.150m.com/

 [9/14] from: louisaturk:coxinet at: 2-Nov-2002 13:15


Hi Gregg, Andrew, Graham, and Carl, Many thanks for your help. I was really looking forward to learning how to multi-task, but as it turns out Gregg and Andrew solved my problem. Both of their save solutions were much faster than what I was using, and Gregg's simple write/lines was fastest, taking only a second or two! compared to well over two minutes before. Quite amazing! Again, many thanks! Louis At 04:39 PM 11/1/2002 -0700, you wrote:

 [10/14] from: joel:neely:fedex at: 2-Nov-2002 13:44


Hi, Louis, Thanks for the thought-provoking question! Louis A. Turk wrote:
> > > In a certain script I must manually insert data then save > > > the file. The file is huge and takes over two minutes to > > > save, while I just sit there looking at the screen...
...
> Here is what is happening: >
...
> foreach line lines [ > write/append %i-8.txt join line newline > ]
...
> >How large is the file? > > 8 MB >
Let me focus on just the part that takes a block (of strings, I'll assume) and writes them into a disk file. I kludged up a little test, which began by creating a block of strings parameterized on the value of N as follows ... stuff: make block! n repeat i n [ insert tail stuff rejoin [ "abcdefghijklmnopqrstuvwxyz" "-" i "-" "ABCDEFGHIJKLMNOPQRSTUVWXYZ" ] ] ... then ensuring that my output files didn't exist ... if found? info? wa: %wa.data [delete wa] if found? info? wl: %wl.data [delete wl] if found? info? wj: %wj.data [delete wj] if found? info? wk: %wk.data [delete wk] ... then tried some different approaches to writing the block's content to a file (a different file for each approach). The first approach resembles your post AFAICT by appending each item in the block to the end of a file ... foreach line stuff [ write/append wa join line newline ] ... the next one explicitly assembles a big string containing all block items (with JOINed NEWLINEs) and then writes that string as a single action ... jj: copy "" foreach line stuff [ insert tail jj join line newline ] write wj jj ... the next one shows the Power of Thinking Ahead, by figuring out how long the result string will be and then preallocating that much space before inserting the lines (and newlines) into the string ... kl: 0 foreach line stuff [kl: kl + 1 + length? line] kk: make string! kl foreach line stuff [ insert insert tail kk line newline ] write wk kk ... the last one lets REBOL do all of the dirty work ... write/lines wl stuff The exact timings are *very* dependent on your system (CPU speed, memory, disk I/O throughput) and the value of N , but for the sake of reference my dog-slow benchmark box (200 MHz Pentium, with 112 Mb RAM and w95) There are a few things we know, but their combined effect may still come as a surprise (it did for me! ;-) * Disk I/O is slow compared to computation. * Opening and closing files is very slow compared to computation. * JOIN is expensive, as it must allocate a new STRING! for each evaluation. * Preallocating series space can significantly improve speed. A little testing shows the following: * WRITE/APPEND was slowest (barely) for N=50000, requiring about 280 times as long as WRITE/LINES does. * WRITE/APPEND was slow, but apparently linear, so that its time increases proportionally to N. * FOREACH ... [INSERT TAIL ... JOIN ...] WRITE ... was next slowest for N=50000, requiring about 255 times as long as WRITE/LINES ... * ... but it's *not* linear! Without preallocation, its time appeared to grow quadratically with N, so that a little beyond N=50000, this becomes the slowest approach. * The Thinking Ahead approach would seem to have a speed handicap because of the double pass across the block, but ... Surprise! It needed only about 11 times as long as WRITE/APPEND did. The pass to calculate total length didn't require memory management, and the big buffer only got allocated once. After that, the second loop was INSERTing into space guaranteed to be available, and so could move more quickly. (Did you notice that it did a double-INSERT instead of INSERT...JOIN? That change alone cut the run time roughly in half.) Morals for me are: * Do as much as possible in memory to avoid I/O. * Preallocate series space whenever possible, even if it takes a little work to figure out how much to allocate! Thanks again for stimulating some pondering! -jn- -- ; Joel Neely joeldotneelyatfedexdotcom REBOL [] do [ do func [s] [ foreach [a b] s [prin b] ] sort/skip do function [s] [t] [ t: "" foreach [a b] s [repend t [b a]] t ] { | e s m!zauafBpcvekexEohthjJakwLrngohOqrlryRnsctdtiub} 2 ]

 [11/14] from: louisaturk:coxinet at: 2-Nov-2002 19:37


Hi Joel, At 01:44 PM 11/2/2002 -0600, you wrote:
>Hi, Louis, > >Thanks for the thought-provoking question!
I'm just glad you take my questions as thought-provoking, and not burdensome. Without you and others on this list, I wouldn't be getting very far. I owe you all a lot, and will probably never be able to repay as you guys are progressing so much faster then me that I'll probably never be of much help to you. Many thank to all of you! Louis

 [12/14] from: al:bri:xtra at: 3-Nov-2002 18:38


Louis wrote:
> I owe you all a lot, and will probably never be able to repay as you guys
are progressing so much faster then me that I'll probably never be of much help to you. Best way for you to help, Louis, is to help others like you were helped. :) Andrew Martin Spreading the Rebolution... ICQ: 26227169 http://valley.150m.com/

 [13/14] from: carl:cybercraft at: 3-Nov-2002 20:14


On 03-Nov-02, Andrew Martin wrote:
> Louis wrote: >> I owe you all a lot, and will probably never be able to repay as
<<quoted lines omitted: 3>>
> Best way for you to help, Louis, is to help others like you were > helped. :)
Oh yes - you'll find it works out 25%/75% in your favour. And this applies regardless of whether you mostly ask questions or mostly answer them. I've probably learn't as much by trying to answer questions as by asking them. -- Carl Read

 [14/14] from: louisaturk:coxinet at: 3-Nov-2002 1:32


Andrew, At 06:38 PM 11/3/2002 +1300, you wrote:
>Best way for you to help, Louis, is to help others like you were helped. :) > >Andrew Martin >Spreading the Rebolution... >ICQ: 26227169 http://valley.150m.com/
Yes, I'm watching for opportunities, but others seem to get to the few I can help before I have a chance. :>) Louis

Notes
  • Quoted lines have been omitted from some messages.
    View the message alone to see the lines that have been omitted