Mailing List Archive: 49091 messages
  • Home
  • Script library
  • AltME Archive
  • Mailing list
  • Articles Index
  • Site search
 

Rebol for real world applications

 [1/10] from: sf::sabufrancis::com at: 19-Nov-2003 15:08


Hi: I have been developing a real world mailing list CGI application (i.e. not the "Hello World" kind :-) ) layer by layer, testing each part for correctness. It had been going quite well till I realized that my program is running out of memory when it is given real world data :-) and now even the rebol interpreter is crashing (The Encapped version crashes even with lesser data for some reason) I removed unwanted network protocols from my encapped application, and that improved the situation slightly. But when I feed large emails to my mailing list app, it crashes. I see two reasons for this to happen. One is that I am reading a fairly large configuration file where parameters for the mailing list are set (Things like subscription welcome email, moderation email... stuff that the mailing list system will send out to subscribers) That file is a human readable Rebol code file. In order that the users of my mailing list app does not put some Rebol code inside that configuration file and screw up my license with Rebol, I use the following func which converts the file into an object as shown below: ;;;;Warning: UNTESTED CODE cfg: make object! [] ; global variable. Is this needed? pp: [] ; global variable. Is this needed? loadCfg: function [cfgfile] [kk] [ either error? try [ kk: read cfgfile pp: to-block kk ] [ return false ] [ cfg: make object! pp clear pp; return true ] ] Once the configuration file becomes an object, then I pick up the configuration parameter I want by using the appropriate path into that object. The second part of the reason for these crashes could be that each email is opened up, footers are inserted into the email and then given over to another function to be sent to the mailing list subscribers. All that must be taking up memory. The question here is: How does memory management work in Rebol? Are global variables better than local ones? I would appreciate any help. The thing is driving me nuts :-) (As you can see, there are too many smileys already in my post) Regards Sabu Francis

 [2/10] from: maximo:meteorstudios at: 19-Nov-2003 17:14


how much ram is your rebol taking... I've had it over 700MB without problems. how many different objects are you creating. there may be a limit amount of objects (contexts) per application, but given that I have applications which generate hundreds of view faces and corresponding data objects, I doubt this is a small number.
> -----Original Message----- > From: Sabu Francis [mailto:[sf--sabufrancis--com]] > Sent: Wednesday, November 19, 2003 4:39 AM > To: [rebol-list--rebol--com] > Subject: [REBOL] Rebol for real world applications
[... text snipped ...]
> the following func which converts the file into an object as > shown below: > > ;;;;Warning: UNTESTED CODE > > cfg: make object! [] ; global variable. Is this needed? > pp: [] ; global variable. Is this needed?
NOT needed! everything is global unless explicitely local.
> loadCfg: function [cfgfile] [kk] > [
<<quoted lines omitted: 8>>
> ] > ]
take the habbit to ALWAYS do: pp: COPY [] ; explained below in more detail. also clear pp will only clear from current point of pp, so if pp isn't at the head of the block, it won't really clear it. you should: clear HEAD pp but this depends if pp ever gets offset or not (in the above code It shouldn't).
> Once the configuration file becomes an object, then I pick up > the configuration parameter I want by using the appropriate > path into that object.
this should not be a problem.
> The second part of the reason for these crashes could be that > each email is opened up, footers are inserted into the email > and then given over to another function to be sent to the > mailing list subscribers. All that must be taking up memory. > The question here is: How does memory management work in > Rebol?
any unsused data (no words point to it, or the data is not in a block) will eventually be cleaned up. Another point is that contexts (objects and blocks on which you call 'BIND or 'CONTEXT) will only disapear if no words are pointing to them or any of their members. this is called garbage collection. I've had a problem where I had done this: my-setup: function [arg] [blk][ blk: [] append blk arg ] instead of: my-setup: function [arg] [blk][ blk: COPY [] append blk arg ] even if blk is a local variable, it is still the same block each time the function is called. What will happen if you iterate through the block and setup another piece of code with every value in it, is that you will accumulate all the previous setups and RE-apply them over and over. in the end, it works, but your application will get slower and slower. This also had the side effect that the blk will eventually be HUGE. When doing tests on scientific data, I had HUGE datasets of 40 000 rows by 100 columns and rebol never winced. BUT at some point I forgot ONE copy and, I accumulated the 1-40000 rows over and over, this would eventually bust a few GBs of ram... so this might be what is happening to you to...
>Are global variables better than local ones?
Not in terms of memory management. most local variables end up hanging around, unused in between each call of your function, anyways.
> I would > appreciate any help. The thing is driving me nuts :-) (As you > can see, there are too many smileys already in my post)
HTH! -MAx

 [3/10] from: rotenca:telvia:it at: 20-Nov-2003 0:54


Hi Sabu, Memory is auto-recycled. You can force recycling with the recycle function. You can test the memory allocated by your program with system/stats You can have an idea of memory used by some code with these routines: http://www.rebol.org/cgi-bin/cgiwrap/rebol/search.r?find=mem2
> cfg: make object! [] ; global variable. Is this needed?
No.
> pp: [] ; global variable. Is this needed?
No. Non locals are made globals in Rebol when you load a file.
> loadCfg: function [cfgfile] [kk] > [ > > either error? try [ kk: read cfgfile > pp: to-block kk > ]
You should use load/all. To-block works, but it does not load words in the global context. Un-loaded words are buggy in Rebol and can lead to crashes.
> [ return false ] > [
<<quoted lines omitted: 3>>
> ] > ]
your function can become: loadCfg: func [cfgfile][ not error? try [ cfg: make object! load/all cfgfile ] ]
>> loadcfg "a: 2"
== true
>> probe cfg
make object! [ a: 2 ] You should know that make object! executes the body block (the loaded block), so it is not secure. --- Ciao Romano

 [4/10] from: sf:sabufrancis at: 20-Nov-2003 10:45


Hi: Thank you Max and thank you Romano. I fixed the problem this way. I gave up the idea of using an object, because as Romano had pointed out; it can be insecure when the block is converted to an object, the code will get executed. I cannot afford to have my users put up some arbitrary Rebol code inside the configuration file -- say to mail out out the secret recipe for everlasting youth which is residing on my server :-) to them or do something more sinister. So now I'm using the following code ;;;Warning: Untested code loadCfg: function [cfgfile] [pp] [ either error? try [ pp: load/all cfgfile] [ return false ] [ forskip pp 2 [set first pp second pp] pp: none return true ] ] The cfgfile contains parameters that are written in name value pairs using Rebol syntax, thus: a: {Something in the way she moves attracts me like no other lover } c: "Something in the way" c: ["she" "woos" "me"] d: 9 e: [george--something--com] Using the forskip statement in my code, the appropriate global variables are setup. Though I dont like global variables generally (a habit picked up while programming in other languages) I guess I'll live with that for now. I am hoping that the way the globals are setup using the forskip statement, I would be preventing people from putting executable code into the configuration file Or have I got that wrong? Is there a better way of doing it? Max, your idea of using instead of using pp: copy [] I am using pp: none after the routine has finished using pp. This idea I got from a post from Carl who said that the reference to the first frame of a function (i.e. when the first time a local variable is used) hangs around if it is not reset by another call to the function. In fact, I've gone around setting the local variables in all my functions to none -- and that seems to have improved the situation fairly. Regards Sabu P.S: I like Beatles -- their songs are part of the secret recipe ;-) ----- Original Message ----- From: "Maxim Olivier-Adlhoch" <[maximo--meteorstudios--com]> To: <[rebol-list--rebol--com]> Sent: Thursday, November 20, 2003 3:44 AM Subject: [REBOL] Re: Rebol for real world applications
> how much ram is your rebol taking... > > I've had it over 700MB without problems. > > how many different objects are you creating. > > there may be a limit amount of objects (contexts) per application, but
given that I have applications which generate hundreds of view faces and corresponding data objects, I doubt this is a small number.
> > -----Original Message----- > > From: Sabu Francis [mailto:[sf--sabufrancis--com]]
<<quoted lines omitted: 30>>
> pp: COPY [] ; explained below in more detail. > also clear pp will only clear from current point of pp, so if pp isn't at
the head of the block, it won't really clear it. you should:
> clear HEAD pp > > but this depends if pp ever gets offset or not (in the above code It
shouldn't).
> > > > Once the configuration file becomes an object, then I pick up
<<quoted lines omitted: 9>>
> > Rebol? > any unsused data (no words point to it, or the data is not in a block)
will eventually be cleaned up. Another point is that contexts (objects and blocks on which you call 'BIND or 'CONTEXT) will only disapear if no words are pointing to them or any of their members.
> this is called garbage collection. > I've had a problem where I had done this:
<<quoted lines omitted: 8>>
> ] > even if blk is a local variable, it is still the same block each time the
function is called.
> What will happen if you iterate through the block and setup another piece
of code with every value in it, is that you will accumulate all the previous setups and RE-apply them over and over. in the end, it works, but your application will get slower and slower. This also had the side effect that the blk will eventually be HUGE.
> When doing tests on scientific data, I had HUGE datasets of 40 000 rows by
100 columns and rebol never winced.
> BUT at some point I forgot ONE copy and, I accumulated the 1-40000 rows
over and over, this would eventually bust a few GBs of ram... so this might be what is happening to you to...

 [5/10] from: nitsch-lists:netcologne at: 20-Nov-2003 12:12


Am Donnerstag, 20. November 2003 06:15 schrieb Sabu Francis:
> Hi: > Thank you Max and thank you Romano. I fixed the problem this way.
<<quoted lines omitted: 37>>
> the configuration file > Or have I got that wrong? Is there a better way of doing it?
On newer rebols there is 'construct. which creates an object but executes nothing.
>> probe construct[hehe: print "hacked" name: "me" block: [1 2 3] object:
#[object! [a: none]]] make object! [ hehe: 'print name: "me" block: [1 2 3] object: make object! [ a: 'none ] ] A eventuall drawback is, using the #[object! []] a lot seems to crash sometimes. Gabriele reported problems when using it for lots of messages. I guess its related to the problems with unbound words. But it crashes then completely, does not execute code. so your youth would be save :) You get the #[object![]]-stuff when using save/all or mold/all, so you can recreate objects without executing code. Words in such objects are not bound, like to-block does. so functions there may start, but trigger an error immediate. -Volker

 [6/10] from: g:santilli:tiscalinet:it at: 20-Nov-2003 10:46


Hi Sabu, On Thursday, November 20, 2003, 6:15:27 AM, you wrote: SF> The cfgfile contains parameters that are written in name value pairs using SF> Rebol syntax, thus: You can use the new native CONSTRUCT, then. It creates an object without evaluating the block. Regards, Gabriele. -- Gabriele Santilli <[g--santilli--tiscalinet--it]> -- REBOL Programmer Amiga Group Italia sez. L'Aquila --- SOON: http://www.rebol.it/

 [7/10] from: greggirwin:mindspring at: 20-Nov-2003 9:35


Hi Sabu, Glad to hear things are improving. On a side note, the more code you can post to show us exactly what it's doing, the easier it is for folks here to help. I know that's not always possible, but it will help you to get more accurate answers in most cases. Best of luck! -- Gregg

 [8/10] from: sf:sabufrancis at: 21-Nov-2003 2:07


Hi: Thank you for your wishes. I do agree that I should supply some more code. I toyed with the idea of making my entire application is made open source but there are other factors that are currently preventing me from doing so. I have kept in mind that there are busy people on this board who should be given accurate information to work upon. ... and I also plan to contribute something worthwhile back too :-) (apart from my notes on Beatles) Currently I am a newbie in Rebol (about a week of use, actually) and I'm learning the language like the proverbial hot knife through butter. I quite thrilled in fact with the progress that I've made so far in my mailing list application. (Thanks mainly to many of this board's posts and Brett's codes) Regards Sabu

 [9/10] from: greggirwin:mindspring at: 20-Nov-2003 16:13


Hi Sabu, SF> and I also plan to contribute something worthwhile back too :-) SF> (apart from my notes on Beatles) :) I try to give back too; but I think I'll always receive more than I can give in this community. :) -- Gregg

 [10/10] from: AJMartin:orcon at: 24-Dec-2003 22:49


> SF> and I also plan to contribute something worthwhile back too :-) > SF> (apart from my notes on Beatles) > > :) I try to give back too; but I think I'll always receive more than I can
give in this community. :) Let's keep this emerging gift culture going! :) Andrew J Martin Speaking in tongues and performing miracles. ICQ: 26227169 http://www.rebol.it/Valley/ http://valley.orcon.net.nz/ http://Valley.150m.com/

Notes
  • Quoted lines have been omitted from some messages.
    View the message alone to see the lines that have been omitted