Rebol for real world applications
[1/10] from: sf::sabufrancis::com at: 19-Nov-2003 15:08
Hi:
I have been developing a real world mailing list CGI application (i.e. not the "Hello
World" kind :-) ) layer by layer, testing each part for correctness.
It had been going quite well till I realized that my program is running out of memory
when it is given real world data :-) and now even the rebol interpreter is crashing
(The Encapped version crashes even with lesser data for some reason)
I removed unwanted network protocols from my encapped application, and that improved
the situation slightly. But when I feed large emails to my mailing list app, it crashes.
I see two reasons for this to happen. One is that I am reading a fairly large configuration
file where parameters for the mailing list are set (Things like subscription welcome
email, moderation email... stuff that the mailing list system will send out to subscribers)
That file is a human readable Rebol code file. In order that the users of my mailing
list app does not put some Rebol code inside that configuration file and screw up my
license with Rebol, I use the following func which converts the file into an object as
shown below:
;;;;Warning: UNTESTED CODE
cfg: make object! [] ; global variable. Is this needed?
pp: [] ; global variable. Is this needed?
loadCfg: function [cfgfile] [kk]
[
either error? try [ kk: read cfgfile
pp: to-block kk
]
[ return false ]
[
cfg: make object! pp
clear pp;
return true
]
]
Once the configuration file becomes an object, then I pick up the configuration parameter
I want by using the appropriate path into that object.
The second part of the reason for these crashes could be that each email is opened up,
footers are inserted into the email and then given over to another function to be sent
to the mailing list subscribers. All that must be taking up memory. The question here
is: How does memory management work in Rebol? Are global variables better than local
ones? I would appreciate any help. The thing is driving me nuts :-) (As you can see,
there are too many smileys already in my post)
Regards
Sabu Francis
[2/10] from: maximo:meteorstudios at: 19-Nov-2003 17:14
how much ram is your rebol taking...
I've had it over 700MB without problems.
how many different objects are you creating.
there may be a limit amount of objects (contexts) per application, but given that I have
applications which generate hundreds of view faces and corresponding data objects, I
doubt this is a small number.
> -----Original Message-----
> From: Sabu Francis [mailto:[sf--sabufrancis--com]]
> Sent: Wednesday, November 19, 2003 4:39 AM
> To: [rebol-list--rebol--com]
> Subject: [REBOL] Rebol for real world applications
[... text snipped ...]
> the following func which converts the file into an object as
> shown below:
>
> ;;;;Warning: UNTESTED CODE
>
> cfg: make object! [] ; global variable. Is this needed?
> pp: [] ; global variable. Is this needed?
NOT needed!
everything is global unless explicitely local.
> loadCfg: function [cfgfile] [kk]
> [
<<quoted lines omitted: 8>>
> ]
> ]
take the habbit to ALWAYS do:
pp: COPY [] ; explained below in more detail.
also clear pp will only clear from current point of pp, so if pp isn't at the head of
the block, it won't really clear it. you should:
clear HEAD pp
but this depends if pp ever gets offset or not (in the above code It shouldn't).
> Once the configuration file becomes an object, then I pick up
> the configuration parameter I want by using the appropriate
> path into that object.
this should not be a problem.
> The second part of the reason for these crashes could be that
> each email is opened up, footers are inserted into the email
> and then given over to another function to be sent to the
> mailing list subscribers. All that must be taking up memory.
> The question here is: How does memory management work in
> Rebol?
any unsused data (no words point to it, or the data is not in a block) will eventually
be cleaned up. Another point is that contexts (objects and blocks on which you call
'BIND or 'CONTEXT) will only disapear if no words are pointing to them or any of their
members.
this is called garbage collection.
I've had a problem where I had done this:
my-setup: function [arg] [blk][
blk: []
append blk arg
]
instead of:
my-setup: function [arg] [blk][
blk: COPY []
append blk arg
]
even if blk is a local variable, it is still the same block each time the function is
called.
What will happen if you iterate through the block and setup another piece of code with
every value in it, is that you will accumulate all the previous setups and RE-apply them
over and over. in the end, it works, but your application will get slower and slower.
This also had the side effect that the blk will eventually be HUGE.
When doing tests on scientific data, I had HUGE datasets of 40 000 rows by 100 columns
and rebol never winced.
BUT at some point I forgot ONE copy and, I accumulated the 1-40000 rows over and over,
this would eventually bust a few GBs of ram... so this might be what is happening to
you to...
>Are global variables better than local ones?
Not in terms of memory management. most local variables end up hanging around, unused
in between each call of your function, anyways.
> I would
> appreciate any help. The thing is driving me nuts :-) (As you
> can see, there are too many smileys already in my post)
HTH!
-MAx
[3/10] from: rotenca:telvia:it at: 20-Nov-2003 0:54
Hi Sabu,
Memory is auto-recycled.
You can force recycling with the recycle function.
You can test the memory allocated by your program with system/stats
You can have an idea of memory used by some code with these routines:
http://www.rebol.org/cgi-bin/cgiwrap/rebol/search.r?find=mem2
> cfg: make object! [] ; global variable. Is this needed?
No.
> pp: [] ; global variable. Is this needed?
No. Non locals are made globals in Rebol when you load a file.
> loadCfg: function [cfgfile] [kk]
> [
>
> either error? try [ kk: read cfgfile
> pp: to-block kk
> ]
You should use load/all.
To-block works, but it does not load words in the global context.
Un-loaded words are buggy in Rebol and can lead to crashes.
> [ return false ]
> [
<<quoted lines omitted: 3>>
> ]
> ]
your function can become:
loadCfg: func [cfgfile][
not error? try [
cfg: make object! load/all cfgfile
]
]
>> loadcfg "a: 2"
== true
>> probe cfg
make object! [
a: 2
]
You should know that make object! executes the body block (the loaded block),
so it is not secure.
---
Ciao
Romano
[4/10] from: sf:sabufrancis at: 20-Nov-2003 10:45
Hi:
Thank you Max and thank you Romano. I fixed the problem this way.
I gave up the idea of using an object, because as Romano had pointed out; it
can be insecure
when the block is converted to an object, the code will get executed. I
cannot afford to have my
users put up some arbitrary Rebol code inside the configuration file -- say
to mail out
out the secret recipe for everlasting youth which is residing on my server
:-) to them or
do something more sinister.
So now I'm using the following code
;;;Warning: Untested code
loadCfg: function [cfgfile] [pp]
[
either error? try [ pp: load/all cfgfile]
[
return false
]
[
forskip pp 2 [set first pp second pp]
pp: none
return true
]
]
The cfgfile contains parameters that are written in name value pairs using
Rebol syntax, thus:
a: {Something in the way she moves attracts me like no other lover }
c: "Something in the way"
c: ["she" "woos" "me"]
d: 9
e: [george--something--com]
Using the forskip statement in my code, the appropriate global variables are
setup. Though
I dont like global variables generally (a habit picked up while programming
in other languages)
I guess I'll live with that for now. I am hoping that the way the globals
are setup using the forskip
statement, I would be preventing people from putting executable code into
the configuration file
Or have I got that wrong? Is there a better way of doing it?
Max, your idea of using instead of using
pp: copy []
I am using
pp: none
after the routine has finished using pp. This idea I got from a post from
Carl who said that
the reference to the first frame of a function (i.e. when the first time a
local variable is used) hangs around if it is not
reset by another call to the function. In fact, I've gone around setting the
local variables in all my functions
to none -- and that seems to have improved the situation fairly.
Regards
Sabu
P.S: I like Beatles -- their songs are part of the secret recipe ;-)
----- Original Message -----
From: "Maxim Olivier-Adlhoch" <[maximo--meteorstudios--com]>
To: <[rebol-list--rebol--com]>
Sent: Thursday, November 20, 2003 3:44 AM
Subject: [REBOL] Re: Rebol for real world applications
> how much ram is your rebol taking...
>
> I've had it over 700MB without problems.
>
> how many different objects are you creating.
>
> there may be a limit amount of objects (contexts) per application, but
given that I have applications which generate hundreds of view faces and
corresponding data objects, I doubt this is a small number.
> > -----Original Message-----
> > From: Sabu Francis [mailto:[sf--sabufrancis--com]]
<<quoted lines omitted: 30>>
> pp: COPY [] ; explained below in more detail.
> also clear pp will only clear from current point of pp, so if pp isn't at
the head of the block, it won't really clear it. you should:
> clear HEAD pp
>
> but this depends if pp ever gets offset or not (in the above code It
shouldn't).
> >
> > Once the configuration file becomes an object, then I pick up
<<quoted lines omitted: 9>>
> > Rebol?
> any unsused data (no words point to it, or the data is not in a block)
will eventually be cleaned up. Another point is that contexts (objects and
blocks on which you call 'BIND or 'CONTEXT) will only disapear if no words
are pointing to them or any of their members.
> this is called garbage collection.
> I've had a problem where I had done this:
<<quoted lines omitted: 8>>
> ]
> even if blk is a local variable, it is still the same block each time the
function is called.
> What will happen if you iterate through the block and setup another piece
of code with every value in it, is that you will accumulate all the previous
setups and RE-apply them over and over. in the end, it works, but your
application will get slower and slower. This also had the side effect that
the blk will eventually be HUGE.
> When doing tests on scientific data, I had HUGE datasets of 40 000 rows by
100 columns and rebol never winced.
> BUT at some point I forgot ONE copy and, I accumulated the 1-40000 rows
over and over, this would eventually bust a few GBs of ram... so this might
be what is happening to you to...
[5/10] from: nitsch-lists:netcologne at: 20-Nov-2003 12:12
Am Donnerstag, 20. November 2003 06:15 schrieb Sabu Francis:
> Hi:
> Thank you Max and thank you Romano. I fixed the problem this way.
<<quoted lines omitted: 37>>
> the configuration file
> Or have I got that wrong? Is there a better way of doing it?
On newer rebols there is 'construct. which creates an object but executes
nothing.
>> probe construct[hehe: print "hacked" name: "me" block: [1 2 3] object:
#[object! [a: none]]]
make object! [
hehe: 'print
name: "me"
block: [1 2 3]
object:
make object! [
a: 'none
]
]
A eventuall drawback is, using the #[object! []] a lot seems to crash
sometimes. Gabriele reported problems when using it for lots of messages.
I guess its related to the problems with unbound words.
But it crashes then completely, does not execute code.
so your youth would be save :)
You get the #[object![]]-stuff when using save/all or mold/all, so you can
recreate objects without executing code.
Words in such objects are not bound, like to-block does. so functions there
may start, but trigger an error immediate.
-Volker
[6/10] from: g:santilli:tiscalinet:it at: 20-Nov-2003 10:46
Hi Sabu,
On Thursday, November 20, 2003, 6:15:27 AM, you wrote:
SF> The cfgfile contains parameters that are written in name value pairs using
SF> Rebol syntax, thus:
You can use the new native CONSTRUCT, then. It creates an object
without evaluating the block.
Regards,
Gabriele.
--
Gabriele Santilli <[g--santilli--tiscalinet--it]> -- REBOL Programmer
Amiga Group Italia sez. L'Aquila --- SOON: http://www.rebol.it/
[7/10] from: greggirwin:mindspring at: 20-Nov-2003 9:35
Hi Sabu,
Glad to hear things are improving. On a side note, the more code you
can post to show us exactly what it's doing, the easier it is for
folks here to help. I know that's not always possible, but it will
help you to get more accurate answers in most cases.
Best of luck!
-- Gregg
[8/10] from: sf:sabufrancis at: 21-Nov-2003 2:07
Hi:
Thank you for your wishes. I do agree that I should supply some more code.
I toyed with the idea of making my entire application is made open source
but there are other factors that are currently preventing me from doing so.
I have kept in mind that there are busy people on this board who
should be given accurate information to work upon. ... and I also plan
to contribute something worthwhile back too :-) (apart from my notes on
Beatles)
Currently I am a newbie in Rebol (about a week of use, actually) and I'm
learning the language like the proverbial hot knife through butter. I quite
thrilled
in fact with the progress that I've made so far in my mailing list
application.
(Thanks mainly to many of this board's posts and Brett's codes)
Regards
Sabu
[9/10] from: greggirwin:mindspring at: 20-Nov-2003 16:13
Hi Sabu,
SF> and I also plan to contribute something worthwhile back too :-)
SF> (apart from my notes on Beatles)
:) I try to give back too; but I think I'll always receive more than I
can give in this community. :)
-- Gregg
[10/10] from: AJMartin:orcon at: 24-Dec-2003 22:49
> SF> and I also plan to contribute something worthwhile back too :-)
> SF> (apart from my notes on Beatles)
>
> :) I try to give back too; but I think I'll always receive more than I can
give in this community. :)
Let's keep this emerging gift culture going! :)
Andrew J Martin
Speaking in tongues and performing miracles.
ICQ: 26227169
http://www.rebol.it/Valley/
http://valley.orcon.net.nz/
http://Valley.150m.com/
Notes
- Quoted lines have been omitted from some messages.
View the message alone to see the lines that have been omitted