World: r3wp

Diffferent, but related......memory may be deallocated within REBOL's 
sphere of influence one the items have no more references; but it 
is not (it seems) dellocated back to the opsys until the REBOL thread 
closes.


That can create several problems ..... eg an application that uses 
a huge peak memory load at startup, and would like to hand it back 
for the next 8 hours of running. It's all in the arena of efficient 
memory management.

There is a difference between loading image using 'load function 
and loading using 'load-image function (which is used by VID). The 
second one stares data in cache.

And I see no problem with the first one:
>> recycle stats
== 4384079
>> x: load %/f/IMG_0783.jpg
== make image! [1024x768 #{
4E707A59767C5D7172565F5A4D4E464E4A3F554D4258504558494254453E
4F40394E3D364A373148332E4A35304F3A35483E34...
>> recycle stats
== 7531902
>> x: none
== none
>> recycle stats
== 4384079
>> load %/f/IMG_0783.jpg
== make image! [1024x768 #{
4E707A59767C5D7172565F5A4D4E464E4A3F554D4258504558494254453E
4F40394E3D364A373148332E4A35304F3A35483E34...
>> recycle stats
== 7531950
>> recycle stats
== 4384127

Maxim, you said that the problem occurs when the image has been used 
in View. What if the face object that was created for the image was 
not properly unset, like we've talked about above?

Maxim, do you know this:
>> view layout [image %/f/IMG_0783.jpg]
>> second second :load-image
== [%/f/IMG_0783.jpg make image! [1024x768 #{
4E707A59767C5D7172565F5A4D4E464E4A3F554D4258504558494254453E
4F40394E3D364A373148332E...

?? load-image

oldes... no I didn't know this.

but I never let view load images on its own, I always control that. 
 I didn't even know about load-image  ;-)

henrik, if a face isn't unset properly, for sure the image will stay 
stuck in ram... since one reference to it exists.

BUT


I've previously realised that using the stylesheet system actually 
runs init on each face when you build the stylesheet.  as the stylesheet 
is linked within the faces, each stylesheet will effectively constitute 
a memory reference on some resources.

oldes, about the load ... its not totally obvious why sometimes the 
images stay stuck in ram... but in this case it was huge png images. 
 and I did every recycle trick I new of... I tried solving this for 
2 hours... but nothing worked... not even allocating other stuff, 
and freeing it, to make sure the GC really did do a cleanup (in case 
is was count based).

now that I think about it... its not loading the images, it was using 
make image which caused the memory leak... if you call 400MB a "leak".

this is for a pretty insane project I have which will use up about 
30-40GB of image data (once loaded pictures take up much more space 
than on disk, obviously), per output picture.

Maybe it's the content of PNG images what couses problems with GC. 
What do you do with so many images?

I use ImageMagick as well. It allows much more than pure Rebol. But 
of course I call it from Rebol:)

hehe I did the same :-)  I have an automated picture catalogue builder...

its part of a huge visual arts project I have which requires thousands 
of photos... all in high quality... the end goal is a 1.75 x 6m wide 
printout. which use many overlaid independent images.

by many, I expect at least 10000.

Interesting:)

output res should be at 600dpi.  the trick is to find a print shop 
which has a large-print printer with enough ram in its module... 
most shops top out at 2-3m

but i need to build the software first, and right now.. that depends 
(again) on R3... I will be using OpenGL to build the output image 
setups, then dumping the info to a monstrous image magic process, 
which in the end might actually process more raw image data than 
my HD actually can contain.  I expect render times in number of days. 
 (4 pictures on a much smaller canvas took more than an hour).

the processing will be spread out on a farm and reintegrated, as 
tiles render-out.

the subjects of the system can change at will, this whole system 
is a process, not a specific one.  in the best of worlds, I'd find 
a philantropist that would allow me to build the full project (at 
a much larger scale), which costs quite a few thousand dollars in 
printing... one day...

but the technology, once built... can be reused for any project  
 at any scale.

what is the source of images?

the current subject is a church next to my house. I already have 
around 3000 pictures of it at various periods... but the real project 
is a 24hour shooting session, with a top-of the line rented digital 
cam.

I'd like to use hdri imagery.  but with image size that can reach 
in exces of 100mb per picture... its a bit prohibitive on the amount 
of pictures you can take before offloading.

so actual details will vary on time and budget when the final step 
of the project is green-lit... not this year for sure.

like any serious art project... it takes time to build up the hype 
around it too, and possibly get subsidies to pay for part of it.

I need to pay the RAID tower, the hours I'll pour on programming, 
photography, img tagging and output image synthesis qualities, etc. 
 publicity and find a few places to show off the end-results... which 
sometimes has to be paid in advance...  


life is so simple when you have no interests... sometimes I'd like 
to be that way for just a few days a year... hehehe.

Sunanda: if an app uses a huge peak at startup and then does not 
use that memory anymore, that memory just gets swapped to disk. So, 
it's not really in the way of other applications.

that still slows things down, doesn't it?

It may not seem obvious, but 

	view layout [image %file.png]

does make use of LOAD-IMAGE, as Oldes pointed out.
Here is where the IMAGE style does it:

	print mold get in svv/vid-styles/image/multi 'file

Gabriele: that may be true in an ideal world, but not in a Windows 
one.


I tried the test code below in R2 on a freshly booted machine.....I'd 
start a REBOL session, paste the code in,ánd wait until it gave me 
the timing. Then start another session (leaving the old one active) 
and repeat.


I only needed five console sessions to exhaust all physical memory, 
and have Windows behaving in an unstable manner. Killing the console 
sessions returned things to almost normal.


Basically, if this code modeled an application's start-up memory 
usage, it would be unwise to run it under Windows:

rebol []
    t: now/time/precise
    b: copy ["a"]
    attempt [
        forever [
          append b b
          append b join last b last b
          ]
     ]
     print length? b
     foreach a b [clear a]
     clear b
     recycle
     print now/time/precise - t
     halt

Is it REBOL's fault that Windows sucks? ;)

Maybe Carl should treat Windows as a special case...

but I've been told before that rebol doesn't deallocate ram, it only 
frees it from the pool.

so this might not just be a windows issue.

and using swap still takes up ram, it prevents other apps from using 
swap, so its a bad idea on any platform to occupy several hundred 
megs of unneeded ram.

on windows, the detail is that even the OS uses swapable RAM by default. 
 there is registry setting to prevent this, and it makes a hell of 
a big difference in windows itself.  when the machine is under load, 
the OS stays responsive, as opposed to becoming a hog.

I remember doing 3D rendering where up to 4GB of swap can be needed, 
just to get the scenes to render... had I known about this switch 
then, It would have made such a difference.  waiting 30 seconds for 
the task manager to open, cause it has to free it from swap is just 
not feasible when you are charging clients per hour.

In Windows you want virtual memory to be turned on, but you can set 
it to not use a swap file if you like, without touching the registry. 
Virtual memory is used to support memory-mapped files and speeds 
program loading and use (which is done with memory-mapping). The 
computer I am on right now has virtual memory and no swap file since 
it has an SSD, and it runs fine.

You better be sure you have enough RAM though, and for some purposes 
you might need 64bit Windows to even support enough RAM.

In my experience, even with 2GB of ram on a machine with xp and one 
or two software, you will get OS out of memory errors if you turn 
off swap files.  this, with 1GB physical RAM still available... I've 
tried it a few times before and after a short period I had to reboot 
my machine.   The kernel is built in such a way that it expects it 
and it just can't really cope without it.

just to get the scenes to render
 - this might explain your experience :)

the registry tells the OS not to use ram which can be swapped to 
disk, eating up about 500MB.  its still virtual, it just doesn't 
swap to files.

no that is different usage... hehe

the registry *hack*.

I get out-of-memory errors sometimes even with a swap file :)

increase it and it will disapear... and don't let the OS auto-increase 
the size.... that leads to other out-of memory conditions when your 
disks are near full.

However, I tend to not get out-of-memory errors on this computer, 
which has 1GB of RAM and no swap file. This is because I am careful 
about which programs I use (no IE, no Firefox).

has anyone been successful in compressing data in php and decompressing 
it again in R2? I see I asked such a question 7 years ago on the 
mailing list, but no solution was found back then. :-)