AltME groups: search

Would it be possible to have a recycle feature for the symbols registry?

You can turn off garbage collecting and do it in the "coffee breaks", 
with this.

recycle/off

Will this work?

    LPINT-def: [value [integer!]] none
    LPINT: make struct! LPINT-def none

    make-LPINT: does [make struct! LPINT-def none]

    get-dereferenced-data: func [

        {Given a pointer to memory, copy the target data into a REBOL struct.}
        pointer [struct!]   "LPINT structure"

        struct-def [block!] "Contains a sub-struct that is the real struct 
        you want."
        /local struct data orig-pointer result
    ] [

        struct: make struct! compose/deep/only [ ; make wrapper struct
            sub [struct! (struct-def)]
        ] none

        orig-pointer: third struct              ; store original inner pointer

        change third struct third pointer       ; change inner pointer to 
        ref'd data

        data: copy third struct/sub             ; copy data from the inner 
        struct
        ;print mold data

        change third struct orig-pointer        ; restore inner pointer

        result: make struct! struct-def none    ; make result struct

        change third result data                ; change data in result struct
        ;probe result
        struct: data: orig-pointer: none
        ;recycle
        result
    ]

It's from some old code, so no guarantees.

for 3 consecutive days, I have experience recycle bug in AltMe. Are 
there anything specific that causes this?

Causes may include clicking before the resync has completed.

But it also seems to be related to the size of the world -- I've 
been getting the recycle bug in the last few days, so possibly Rebol3 
is getting too large.

I recently got the recycle bug  on a single group of a small world 
(repeatdly ) untill someone else posted to that group.  (note my 
client was running on the same machine the world was running on)

So our next update is coming along, aside from any features we are 
planning to include in the next build of AltME, we are looking for 
a new top 10 list of features or bugs for an interim release.


Please take some time to think about this before posting, in an attempt 
to keep this post clean and simple.


Even if you have posted to feedback in the past, I'm look for what 
is most important to you.


Also, the Recycle bug is being fixed, but can't be for this interim 
build, since it is part of the new architecture.

0) multiple fallback mechanisms for getting/staying connected.

1) absolute control of what is actualy removed from local machine... 
ever.

2) better support for using a single account from several machines.
3) better support for using several worlds from one computer.

4) finer control of what posts you are exposed to. (groups and users 
and users in groups) 
5) filesharing
6) api,locks, so we can muck with the filesets.

7) calendar,just fix it up. hour/day/week views, sorting,repeated,various 
alerts ...
8) adjustable font sizes

9) search should return hits (the text) from further back than your 
message buffer limit.
10) OSX support & recycle bug

Thanks.
But that's a little confusing.

The March announcement was not tied to View13 (it explicitly exluded 
the recycle bug for which a new version of View was needed).

It was an announcement, if I read it right, of an interim release 
on the existing SDK.

I start getting "invalid datatype during recycle" messages with Altme. 
Twice during the last two weeks. Is this a known problem???

I'm getting lots of recycle errors with Altme these days ...anyone 
else?

Reichart - we need new Altme or at least please encap old altme in 
1.3 SDK. I got at least 5 or 6 crashes (invalid datatype during recycle) 
in last week ...

he probably means the recycle error ..

yes, the recycle error. I use only this one world ...

A not about the other big big bug...the recycle bug.  when I set 
my may messages to 100, I simply never see that bug.  I left my settings 
at over 1000 recently, and I knew it because I started seeing the 
recycle bug.  If this is true for others, we will change our defaults 
to start.

I have my settins at 1000, and I see the recycle bug once in a blue 
moon.

<<when I set my may messages to 100, I simply never see that bug.>>
Good tip.  I hasn't seen the recycle bug for a while.
Then, last week, I upped my message limit.
And started getting recycle crashes.
And failed to make the connection :-(

On the Recycle bug,  I often leave my message limit at at least 1000 
I've turned it up to 5000 a couple times because messages I was searching 
for were more than 1000 messages in the past.   If I leave it at 
5000 and then don't log in for about a week then I almost always 
get a recycle bug by logging in and clicking on the first group to 
sync while it is syncing the rest of the groups.  Waiting for syncing 
to complete will usually avoid the Recycle bug.  The number of unread 
messages in the group your switching to while it is syncing seems 
to affect whether or not you see the bug, more unread messages seems 
to increase the chance that you will see it.

On the dot bug, on our desktop that's connected to the LAN by wire, 
I nearly never notice anything, then on my laptop wirelessly (weaker 
connection) it seems to just lag, then give up.


I haven't seen the recycle bug in AltME ever since I left the number 
of messages at 100.  I saw it twice in a span of several months, 
though, with it set at 500 messages.

Reichard, 

1) can you try an artificial delay in group-switching? no 'wait, 
just a big delay-loop? to test if the big delay affects networking?

2) no delay, but trigger a recycle? but create a lot of garbage which 
will likely trigger a recycle the next time?

I'm getting a lot of  "invalid data type during recycle" messages 
in the new Altme.  I got them like 4 times in a row when trying to 
click on the !Altme world and then decided to click another world 
and then come back into this one and was finally able to.

Carl - that is strange ... becuase last weeks, I got really two-time-the-week 
recycle crashes, mainly while still syncing and pressing groups quickly 
to "unred" them :-) No such problem with new beta yet .... but maybe 
it is too early to tell, however surprised that it is build on 1.2.10 
....

Pekr: on no recycle, just luck. It is not gone yet.

I just had "invalid datatype during recycle" error:-(

I know it's a known bug, but it happened first time for me: strange 
datatype during recycle.
While clicking a red group during syncing.
AltME 1.1.28 Windows XP

Is this beta client using an old version of encap still?  I got the 
recycle error again.

Altme 1.1.28 WinXP -> pressed a group after startup and got this 
-> Invalid data type during recycle
** Press enter to quit...

I'm starting to see regular recycle errors again ...:(

I've been getting lots of these errors lately for some reason ( Never 
had this problem before):

Invalid data type during recycle
** Press enter to quit...

I would also want a browse mode where i could broese all history 
--- and that 100 is pretty small 500 or 1000 should work. (and if 
they dont there is something else learned about the bug) but I am 
sure I have never run with less than 1000 lines of buffer and only 
rarely had the recycle bug  and then most likely because of searching 
prompted me to increase my buffer and I would only decrease it if 
things got too sluggish

note, that 1.2.3 most probably fixes the "recycle error" crashes 
many people have seen. if you used to get that problem, please try 
this new version and let us know if it still crashes. :)

My first recycle error with 1.1.28 beta ... on hitting a group I've 
already read.  400 setting.  Changed it back to 100.

How safe is catch?
	

I have some rebol applications serving message communication (around 
1000 to 2000 messages per day mostly) running for more than half 
a year on Windows2000 Server without interruption since the last 
update of the OS for security reasons.

Recently I had to add some message splitting:
one-message -->  [message-part-1 message-part-2 message-part-3]

I used a construct similar to this

forever [
	until [new-messages-available]
	foreach message new-messages [
		catch [
			if not important [throw]

   do-some-heavy-message-processing-and data-completion-using-odbc
			if some-tests [throw]
			message-parts: split-messages message
			until [
				catch [
					message: first message-parts
					do-more-conversions
					if other-tests [throw]
					deliver message
					emtpy? message-parts: next message-parts
				]
			]
		]
	]
]

Now I saw two crashes in one day.
I was somehow able to reproduce the crash 
Invalid data type during recycle
 

by playing again the history of one to two weeks. But the crash happened 
always processing another message.

my most hated bug in Rebol: Invalid data type during recycle

g: does [o/f: does [recycle print "o/f"] o/f]
o: make object! [f: does [g print mold disarm try [1 / 0]]]

o: make object! [f: does [g print "version 1"]]
g: does [o/f: does [print "version 2"] recycle]
o/f

f: does [g print "version 1"]
g: does [f: does [print "version 2"] recycle]
f

f: does [g print "version 1"]
g: does [unset 'f recycle]
f

f: does [g print "version 1"]
g: does [unset 'f recycle]

doer: func [f [function!]] [f]
doer :f

to #3885

It might be useful to add the info, that

    f: does [g print "version 1"]
    g: does [unset 'f recycle]
    do :f

circumvents the crash

Gabriele: I submitted a "Read beyond network port" crash ticket. 
(see Recycle bug group discussion)

filet!: context [filet: none]
spec: [filet: 1]
fil: make filet! spec
recycle probe stats
repeat i 10000[
 fil: make make filet! spec fil
]
recycle probe stats
repeat i 10000[
 fil: make make filet! spec third fil
]
recycle probe stats

Its three memory-stats (after recycle).

Graham: I might not quite be getting it but this style works for 
me.  Wrap your code in a "go" function and then ; Loop forever
forever [
    if error? result: try [go] [
        errobj: disarm result
        errorlog [mold errobj]
        either noask [
            alert reform ["There has been an error logged"
                  newline copy/part mold errobj 200

                  newline "See File/Configure/View Error File/ for more info"]
        ][

            unless question/title reform ["There has been an error logged."

                                  newline copy/part mold errobj 200] "Continue?" [
                print ["Type Q to quit, GO to restart FirM"]
                halt
            ]
        ]
    ]
    ; halt
    recycle
]

Could someone with RAMBO access update 4063 ("Garbage characters 
appear in molded strings sent to SQLite via routine!") to note that 
the problem does not occur if a periodic recycle is performed (and 
that the referenced driver now does this, so the recycle must be 
removed to reproduce the problem). I think this narrows down a whole 
class of problems related to unexpected results when repeatedly calling 
a routine.

>> a: make string! 15
== ""
>> insert a "a"
== ""
>> a
== "a"
>> str: make struct! [s [string!]] reduce [a]
>> str/s
== "a"
>> insert/dup tail a "b" 14
== ""
>> a
== "abbbbbbbbbbbbbb"
>> str/s
== "a"
>> recycle
>> str/s
== "¸^D^G^A"

the GC does not imediately recycle memory.

I'm hitting something that causes "invalid datatype during recycle" 
sometimes. I don't know yet what it is, but I thought the recycle 
bug was gone?

i had such a problem with massive gui and async. Workarounded the 
following way: recycle is off permanently. there is a thread (do-every 
or such) which checks how much memory was allocated and when it is 
to much it recycled. crashing stopped.

you may try  recycle/off   and do it yourself

;like

if 20 * 1000 * 1000 + stats > last-mem [  recycle .  last-mem: stats 
]
;and that every 0.01 second or so.

with recycle forced off, it seems to be running OK for now

just make sure you recycle sometimes... if it's long running process

Good news!
So you have a few days to fix the recycle problem for real :-)

recycle/torture

I'm studying memory usage and recycle for a bit. Whenever I'm adding 
a block or doing an operation, REBOL might consume small chunks of 
memory continuously, like 16-32 kb per second. whenever recycle is 
applied, it just stops. Why is that?

I must point out that recycle specifically was turned off and so 
the number would just keep growing for hours eating up 100s of MB. 
Recycle is probably normally invoked, if you don't specify that it 
has to be turned off. The fact that the memory usage just keeps growing 
in an idle situation seems just like a memory leak to me.

henrik, with recycle/off, no memory is ever reused, and obviously 
rebol is constantly allocating memory for temporary values and so 
on. so used memory grows. when you do a recycle, the gc will collect 
all the garbage and start reusing it for later allocations, so that 
memory used stops to grow until you get to the same point as before 
and rebol needs more memory.

recycle never releases memory to the system (afaik) because that's 
usually inefficient, better to keep it for future usage.

If have a problem with Caching in rebol..seems everytime i need to 
restart view/console to make sure the PICTURE is reloaded to the 
currect on... im using 'IMAGE %file.png to load it... and a recycle 
does not fix the problem

it's a supplement to sunanda's example. if you use newblock: copy 
[] you are correctly creating a new block, but every time you then 
append to that block, REBOL has to spend a little time allocating 
new space for the block as it grows. if you preallocate space with 
say:

newblock: make block! 10000


REBOL won't have to do that. This means that when REBOL must garbage 
collect or when you use 'recycle, it knows that there's one big block 
there that it can just remove. Easier for REBOL. Having more blocks 
inside that block makes the case a bit more complex, but since each 
block inside is only created and manipulated once, there may not 
be an issue.

Ports are closed when the garbage collector gets them. If you are 
batch sending, the next time you call the SEND function it will assign 
none to the smtp-port local variable, rendering the previous reference 
assigned to that variable into garbage. The next time RECYCLE is 
run, any open ports in the garbage will be closed.

Recycle yourself if you're worried. Interesting idea on the return 
value.

even had a recycle..
code: 501
description: access error, Port none not open
near:
[recycle 
wait 0.2
]
where: confirm

MySQL driver release v.1.1.2 Download at : http://rebol.softinnov.org/mysql

- Fix for an infinite loop issue when the server times out the connection 
(unix platforms).

- TCP keepalive option activated by default (for longstanding idle 
connections).

- Send-cmd function optimized to be a little bit faster and use less 
memory.
- Added new-lines markers to resulting recorsets.

- Added option to switch on/off new-lines marker through port/locals/newlines? 
flag.

- Recycle call removed from read-rows function. Should speed up the 
results a little bit.
- Minor source code cleanup.

progress: make face [

  effect: copy [draw [pen blue fill-pen blue box 0x0 0x0]] ; is copy 
  needed?
		data:	0
		font:	none
		para:	none
		feel:   make feel [
			redraw: func [face act pos] [
				if act = 'show [
					face/effect/draw/box: to pair! reduce [

      to integer! face/size/x * face/data: min 1 max 0 face/data face/size/y
					]
				recycle
				]
			]
		]
	]

after the draw we '"clear" the memory using recycle

same for anim adding to the feel a recycle call after the show face 
will stabilize the memory usage

Anton without recycle yes but with it it's stable

mmh, after disabling nearly all widgets, it seems that the events 
who eats memory:

even without 'progress, memory is consumed by 4-8ko steps. (just 
going over 'button eats memory)
for 'anim its more visible ('time events),

recycle in 'progress mean recycle at each 'show, so a recycle at 
window level could do the same.

anim: make face [
		edge:	none
		font:	none
		para:	none
		feel:	make feel [
			engage: func [face act event] [
				if event/type = 'time [
					face/image: first face/data

     face/data: either tail? next face/data [head face/data] [next face/data]
					show face
					recycle
				]
			]
		]
		rate:	1
	]

with recycle in the anim feel I get 6776ko allocated

after some play on forwar/backward icon-button I get 7600 ko of allocated 
memory without  any recycle call ...

i think 'show use some memory, and don't recycle it
it would be bigger else, something like  n * image size
I tried with bigger images (400x400 instead of 40x40)
and only 16k more is allocated at each time

I'm keen to see your findings / conclusion on this one. Nailing "memory 
leaks" early in the design is a high priority.


If the "problem" is with 'show then we can always do something simple 
like:

	show*: :show
	show: func [face [object! block!]][show* face recycle]

Not sure about the performance hit though ...

I'm probably doing something wrong here, but I get an error when 
I try do %gui.r:

Script: "RebGUI system" (25-Mar-2005)
** Script Error: Word show is protected, cannot modify
** Where: context

** Near: set 'show func [face [object! block!]] [show* face recycle]

A 'recycle was done by RebGUI in earlier versions (patched 'show 
function)

the problem of a recycle into show is tha it's performance consumer 
...

In all cases we can solve this by patching the show but this force 
the intensive use a recycle witch was  not designed to do ths ...

Maybe recycle could take an argument, which specifies a word to take 
particular note of.

but the point is that we don't know exactly how recycle works

Earlier builds of RebGUI used the following code to minimise the 
problem:

	context [
		show*: :show
		set 'show func [face [object! block!]] [show* face recycle]
	]

which kind of points to the GC as the problem.

switch on recycle/torture, thats what happens if you empty the trashcan 
immediate.

I think I'll add a recycle to that display timer just to avoid confusion 
...

How does one force rebol to recover memory?

>> stats
== 4202333
>> do %tour.r
Script: "Untitled" (none)
Script: "RebGUI system" (15-Jun-2005)
>> stats
== 14955467
>> recycle
>> stats
== 14922539

If the memory is allocated (in local variables) then you can't. Cos 
it is still in use.
Otherwise, try recycle/torture


But first, try to work out what stats is saying (cos I don't understand 
it) --- look at this:
  stats

  longstring: form now/precise loop 20 [append longstring random longstring]
  stats

Longstring is some 25meg long and not very compressible , but the 
2nd stats returns (for me) a slighly larger number than the 1st time.


Stats has some refinements now -- try help stats -- they might be 
more useful for tracking memory.

And maybe switch to core group for a general discussion.

Graham: If you want to reduce RebGUI's memory footprint just rename 
%dictionary/American.dat ... that alone accounts for more than half 
the memory footprint. ;)

Interestingly enough, adding:

	button "Clear" [clear system/locale/dict]

or

	button "Clear" [system/locale/dict: none]


to %tour.r have lesser and differing effects (from a stats / recycle 
perspective).

I left my smtp service running all night .. about 10 hours .. and 
after the 2190'th spam, I hit the infamous "invalid data type during 
recycle" :(

I was getting async read errors, and at least one data type recycle 
error before switching to  2.5.55.3.1

Yes, there is an automatic garbage collector. (you can force it with 
RECYCLE)

Values are released when there are no more references to them (and 
the GC decides it's time to do a recycle).

time: func [b /local start] [
	start: now/precise
	do b
	print ["Time:" difference now/precise start]
]

rgba-to-int: rebcode [r g b a][
	lsl a 24
	lsl r 16
	lsl g 8
	or a r
	or a g
	or a b
	return a
]


draw-pix: rebcode [r g b a][
	apply a rgba-to-int [r g b a]
	return a
]

draw-pix2: rebcode [r g b a][
	loop 1000000 [
		apply a rgba-to-int [r g b a]
	]
	return a
]

probe stats
recycle
time [loop 1000000 [draw-pix 255 255 255 255]]
probe stats
recycle
time [loop 1000000 [rgba-to-int 255 255 255 255]]
probe stats
recycle
time [draw-pix2 255 255 255 255]
probe stats

This is why I have repeatedly requested details about REBOL's recycle 
algorithm. With the wrong algorithm, REBOL could be subject to the 
same problems the comic discusses, like delays and fragmentation.

And if I use the TABLES command I get:
>> print tables

material CREATE TABLE material (_id INTEGER PRIMARY KEY AUTOINCREMENT,_version,_benchmark,_prev,_next,_product_id,name,price,pcdl,pudl,netto,nudl,scrap,loss,process_loss,scrap_re

cycle_rate,scrap_recycle,srcdl,srudl,process_loss_recycle_rate,process_loss_recycle,plrcdl,plurdl) 
sqlite_sequence CREATE TABLE sqlite_sequence(name,seq)

Success! ... of sorts. If you add a 'recycle as the first line of 
the 'sql func then all seems to work fine; but a lot slowwwwwwer 
(1 minute 48 as opposed to 1.5 seconds in Pekr's test case).


But, if you recycle every 100 statements it still works and only 
increases the runtime to 1.85 seconds. I'll do a few more tests before 
uploading a new version with this change.

beer-client-loop: func [/local last-mem mem-jetzt ports-bak] [
    recycle/off 
    last-mem: stats 
    forever [
        if error? set/any 'error try [
            wait 0.1
        ] [
            write %autsch.txt mold disarm error 
            win-log 
            print "-----------------" 
            print disarm error
        ] 
        if 20 * 1000 * 1000 + last-mem < mem-jetzt: stats [
            recycle 
            last-mem: stats 

            /print [now/time mem-jetzt - last-mem mem-jetzt last-mem]
        ]
    ]
]

problem does not occur if a periodic recycle is performed

 should read "problem occurs less frequently if a periodic recycle 
 is performed"

IIRC there is a bug in Rebol's GC that can show up when using big 
datasets and corrupts data. Have you tried to play around with RECYCLE

Robert, 'recycle' worked like a charm!
Thanks a lot.

I had the same problem when reading from the DB so I used 'recycle 
again in hte query loop....

This had the added benifit of of sealing amemory leak with the images 
too.

Memory would increase evrytime a new image was displayed but now 
it tops out at an acceptable level..

cool ,  you can recycle some/most of it.