callbacks

hi guys, I have been trying to use callbacks today and well, its a totally random experience so far. my rebol func actually gets called, but its interface, how I spec it, is irrelevant... whatever I write nothing changes. I can even write a call-back spec which is empty, change the types, anything... the rebol func being called never reacts differently. anyone done anything with callbacks and can give me a few pointers? another strange issue, in some examples, I see callbacks defined like so: test: make routine! [ a [int] b [int] c [callback! [int int return: [int]]] return: [int] ] test-lib "test" but I get an error about the 'CALLBACK! word. If I replace it with 'CALLBACK then the error goes away, but what gets called in the actuall callback is oblivious to anything I write in the spec . indeed I can even add as many args in the rebol func and they just get pumped with random values (probably raw mem addresses). The other strange thing is, in my specific useage, the first and second value which get passed to the rebol func never change, and type also has nothing to do with routine spec nor function spec! help! I'm using the 2.7.5 view pro (almost the last unofficial release ;-) on windows -MAx

well, seems I sent this post 5 minutes too soon. As part of my last ditch attempts, I just started moving thing around related to my dll's library documentation and includes files, expecting crashes and MS errors... and curiously, it seems, something is not calling the proper callback in the first start... so it seems the issue does not lie with REBOL finally... if I change the spec of a completely unrelated callback, the interface is changing... so it seems, the dll is not calling what its supposed to, according to its specs and its init funcs... thanks anyways. -MAx On 10/29/07, Maxim Olivier-Adlhoch <moliad-gmail.com> wrote:

ATTENTION: This is a report about a bug I am having with the callback system within rebol. Your experience may be different, but this just happened to me. Its good to know, so I am posting this on the Mailing list, so someone searching about callbacks might find it. its a long and detailed post, sorry, but this just couldn't be explained properly within a simple 3 line post without quite a few questions popping up . note, the code below is simplified and does not run by itself, it just illustrates the exact problem without clutter. here we go: this code (the extended version of this code) crashes whenever the func-a is called by the external lib! ;----------------------------------------------- rebol [] my-lib: load/library %some-bogus-lib do-init: make routine! [ "pass the supplied arguments as callbacks to lib." number [long] "this shall always be set to 0 (void* = NULL)" spec-a [callback [long ]] spec-b [callback [char]] spec-c [callback [char*]] ] my-lib "doInit" func-a: func [value [integer!]][print value] func-b: func [letter [char!][print letter] func-c: func [sentence [string!]][print sentence] do-init 1 :func-a :func-b :func-c ;----------------------------------------------- I was getting an "illegal memory read at address 0x00000001", hard core windows crash. (but thankfully not a blue screen :-) I thought I had my callbacks in bad order, but no, it would effectively fire off the wrong callbacks if I changed the order supplied to do-init, so I had to put things back... I was a bit stumped. then I looked at the address MS was giving me... hum ...the number looked too clean... like an enum value, so I could figure out a callback value actually was being supplied somewhere. I tried many things, tripple-checked everything but finally, to have the code working, I had to switch the callback specs, without changing the actuall callback submission order, cause remember, its actually calling the proper callback function itself! what's happening, is that its in fact (for some VERY ODD reason) using spec-c and being a pointer to string, its trying to reference mem addres at value! within the lib, func-a really IS being called, its just REBOL which is trying to apply the wrong trampoline specification, in order to convert the mem address into a rebol string. Cause in fact, func-a shouldn't be referecing the address, it should just use the integer directly and supply that to func-a. the above illustrates the change relative to the example above. ;----------------------------------------------- rebol [] my-lib: load/library %some-bogus-lib do-init: make routine! [ "pass the supplied arguments as callbacks to lib." number [long] "this shall always be set to 0 (void* = NULL)" spec-c [callback [char*]] spec-b [callback [char]] spec-a [callback [long ]] ] my-lib "doInit" func-a: func [value [integer!]][print value] func-b: func [letter [char!][print letter] func-c: func [sentence [string!]][print sentence] do-init 1 :func-a :func-b :func-c ;----------------------------------------------- so beware, there is some oddity with the callback system. now I'm about to use Ladislav's get-mem powertool, cause I am receiving variable length arrays of varying types.. :-) I should be having loads of fun tonight :-) -MAx

Hi Max, If I remember correctly, callbacks have less supported types available to use than routines. I'm not sure, but string might not be supported, so that might be the problem. And yes, you could probably work around it using get-mem. Regards, Anton. Maxim Olivier-Adlhoch wrote:

hi anton, note that within the routine, I am using char* which is one of the 3 types supported within callbacks... within the callback func, whatever you put as the argument type is irrelevent. it totally ignores the interface you try to impose. I just discovered that the callbacks also ignore the error trapping completely, and this is very hard to detect! nothing happens, the function just quits silently... I'll try putting an attempt within the func another detail I have been trying to understand is that the GC is totally trashing the stack or something while callbacks are being called. Unless I use recycle/off, at the first recycle occurence... rebol crashes. once I am done with the callbacks I can put recycle/on, but at that point, I dont reclaim any ram which was lost. in fact, it really seems like the GC in rebol is often useless, the moment something makes it peak. often, an app just grows to incredible sizes in the simplest of uses, even without view. -MAx On 10/29/07, Anton Rolls <anton-wilddsl.net.au> wrote:

hi all, well, seems the instability really is related with the GC being sort of pre-emptive hehe it thinks the ram is not usefull anymore... but we are inside the callback itself and bam... it crashes. the moment we play with a callback's arguments, setting one to none or even within a function which is called by the callback... so I had to allocate a Global word and set one of the callback's values to that word, as the first operation of the callback... I even had to convert the interface, set one to char* so rebol's GC would track its pointer use... from that point on, the whole system became totally stable, even with recycle/on being called repeteadly. because I set one of the interace's values to a char* I had to extract the address of the string and use two other methods from peekpoke to retrieve the original value which was a double float array to start with! so, overall, the callback is not for the faint of heart, but if you are patient, and tweak here and there, you can eventually get you want. note that you will not do much, if you do not also use the EXCELLENT peekpoke.rpower library over at ladislav's site. Cause in many cases, the callbacks use much more complex datastructures than what is available out of the box in rebol. I am able to convert double float variable length arrays being sent to me and other nice things, but without the peekpoke lib, I'd be quite lost indeed. -MAx On 10/30/07, Maxim Olivier-Adlhoch <moliad-gmail.com> wrote:

I'm about to go mad! turns out the char* trick below, also trips up the address when some values are being passed. but not always. So I had to stop that. It also seems further testing might invalidate my previous observations. it seems the use of the memory? function within the peekpoke.r file causes the GC to go wild and eventually crashes rebol when callbacks are being used. so far this is pretty constant. if I do recycle/off, no crash occurs, if I let recycle/on and remove the call to memory? there are no crashses either (but then I'm pretty much at step 0!). damn, this is just too frustrating! the damned thing, is that a part from the GC crashing rebol, all is working flawlessly. any ideas/experience welcome -MAx On 10/30/07, Maxim Olivier-Adlhoch <moliad-gmail.com> wrote:

Did you try to use char-array in order to be on the safe side ? http://www.rebol.com/article/0141.html Maxim Olivier-Adlhoch wrote:

hi, I saw that ... but althouh it might fix the issue, my problem is that I have no idea about the length of the returned data before the call... and it can be huge. I could have to make 100kb or even larger copies. in the end, if that's the only thing that makes the system stable, I'll end up doing this (but I still have to check if it really alleviates the problem, cause its not totally clear why and how the GC trips up like that). I'm still open to ideas about why this all happens in the first place and how to solve it. -MAx On 10/30/07, sqlab <sqlab-gmx.net> wrote:

Maxim Olivier-Adlhoch napsal(a):

> hi, > I saw that ... but althouh it might fix the issue, my problem is that I have >>> >>

aha, I was hopping you'd pop in. I was thinking today, that the GC might be trying to de-allocate memory from the struct, which is not its own. maybe if I reuse the structs used withing the memory? func or nullify the pointer just before leaving the memory? func. I'll checkout the link you gave me to get other ideas. -MAx On 10/31/07, Ladislav Mecir <lmecir-mbox.vol.cz> wrote:

well, that was it. I have read that little paragraph a while back, but had forgotten about the [save] option on struct spec. its just badly placed within that doc... its part of the routine! description... it really should have been put under the struct! or maybe as part of its own topic "Preventing Garbage Collection" It was pretty easy to update address-as-struct so it creates a safe struct. (adding it in the memory? struct was not enough for some reason). thanks ladislav. -MAx On 10/31/07, Maxim Olivier-Adlhoch <moliad-gmail.com> wrote:

hi Lad, although its fixing the crashing, safe structs now act as a huge memory leak. its about half the amount of doing recycl/off... in my tests so far. -MAx On 11/1/07, Maxim Olivier-Adlhoch <moliad-gmail.com> wrote:

Maxim Olivier-Adlhoch napsal(a):

> hi Lad, > > although its fixing the crashing, safe structs now act as a huge memory > leak. its about half the amount of doing recycl/off... in my tests so far. > > -MAx >

Hi Ladislav, yes, I understand the whole issue quite clearly. I know exactly what is going on. I was just mentioning, that the [save] option when used in struct is akin to half the memory leak of shutting the GC off completely. in this case, REBOL was trying to free ram for which, it was not the original allocator. IMO it seems the space used for the original struct is being lost somewhere, since we are changing its pointer AFTER its been allocated, and telling REBOL to not recyle it after. -MAx On 11/1/07, Ladislav Mecir <lmecir-mbox.vol.cz> wrote:

hi all, this is the FINAL mail about this topic from me, and its a bit surprising. Its a happy conclusion, at least. I did a merge of two of the functions within peekpoke.r added a few tricks to it and called it memcopy. I include it here and I hope ladislav will include it withing the official distribution. there are two reasons why this function is a complement to what is already there: * it solves the issue of the [save] being a memory leak completely. in fact, I do not use [save] in the function at all. this was the solution I intended to test tonight as a fix to the GC issue. and its completely and 100% GC friendly. In fact, the GC really does recycle the structs itself and the original memory address is in fact allocated by an external source (best of both worlds). * its MUCH faster than calling the memory? function using [save] option within struct even though the basis of the memcopy function uses all of the same code! don't ask me why, I've been splitting my head enough for the last 3 days. ;-) in my tests which use this new function, the application in a tight loop (with no console activity) doing requests on the relavance database (shown at devcon 2007) was 20% faster, but this includes the actual query itself, which takes most of the time, so its possible that the function by itself, is like twice as fast (I don't have time for further benchmarking). so, Ladislav, if you want to add it, do so, I'll be happy if I did a little part in your powertool library :-) -MAx On 11/1/07, Maxim Olivier-Adlhoch <moliad-gmail.com> wrote:

oops, forgot the func... here it is :-) note, this is a stand-alone function, it does not expressly rely on any of the funcs within peekpoke.r memcopy: func [ {copies a region of ram given at address with sizeof length.} address [integer!] length [block!] /local struct-struct buffer addr ] [ address: make struct! [address [integer!]] reduce [address] struct-struct: make struct! compose/deep [ struct [struct! [[save] (spec)]]] none addr: copy third struct-struct change third struct-struct third address buffer: copy third struct-struct/struct address head insert/dup copy [] [c [char]] length change third struct-struct third addr buffer ] -MAx On 11/1/07, Maxim Olivier-Adlhoch <moliad-gmail.com> wrote:

Maxim Olivier-Adlhoch napsal(a):

> Hi Ladislav, > yes, I understand the whole issue quite clearly. I know exactly what is > REBOL to not recyle it after. > -MAx

Hi Max, I did something similar a few years ago. Not callback related, but filling a struct with data allocated by an API (IIRC). get-dereferenced-data: func [ {Given a pointer to memory, copy the target data into a REBOL struct.} pointer [struct!] "LPINT structure whose /value is the data pointer" struct-def [block!] "The struct you want returned with data" /local struct data orig-pointer result ] [ struct: make struct! compose/deep/only [ ; make wrapper struct sub [struct! (struct-def)] ] none orig-pointer: third struct ; store original inner pointer change third struct third pointer ; change inner pointer to ref'd data data: copy third struct/sub ; copy data from the inner struct change third struct orig-pointer ; restore inner pointer result: make struct! struct-def none ; make result struct change third result data ; change data in result struct struct: data: orig-pointer: none result ] -- Gregg

hi all, I'm sorry to say I submitted something odd as the memcopy function. its actually only 0.5% slower than ladislav's but has two differences, *its returns a binary type *it should be 100% GC safe here is the proper version: memcopy: func [ {copies a region of ram given at address with sizeof length.} address [integer!] length [integer!] /local struct-struct buffer addr ] [ address: make struct! [address [integer!]] reduce [address] struct-struct: make struct! compose/deep [ struct [struct! [ (head insert/dup copy [] [c [char]] length)]]] none addr: copy third struct-struct change third struct-struct third address buffer: copy third struct-struct/struct address change third struct-struct third addr buffer ] I realized that its VERY similar to what gregg sent in when I looked at his submission. its probably equivalent, in fact. -MAx On 11/1/07, Gregg Irwin <gregg-pointillistic.com> wrote:

Hi Max,

> hi all, > > I'm sorry to say I submitted something odd as the memcopy function. > > its actually only 0.5% slower than ladislav's but has two differences, > *its returns a binary type > *it should be 100% GC safe >

Lad, I allocate nothing, its a callback. the lib calls my func. I never build a struct except to supply the callbacks upon loading the lib, and that is safe. Maybe its possible the gc could recycle in the window of time in between change pointer address and doing the copy, but if this is the case, both versions I guess suffer the same fatality. maybe adding a recycle/off recycle/on pair within the func could prevent this possibility. previously, anytime the GC would pick up the struct it would crash. Delay was no issue, the stuct really is left corrupted, my guess is that the GC attempts a free() on the memory pointed to by address. There are a few reasons why structs could actually be allocated manually, outside of the normal recycling pool, and since there is a GC managing the struct rebol wrapper anyways, the rule that everything eventually gets freed is still applicable. wrt speed, my tests where within my whole environment. I'm sorry if a pure test shows you're function almost twice as fast, and I insinuated otherwise. just a question, can you add a to-binary to your memcopy and see the speed impact that has? cause your function returns a string. I don't expect it to be major, but with REBOL I don't assume anything anymore. I'm just curious. -MAx On 11/6/07, Ladislav Mecir <lmecir-mbox.vol.cz> wrote:

Hi Max,

> I allocate nothing, its a callback. the lib calls my func. I never build a > struct except to supply the callbacks upon loading the lib, and that is > safe.

> Maybe its possible the gc could recycle in the window of time in between > change pointer address and doing the copy, but if this is the case, both > versions I guess suffer the same fatality. maybe adding a recycle/off > recycle/on pair within the func could prevent this possibility. >

> previously, anytime the GC would pick up the struct it would crash. Delay > was no issue, the stuct really is left corrupted, my guess is that the GC > wrt speed, my tests where within my whole environment. I'm sorry if a pure > test shows you're function almost twice as fast, and I insinuated otherwise.

> just a question, can you add a to-binary to your memcopy and see the speed > impact that has?

hi, wrt the crashing, strangely, I was having problems yesterday also, I don't know why this appeared suddenly. its possible I corrupted the version I had been using reliably just prior to sending a strange version last week. last week I had done tests which lasted well above 5minutes of non-stop querying (at over 50 queries a second) and all was well, this week it seems only your version is safe, so I'll stop trying to fix mine and use yours all the time. I'm just happy this whole debate ended up in exploring and fixing this unpredictable REBOL behaviour. It seems that now, finally, we have an easy and 100% safe way of using external data within REBOL. I noticed in your latest version, you change the address back to 0. My guess is this is the way rebol handles things inside, it assumes NULL * as an unallocated struct, so the GC stops trying to free it. This was to be my next test. -MAx