World: r3wp

Have you looked into Cyphre's suggested CTYPES?

@ all,   since there is no struct! or run-time generated datatype 
in R3 we basically have to start from scratch... we do now have a 
handle! datatype which is just like a pointer, which can be used 
as a reference to /library allocated RAM.


to help with visualizing my ideas here is an idea of how I see the 
/library import process happening:

maybe it will help relax apprehensions I have created earlier  ;-)



;-------------------------------
; importing libs
;-------------------------------

user32-lib: retrieve-library 'user32 [
	GetDesktopWindow:	:int32 ]
	OpenClipboard:		:int32 [Wnd: int32]
	close-clipboard:		:int32 "CloseClipboard"  ; note: renamed!  
]

my-lib: retrieve-library 'myown.dll [
	do-this:		:char* [my-arg: complex-struct* ] "DoThis"
	get-that:	:super-simple-struct* "GetThat"  
]



;------------------------------
; declaring structs:
;-------------------------------
super-simple-struct: [
	x: int32
	y: int32
]
complex-struct: [
	value: char*  ; a C string
	value2: int32
	int-array: int32 [50]
	buffer-ptr: byte* [4096]
	struct-vals: sub-struct [
		value3: int64
		value4: int32*
	]
	sub-struct: super-simple-struct
	struct-ptr: super-simple-struct*
	struct-array-ptr: super-simple-struct* [50]
]

;---------------------------------
; calling library stubs
;---------------------------------
window: user32-lib/GetDesktopWindow
user32-lib/OpenClipboard window
user32-lib/close-clipboard

;---
; with structures
;---

; on alloc, all members which aren't explicitely set are either set 
to  0, or  point to 0 filled arrays and structs.
my-lib/do-this alloc-struct complex-struct [
	value2: 2995	
]

coordinates: my-lib/get-coords


;-----------------------------------------------------------------------------

that's what I mean by simple  :-)


This is just a plan, an idea... its not a specification nor is it 
set in stone in any way.

surfed quickly about CTYPES... seems similar to my !Sea source analyser/compiler. 
 The idea behind !Sea is that it would dump the above code, by reading 
C header files.

people have to realize that R2 has routine! and struct! types.  we 
don't in R3.  so it a clean slate, I can't just improve the R2 system... 
there are fundamental differences in the API which add features, 
but also remove some...  due to this fact.

@ shadwolf, wrt memory dumping of structs... yes... easy.  they will 
be flat memory chunks in RAM, managed with malloc/free  so we could 
easily just dump the bits to a file.

note that the /library extension will NOT trample or play within 
the R3 memory/GC if I can prevent it.  this is to sidestep the MANY 
stability issues (some incurable) which I have had to deal with when 
using R2 struct! types in lib calls.


the fact that the /library lives outside of the core is a very welcome 
improvement IMHO.  it does mean that we will be duplicating/copying 
RAM... but this happens in other languages as the default for any 
series manipulation... 


we'll see how it evolves, but it may be possible to share some memory 
intensive datatypes, like image!, and vector!... That will depend 
on the evolution of the extensions system itself.

---------------------------------------------------------------------
C-Header

---------------------------------------------------------------------
typedef struct
{
 double  *data;
 long     size;
 long     datasize;
 long     firstvalid;
} Array;

typedef struct
{
 Array  dt, op, hi, lo, cl, vol, oi;
 long     size;
 long     datasize;
 long     reccnt;
 char     path[256];
 char     name[16];
 char     description[48];
 char     symbol[16];
 char     cusip[12];
 double   begindate;
 double   enddate;
 long     type;              
 long     frequency;        
 long     datatype;          
 long     optiontype;       
 double   deliverydate;
 double   strikeprice;
} Bars;

---------------------------------------------------------------------

maxim, here e.g. is the problem with my lib.
I can allocate, read and write data to the first Array struct.
One strange thing here is it works only if I pass the pointer
as third array binary! to the lib, the requested double fails.


the array is also part within the bars struct. in this case rebol 
is passing the substructures as pointer which fails too. 
char arrays for name desc.. and cusip fails either.


my 'must have' requirement would be that  /library should able to 
handle
these standard structs without going 7 corners.

so working with pointers is a pain and nested strucs and char-arrays
are not existend...

thanks tom... that is a very nice real-world example I can work with. 
  

Do you understand the quick and dirty examples I gave above?  


 looking at it and without any other explanation, do you think you 
 would be able to map your example struct and would it solve all your 
 current requirements (assuming all the types are supported, of course)?


the one thing I DO NOT plan on supporting right now are unions... 
they just make a simple thing complex for no reasons... and they 
aren't that often used in the field anyways (for that very reason).

well I don't understand all but some parts I like to see..e.g.  this 
here -> value: char*  | and of course this -> struct-array-ptr: super-simple-struct* 
[50] :-))

well, you groked the most complex part of the struct  :-)

with a functional /library interface stuff like this will be
possible in rebol, quite interesting. 
If you like visualisation take a look here:

http://www.panopticon.com/demo_gallery/d_bats_usa_ex_demo.htm

for an overview:

http://www.panopticon.com/demo_gallery/index.php

again, there are tons of very usefull libs out there
for so many cool things like mpi, concurrent programming,
ai, ea and so on but we can't use them. what kind of professional
or even commercial software is creatable with rebol without 
having access to resources like that?

I can tell you:   the micro/tiny/fabulous/boring/oneliner bulletin 
board version 5243?
(sorry but I couldn't resist :-))

It occurs to me that if you want to go the LOAD/library way, you 
could have the library spec be a parameter to the /library option. 
It could then return a module that wraps the library, kind-of a mezzanine 
extension.

and to complete it before I promise to stop bothering
with this again, a blocking user interface in 1985
was acceptable but today? (task! please!)

Transforming the C side from R2 to R3 is pretty simple. I have enhanced 
my "in-house" DLL with a R3 interface in a couple of hours. So, now 
it can be used from R2 and R3. Same interface, some functions.

So a good way could be to generate a R3 C based extension wrapper 
around the R2 used function and use the R3 extension interface. It's 
much better and simpler to use.

Robert, we can do that already. This group is discussing a project 
that is taking another approach, though is built on R3 extensions.

A quick note of success: I've hooked up dyncall as extension to a 
properly linked hostkit

@ Robert, the reason for this /library (which is an extension) is 
that most REBOLers do not want to mangle with compiling C stuff (most 
probably don't even now where to begin).  and for most tasks, the 
speed hit isn't really noticeable.

cool, I was going to start working on that very concept this week 
end.  I've looked at all the Dynamic function linkers and its really 
the best best designed one out there.

It works nicely, here's an example of how it currently looks:
  import %ext/dyncall.so
  dyncall %libm.so 'cdefault "sqrt" "d)d" [64.0]
  ; == 8.0

dyncall %libm.so 'cdefault "pow" "dd)d" [2.0 8.0]
; == 256.0

Currently I only handle decimals and ints, both as argument and return 
type

@ BrianH, if you looked at my /library API example, you see it would 
be pretty easy to wrap the call into LOAD/library ... it would just 
be a question of supporting the refinement and calling the extension's 
function... but it would have to check for the extension's availability.

Andreas, its nice to know that it works.

:-)  was it hard to setup?  it looked pretty easy when I looked at 
the dyncall docs (and source code) earlier this week

Nope, it's a very nice library

_and_ it's quite small as well

no changes to the hostkit necessary

I'll be using the argVM directly, its easier to setup within a parsed 
spec.

so do I

my dyncall.so extension which has all the necessary dyncall parts 
statically linked into it is currently only 22K

for comparison: libr3.so is 405K, the hostkit binary is 31K

would you agree to share the work you've done so far?  It will save 
me some time for sure  ' :-/  and I'll be working on a Win version, 
so its going to be a parrallel effort in any case.

sure

i'll compile this thang for windows somewhen early next week

should work as-is

so, where do i claim part of the bounty :) ?

ehehe... the dyncall is the easy part hehe...  ;-)

if you can send me the code you have so far, I'll be working on it 
this very week-end, on windows and that means starting to build the 
/library API itself, including struct support and stub function mapping 
and all.

my email address here is the best way to send me stuff, if you are 
so inclined .

I've put up my current code for the dyncall extension at: http://github.com/earl/rebol3/blob/master/extensions/dyncall.c

cool thanks, will give it a look.

Maxim asked me to take a look here...  it would be good to throw 
the main parts of this into www.rebol.net/wiki -- because it's hard 
to dive into the details in an evolving discussion, do you agree?

yep... but I'm mainly interested in a quick critique of the code 
snippet I post above  (the long post)

once I have parts of this working, I'll do a wiki post and release 
alphas so people can test.

Should the spec block of a lib func be similar to the spec of a normal 
func ?

If so, then all the help features might work ok.

I should note that there's a pending change to func specs.  Namely, 
the use of set-words for special options.

For example, there's no way to describe the return value of a function, 
or trace options, or catch/throw handling.