World: r3wp

I agree as well... mostly peope would like to use /library instead 
of extensions to do simple things like:
kernel32.dll: load/library %kernel32.dll
set 'MoveFile make routine! [
		"Moves file using OS"
    lpExistingFileName  [string!]
    lpNewFileName [string!]
] kernel32.dll "MoveFileA"

We don't want to download MS sources from MSDN to parse headers and 
integrate everything what's in kernel32.dll. Of course it may be 
cool in some cases, but we should keep it simple where possible.

Maxim your feeling around the R2/load library is the feelling we 
all get that's why in R3 so much work have been done to improve it 
and i think it's a real good path.

I was recently working on Imagick dll. Firsty I have got almost all 
routines but than and I ended with routine preprocessor which makes 
only routines I really need to load/save/scale images instead including 
all of them in my script.

for example actually being lua ruby or python most of their "regular" 
use  are to be merge as plugin into a host application that shares 
data with them Allowing to set up a base that will not change and 
an extention that will be faster to create ... This point is still 
in my opinion a strutural problem in rebol since in rebol data structure 
are hum ... special  and cool. 


One thing you can't do in rebol and that will miss us alot is for 
example the hability to create a ready made structure  in memory 
and map a file content directly to it. (For example in case of  "memory 
dumped files" in C ...) I could provide a detailled example but i 
think most of you saw what was my point...

Maxim, I'd suggest to look how CTYPES module for Python works. I 
think the have iplemented some interesting ideas which could be a 
good inspiration...http://starship.python.net/crew/theller/ctypes/

cyphre indeed

thanks for all your comments, they are all appreciated.

replies in order:


@ pekr,   I am well aware that extensions and /library are different. 
 R2 is not such a very easy way... I've cursed so many times trying 
to use it in "real life".

sea would make your life importing a comple API much easier but its 
not required... the source tree format is the basic interface.  which 
you can submit directly, just like in R2.

comple=complete

I agree with other guys, that if improvements suggested by Ladislav 
would be done, we would OK for many cases. But I've got the feeling 
you know what do we need :-)

Have you looked into Cyphre's suggested CTYPES?

@ all,   since there is no struct! or run-time generated datatype 
in R3 we basically have to start from scratch... we do now have a 
handle! datatype which is just like a pointer, which can be used 
as a reference to /library allocated RAM.


to help with visualizing my ideas here is an idea of how I see the 
/library import process happening:

maybe it will help relax apprehensions I have created earlier  ;-)



;-------------------------------
; importing libs
;-------------------------------

user32-lib: retrieve-library 'user32 [
	GetDesktopWindow:	:int32 ]
	OpenClipboard:		:int32 [Wnd: int32]
	close-clipboard:		:int32 "CloseClipboard"  ; note: renamed!  
]

my-lib: retrieve-library 'myown.dll [
	do-this:		:char* [my-arg: complex-struct* ] "DoThis"
	get-that:	:super-simple-struct* "GetThat"  
]



;------------------------------
; declaring structs:
;-------------------------------
super-simple-struct: [
	x: int32
	y: int32
]
complex-struct: [
	value: char*  ; a C string
	value2: int32
	int-array: int32 [50]
	buffer-ptr: byte* [4096]
	struct-vals: sub-struct [
		value3: int64
		value4: int32*
	]
	sub-struct: super-simple-struct
	struct-ptr: super-simple-struct*
	struct-array-ptr: super-simple-struct* [50]
]

;---------------------------------
; calling library stubs
;---------------------------------
window: user32-lib/GetDesktopWindow
user32-lib/OpenClipboard window
user32-lib/close-clipboard

;---
; with structures
;---

; on alloc, all members which aren't explicitely set are either set 
to  0, or  point to 0 filled arrays and structs.
my-lib/do-this alloc-struct complex-struct [
	value2: 2995	
]

coordinates: my-lib/get-coords


;-----------------------------------------------------------------------------

that's what I mean by simple  :-)


This is just a plan, an idea... its not a specification nor is it 
set in stone in any way.

surfed quickly about CTYPES... seems similar to my !Sea source analyser/compiler. 
 The idea behind !Sea is that it would dump the above code, by reading 
C header files.

people have to realize that R2 has routine! and struct! types.  we 
don't in R3.  so it a clean slate, I can't just improve the R2 system... 
there are fundamental differences in the API which add features, 
but also remove some...  due to this fact.

@ shadwolf, wrt memory dumping of structs... yes... easy.  they will 
be flat memory chunks in RAM, managed with malloc/free  so we could 
easily just dump the bits to a file.

note that the /library extension will NOT trample or play within 
the R3 memory/GC if I can prevent it.  this is to sidestep the MANY 
stability issues (some incurable) which I have had to deal with when 
using R2 struct! types in lib calls.


the fact that the /library lives outside of the core is a very welcome 
improvement IMHO.  it does mean that we will be duplicating/copying 
RAM... but this happens in other languages as the default for any 
series manipulation... 


we'll see how it evolves, but it may be possible to share some memory 
intensive datatypes, like image!, and vector!... That will depend 
on the evolution of the extensions system itself.

---------------------------------------------------------------------
C-Header

---------------------------------------------------------------------
typedef struct
{
 double  *data;
 long     size;
 long     datasize;
 long     firstvalid;
} Array;

typedef struct
{
 Array  dt, op, hi, lo, cl, vol, oi;
 long     size;
 long     datasize;
 long     reccnt;
 char     path[256];
 char     name[16];
 char     description[48];
 char     symbol[16];
 char     cusip[12];
 double   begindate;
 double   enddate;
 long     type;              
 long     frequency;        
 long     datatype;          
 long     optiontype;       
 double   deliverydate;
 double   strikeprice;
} Bars;

---------------------------------------------------------------------

maxim, here e.g. is the problem with my lib.
I can allocate, read and write data to the first Array struct.
One strange thing here is it works only if I pass the pointer
as third array binary! to the lib, the requested double fails.


the array is also part within the bars struct. in this case rebol 
is passing the substructures as pointer which fails too. 
char arrays for name desc.. and cusip fails either.


my 'must have' requirement would be that  /library should able to 
handle
these standard structs without going 7 corners.

so working with pointers is a pain and nested strucs and char-arrays
are not existend...

thanks tom... that is a very nice real-world example I can work with. 
  

Do you understand the quick and dirty examples I gave above?  


 looking at it and without any other explanation, do you think you 
 would be able to map your example struct and would it solve all your 
 current requirements (assuming all the types are supported, of course)?


the one thing I DO NOT plan on supporting right now are unions... 
they just make a simple thing complex for no reasons... and they 
aren't that often used in the field anyways (for that very reason).

well I don't understand all but some parts I like to see..e.g.  this 
here -> value: char*  | and of course this -> struct-array-ptr: super-simple-struct* 
[50] :-))

well, you groked the most complex part of the struct  :-)

with a functional /library interface stuff like this will be
possible in rebol, quite interesting. 
If you like visualisation take a look here:

http://www.panopticon.com/demo_gallery/d_bats_usa_ex_demo.htm

for an overview:

http://www.panopticon.com/demo_gallery/index.php

again, there are tons of very usefull libs out there
for so many cool things like mpi, concurrent programming,
ai, ea and so on but we can't use them. what kind of professional
or even commercial software is creatable with rebol without 
having access to resources like that?

I can tell you:   the micro/tiny/fabulous/boring/oneliner bulletin 
board version 5243?
(sorry but I couldn't resist :-))

It occurs to me that if you want to go the LOAD/library way, you 
could have the library spec be a parameter to the /library option. 
It could then return a module that wraps the library, kind-of a mezzanine 
extension.

and to complete it before I promise to stop bothering
with this again, a blocking user interface in 1985
was acceptable but today? (task! please!)

Transforming the C side from R2 to R3 is pretty simple. I have enhanced 
my "in-house" DLL with a R3 interface in a couple of hours. So, now 
it can be used from R2 and R3. Same interface, some functions.

So a good way could be to generate a R3 C based extension wrapper 
around the R2 used function and use the R3 extension interface. It's 
much better and simpler to use.

Robert, we can do that already. This group is discussing a project 
that is taking another approach, though is built on R3 extensions.

A quick note of success: I've hooked up dyncall as extension to a 
properly linked hostkit

@ Robert, the reason for this /library (which is an extension) is 
that most REBOLers do not want to mangle with compiling C stuff (most 
probably don't even now where to begin).  and for most tasks, the 
speed hit isn't really noticeable.

cool, I was going to start working on that very concept this week 
end.  I've looked at all the Dynamic function linkers and its really 
the best best designed one out there.

It works nicely, here's an example of how it currently looks:
  import %ext/dyncall.so
  dyncall %libm.so 'cdefault "sqrt" "d)d" [64.0]
  ; == 8.0

dyncall %libm.so 'cdefault "pow" "dd)d" [2.0 8.0]
; == 256.0

Currently I only handle decimals and ints, both as argument and return 
type

@ BrianH, if you looked at my /library API example, you see it would 
be pretty easy to wrap the call into LOAD/library ... it would just 
be a question of supporting the refinement and calling the extension's 
function... but it would have to check for the extension's availability.

Andreas, its nice to know that it works.

:-)  was it hard to setup?  it looked pretty easy when I looked at 
the dyncall docs (and source code) earlier this week

Nope, it's a very nice library

_and_ it's quite small as well

no changes to the hostkit necessary

I'll be using the argVM directly, its easier to setup within a parsed 
spec.

so do I

my dyncall.so extension which has all the necessary dyncall parts 
statically linked into it is currently only 22K

for comparison: libr3.so is 405K, the hostkit binary is 31K

would you agree to share the work you've done so far?  It will save 
me some time for sure  ' :-/  and I'll be working on a Win version, 
so its going to be a parrallel effort in any case.

sure

i'll compile this thang for windows somewhen early next week

should work as-is

so, where do i claim part of the bounty :) ?

ehehe... the dyncall is the easy part hehe...  ;-)

if you can send me the code you have so far, I'll be working on it 
this very week-end, on windows and that means starting to build the 
/library API itself, including struct support and stub function mapping 
and all.