World: r3wp

I just created this group, so we can focus on this specific and community 
sponsored extension.

At this point, any ideas about how people want to use /library functionality 
within REBOL 3 is open for discussion.  :-)  

Please post any ideas, wishes, gotchas, requirements here.


The more I know now, the easier it will be to design the system appropriately.

some ideas regarding a possible STRUCT! dialect is more than welcome. 
  
short rebol code examples welcome!


This will allow me to tailor the engine to how people want to use 
it.

food for thought:

http://www.rebol.net/wiki/DLL_Interface

great, will prepare some requirements...

if you can separate them as:
must have:
nice to have:


and provide them in the order which you can test them on their own, 
then I'll have a good feel for what I should try to tackle first 
wrt the /library API.

ok, will do so.

cool, thanks  :-)

I want to mention this openly:


The name of this altme group is intentionally meant to look RT official. 
 The end goal is for this extension to become THE standard for mapping 
libraries dynamically to REBOL 3.


I want to work WITH the whole community AND with Carl, so that it 
becomes part of the official "sanctified" R3 toolset. 


It will be an ongoing project and will probably, eventually,  be 
maintained by several people.  So its best to start with a name which 
implies this, if only so that the group doesn't get renamed later.


Please understand that as the extension and host kit capabilities 
are improved and changed, this extension will try to keep up.  Its 
even possible that noteworthy efforts in this project might ultimately 
alter REBOL3 itself, so please feel free to participate, even if 
you aren't intent on testing/using the extension right away!


I resisted in giving this project any special name... its as plain 
and obvious as can be... so it should be obvious that ITS AS MUCH 
YOUR PROJECT THAN MINE.

Maxim for more lasting documentation, please create a wiki page for 
what design decisions are made and for general documentation. Thanks.

Henrik, Good point.  

Will do when a prototype is doing at least "something" .


I will note, though, that if someone wishes to *officially* take 
up this task (especially testers which will have their hands at it), 
I will be more than happy.  My time is sparse, and documentation 
is time consuming.

maxinm, since english is not my native language, documentation

is quite hard for me. but if this project e.g. need a public server
for datasharing or whatever I can provide a virtual host with 
slicehost or linode. just mentioned in case of...

For me the good start would be to have the functionality as is in 
R2. Maybe with some more elegant way how to for example get string 
from pointer etc.

Oldes, I'll admit I never liked the R2 /library API.   its difficult 
to understand and feels more like a permanent work-around than a 
complete solution.


I'd really like to allow access to external libraries with a lot 
less fuss.

One idea I have is to use C-like source strings as the interface 
to describe the functions (so we could import .h files directly, 
when possible).  but that will be for a release down the road.

But first, I'd use a native rebol ANSI C90 source tree format (tokenized, 
pre-processed, un-wound error-free source tree data).   

The /library extension would then only manage declarations (funcs 
and structs, for now), assuming funcs to be external library declarations.

we could then use tools like the !sea source parser and simply provide 
an easier way to declare the library functions (using the actual 
C source itself), as like I hope to be able to do.

the source tree format could also be used by Cyphre's jit, an LLVM 
extension, Rebcode, or things like that, as a common target.  so 
we can all share efforts.

maxim - what is 'sea? Please beware, that /library interface is supposed 
to NOT replace Extensions. It is supposed to allow us to link to 
libraries in very easy, R2 like way. Requiring any other tool should 
be prohibited .... just structs, routines and dtype conversions ....

maxim, from the userside (this is how I can talk about /library) 
the wiki from ladislav sounds very good. containing all solutions

for the current problems (pointers, fixed length arrays, nested structs,
conversions). the pointer conversion is cool too. 


what about taking the current solution and these ideas from ladislav 
as 

a clean base and built later additional higher abstraction layers 
for e.g.
automated C header or calling conversion?

as said, for me as a 'normal user' I would like to shielded from 
lower level

languages like using MASM or C as long as I only intend to use precompiled

external libs which I guess is the most using scenario for /library.

I agree with TomBon ... nothing fancy, just remove some obstacles 
of R2 /library interface imo ...

I agree as well... mostly peope would like to use /library instead 
of extensions to do simple things like:
kernel32.dll: load/library %kernel32.dll
set 'MoveFile make routine! [
		"Moves file using OS"
    lpExistingFileName  [string!]
    lpNewFileName [string!]
] kernel32.dll "MoveFileA"

We don't want to download MS sources from MSDN to parse headers and 
integrate everything what's in kernel32.dll. Of course it may be 
cool in some cases, but we should keep it simple where possible.

Maxim your feeling around the R2/load library is the feelling we 
all get that's why in R3 so much work have been done to improve it 
and i think it's a real good path.

I was recently working on Imagick dll. Firsty I have got almost all 
routines but than and I ended with routine preprocessor which makes 
only routines I really need to load/save/scale images instead including 
all of them in my script.

for example actually being lua ruby or python most of their "regular" 
use  are to be merge as plugin into a host application that shares 
data with them Allowing to set up a base that will not change and 
an extention that will be faster to create ... This point is still 
in my opinion a strutural problem in rebol since in rebol data structure 
are hum ... special  and cool. 


One thing you can't do in rebol and that will miss us alot is for 
example the hability to create a ready made structure  in memory 
and map a file content directly to it. (For example in case of  "memory 
dumped files" in C ...) I could provide a detailled example but i 
think most of you saw what was my point...

Maxim, I'd suggest to look how CTYPES module for Python works. I 
think the have iplemented some interesting ideas which could be a 
good inspiration...http://starship.python.net/crew/theller/ctypes/

cyphre indeed

thanks for all your comments, they are all appreciated.

replies in order:


@ pekr,   I am well aware that extensions and /library are different. 
 R2 is not such a very easy way... I've cursed so many times trying 
to use it in "real life".

sea would make your life importing a comple API much easier but its 
not required... the source tree format is the basic interface.  which 
you can submit directly, just like in R2.

comple=complete

I agree with other guys, that if improvements suggested by Ladislav 
would be done, we would OK for many cases. But I've got the feeling 
you know what do we need :-)

Have you looked into Cyphre's suggested CTYPES?

@ all,   since there is no struct! or run-time generated datatype 
in R3 we basically have to start from scratch... we do now have a 
handle! datatype which is just like a pointer, which can be used 
as a reference to /library allocated RAM.


to help with visualizing my ideas here is an idea of how I see the 
/library import process happening:

maybe it will help relax apprehensions I have created earlier  ;-)



;-------------------------------
; importing libs
;-------------------------------

user32-lib: retrieve-library 'user32 [
	GetDesktopWindow:	:int32 ]
	OpenClipboard:		:int32 [Wnd: int32]
	close-clipboard:		:int32 "CloseClipboard"  ; note: renamed!  
]

my-lib: retrieve-library 'myown.dll [
	do-this:		:char* [my-arg: complex-struct* ] "DoThis"
	get-that:	:super-simple-struct* "GetThat"  
]



;------------------------------
; declaring structs:
;-------------------------------
super-simple-struct: [
	x: int32
	y: int32
]
complex-struct: [
	value: char*  ; a C string
	value2: int32
	int-array: int32 [50]
	buffer-ptr: byte* [4096]
	struct-vals: sub-struct [
		value3: int64
		value4: int32*
	]
	sub-struct: super-simple-struct
	struct-ptr: super-simple-struct*
	struct-array-ptr: super-simple-struct* [50]
]

;---------------------------------
; calling library stubs
;---------------------------------
window: user32-lib/GetDesktopWindow
user32-lib/OpenClipboard window
user32-lib/close-clipboard

;---
; with structures
;---

; on alloc, all members which aren't explicitely set are either set 
to  0, or  point to 0 filled arrays and structs.
my-lib/do-this alloc-struct complex-struct [
	value2: 2995	
]

coordinates: my-lib/get-coords


;-----------------------------------------------------------------------------

that's what I mean by simple  :-)


This is just a plan, an idea... its not a specification nor is it 
set in stone in any way.

surfed quickly about CTYPES... seems similar to my !Sea source analyser/compiler. 
 The idea behind !Sea is that it would dump the above code, by reading 
C header files.

people have to realize that R2 has routine! and struct! types.  we 
don't in R3.  so it a clean slate, I can't just improve the R2 system... 
there are fundamental differences in the API which add features, 
but also remove some...  due to this fact.

@ shadwolf, wrt memory dumping of structs... yes... easy.  they will 
be flat memory chunks in RAM, managed with malloc/free  so we could 
easily just dump the bits to a file.

note that the /library extension will NOT trample or play within 
the R3 memory/GC if I can prevent it.  this is to sidestep the MANY 
stability issues (some incurable) which I have had to deal with when 
using R2 struct! types in lib calls.


the fact that the /library lives outside of the core is a very welcome 
improvement IMHO.  it does mean that we will be duplicating/copying 
RAM... but this happens in other languages as the default for any 
series manipulation... 


we'll see how it evolves, but it may be possible to share some memory 
intensive datatypes, like image!, and vector!... That will depend 
on the evolution of the extensions system itself.

---------------------------------------------------------------------
C-Header

---------------------------------------------------------------------
typedef struct
{
 double  *data;
 long     size;
 long     datasize;
 long     firstvalid;
} Array;

typedef struct
{
 Array  dt, op, hi, lo, cl, vol, oi;
 long     size;
 long     datasize;
 long     reccnt;
 char     path[256];
 char     name[16];
 char     description[48];
 char     symbol[16];
 char     cusip[12];
 double   begindate;
 double   enddate;
 long     type;              
 long     frequency;        
 long     datatype;          
 long     optiontype;       
 double   deliverydate;
 double   strikeprice;
} Bars;

---------------------------------------------------------------------

maxim, here e.g. is the problem with my lib.
I can allocate, read and write data to the first Array struct.
One strange thing here is it works only if I pass the pointer
as third array binary! to the lib, the requested double fails.


the array is also part within the bars struct. in this case rebol 
is passing the substructures as pointer which fails too. 
char arrays for name desc.. and cusip fails either.


my 'must have' requirement would be that  /library should able to 
handle
these standard structs without going 7 corners.

so working with pointers is a pain and nested strucs and char-arrays
are not existend...

thanks tom... that is a very nice real-world example I can work with. 
  

Do you understand the quick and dirty examples I gave above?  


 looking at it and without any other explanation, do you think you 
 would be able to map your example struct and would it solve all your 
 current requirements (assuming all the types are supported, of course)?


the one thing I DO NOT plan on supporting right now are unions... 
they just make a simple thing complex for no reasons... and they 
aren't that often used in the field anyways (for that very reason).

well I don't understand all but some parts I like to see..e.g.  this 
here -> value: char*  | and of course this -> struct-array-ptr: super-simple-struct* 
[50] :-))

well, you groked the most complex part of the struct  :-)

with a functional /library interface stuff like this will be
possible in rebol, quite interesting. 
If you like visualisation take a look here:

http://www.panopticon.com/demo_gallery/d_bats_usa_ex_demo.htm

for an overview:

http://www.panopticon.com/demo_gallery/index.php

again, there are tons of very usefull libs out there
for so many cool things like mpi, concurrent programming,
ai, ea and so on but we can't use them. what kind of professional
or even commercial software is creatable with rebol without 
having access to resources like that?

I can tell you:   the micro/tiny/fabulous/boring/oneliner bulletin 
board version 5243?
(sorry but I couldn't resist :-))

It occurs to me that if you want to go the LOAD/library way, you 
could have the library spec be a parameter to the /library option. 
It could then return a module that wraps the library, kind-of a mezzanine 
extension.

and to complete it before I promise to stop bothering
with this again, a blocking user interface in 1985
was acceptable but today? (task! please!)

Transforming the C side from R2 to R3 is pretty simple. I have enhanced 
my "in-house" DLL with a R3 interface in a couple of hours. So, now 
it can be used from R2 and R3. Same interface, some functions.

So a good way could be to generate a R3 C based extension wrapper 
around the R2 used function and use the R3 extension interface. It's 
much better and simpler to use.

Robert, we can do that already. This group is discussing a project 
that is taking another approach, though is built on R3 extensions.

A quick note of success: I've hooked up dyncall as extension to a 
properly linked hostkit