AltME groups: search

so I've been looking into this a litle bit, and it seems like a portion 
of the /library extension might be a pain to implement... so far 
it seems like I will have to add in-line assembly, since I haven't 
found any C routines or macros which handle the puch/call/pop aspect 
of the program stack (not saying it doesn't exist, just that I haven't 
found any).

and since this code is totally compiler dependent... it means, the 
extension will be tailored to be compiled on one specific compiler... 
(but will be useable by others AFAICT).

I'm still looking for a definitive dll calling convention example... 
AFAIK this has to be pretty set in stone... or dlls would quickly 
be incompatible between vendors.  

Then again, its possible the DLLs contain some calling convention 
parameter...   if anyone has a bit of experience in this (and can 
point me to understandable docs) I'd welcome a few pointers (pun 
intended ;-).

and yes, that's extremely platform-dependent stuff

for windows, the calling conventions used by the OS seem to be pretty 
standardized and simple.  which is nice.  So far it seems to be a 
bit more sprawled for linux.


first release of the /library extension will be 32 bit only, but 
when we have a 64 bit REBOL it will be pretty easy to adapt... in 
fact, MS/intel did their homework for 64bit, it seems, and there 
is only ONE calling convention for that platform.

afaik microsoft has different calling conventions for x86 and x64

and iirc they have at least 3 support cconvs anyway (cdecl, stdcall, 
fastcall; maybe more)

yep, the x64 is better in fact.  x86 has three official conventions, 
some on the stack, some using registers, and different cleanup.  
 the x64 has only one and is based on __fastcall

maxim, any change the extension will be capable to handle nested 
strucs and pointers?

dyncall is BSD and should even work with MSVC

tom, yes.    that will start out simple, but will be handled in the 
rebol & extension part of the code.  


once I have the actuall subroutine jump (ASM call) code in place 
with an unlimited number of run-time functions registering, we'll 
see how the community wants to evolve the struct! dialect.


I am thinking of a rebol to struct converter and a way to keep handles 
to those structs, and even use them within sub structs, so that we 
can easily reuse allocated structures without doing the convertion 
over and over.

so you can work like draw, keeping just the rebol version and convert 
it at every call, or manually convert it once, and then reuse it 
over and over.  

this will allow us to chose between ease of use or raw speed (with 
a bit more management in our code).

Andreas, I won't need an external lib, as we are in effect building 
a dyncall specifically for rebol.  which makes it simpler and faster... 
and it seems very simple to implement ourself, in fact.

hehehe  I'll do my first tests, and then I'll see If I can wrap the 
platform-specifc parts of the process through dyncall's source code. 
 this way we will be able to compile the /library extension for any 
platform without any source code changes.


my design is to use header file macros which actually map the calling 
conventions based on your platform... THAT I can definitely borrow 
from dyncall if it BSD licensed :-)

I'm officially a geek... I actually laughed when I read a bit of 
ASM that had the   lea   instruction in it.. and instantly remembered 
what it meant hehehe  


three little characters, pop up 17 years later and my mind does a 
180 degree shift    :-)

Max, I don't know what you do but the overall concept would be:
1. Loading the DLL dynamically

2. Finding all requested functions dynamically (Windows has a call 
for this where you provide a string of the function name (just can't 
remember the call name)).

3. Implementing a generic R3 extension function that will be marshalled 
on Rebol site so that the first parameter will the function name 
and than all paraemters (I would use a block here).
4. Rearrange all this to call the Win32 function.

Robert, three problems with that approach (even just on Windows):

- Not all DLLs export that info, some just export numbered entry 
points ane expect you to use a .def file or some such.

- Even for the DLLs that do export names, the C-compatible APIs don't 
have info about function arguments, data structures, ...

- The languages that have real reflection models and encoded info 
(Delphi, C++, .NET, Java) aren't compatible with an R2-style model.

Which is what Maxim is working on. You can't use reflection with 
C-style APIs, and API models that you can use reflection on can get 
their own extensions to access them :)

Yes, and that can only come from docs of some kind.

Robert,  even if you know the function arguments, how do you call 
the function itself?


the extension will be compiled, so we can't create generic function 
calls.   The calling convention is part of a function's compilation 
process, on Windows any DLL may have three different argument passing 
methods simultaneously!!


get the problem is that you have to push arguments on the stack and 
retrieve them.  there are NO C functions which manipulate the stack 
directly.  these are always done via assembly, either inline or libraries.

And the associated new extension model.

The command! type is the function type that you export from extensions 
into REBOL. There are supposed to be many enhancements to the command! 
type in the next host kit, and extensions are supposed to be embeddable 
in the host as well. And DELECT is going to be redone too. With all 
of these changes, more enhancements to the extension model are likely 
as well.

Sorry, do you mean that you want to return new, non-string REBOL 
data in a block that you make? Right now you can do that (sort-of) 
with a limited set of datatypes supported, and it gets marshalled. 
There may be hacks to create more, but others would have to answer 
that with the current system.

Example: Rebol code sends [a b [c d] [e [f g]] h i] to extension. 
The extension makes a string out of this and stores it. Later the 
extension should take the string and make block out of it, so that 
the script gets a block! and can do result/1, result/b etc.

And you don't have to export the command if you don't want to.

Just export the wrapper function from the extension's inner module 
and the calling code won't have to know the difference.

Up to now I don't have used a Rebol module to wrap the extension. 
The code just loads the extension. Just to be sure I understand your 
proposal: You would create a Rebol module that the user code uses, 
and this modul would deal with the extension?

There is a module inside the extension, and it gets wrapped around 
the extension automatically at LOAD time. That REBOL code in the 
string constant that you return from RX_Init is the module that I 
am talking about.

So, decision made: Serializing BLOCK within in Extensions is a PITA. 
As long as you want to walk a BLOCK and don't have to deal that much 
with nested BLOCKs it's OK.


But it's not possible to make a to-string block! within an extension 
with a simpe to-string call.


Overall I think we need a way to serialize Rebol datatype into a 
char* or void* in a way that it can be stored and brought back into 
Rebol.

Workaround: Catch blocks on the Rebol side, make strings out of it 
and than go to the extension. Same for the other way: Extension devlivers 
string and Rebol code creates Rebol conform datatypes.

There has been some speculation that the new extension and command! 
model that is coming with the next host kit will help solve this 
kind of thing. We shall see.

As mentioned a few times in the past, I don't think that user-defined 
devices are at all possible, with the current hostkit. And yes, I'd 
love to be proven wrong :)

Ok, since we are not sure, I will take a deeper look into the code 
and try to ask Carl.

I agree with Andreas, when I looked into the hostkit code it looks 
that table of 'names' of the devices is hardcoded in the r3.dll at 
the moment. My only idea (workaround) was to try use for example 
'clipboard device name and try to replace the internal code with 
custom one for testing purposes. I haven't tried that though from 
that time yet. Not sure if it is worth the time. Better wait for 
new hostkit ;)

has anyone tried to build R3 host as a dll, and use it within R2 
 as a routine?

yeah, but some tricks are MUCH easier and probably quite faster to 
do in R3.

we are talking an expert PARSE user, and so far , this would be his 
main reason to move to R3, but then all his prior code isn't upgraded 
so its a bit daunting to just plunge right in the middle of a project.

You are thinking about scale relative to the wrong number. It's not 
a matter of how many lines the parse rules have, it's a matter of 
how many additions there are in R3 parse, and there aren't that many. 
Make the compiler once, and it will compile quickly. It could even 
do so at runtime.

R3 Host-Kit A100 has been uploaded to www.rebol.com.  This release 
can build CORE or VIEW (with externalized graphics lib.)


However, although the graphics lib is there, the DRAW commands are 
still in the process of conversion (Cyphre will be working on that 
part).


It's also an example (and a fairly simple one to learn from) of how 
to build a host-based extension module, including a few scripts that 
show how to process the module source to embed it in the boot.

Although Cyphre and I have both been able to build it using Mingw, 
other's have not been able to do so, and we're not sure why as of 
yet.


Therefore, we're not quite ready to distribute the host-kit to everyone, 
because we need more testing on it. However, if you want to actually 
test it (windows only), then ask Henrik for the download URL.

The host kit comes with all files necessary to build itself.  In 
theory, all you need to do is type "make" and it will produce R3/View 
(as r3.exe).  Also, make core will produce a core exe, w/o graphics.

The code includes some new host source files, for example to handle 
windowing and the related events.


Although this release comes with the AGG sources needed to do the 
build, note that these sources are pre-GNU versions, so are a bit 
older.


In addition, I suspect Maxim or someone may want to try creating 
an OpenGL version.


Soon, we plan to take a long look at making PAIR! into a float-based 
point, to allow better control with graphics. This somewhat non-trivial 
due to the assumptions in code that PAIR! is integer only with truncation 
for math operations.

Ok by me, but someone will need to collect the comments and issues. 
 You want to do that?

No, I mean someone needs to digest the issues and fixes, and submit 
them back.

The host-kit can be built by any reasonable C and C++ compiler/linker.

Graham: I got the message on protocols and it's on the list.

A96 hostkit on Linux and Win32.

The makefile is auto-generated and changes with each release. 


Also, there are big variations in make between target platforms (some 
are really horrible.) So, the makefile is kept very "dumb" to allow 
those builds.

Second, the config controls are already part of R3, and are triggered 
by the reb-to.h file.

I used CMake, which is a cross-plattform tool that automatically 
generates "project" files for a variety of targets (such as GNU Make, 
Eclipse project files, Visual Studio project files). It works great 
on Linux, Win32, OSX and is used by several high-profile projects. 
But I'm well aware that third-party tools are most likely no-go for 
REBOL, so I'm fine with all that, and can easily look into it, as 
long as there' s at least a Linux hostkit available.

Also, problematic builds are Sun and HP Unix. But, anyway, if people 
are willing to add them, it's ok with me.

( the current version of Os/2 is ecomworkstation  and is positioned 
as a secure enterprise client environemnt )

CMake provides the benefit that we don't have to reinvent the wheel 
for now. It has, for example, a nice installer for Win32 and can 
automatically generate Visual Studio project files from the above.

CMake takes the above as input and generates e.g. a Makefile as output.

The method I use for R3 simply uses REBOL to generate the host-kit, 
the makefile, and other related files at the same time.

With CMake the workflow of a user who wants to build the hostkit 
on Win32 would be as follows:
- Download and install a toolchain (e.g. MingW or MSVC)
- Download and install CMake
- Download the R3 Hostkit

- Generate a R3 build script for your preferred toolchain using CMake. 
E.g. use CMake to generate a GNU make Makefile for MingW
- Build the R3 Hostkit (using the generated build script)

Without CMake, those steps would be the same, except for the extra 
installation of CMake and the generation of a local build script. 
But it would also either limit the user to a single toolchain (such 
as MingW, currently) or you would need to replicate some of the functionality 
of CMake in your make-build.r

there's a GUI front end for cmake too, though I guess one could be 
made for REBOL just as easily - this lets you resolve env var issues 
and other things

Further, at least for the A96 hostkit, Linux and Win32 hostkits are 
completely separate packages, and that's an incredible nuisance for 
someone doing cross-platform work.

After adding MinGW\bin and msys\1.0\bin to the windows command path 
and also copying rebol.exe to msys\1.0\bin folder, I was able to 
build it from a normal command window without modifying the makefile.

Carl: in any case, I don't care much about CMake. Just shipping the 
hostkit with your make-build.r instead of a pre-built makefile would 
most likely do just fine for getting an integrated cross-platrform 
build working. (Plus, of course, bundling the libraries and sources 
for all platforms in the hostkit.)

And just in case: _THANKS_ a million for the new hostkit code drop, 
Carl (and everyone else who helped make it happen)!

and now is probably as good a time as any to move this discussion 
to !REBOL3 Host Kit

Robert, if you read above, Carl did not want to make a public release 
unless someone gathered all the comments ... and Andreas & I volunteered.

The curecode tracker does not have a hostkit project ... so I used 
my new Jira tracker.  Simple as that.

Most of the host kit problems are R3 problems as well. We have categories 
in R3's Curecode project, and we can add a Host-Kit category if needed 
(I will do so right now). Multiple trackers are not a good idea, 
in general. We'll see if they become a problem in this specific case.

I consider the current hostkit drop to be hostkit version A100. The 
next hostkit to be released should (and hopefully will) be A101 or 
later.

Carl calls it a release, so I consider it a release. And yes, should 
be in R3 Host Kit

I too vote for an  A101 numbering the next time a package is released... 
its going to be MUCH easier to follow for everyone.   IIRC the A97 
created this strange pre-release condition too and it had to be specified 
each time we talked about what we downloaded.

Robert, did you ever upload your macros stuff that you mentioned 
in Jan?  I looked in the extensions group and didn't see anything

And your sqlite3 extension?

thanks for the test and the feedback, Graham

// 	string to convert to R3 string
//  R3 frame to add string too
REBSER* RXI_Make_String(const char *string){
		// build string
		int string_len = string == NULL ? 0 : strlen(string);

  odprintf("RXI_Make_String with length = %d, string = %s", string_len, 
  string);
		REBSER *s = RXI_MAKE_STRING(string_len, 0);
		if(string_len == 0) return s;

		// and set single chars if there are some
		for(int x=0; (*(string+x)) != '\0'; x++){
			RXI_SET_CHAR(s, x, *(string+x));
		}

		odprintf("RXI_Make_String: done");
		return s;
}


void RXI_Return_String(const char *string, RXIFRM *frm, int frm_position) 
{
		REBSER *s = RXI_Make_String(string);

		// set parameter 1 (=return value?) to constructed string
		// set index to 0
		// set type to string
		RXA_SERIES(frm, frm_position) = s;
		RXA_INDEX(frm, frm_position)  = 0;
		RXA_TYPE(frm, frm_position)   = RXT_STRING;
}

continuing Request for object lookup within host/extensions here....



Here is a proposal (using example C code) for OBJECT access within 
extensions  this uses the EXACT same API as commands, and with just 
a little bit of work on the extensions API by Carl and a single hook 
within the r3core, we could allow callbacks within objects.


the advantage is that we re-use the same safe sand box as commands, 
and don't require to do much coding to enable it, AFAICT.


since object lookup is performed on demand, we can very rapidly supply 
an object to the command, and it doesn't cause any heavy-handed conversion 
at each command call if the object isn't needed.  more fine-grained 
control could be added so we don't need to frame the whole object, 
but it would be usefull as-is already.

RXIEXT int RX_Call(int cmd, RXIFRM *frm) {
	i64 i;
	
	// objects are frames, exactly like command arguments.
	RXIFRM *obj;
	

 // return values are also frames (with one argument), for type-checking 
 purposes.
	RXIFRM *rval;
	
	if (cmd == 1) {
		// ask the core for a frame which represents object.
		// attributes are returned in order they appear in object
		//

  // the command frame doesn't include the object frame, only a handle 
  to the

  // object which the core can understand. we then ask for the object 
  frame on-demand.
		obj = RXA_OBJ(frm, 1);
		
		switch (RXA_TYPE(obj, 1)) {
			case RXT_INTEGER:
	            i = RXA_INT64(obj, 1);
				break;
			
			// we could build a frame for supplying arguments to the eval
			case RXT_FUNCTION:
				rval = RXA_EVAL(obj, 1);
				if (RXA_TYPE(rval, 1) == RXT_INTEGER){
	            	i = RXA_INT64(rval, 1);
				} else {
					// return error
				}
				break;
			
			// wrong type
			default:
				// return error
				break;
		}
		
		// do something with the value
		my-native-function(i);

	}
}

so what do you guys think?

So objects with more than 7 fields wouldn't be supported? Command 
frames are marshaled, remember, they are not references to stack 
frames. If you want to pass an object and get access to its fields 
perhaps passing it as some kind of handle and getting access to the 
fields through accessor funcs would be better. Unless I misunderstood 
your code and you are doing this already.

the frame size is an artificial limit. 


the frame is just an array, there is no real reason for it to be 
limited to only 7 items, its just an initial limitation, (probably 
to simply the macros and support code).

another approach would be to access a single attribute at a time, 
but anytime data is exchanged, I think re-using the current frame 
code is ideal... 


it already solves all of the type and memory marshalling, separates 
the external data from the core, solving the issues with the GC and 
stuff like that.


as long as the API copies data when its setting values back in the 
core (which I guess its already doing) then I see no point in inventing 
a second API just for objects.

IMHO, we cannot prevent marshalling without exposing a lot more of 
the core into the extensions, and/or making extensions unsecure (able 
to peek directly within the core datasets).

I looked at the example ext-test.c and it lacks the extension header 
as specified in the docs

The extensions docs say that dealing with handles is undocumented 
...  but if it's just an integer, can we just pass that back and 
forth between rebol and the extension?

the idea of handles is for an extension to give a key to a script, 
and make sure the script can't play with it.   by key, I mean an 
arbitrary value which makes sense for the extension (pointer, index, 
value, etc).

re: "GC recycling interruptions"  if we can browse the objects directly 
from extensions a lot of the need to "bake" command access within 
blocks and execute via do-commands is alleviated in the first place.

And, actually, the RXI_API would be RXI_GET_FIELD (because RXI_GET_OBJECT 
should be reserved for later obtaining parameters related to the 
object itself, such as number of values stored, etc.)

if you can add some object reference to the hostkit, I'll be switching 
over all of my tools to R3, may even help with low-level AGG, and 
finally start forgetting about R2.

as an example, we could just represent a draw structure using primitives 
as objects (which can include several AGG primitives) and call AGG 
native code directly from the object tree.

something like 
draw context [
	prim: 'cross-box
	offset: 10x10
	size: 20x20
	thickness: 1
	border: black
	cross-color: red 
]


would draw a box and two cross lines, without the need to reduce 
this at every refresh when its refreshed and data comes from external 
manipulators:

draw compose [
	pen (prim/border)
	fill-pen none 
	line-width (prim/thickness)
	line-pattern none
	box  (prim/offset) (prim/offset + prim/size - 1x1)
	pen (prim/cross-color)		
	line (prim/offset) (prim/offset + prim/size - 1x1)

 line (prim/offset  + (prim/size * 1x0)) (prim/offset  + (prim/size 
 * 0x1))
]

but the first one becomes a persistent object and doesn't need to 
reduce over and over.  


when the structure becomes somethings which goes beyond 50kb of draw 
source,  reducing over and over hits the GC very fast.

also note that I gave this example as a Draw, since we can all quickly 
see what is happening, but the above applies to many other datasets 
which are used libs, and for which objects are much more user-friendly 
than objects.


think of C libs which use deep struct trees. I'd rather keep the 
user interface object related.

yes... but they also allow context on the REBOL source side.


note that with the example above using objects, I don't need to go 
thru a draw block on the native side.  I could inspect the object 
and fire AGG (or other lib ;-) commands directly

Yes, but there is a problem in your example above in that one is 
an apple and the other an orange.

It's funny, I go back and forth a lot on my own designs in regard 
to object vs block.

For the human side, such as providing a style sheet containing graphics 
attributes, object is the winner. However, as that style sheet is 
processed, it is flattened into a sequence of commands sent to the 
AGG rendering engine.


Now, it's probably possible to change our API into AGG to use objects, 
and that's probably find, but I'm not sure that it's really any more 
efficient.

true, but objects can be nested. and a single small object, like 
in the above, may actually represent MANY draw commands.
 

for example... a single block of text strings... may actually represent 
all the items of a text list.  parsing that list to fit things within 
bounds. 

re-creating the whole AGG block as you scroll the list, forces you 
to possibly generate a few hundred draw items in a block.


but you have to build that block using intepreted code, which only 
ends up being an intermediate in order to pass the visuals to the 
rendering.


with an object, constructing that visual can all be handled on the 
native side and will save a lot of work on the interpreter and the 
GC.

the object will contain a few parameters and it represents the list... 
but rendering it will all be managed in the native side.

It seems to me that the such a method result will be a program that 
is many times the size and complexity... and hence cost and brittleness.

Of course, I may not be understanding precisely your method.

at some point, the benefits of context and object instantiation become 
invaluable.

And, mainly at the human edge.

liquid is at the center of this, of course, but without objects, 
I coudn't easily hold state, process , cache and data together.

liquid uses state to control processing.  each node is like a mini 
kernel only aware of itself.  by linking stuff together and with 
messaging, all the nodes cooperate.


right now, liquid is able to completely control a complete GUI forking 
of only the refresh of data which actually changes.  but i cannot 
implement some liquid's low-level code within extensions because 
they can't look up the nodes.

nested draw blocks may be slower in R3, but I see a benefit with 
Maxim's way. It allows easy composing and inheritance of transformations 
in the sub-blocks, like skewing, rotation, translation, scaling.

in R3 I could go far beyond, actually dumping some of liquid's processing 
directly in the native side of an extension.


in fact, I could use AGG or OpenGL directly without even using gobs 
or draw blocks since the interface is built by how you link things 
together and this linking and messaging will automatically fire-off 
AGG commands, without the need to go through an intermediate.

liquid is at a lower level than gobs (its not strictly gfx related) 
and I need object access for liquid (and other things).

Yes, true and useful too.