AltME groups: search

This layer handles all of the nasties for complex math like shadows 
and even some of the SW & HW shader/texture manipulations at a higher-level.

both, scene creation will be dialect based, with a full mapping of 
EVERY single public class, member & method

above: "with a full mapping"  I meant that  "there will *also* be 
a full mapping"

I know... but we can build code around the fact that its symbols 
won't be used for a long time.  like I say, its not something you 
do without knowing what you are doing.  any commands refereing to 
the old lib, can be replaced with no-ops raising errors... this would 
make it safe.

I haven't read everything here--just trying to clear a few groups 
so it's not as overwhelming next time I make it back here, but shouldn't 
it be possible to write the equivalent of R2's library interface 
*as* an extension in R3?

yes, its the way I think it should be done. we could provide pseudo 
type middle ware in the extension to accomodate incompatible types 
like Unsigned integers or structs, much better.


I plan on building a block->struct interface when I start doing my 
more serious extension development... (waiting for next extension 
enhanced release.)

If you are willing to give up support for MAKE, MOLD, built-in reflectors 
and path access, you can implement structs as handles. You would 
access them through accessor commands, and struct and routine declarations 
would be a dialect that would be compiled to calls to extension code.

yeah, that's the idea... using a block to struct dialect to build 
the structs from scratch, then just refer to them after.

I already did a header file scanner a few years ago... but it wasn't 
recursive... nor did it resolve macros.

that is a great idea... generate the intermediate C files, and convert 
them into rebol extensions... using only the top-level headers as 
what is exported as commands and defined structures.

I also thought of a very neat way to map c methods and struct access 
 :-)

It's the JIT compiler that is used as the backend of DotGNU Portable.NET, 
among other projects. LGPL, generates x86, x64 and (experimentally 
so far) ARM. SSA intermediate code. We can make a compatible SSA 
REBOL dialect and build up a compiled dialect from there. The *existing* 
extension model is sufficient, no more changes required.

Tiny (about the same size as TCC), and a good first project that 
could be adapted to wrap LLVM if we like, as long as we keep our 
portion of the extension code license compatible with BSD.

it generated C code compiled by a (very high-eng) compositing software.

In case you were wondering, this is the main reason I was working 
so hard on R3 - so I could work on a project like this. Now that 
the parsing advancements are happening, soon R3 will be good enough 
for me to do my projects. For me, R3 will be released.

the reason this software (shake, now owned by apple) was so much 
faster at rendering, is that it wasn't a rendering software... it 
was just a C disguised C compiler :-)

when you pressed the render button, it actually compiled your project 
(in a matter of a few micro-seconds) and ran it.

Apple is moving rapidly to LLVM because they can't get the GCC project 
to cooperate. They already use it for iPhone development and OpenGL. 
Having a path to get to LLVM would be a good thing.

my callback example, really was similar to a device/port implementation 
IMHO.

It was a good choice :)

PPC to x86 transition. x86 to x64 transition. OpenGL emulation on 
old video cards that don't support new features. Instruction set 
independence for the iPhone. Objective-C support in the front end 
that can be integrated with their development tools without license 
restrictions. A development commuunity that will accept their patches. 
The list goes on.

As for LLVM for a target - what is the plan - to port entire REBOL 
into LLVM? Where can you find LLVM by default? Is it e.g. part of 
Linux distros?

The standard REBOL DO dialect isn't compilable, though in theory 
PARSE could be if it were made less dynamic. What we could do is 
a compilable dialect that would resemble a subset of the DO dialect, 
similar to RPython of the PyPy project.

my point has always been, that IN GENERAL, REBOL code is not compilable. 
that is, for every compiler you write, i can write a piece of REBOL 
code that is perfectly legit but your compiler can't compile. That, 
however, does not mean that you can't compile 99% of existing REBOL 
code. In fact, I do have a compiler that can handle most REBOL code 
(it's unfinished, i need to add some kind of type inference to make 
it useful enough). it can already take REBOL code as input, and produce 
optimized REBOL code as output.

also, JIT compilation is a whole differen thing than static compilation. 
You can probably JIT compile *all* REBOL code, however, it may be 
slower than the interpreter in some cases.

In either case, compiling REBOL requires thinking about REBOL and/or 
compilation differently, and trying to do so would push the envelope 
of compilation as a subject. And thus would make an interesting project 
:)

In general though, REBOL's semantics have been optimized for interpretation. 
As such, compilation may not give you as much of a speedup as you 
might otherwise think. This is why I was thinking of a dialect with 
semantics that would be more appropriate to compile. The syntax could 
be the same (to save on parsing code), but the semantics subtly different.

I tried to compile an extension for R3. Compiled as a C dll, it's 
ok, but I need to compile it as C++,as I will use com calls. So far 
it seems that R3 is not compatible with a C++ compilation of the 
extensions. Any experience on that ?

BrianH can probably give you some pointers on what to try out if 
you're a proficient C++ coder.

It shouldn't be difficult to wrap the function declarations in the 
extension header files in extern C blocks. That would handle the 
C++ incompatibility. You would be able to write your C++ code within 
your extension, and still talk to R3 through the C model. In the 
longer run we intend to tweak the headers so that they are compatible 
with more compilers, so that you can be binary compatible when your 
code is compiled with a different compiler than R3 is.

Maxim, Brian,
your suggestions were correct. Thanks


I succeded with very few modifications to the example given by Carl:
( tool: Visual Studio 2003)

- the source file extension must be changed from .c to .cpp for a 
cpp compilation
- in reb-ext-lib.h change the line 
#define RXIEXT _declspec(dllexport) to
#define extern "C" RXIEXT _declspec(dllexport) 


I have not fully tested the extension, but I was able to do a simple 
Text To Speak using windows SAPI5

sorry , I made a mistake :- in reb-ext-lib.h change the line 
#define RXIEXT _declspec(dllexport) to
#define RXIEXT extern "C"  _declspec(dllexport)

I have coded a simple Text To Speech Extension, using SAPI5:
http://colineau.societeg.com/rebol/R3_extensions.html

I would be interested to know any other experience in R3 extensions

I did a few tests loading up OpenGL/GLut and it worked without a 
hitch... waiting for Carl to add a few features before I can continue.


screen shot of a rotating cube http://www.pointillistic.com/open-REBOL/moa/steel/R3-OGL.png

I remember you posting a similar screenshot a couple of months ago. 
What's the difference now?

I try to keep this as short as possible.

Imagine you have this file, called %test.r:

==file==

REBOL[
    Title: {Simple extension test}
	Name: ext-test
	Type: extension
	Exports: []
]

map-words: command []{
    word_ids = RXI_MAP_WORDS(RXA_SERIES(frm, 1));
    return RXR_TRUE;
}

fibc: command[
    len [integer!]
]{
    RXA_TYPE(frm, 1) = RXT_INTEGER;

    i64 o = 0;
    i64 l = RXA_INT64(frm, 1);
    i64 i;

    for (i = 0; i <= l; i++)
        o = o + i;

    RXA_INT64(frm, 1) = o;

    return RXR_VALUE;
}

add5: command [
    a [integer!]
][
    a: a + 5
    return a
]

==end of file==


And now imagine that in R3 console you are in the directory where 
you have the file %test.r .
Now you type:

>> compile %test.r
>> import %test.dll
>> fibc 10
== 55
>> add5 5
== 10

And that's all.


If you want to try it, you need to have TCC (TinyC Compiler) - get 
it from http://download.savannah.nongnu.org/releases/tinycc/tcc-0.9.25-win32-bin.zip
The script expects it instaled to %/c/tcc/ but it can be changed.
Then go to r3 console and type:

>> do http://box.lebeda.ws/~rebolek/rebol/srec.rip
>> cd %srec/
>> do %srec.r


Then you can try COMPILE etc. (see above). %test.r is included in 
the archive.

SREC is shortcut for Simple REBOL Extension Compiler.
    
Enjoy! (if it works ;)

janko congratulations on your extensions... I knew extensions where 
the door to a whole new world of possibilities for REBOL.

Ass soon as I know how to call a Rebol function from Extension with 
some simple parameters I'm ready to start.

Not that I'm aware of. Callbacks are a separate issue, which is supposed 
to be handled by devices, afaik.

so, Robert, not sure if you understood all of these replies as even 
I had a bit of a tough time to "get" them.


Right now, Extensions only allow REBOL to call functions from a dll. 
  What I would like is to simply improve the extension model so  
it can also call REBOL code, as a callback or something else, but 
there are a few issues which make this a non-trivial thing to do.


So far there seems to be a generalized idea that there should be 
a different kind of extension which allows this, but I see no reason 
why it should be another, different. api.  having one DLL   for  
REBOL -> DLL  and another for  DLL -> REBOL  makes absolutely no 
sense to me.   IMHO we need a single DLL able to do both.  Even if 
it means a little bit more work to design it.

agreed - just don't put another burden on user. By dismissing DLL 
interface, many users will not be able to make otherwise usefull 
things at all. So if it can be done in terms of current extension 
API, even if it would mean some additional work, let's add it. I 
don't want to hear, that in order to do a callback, feature available 
in R2, I hav to do some other tricks.

It's build from the ground. It parses the dialect and translates 
it to c equivalent (a: 5 becomes a = 5 and so on).

Good idea. And you have strong typing rules, right? So if a var is 
defined as an integer, you can't change it to something else along 
the way?

Hm, I'm not sure right now, I think it was possible to change it, 
but I may be wrong. But I think there were no error checks, it was 
just a basic version that can produce something working and I haven't 
much time to improve it since. That has changed recently.

I was looking at making a libtcc extension, which would allow something 
like RebC to be used as a JIT compiler.

Eventually I was going to make a libjit or LLVM backend, but it looks 
like I can get libtcc working sooner.

Not the first thing on my list though (which is HTTP) so if you want 
to take a crack at it first, I can help with the extension/module 
model.

A command! is an indexed dispatch function, and the index has no 
inherent meaning. You could dynamically generate functions with libtcc, 
which would all have the same function signiature because they would 
just take command! call frames. These generated functions could be 
referenced from an array of function pointers. After you generate 
a new function and assign it to a new array slot, return the index 
of that slot to the calling REBOL code (embedded in the libtcc extension) 
and it can then make a command! with the libtcc extension's handle 
and that index. Then that command! can be called like any other REBOL 
function.


A trick though is that the generated C code would need to use the 
extension macros to manipulate the function arguments, rather than 
direct variable access. In other words, your generated functions 
would be extension-style code, not regular C code.

Pekr, the reason callbacks are so tricky in R3 is that you need a 
way to dispatch to the correct task within REBOL. R3 is not going 
to be single-tasking anymore, so direct callbacks will be impossible: 
You have to coordinate between the task/thread of the external code 
and the tasks/threads of R3. Devices manage that coordination and 
synchronization.

Bolek, if you have the commands that are exported from the libtcc 
extension themselves implemented as functions with the same signiature 
as the generated functions, you can call them through the dispatch 
array as well. This would reduce your RX_Call function to just a 
few lines.

Playing with the extension example: IMO it's done  to complicated.


- Why do I need make-ext.r? Do I always need it or just for this 
specific example?

- Why is the init block a const char array and not just a plain ASCII 
text?

How do I return a new string back to Rebol?

I seems I need to construct a RX string and set every single char?

This uniton gives a warning/error with mingw:

typedef union rxi_arg_val {

	i64 int64;

	double dec64;

	REBYTE bytes[8];

	struct {

		i32 int32a;

		i32 int32b;

	};
	struct {

		u32 index;

		void *series;

	};
	void *handle;

} RXIARG;

I commented the struct part and than it works. So either we need 
to give it a name, which will result in a ->struct_name.series sequence. 
Not sure if this makes any difference instead of just putting the 
members into the union.

yes, it makes a difference

(since it is a union)

BTW: I'm currently making a R3 SQLite extension.

Looks like one need to be carful when designing the command interface 
(arguments) because if you insert an argument, the refinements shift 
to a new position. As the access to arguments on the c side is via 
positions you need to adjust your code. Need to think abou a good 
way to abstract this.

is there a document showing how to define refinements in extensions 
?

well in fact I have not found any mechanism to call a function with 
a changing number of arguments, like, for instance: 
my-func a b
my-func a b c
or, 
my-func a b
my-func/my-refinement a b c

use a block of arguments ?

Yes, of course it is a solution, but not a clean on in my case

in fact, you cannot know, from inside the C part of the code, the 
real number of arguments send to the function.

Jocko... you know... it has never been possible in REBOL to define 
functions with a variable number of arguments...

Is there anyway to do a callback? Or trigger R3 to do something? 
At the moment I use a localhost port for this.

Not at the moment. That is as good a method as any for now. Maxim 
has beeen doing some research on this, and the device model is supposed 
to solve this problem in the long run.

I'm not as much help in this as I'd like, since the current API is 
just fine for what I need to do - at least until we get device extensions. 
Maxim has been more help, since his needs aren't met by the current 
system. If you are writing a database API, your experience will likely 
help refine the model too.

I have been waiting for extensions for a decade, and its almost there.

a lot of stuff depends on the improvement of extensions and addition 
of device extensions.  not just for me but for Carl also.

Unfortunately I am not at liberty right now to tell what that is, 
but I can assure you extensions will have to improve in the short 
term because a new player (company) in the REBOL community needs 
this, already.


this company might become one of the levers to propel REBOL into 
adoption in (several) very large corporations (fortune 500) & scientific 
organisations around the world, so RT has vested interest into doing 
as much as it can to make this happen... and right now... the host 
code and extensions is the key to most of it.

Using Roman numerals has always been a challenge for their coding.

Rebolek, that is not really true - the function still takes a fixed 
number of argument, and you're just passing a unset! value to some 
of them (which is a side effect of R2 passing unset! at the end of 
the block, i think R3 does not even do that)

Gabriele, you're right that it's just a R2 side-effect and it's true 
that it does not work in R3. Not that I miss it.

Doing a R3 extension for this would be a no-brainer if the gzip code 
is simple to call. Getting the data to/from Rebol is easy.

Robert, if you are good at C macros and have a good idea about how 
to improve things, make suggestions. Good safe methods for bulk copying 
of string or binary data into the REBOL values to be returned, or 
from values passed in would be great. Look at the existing extension 
source for an idea about how the current macros work. Safety is a 
priority here, so don't forget the bounds checking.

Ok, so I've not yet provided everything that you'll need to do it. 
I divided the
 extensions release into a few stages:

1. simple - 
just simple access to commands and args
2. series - access to series 
values of various types
3. objects - access to objects (of all types)
4. 
codecs - support for codecs
5. host-lib - ability to bundle extensions 
with the host-lib itself.

So, I need to get you a bit more... in 
fact along the lines that you mention.

Re #6156: Pekr, we ARE NOT giving up on dialects!! There are many 
dialects in RE
BOL, and it is one of the main concepts.

What we 
are doing is removing the strong overlap in DELECT and COMMAND. If 
you l
ook at the DELECT method, it is a small subset of full dialects. 
It implements a
 form of function with optional arguments.

So, it's 
better to move that code into COMMANDS, and allow them to work that 
way
. This makes it much easier for people to learn and use. Even 
me!

Also, REBOL/Services will use this same method, because COMMANDS 
are not limited
 to just extensions... ah the secret is out: COMMANDS 
can also be attached to a
context, making them generally useful in 
REBOL code.

I will check the blog comments to make sure it's not 
misunderstood.

Please could someone translate to me, what does it mean that COMMANDS 
can be attached to a context, and that it will make them useful in 
REBOL code? :-)

The way I see it is  that the code inside a command probably can 
be late bound to a context, rather than the global context, as it 
is now.


when extensions will support objects, this can be pretty powerfull, 
since commands can become virtual and private methods for an object 
where the data is stored in a stuct in the binary (C) side... 


which is EXACTLY what I need for liquid, where I need rebol dispatchers 
but native data storage, so it can scale to hundreds of billions 
of nodes, and yes I already have the solution for the storage/memory 
engine if Carl can give me the means.   :-D

I already found a way to make callbacks extension callbacks in the 
current host distribution, even if nothing in the current rebol native 
code supports it  :-D


will be testing this out tonight and will report on this... I hope 
my idea works.  this would reactivate the OpenGL project along with 
other stuff on the backburner.

It's the dispatch. Right now with extensions, when you make a command! 
it makes a function that is dispatched by a function in the extension 
based on a number (which you can think of ay a key), to code that 
handles the command (the value associated with the key). In theory 
this is not that different from an object! grabbing data from one 
of its slots based on the keyword you pass it. Apparently commands 
will be able to dispatch to objects soon, and the functions assigned 
to slots of that object will handle the command code.


The DELECT dialect model was based on rebcode, mostly on its JIT 
binding. DELECT added the out-of-order, possibly optional argument 
handling to the dialect decoding phase, but the dispatch phase was 
mostly left out (I commented on this at the time). The command! type 
has the dispatch model, but uses the function call model for calling 
the commands. The overlap that Carl mentions is in the mapping of 
keys to command handlers.


If you unify the command mapping models between DELECT and command!, 
then that code can be shared. This means that the DELECT function 
could do the out-of-order dialect decoding, then dispatch the operations 
as commands. Values of the command! type would continue to be called 
like regular functions in DO code or by APPLY, and then dispatch 
using the same dispatch code as above. On the other end, commands 
would either dispatch to objects (including modules perhaps) or extensions.


By the sound of it, this might also allow the command! type to serve 
as a method pointer, but we'll have to wait to see about that :)

they would be globally bound, but still, usefull  I wonder how extension 
re-entry from a callback will react , if it even works... the stack 
can get a mighty mangled hehe    :-)

just a single place where we can put data which is accessible by 
extensions.  that would already make the host that much more usefull, 
especially for testing new host models or devices.  which add new 
possibilities for extensions.


the event device is also not useable for my specific task and I'm 
not sure I can really play around with it without breaking the r3lib 
<-> host integrity... testing will provide clues, I guess.

but on we go with the callback hack  I think I'll name this a   "hackback" 
   the back door, like a "hatchback"    :-D

I think I could have something working for non multi-threaded stuff 
in a little while... I'm working on this now... its the time I have 
to do it ... after that...
I return from a sanity preserving week of vacation.


and yes I hope I can help with the development, especially since 
I have two different devices which need to be added.  

with the current host I might make my hackback do some usefull tests 
to help shape a working model aka prototype.

but the coupling with the core run-time is practically abscent.  
there is only one function I can use to have the run-time do anything 
and thats a pretty simple... do_rebol_string() which basically runs 
a block of code in the global space... beyond that I've only got 
network/file like ports, which basically are streamed I/O.  


I can't create data directly and leave it at the port, in a block, 
like I'd do for a proper event queue.   This is currently my pet 
peeve about the host... 


but let's not be judgmental... I'm VERY happy I have the host, so 
I can at least try to rig something up with bailing wire, duct tape, 
pliers, a bit of string & epoxy glue.  


 Extensions & the core allows me to hide this under a nice fiberglass 
 body  ;-)

just don't go looking under the hood... its going to look like a 
honda vtec... hehehe

that is... when the other-def isnt' a trivial integer... but something 
like a definition of a callback function ...

How do I put NONE into a block?

If the c-level side uses UTF-8 strings as well, can I just use the 
Rebol series as is? get_string returns a decoded string.

I will do a stress-test with a triple engine I have for R2. We will 
see.

In R2 the collection loop to build the result was done on the Rebol 
side. Now it's done on the C-side. This is a lot faster.

can I give a guess that the series building part of the request will 
be about 100 times faster now?

@ brian: I understand what you say about working around... but, with 
the host right now, I can't do anything else than hack up a solution 
in order for extensions to have callbacks.  99% of the real work 
is inside the extension.  when the host will be improved, the extension 
work won't change.  I can't wait for things to happen... this solution 
is just for my own use, and it gives me a good perspective on how 
to help with a real solution.  


I'm also brushing up on my C skills (I haven't done any in over a 
decade!) so doing this stuff is a good exercise anyways.


right now, if I can show to Carl how complicated it is to do some 
stuff, he will have an explicit example and have a better reference 
for practical solutions.  Doing anything... its all just theories. 
 look at the notes on delect and command... this is similar... lets 
start with something.  see where that leads us and then, we have 
a reference to pick on and critique.  


We can have better ideas for the api, just by having something bad 
to start with.... and believe me.. I know my hack sucks ... but I've 
got part of the solution complete, and compiling without warnings, 
and some code sharing between extensions api and the host.  :-)

Maxim, are you preparing a Glass extension ?

Maxim, you do realize that the purpose of the current host release 
is to test and improve the host model, right? Not to build final 
projects? If you run into problems in the host model, try to fix 
them, not work around them. Otherwise your work is a waste since 
the host interfacing model is going to change in the next version, 
hopefully based on your and my feedback. And a callback solution 
that doesn't integrate with R3's multitasking model is worse than 
having none at all - since any code that might be written to use 
it would need rewriting, and probably rearchitecting, very soon.

jocko,  yes and no.   ;-)  


Glass is going to be rebol code only, but its going to be based on 
rebogl, the OpenGL extension I am currently working on (as I write 
this).  Rebogl its going to be an evolutionary process, starting 
with simple high-level pre-defined primitives and colors and then 
will get more and more customisable (deformers, animation, textures, 
programmable shaders, etc).


I am still not sure how the Glass engine will evolve, but there is 
a good chance that it will be based on the scene graph technology 
I am working on for the Scream game engine.  This has the benefit 
that Glass can be used to build the interfaces within the games themselves. 
 But it most definitely won't require you to load a complete  (and 
inherently complex) 3d world manager, just to build a window with 
a form.  if possible, I'd like to have window masks, so that the 
3D forms can actually live like 3d models direclty on the desktop... 
so some of the nice 3d feature isn't wrapped within an OS window 
border.

brian, yes we can add our own devices... in fact, it seems quite 
easy, and I will probably be adding a DB trigger device within a 
week or two.  :-)


the thing is that there aren't any exposed or documented *native* 
hooks from the host into the core... 


so far, I've got a callback library (called wire) working which executes 
rebol code in global context using the Reb_Do_String() r3lib.dll 
exported function  :-)


now I just need to use that library within the extension and see 
how it goes... the moment I have *something* which works... I'll 
stop improving the hack... from there on, I'll just work on the architecture 
of the caller and callee, to see how we could make it simple and 
easy to setup, from the extension and within the application using 
that extension... generically.


the code in between can change completely, it wouldn't actually change 
the extension or application code (that's the idea anyways)... just 
a few includes and headers which map how to link to the callback 
system.  I'll also try to build a device, just to see how that can 
be used instead of callbacks... but I still need to use a callback 
from the extension in order to access the host... so for now my hack 
is essential, whatever I do.


in this case, I'll be dispatching the GLUT events within the rebol 
using this architecture... I should have an interactive OpenGL window 
by tomorrow... crossing my fingers.


for now I am busy rebuilding my old OpenGL project within the new 
cleaned-up MSVS solution I've been working on for 2 days now... there 
are soooo many properties, its scary and long to setup... especially 
in this setup where there are several interdependent projects within 
the solution... but now, at least, when I change stuff at any layer 
and build, it builds all the stuff correctly in one step...

Perhaps RT could provide a service to build encapped exe.

Via a a simple web page where we could post a script then the service 
would return an executable (after the choose of a platform).

so, people would not be obliged to control the construction of extensions 
with this simple use case.

No I don't. there are zillions to choose from but I don't know which 
one is really good and leads to a result.

liquid is a dependency engine, its like a kernel but managing individual 
operations (functions/procedures) instead of whole applications (processes/tasks). 
 Scream uses liquid to build data and make sure it stays up to date 
with whatever data it is based on.. if you change sphere radius... 
the 3d model representing that sphere will rebuild itself... no need 
to know how the sphere model itself works.  


If Glass is based on some of the technology within scream, which 
uses liquid, then things like dependencies between input data, their 
 forms, and the result of that input become impossible to break. 
 there is, as such, no action function as we had in VID.  the interconnections 
from data and process is what defines an application.