AltME groups: search

also posted a private msg to you on R3 re this

Maxim, let me find a direct private channel to you.

Check R3 Chat. I'll be sending you a few notes.

The method seems sound, but I think there could be a problem with 
the quantity of tests generated. Currently, with only five (of 56) 
datatypes tested, there are more than 22'000 test vectors, and they 
build exponentially. We may end up with more than a million test 
vectors for the final suite.

it might even provide other insight on running such a heavy script

Or give Carl the impetus to build a grid/ipc/distribution infrastructure.

Graham, SQLite uses over 2 Billion tests before doing a release.

Guys .. I didn't write that .. it's a quote from Carls blog

write %qt2.dll read http://www.compkarori.com/r3extend/qt2.dll
secure [ extension allow]
test


and this brings up a Qt window with a push button .. says "hello 
world"

Looks like Nokia sponsored the development of PySide, the Python 
bindings for Qt4 .. some 300 odd functions etc.  Wonder if they might 
sponsor a R3 binding!

There is a Python port to at least two of Nokia's cellphone operating 
systems.

I have successfully mechanically generated  a complete r3 extension 
and test script, by simply parsing a C language .h file.
  

if you have a makefile ready for your extension, it will even compile 
the extension for you and launch the test script in a single pass 
of the engine.

this is based on the latest r3 A101 build.

congrats maxim, THIS is a real milestone.

Some good SWIG-like tools will helpe people a lot.

I guess there will be a lot of more work and testing ;-)

the current code is a single ~16kb file,  so its not too large right 
now.


obviously as more types are added, this will grow, but since I'm 
using a "rebol-formatted intermediate source tree", its easy to improve.


the parser just generates a version of the C source in this "RFIST" 
format, and the generator only has to convert this into something 
else.


in time, we might even build different emitters for the "RFIST" data.

one fun thing is that we do not have to use the C source types as 
the interfaces for the commands.

my next test is to build a generic object to parameter map.

so, if you have a C func declared as:
int myFunc (int a, int b)

you'll be able to call it from rebol using:

obj: context [a: 1 b: 2]
myFunc obj

and my tests (with a fix or two)  and successfull builds probably 
helped him be a bit more confident.

this is a genetic approach :)

And my other suggestion about releasing a linux version also still 
stands

cause right now you can't compile extensions as DLLs... a small side-effect 
of the implementation of pair floats.... just related to linking.

no the hostkit release broke this capability when you compile the 
DLL, there is a problem in the order of #include... but its fixed, 
will be part of release... which Carl has confirmed is happening 
very soon (probably today)

its funny because the #include bug left me in rage as I was trying 
to fix my makefile (I'm VERY poor at this stuff) .


but 2 hours later, editing my scripts, makefiles and stuff, I can 
say that I've grown to understand make quite well... 

so this bug ended up being a good thing  :-)

I feel like this release is my own pet release... Carl added the 
object! handling in extensions just for me... so I felt a certain 
responsability in working with carl to test it.

Is there no way for an extension to call a R3 function?

man exceptions are quick. 


more than 4 million adds a second, even though the values are cast 
to 32 bits and its using a stub (it jumps into a C function from 
a library) .

But I guess you're going to have to open and close a new connection 
for each query which is not very satisfactory

and it has to run faster than the PC version which has a 1000 times 
faster CPU  ;-D

If you import one extension which has an exported function such as 
test and then import a different extension which has the same test 
function, it only recognizes the first.

that's a drag

its nice that the extensions use a module to negotiate the access 
of a lib.

can we force an extension not to export anything?

It's not a problem with host-embedded extensions because they are 
not dlls in the first place.

I notice that as long as my r3 instance is running, the dll I imported 
has a file lock on it.

For example, in the case of two modules with same function names, 
you can use a direct reference to clarify which one you're referring 
to, or bind your code to a specific module. Easy to do.

There's no unloading, but it's possible, however, very low on the 
todo list considering that most extensions are long-lived.  The locked 
DLL G refers to above is because the code segments are loaded. (That's 
more of an OS implementation issue, a poor one IMO.)

I've read all the docs .. as to whether I remember the content, that's 
a different issue

Has anyone thought in detail about what is wanted for callbacks, 
or is it just a general idea that is wanted?

We have tasks now. Just because they don't work properly doesn't 
mean they're not there, and certainly they need to be a consideration 
eventually.

If I bring up a GUI in Qt or something .. I'll need a way to call 
rebol from the GUI as otherwise I have no way of communicating except 
on exiting the GUi

That's a messaging request.

So, you could use a socket or IPC, etc.

don't know if this has been discussed wrt callbacks, but it would 
be good to be able to call back into REBOL for purposes of controlling 
execution, say for debugging from a non REBOL editor/IDE

That's more of a reverse callback, right?

Graham, if it's a task (OS thread), then no, it's not blocked and 
has its own stack.

But we can't do that at present ... start a thread to call our extension

I am having a slight policy problem with the delayed module model, 
which could some feedback.

Callbacks work both ways, which has been my point for a whle now. 
But, we'll need to know what we want.

What we probably should do is designate a PORT type for intertask 
messaging, and make it easy to use.

a Rrexx port

If you are adding a module to the module list, and there is an existing 
module of that name, then the new module either overrides it, replaces 
it, or doesn't get added (possibly with an error triggered, but so 
far not). The question is which one to do in the particular circumstances. 
The factors are whether it is the same module, for whatever "same" 
means here considering it might be reloaded or still source; whether 
the versions are the same or greater; whether the existing module 
has already been made or is still source, and the same for the module 
to be added.

simple example 

view [ button "Print" print "hello" button "QT" do [ test ]]


when I click on the QT button, I bring up a QT window but event processing 
stops on the Rebol side .. the Print button no longer responds

all the test function does is open up a qt window with a button and 
starts qts event handler

quick heads up: I built A101 just fine (a Win32 build cross-compiled 
on Linux) and the "go.r" demo still works. so thanks for A101, Carl!

So, the go.r demo is a heads up display?

the main difference between a message based callback and function 
stack callback are in synchronisation and locking.


they both have advantages, but many libraries require function stack 
callbacks.

Brian, my callbacks are thread safe in regards to tasks and threads 
(I read extensively on the subject when I did my hack).


but I talked with Carl, and in fact they are not safe within a single 
process to begin with

to make it work, carl would have to make a purpose-built C function 
which makes sure that memory used by a command is safely preserved 
for the length of the command, or something like that.

We have a task! type now, and have had it for a while. It doesn't 
work well but when last I checked it does work. Of course none of 
the system supports its use, and error handling in a task will crash 
R3, but that all is fixable.

I am hoping for a certain level of non-shared memory and intertask 
communication, but we'll have to see how much we can get away with.

I'd build a device model for the tasks, this way it would be async. 
 just don't inlude stupid OS based limitatios like in python, where 
the thread which creates a thread, has no way of forcefull closing 
it.  so if a tread hangs.  your whole app can hang with it.  :-(

Kind of a linear thing. If a task takes ownership, it becomes copy-on-write 
for other tasks.

internally that can work, but the issue arises when two threads share 
a series. if they write at the same time, even with a copy, things 
like indexes might not update atomically to the copy operation.

the way I see it, when a thread is launched, it should Copy the whole 
environment and become completely independent.  data exchange between 
tasks is done explicitely, and no GC allocated data should be able 
to cross thread boundaries.  obviously using external libs you may 
have some fun, but then its OS allocated and not managed by the core, 
so its your call.

utility functions like  synchronise would be nice. but they would 
be something you call manually.   I find its always better when the 
code actually implements the paradigm, rather than trying to hide 
it.   a bit like Apple's GDC.  you explicitely create little tasks 
which you launch and wait for completion.  its simple, obvious and 
highly scalable.

A model with some shared data can be more memory-friendly on small 
devices too. While the native code can just use shared pages, the 
mezzanine code needs to be per-process. And from a memory standpoint, 
mezzanine code takes up a lot more room than native code. Shared-nothing 
would kill R3 on portable devices.

but the loaded function is now immutable in R3, so it can be shared, 
as long as there are no "static" or global variables in the code. 
as opposed to a string being hacked away within that function.

it would also be nice to have something like GDC in the thread API. 
 something specifically designed for short-running bursts.  something 
like....

draw this image, read this web page, etc. this way, an app could 
divide a process into parrallel tasks, linked in a port or something 
so that you just need to wait on completion of some or all tasks.

breakup rendering of an sequence into several threads and instead 
of messaging long running threads and having to build a complex application... 
you just something like

render-sequence: func [data frames] [ 
	repeat frame frames [
		dispatch 'pic-renders img: image-render data frame
	]  
	wait pic-renders 
	dispatch 'encode-movie do-encode %movie.mp4 img
]


the thing here is that the dispatch setup is external to the source 
code using bursts, and could be configured to use more or less threads.

you've got two independent bursts with a dependency

As for callbacks - I surely don't understand the issue properly, 
but in overall - we are a messaging language. We should aim for message 
passing based kernel, as Amiga was, and as QNX is. Let's use events, 
ports for IPC, when appropriate ...


As for tasks - my understanding is, that Carl still plans on threads 
internally. Didn't BrianH said yesterday, that in modern/future multicore 
architectures OS tasks are better way to go?

note here that we DON'T wait for the second dispatch to finish, we 
just make sure its got all the frames before  dispatching it.


then you could have a burst config which allocates different number 
of threads based on dispatch labels... maybe there's 4 rendering 
threads and only one encoding thread.

but you could dispatch that  function  too...

render-scene: func [shots][
	foreach shot shots [
		dispatch render-sequence shot/data shot/length
	]
]


so here you just send the whole scene to render, and it would only 
use the allocated number of 'pic-render threads allowed accross all 
shots.  :-)

Max - we should "copy" Amiga/QNX, and not a Python - simply put - 
REBOL uses its own OS-like advanced mechanisms - ports, devices. 
Codecs are turning back to ports too. And as for tasks and IPC, we 
should not just wrap OS, but use the REBOL way of doing things once 
again ....

with a few polling function it could be really nice:


completed-bursts 'pic-renders  ; reports progress done on overall 
pic-renders for all shots.

cancel-bursts 'pic-renders ; stops all pic-renders and bursts which 
depend on them.

interrupt-bursts 'pic-rendrs ; puts bursts (and all depencies) into 
a wait list, to be restarted later.

this is NOT os way, its an api over a thread model which doesnt require 
the programer to know about tasks and deal with messaging, IPC, bla 
bla.


in the vast majority of cases, threads don't need to be persistent, 
and is in fact a burden to have to manage.

using OS threads is the only way we can use multiple hardware threads 
in the same application.  otherwise, they will be completely different 
processes, and that makes a BIG difference in memory consumption, 
especially on GUI Oses which open up a lot of libs on startup.

What we probably should do is designate a PORT type for intertask 
messaging, and make it easy to use.

Yes please. :-)

this C header file:

//---------------------------------
// r3t_integer_add
//
// test: print [ r3t-integer-add 1 0 " > expecting: " 1]
// test: print [ r3t-integer-add 2 2 " > expecting: " 4]
// test: print [ r3t-integer-add 2 3 " > expecting: " 5]
// test: print [ r3t-integer-add 0 0 " > expecting: " 0]
// command-format: [object!]
extern int r3t_integer_add(int a, int b)


will tell the tool, to provide an object interface to the function 
rather than to expect two integers.

there is also a lot of work to do in order to provide support for 
more datatypes, but that can be done concurrently by a few of us, 
once the tools is public.

I also want to add A LOT of comments to make the source more litterate. 
 in fact I expect the source to contain more comment bytes than code.

#define RXI_SET_FIELD(a,b,c,d)      RXI->set_field(a,b,c,d)

its just a massive amount of comments, the reason being that I want 
this tool to be easy to maintain in the long run and or it to be 
easy to modify for people who have special needs.

Max - please define, WHEN are you going to be ready with your "last" 
feature, because judging upon your Glass project, which should be 
delivered months ago, we can only see a constant teasing, but nothing 
conrete being delivered ...

Wow, I was gone for a few days.  Really exciting developments!

and since we can specify as many alternate command formats as we 
want per C function, we can build a library of parameter-less rebol 
commands which call the C functions with usefull values built-in 
to the extension.

Has anyone else considered what needs to be done for intertask communications 
... are we thinking maybe something like a RPC on the Rebol side?

Sounds like a dinosaur approach ...

I've thought about it on and off. There are a million models to choose 
from. I don't think RPC is the way to go at all though. I'm for channels 
and messages. On a larger scale, I like the tuplespace model.

I'd like a tuplespace implementation too - I've had a good experience 
using GigaSpaces Jini/JavaSpaces implementation.  Very intuitive 
(and simple in terms of API) concept.

as a testament to what is now possible with the new assimilation 
engine, just look at how insanely flexible it is at allowing you 
to provide just about any interface to a command...

using this information in the C header...

// command-template: [obj [object!]] prefix: obj- suffix: 3
// command-name: rxt-iadd
// command-template: [a [integer! decimal!] b [integer!]]
// command-template: [obj [object! x y]] prefix: xy-
// command-name: plus10
// command-template: [10 [litteral!] value [integer!]]

extern int r3t_integer_add(int a, int b)


;--------------------------------------

one ends up with this command interface... all calling the same C 
function.

    rxt-module: {REBOL [
        Title: {r3-test-extension extension}
        Name: r3-test-extension
        Type: extension
        Exports: []
    ]
    rxt-obj-r3t-integer-add3: command [ obj [object!]]
    rxt-rxt-iadd: command [ a [integer! decimal!] b integer!]
    rxt-xy-rxt-iadd: command [ obj [object!]]
    rxt-plus10: command [ value integer!]

    rxt-r3t-integer-mult: command [ a [integer! decimal!] b [integer! 
    decimal!]]

;----------------------------------------


note that in this setup, auto-exports is set to false and an rxt-prefix 
was added to every command name automatically.

also note that the command templates are not required, they just 
allow you to manipulate how a C function is mapped to REBOL.  there 
are still many features to come, but right now I am working on finishing 
the last phase of assimilation which is the actuall extensigon C 
source code generation.

btw, this command  


rxt-r3t-integer-mult: command [ a [integer! decimal!] b [integer! 
decimal!]]


was automatically generated by a function without any command-template, 
later in the header file.

well, the fact that you don't have to write the command extension 
code itself in order to implement those 5 different interfaces is 
already a benefit.

It's a discussion page, so it's not in any way complete yet.

Graham, on IPC, I think Carl has to be involved in the initial spec. 
I've done point-to-point systems, using TCP ports, but that's not 
a general solution. I've done a tuplespace over Rugby, but it required 
polling. I wanted to revisit Uniserve to replace Rugby in that, but 
it fell off my radar.


Ultimately, I think we need a central process--the tuplespace--that 
apps communicate through. To be general, scalable, and lightweight 
while allowing us to build something like GigaSpaces on top of it 
takes more net guru skills and design effort than I have mustered.

This design-with-discussion model we've been doing lately has worked 
pretty well for modules. Perhaps it can work for callbacks and IPC 
as well. All you need is a lead designer (so it isn't design-by-committee) 
and some quality feedback.

isn't Carl an author of AmigaOS, a message passing based OS? Aren't 
IPC techniques well known and defined concepts nowadays? Maybe we 
should not come-up with something completly new, just choose what 
fits our model. E.g. we have got async devices ... what about having 
IPC/messaging device?

If we do IPC through the port model (likely) then there will be a 
device as part of that.

I think interop is important, but I don't know if there's a clear 
cross-platform choice. I looked at http://www.spread.org/some time 
back, but it doesn't support Windows and is probably best accessed 
as an extension. 


I can pull notes together for discussion, but I don't want to spend 
time on it without knowing that it has some chance of it being helpful 
and making a difference.

IMO IPC shouldn't be a default thing as there are zillions of ways 
how to do it.

There are zillions of ways, and having choices is good, but if we 
have a standard it will be much easier to write tools that can coordinate 
with each other. If we have rebrowser (or, even better, a generic 
plug-in based app framwork that rebrowser is built on :-), it may 
be something that gets built anyway.

Out of the zillions of options, and with Carl's background and knowledge, 
what is the best option for REBOL (or maybe the top 3)? If Carl has 
ideas/opinions/designs already, could he write up a paragraph that 
outlines how *he* thinks it should be done?