AltME groups: search

And then it magically reads your mind and chooses REBOL, rather than 
the R that it does support. We could use that ability in REBOL - 
it would allow us to implement half of the CureCode tickets we've 
had to dismiss :)

Maxim: Get and set of object fields has been added to the extension 
API in A101. So, great things can happen now?


But... before A101 is released I need a decent test for extensions, 
and Robert will be coordinating that project.

This loads up the C++ CImg Image Processing library and prints "hello 
world" in purple

Now to see if I can get tetris up and running

It's good, really. It means that you can secure a sandbox and still 
allow extensions, just by prohibiting writing to disk.

Here's another test 
download  http://www.compkarori.com/r3extend/img.zip
unpack it
run the r3.exe

secure [ extension allow ]
import %3d.dll
test

and it shows a 3d figure you can rotate with the mouse

There are changes in the design, files, and interfaces are made, 
so that, you will have to spend some time for migratintg your applications. 
I tried to maintain is as painless as possible, but still some concepts 
have changed. There some files were removed from the package, but 
not functionality.
 - probably ...

@brian .. what I meant was that many of the C programs I've come 
across use argv, argc in main.  Instead i would pass these parameters 
in the function that is being exported.  Just need to rewrite all 
those c functions that access argc, and argv

Graham: host-args.c -- clean and easy to modify or replace with what 
you need. And special args can still be passed in the arg struct 
if R3 needs it.

(And no, we'll not just dump it to the group, because I don't want 
to flood Curecode due to non-tested status. Best to wait on it.)

Two ... one in May, and one in April 2010

We need a way to test extensions with each release.


Tests need to be able to handle normal extensions as well as embedded 
extensions (because the difference is only a few #ifdef statements.)

There are a few "dimensions" to test the basic mechanisms:

1. some commands (in a named extension module)
2. commands that use and verify all possible argument datatypes
3. commands that return all different datatypes
4. defining and setting words (variables)
5. accessing values in a block

6. accessing values in an object (new feature, but this is one reason 
why we need the test)

I'm sure there are a few other misc items.


We think that the entire tests can be generated with a REBOL script, 
and save a lot of typing. However, that's not a requirement.

And, of course the host-kit that needs to be tested needs to be released. 
IMO we will release a RC to get feedback on the extension part.

Don't you release and then test?  Isn't this back to front?

The concept of "just upload it" does not work.  Here's why:


1. Parts of the API may not work at all, because of course, they 
are not tested!


2. Parts may work incorrectly... and therefore are just going to 
confuse most developers, and they'll get frustrated.


3. Because the code is not tested, it floods CureCode with tickets, 
most of which should not even be posted there.


4. Without a testing methodology, there will be no consistent test 
suite over time... just a bunch of hacked together code pieces from 
various developers.

btw, I have volunteered with robert, but have not gotten reply yet.


I have client work which deals with extensions and have started some 
extension building automation... 

since this testing goes hand in hand with actual work I'm doing so 
I guess Its logical for me to step up and take the task.

And surely the testing argument applies to R3 ... so where is the 
test suite?

The method seems sound, but I think there could be a problem with 
the quantity of tests generated. Currently, with only five (of 56) 
datatypes tested, there are more than 22'000 test vectors, and they 
build exponentially. We may end up with more than a million test 
vectors for the final suite.

The point is that 22'000 test vectors isn't much, and can be generated 
for the most part. We don't have to write them by hand.

And, it's all about test-coverage, not about high number of tests 
triggering always the same code.

write %qt2.dll read http://www.compkarori.com/r3extend/qt2.dll
secure [ extension allow]
test


and this brings up a Qt window with a push button .. says "hello 
world"

I have successfully mechanically generated  a complete r3 extension 
and test script, by simply parsing a C language .h file.
  

if you have a makefile ready for your extension, it will even compile 
the extension for you and launch the test script in a single pass 
of the engine.

this is based on the latest r3 A101 build.

using comments in the C source, you can give instructions to the 
engine on alternate bindings and rebol test code you wish to compile.


the advantage is that these comments are side-by-side with the original 
C source file, so its easy to maintain.

I guess there will be a lot of more work and testing ;-)

what current limitations you have and in what timeframe you could 
solve them?

the current code is a single ~16kb file,  so its not too large right 
now.


obviously as more types are added, this will grow, but since I'm 
using a "rebol-formatted intermediate source tree", its easy to improve.


the parser just generates a version of the C source in this "RFIST" 
format, and the generator only has to convert this into something 
else.


in time, we might even build different emitters for the "RFIST" data.

limitiations, well, just the scope of supported datatypes, both in 
C and in REBOL commands.

@Carl .. you called my suggestion insane, and now you post this blog 
http://www.rebol.net/r3blogs/0327.htmlsaying that you will do it 
anyway!

and my tests (with a fix or two)  and successfull builds probably 
helped him be a bit more confident.

And my other suggestion about releasing a linux version also still 
stands

The linux users tend to be more experienced and yet they are being 
excluded in this process

The rebol 2.7.7 View problems on linux were reported way back and 
it' s only now that they are being addressed ...

We should parallelize testing and development.

my extension building engine will probably be released tomorrow. 
  its currently building extensions which are 100% functional.


now I will add more types and implement the command interface directive 
so that we can mutilate how C code is bound on the REBOL side.

and yes probably, but I can't talk for carl, but he seems commited 
to doing it.

Carl is doing the release because Maxim committed to doing the testing. 
This means Maxim gets the prereleases, and we get the releases.

its funny because the #include bug left me in rage as I was trying 
to fix my makefile (I'm VERY poor at this stuff) .


but 2 hours later, editing my scripts, makefiles and stuff, I can 
say that I've grown to understand make quite well... 

so this bug ended up being a good thing  :-)

man exceptions are quick. 


more than 4 million adds a second, even though the values are cast 
to 32 bits and its using a stub (it jumps into a C function from 
a library) .

But I guess you're going to have to open and close a new connection 
for each query which is not very satisfactory

And, thank Maxim for his help on testing it.

And see my general rant at:
http://www.rebol.net/r3blogs/0327.html

and it has to run faster than the PC version which has a 1000 times 
faster CPU  ;-D

And least useful too

If you import one extension which has an exported function such as 
test and then import a different extension which has the same test 
function, it only recognizes the first.

That's what makes in-place updating and manual conflict resolution 
possible.

no... just dont export them and do this:

lib: import %r3-test-extension.dll

lib/my-command arg arg arg

It is still referenced in system/modules if it is imported. Start 
over, just LOAD-EXTENSION (ignoring what it returns) and recycle. 
See if the DLL is still locked.

Fot that matter, unsetting the function wouldn't work unless you 
unset it in both the user context and the exports context.

We have tasks now. Just because they don't work properly doesn't 
mean they're not there, and certainly they need to be a consideration 
eventually.

And to review the existing task-related tickets.

So, all the examples I've seen that I remember .. Rebol executes 
an imported function...and control returns Rebol.

Graham, if it's a task (OS thread), then no, it's not blocked and 
has its own stack.

What we probably should do is designate a PORT type for intertask 
messaging, and make it easy to use.

If you are adding a module to the module list, and there is an existing 
module of that name, then the new module either overrides it, replaces 
it, or doesn't get added (possibly with an error triggered, but so 
far not). The question is which one to do in the particular circumstances. 
The factors are whether it is the same module, for whatever "same" 
means here considering it might be reloaded or still source; whether 
the versions are the same or greater; whether the existing module 
has already been made or is still source, and the same for the module 
to be added.

all the test function does is open up a qt window with a button and 
starts qts event handler

quick heads up: I built A101 just fine (a Win32 build cross-compiled 
on Linux) and the "go.r" demo still works. so thanks for A101, Carl!

the main difference between a message based callback and function 
stack callback are in synchronisation and locking.


they both have advantages, but many libraries require function stack 
callbacks.

Brian, my callbacks are thread safe in regards to tasks and threads 
(I read extensively on the subject when I did my hack).


but I talked with Carl, and in fact they are not safe within a single 
process to begin with

the way I see it is that for callbacks to be safe, the GC has to 
be momentarily locked.  otherwise memory swapping might occur within 
the core which isn't synchronised in the command (you get some pointers 
from the core) and those pointers might not exist anymore when the 
callback returns.

so you end up changing memory which has been recycled and asking 
for trouble.

We have a task! type now, and have had it for a while. It doesn't 
work well but when last I checked it does work. Of course none of 
the system supports its use, and error handling in a task will crash 
R3, but that all is fixable.

I am hoping for a certain level of non-shared memory and intertask 
communication, but we'll have to see how much we can get away with.

its the same issue we have with commands and callbacks... there is 
no data "ownership" AFAICt.

the way I see it, when a thread is launched, it should Copy the whole 
environment and become completely independent.  data exchange between 
tasks is done explicitely, and no GC allocated data should be able 
to cross thread boundaries.  obviously using external libs you may 
have some fun, but then its OS allocated and not managed by the core, 
so its your call.

utility functions like  synchronise would be nice. but they would 
be something you call manually.   I find its always better when the 
code actually implements the paradigm, rather than trying to hide 
it.   a bit like Apple's GDC.  you explicitely create little tasks 
which you launch and wait for completion.  its simple, obvious and 
highly scalable.

The advantage to the thread-like tasking model is that the built-in 
functions and data structures can be made copy-on-write, or copy-on-UNPROTECT. 
Protectecd series can be sharable.

A model with some shared data can be more memory-friendly on small 
devices too. While the native code can just use shared pages, the 
mezzanine code needs to be per-process. And from a memory standpoint, 
mezzanine code takes up a lot more room than native code. Shared-nothing 
would kill R3 on portable devices.

Only if the series and other data structures in its code block are 
marked copy-on-write.

breakup rendering of an sequence into several threads and instead 
of messaging long running threads and having to build a complex application... 
you just something like

render-sequence: func [data frames] [ 
	repeat frame frames [
		dispatch 'pic-renders img: image-render data frame
	]  
	wait pic-renders 
	dispatch 'encode-movie do-encode %movie.mp4 img
]


the thing here is that the dispatch setup is external to the source 
code using bursts, and could be configured to use more or less threads.

you've got two independent bursts with a dependency

As for callbacks - I surely don't understand the issue properly, 
but in overall - we are a messaging language. We should aim for message 
passing based kernel, as Amiga was, and as QNX is. Let's use events, 
ports for IPC, when appropriate ...


As for tasks - my understanding is, that Carl still plans on threads 
internally. Didn't BrianH said yesterday, that in modern/future multicore 
architectures OS tasks are better way to go?

note here that we DON'T wait for the second dispatch to finish, we 
just make sure its got all the frames before  dispatching it.


then you could have a burst config which allocates different number 
of threads based on dispatch labels... maybe there's 4 rendering 
threads and only one encoding thread.

but you could dispatch that  function  too...

render-scene: func [shots][
	foreach shot shots [
		dispatch render-sequence shot/data shot/length
	]
]


so here you just send the whole scene to render, and it would only 
use the allocated number of 'pic-render threads allowed accross all 
shots.  :-)

Max - we should "copy" Amiga/QNX, and not a Python - simply put - 
REBOL uses its own OS-like advanced mechanisms - ports, devices. 
Codecs are turning back to ports too. And as for tasks and IPC, we 
should not just wrap OS, but use the REBOL way of doing things once 
again ....

with a few polling function it could be really nice:


completed-bursts 'pic-renders  ; reports progress done on overall 
pic-renders for all shots.

cancel-bursts 'pic-renders ; stops all pic-renders and bursts which 
depend on them.

interrupt-bursts 'pic-rendrs ; puts bursts (and all depencies) into 
a wait list, to be restarted later.

this is NOT os way, its an api over a thread model which doesnt require 
the programer to know about tasks and deal with messaging, IPC, bla 
bla.


in the vast majority of cases, threads don't need to be persistent, 
and is in fact a burden to have to manage.

using OS threads is the only way we can use multiple hardware threads 
in the same application.  otherwise, they will be completely different 
processes, and that makes a BIG difference in memory consumption, 
especially on GUI Oses which open up a lot of libs on startup.

What we probably should do is designate a PORT type for intertask 
messaging, and make it easy to use.

Yes please. :-)

I'm hip deep in it right now...  I'm implementing the last "feature" 
which is the ability to format command arguments differently than 
the original C function parameters.


this will allow templating for extensions, just like in C++, and 
will also allow us to put litterals in the spec, so that one doesn't 
need to provide ALL parameters from the REBOL side.

Maxim, well, not too literate... or it gets in the way and starts 
looking like XML.

its just a massive amount of comments, the reason being that I want 
this tool to be easy to maintain in the long run and or it to be 
easy to modify for people who have special needs.

things are explained everywhere so that little is left "hanging in 
the air" .  I want cause and effect to be explicit everywhere in 
the source.

the assimilator is getting quite powerfull, we can now specify litteral 
values in the alternate command formats!


this allows C functions to be called with some of the arguments filled-in 
by default and requires one-less argument from the REBOL side of 
things.

and since we can specify as many alternate command formats as we 
want per C function, we can build a library of parameter-less rebol 
commands which call the C functions with usefull values built-in 
to the extension.

I've thought about it on and off. There are a million models to choose 
from. I don't think RPC is the way to go at all though. I'm for channels 
and messages. On a larger scale, I like the tuplespace model.

I'd like a tuplespace implementation too - I've had a good experience 
using GigaSpaces Jini/JavaSpaces implementation.  Very intuitive 
(and simple in terms of API) concept.

as a testament to what is now possible with the new assimilation 
engine, just look at how insanely flexible it is at allowing you 
to provide just about any interface to a command...

using this information in the C header...

// command-template: [obj [object!]] prefix: obj- suffix: 3
// command-name: rxt-iadd
// command-template: [a [integer! decimal!] b [integer!]]
// command-template: [obj [object! x y]] prefix: xy-
// command-name: plus10
// command-template: [10 [litteral!] value [integer!]]

extern int r3t_integer_add(int a, int b)


;--------------------------------------

one ends up with this command interface... all calling the same C 
function.

    rxt-module: {REBOL [
        Title: {r3-test-extension extension}
        Name: r3-test-extension
        Type: extension
        Exports: []
    ]
    rxt-obj-r3t-integer-add3: command [ obj [object!]]
    rxt-rxt-iadd: command [ a [integer! decimal!] b integer!]
    rxt-xy-rxt-iadd: command [ obj [object!]]
    rxt-plus10: command [ value integer!]

    rxt-r3t-integer-mult: command [ a [integer! decimal!] b [integer! 
    decimal!]]

;----------------------------------------


note that in this setup, auto-exports is set to false and an rxt-prefix 
was added to every command name automatically.

Graham, on IPC, I think Carl has to be involved in the initial spec. 
I've done point-to-point systems, using TCP ports, but that's not 
a general solution. I've done a tuplespace over Rugby, but it required 
polling. I wanted to revisit Uniserve to replace Rugby in that, but 
it fell off my radar.


Ultimately, I think we need a central process--the tuplespace--that 
apps communicate through. To be general, scalable, and lightweight 
while allowing us to build something like GigaSpaces on top of it 
takes more net guru skills and design effort than I have mustered.

This design-with-discussion model we've been doing lately has worked 
pretty well for modules. Perhaps it can work for callbacks and IPC 
as well. All you need is a lead designer (so it isn't design-by-committee) 
and some quality feedback.

hehe ... I just realized that Carl and I are editing the page simultaneously...

isn't Carl an author of AmigaOS, a message passing based OS? Aren't 
IPC techniques well known and defined concepts nowadays? Maybe we 
should not come-up with something completly new, just choose what 
fits our model. E.g. we have got async devices ... what about having 
IPC/messaging device?

Obviously, we would be best served by Carl as the lead designer on 
IPC and callbacks :)

is choosing callback and IPC method close to get tasking implemented 
too? :-) That would be cool - the core would feel complete that way 
finally ... I mean - for beta status ...

callback not directly, but threading has some design considerations 
wrt callbacks which are probably all bubbling in Carl's mind.

Inter thread and inter application IPC will most likely be built 
using the same design.

ok, that one uses System V signals and Windows doesn't expose signals 
( i think )

I think interop is important, but I don't know if there's a clear 
cross-platform choice. I looked at http://www.spread.org/some time 
back, but it doesn't support Windows and is probably best accessed 
as an extension. 


I can pull notes together for discussion, but I don't want to spend 
time on it without knowing that it has some chance of it being helpful 
and making a difference.

And the very first IPC mechanism I would focus on would be STDIO.

There are zillions of ways, and having choices is good, but if we 
have a standard it will be much easier to write tools that can coordinate 
with each other. If we have rebrowser (or, even better, a generic 
plug-in based app framwork that rebrowser is built on :-), it may 
be something that gets built anyway.

Out of the zillions of options, and with Carl's background and knowledge, 
what is the best option for REBOL (or maybe the top 3)? If Carl has 
ideas/opinions/designs already, could he write up a paragraph that 
outlines how *he* thinks it should be done?

Gregg... you're suggesting stdio pipes, right?  I'm not sure of the 
quality of their support over all platforms, but perhaps it's possible. 
 Of course, there are no easy alternatives.  IOS had IPC, and it 
was implemented differently for each OS model.  I was going to reuse 
most of that code for R3.  But, it does use sockets for posix, but 
I think they work well.

+ lots of points for having a code base that's already done and tested!

... and to receive them of course.