AltME groups: search

Also, I don't think that triggering platform-specific suffix replacement 
by using %filename.rx is a particularly good idea (too much magic). 
But it is one option.

I don't think using a platform-specific suffix for a cross-platform 
extension and then expecting code that uses it to be cross-platform 
is a good idea. YMMV.

The extension file itself is not cross-platform (in most cases, at 
least), therefore using a standard suffix is extremely appropriate.

Yup. But that's why extension lookup is implemented in a platform-specific 
way, internally, to gloss over those differences.

Linux's loader in general couldn't care less about the extension 
of a shared object. That doesn't make it any less prudent to stick 
with the standard .so extension.

prudent to stick with the standard .so extension

 - I'm having a little trouble understanding this, but maybe that 
 comes from too much experience with platforms where it has been demonstrated 
 that there are advantages to distinguishing between different dynamic 
 library types by using different file suffixes. The "standard .so 
 extension" means that these differences get put somewhere else in 
 the filename. But I can see your point with libraries like System.Data.Sqlite 
 that support more than one calling convention in the same file, so 
 that the R3 extension API would just be added to a system dynamic 
 library that is otherwise meant to be called by non-REBOL code - 
 incorporating its own wrapper extension.

Remember, on many Linuxes you just want to load libc, but the actual 
file is something like libc.so.6 or something (I'm paraphrasing here). 
Having LOAD-EXTENSION translate .rx is no different from that. You 
can even have a filename map on your platform if you like.

There is no further translation happening on Linux from libc.so to 
libc.so.6 as that is just a symlink in the file system

It's main purpose is to translate a "generic" extension suffix (%.rx 
per default) to platform-specific extension suffixes (yes, potentially 
many).

I hope extload to be a suitable foundation upon which to further 
discuss these matters.

You can try them in the order the suffixes appear in system/options/file-types. 
That way your code doesn't have to special-case per platform; let 
the host code special-case it instead. Just in case .rx isn't supported 
on a platform, you might consider searching for 'extension and backtracking 
to get the file! suffixes.

Do we need a system/options/extension-paths setting? You could add 
it to system/options in your %extload.r3, and it would let you preconfigure 
your system with a directory of shared extensions, if such a thing 
exists. Or it could just be assigned the same block as system/options/module-paths 
by default.

An extension-paths setting and searching this path would be a good 
option (esp. on platforms with weaker loaders). But more about this 
later, gotta run :)

Or is there a better way to map the error message back to the source 
file?

OMG... this is a good example of seeing RXARG_SERIES and understanding 
RXA_SERIES... I though I did this already.

But RXARG (the old one) was a shortcut to access the .series union 
member. The new RXA_SERIES one needs a frame pointer and a number. 
Which I don't need. So, the new approach is to access the union members 
directly?

Just a question: with the extension model complete is it possible 
to interface to .NET ?

It would be useful to have a general .NET wrapper extension that 
uses Reflection, ActiveX or something to call arbitrary .NET code.

Must be a function of the system's loader

afaik, .net is a dll too.  so its probably just loading the .net 
libs.    


in fact, looking at some stuff related to C# and p/invoke I realized 
that the rebol .dll and  .net stuff are very similar in scope and 
function  :-)

IIRC as soon as you load a .NET based DLL, this will trigger the 
startup of the .NET runtime. So, it's hidden. I don't know if a .NET 
DLL has a dynamic link to the runtime DLL and calls an init code 
sequence or if this is done inside the Windows loader.

The only thing you can't do is fully strip such a library, because 
then the OCaml bytecode is gone

What is error: ** Script error: commaned word is not bound to a context

ech.. just a typo!

The command call process does a lot of marshalling to get the values 
into the native code. You only have limited access to the original 
values in REBOL memory.

The index is simply not part of a series at the implementation level

What appears to be a series with an index in REBOL is actually a 
reference with an index. That's exactly what an RXIARG is. What the 
hostkit level calls a REBSER is the actual series, not a reference, 
so that doesn't have an index

I personally think of the RXIARG (which is a neutered REBOL "value") 
as "the series", and of the REBSER as a subordniate helper structure 
(holding width/size/left and data).

Yes, at first REBSER seems an incorrect name, but the reality is 
that RXIARG can be though of as a "REBREF"

Has anyone done some code using the RL_words_of_object, RL_Get_Field, 
RL_Set_Field for extensions? I would like to take a look at the code 
how to best use this stuff.


And, is it faster to seperate words & values on the C-side or to 
call an extension function just with two blocks: words, values?

Has anyone a simple callback example?

For callbacks to set the callback function, your C code must have 
a pointer to the object where it resides (an object you made or some 
other system context) and the name of the function, as a word id.

Those need to be set in the CBI structure:
cbi->obj = obj;
cbi->word = word;


The word ID can be retrieved with RL_MAP_WORD. And a self created 
context in the same way. But how do I rerieve the ID of the global 
context (do we still have such a thing?)?

You need to pass in a reference to the extension. There is no overarching 
global context, though there are a few centrally referenced contexts.

You need to pass in a reference to the extension. -> You need to 
pass to the extension a reference to the context containing the function.

Most words won't be "global" unless they are exported from a module 
(at which point they will be in the lib context) or system internal 
(sys context). User script words are in a task-local user context 
(once we get that working), and module words are in module-local 
contexts. I'd be shocked if most callback functions weren't at least 
module-local, or often in hidden inner contexts.

There's a callback example in the cURL binding, for progress info

cbi->obj is of type REBSER* so this suggest to pass in a series. 
A string with the name of the context?


Brian, how does a  "reference to the context containing the function" 
look for this:

rebol []

myfunct: does []

What do I pass to the extension here?

I have a problem with a callback providing a string.

The RXT_INTEGER part works. But when using a RXT_STRING, R3 crashes.

The callback from the C side comes our from a multithreaded DLL. 
Maybe this is a problem.

Is it possible for the callback to be executed simultaneously by 
multiple threads? Then your allocated REBOL string can be recycled 
if it has not reached the status of a reference in REBOL yet

I added a mutex around the callback and the string! creation. No 
effect, still crashes.

A shared library in two processes only shares the static code sections. 
The data sections are created for each process, so they can be considered 
independent. The only issue is multithreading within one process

You could try to have the progress callback in the cURL binding pass 
a series. cURL is singlethreaded, so this would establish whether 
it's a threading problem or an R3 memory management/callback problem

As said I can use a string from an other "not threaded" area in the 
code.

The problem is that I can't investigate further without Carl taking 
a look at it.

Ok, so as expected I get two different thread IDs. The callback is 
triggered from a different thread ID than the R3 host interpreter 
is running from.

Is there a simple way to call a function in the context of an other 
thread from the same process?

There are many precautions that need to be taken to make a system 
thread safe, including memory access. Apparently, R3 isn't

Thread synchronisation is also a complex topic. You can't just execute 
a piece of code in the context of another thread. The other thread 
has to execute it, and to make it do so you need a framework for 
that, such as a messaging infrastructure

Well, this gives me an idea... I create the CBI structs in thread 
A and use them from B... that sounds like a good candidate to fix.

There you have created a memory clash between threads

Where is a memory clash?

And as said, I now moved all callback structur related code to a 
single thread (not the one the interpreter runs in) and it crashes 
too.

I can ensure that it's not a thread synchonisation problem. The memory 
for the callback is never used by anything on the c-side after setup.

It's probably a thread synchronisation problem within R3

the immediate solution is to have a piece of code, running in the 
same thread as the R3 host, which is used to initiate all callbacks

Well, to get this code started I need to call it from R3, but than 
I can't return to R3 otherwise I get a new thread dealing with it. 
So this is not possible.

But I could trigger a poll via callback sending an integer! :-) It's 
two rounds than but that would work.

i assume you currently have roughly the following flow:
- R3 calls into extension (via RX_Call)

- extension calls into library, passing an extension-internal "activity" 
handler along
- the library calls the extension's activity handler
- the extension activity handler initiates a callback into R3

- R3 call init-lib

- init-lib installs a listener, setups a C-level callback handler 
(CBH) and returns
- the listener is sometimes triggered, calls the CBH (new thread)

- CBH prepares Rebol callback, calls Rebol callback (and here it 
crashes with string!, not with integer!), continues and ends

To complete the sequence from above:

- Rebol exectues the rebol side callback function, which make a synchronous 
call to the lib, getting all buffered messages and continues to process 
them

I have a very strange effect: The init_block gets a c filename attached. 
The .r file that is used to generate the .h header file of course 
doesn't inlcude it. And the generated init_block numbers don't include 
it too.

How can this be? I mean it's a const-char array.

Seems the R3 script I use has a bug, there is a strange line:
to-cstr: either system/version/4 = 3 [
	; Windows format:
	func [str /local out] [
		out: make string! 4 * (length? str)
		out: insert out tab
		forall str [
			out: insert out reduce [to-integer first str ", "]
			if zero? ((index? str) // 10) [out: insert out "^/^-"]
		]
		;remove/part out either (pick out -1) = #" " [-2][-4]
		head out
	]
][

btw it's really strange.. I would thought that that's a basic thing. 
To be able easily define a string.

Then there's also a limit how big concatenated strings can get (i.e. 
"..." "..." "..."). Which is 65k in MSVC, IIRC.

So, if you don't do extremely long lines and a 65k+ init string, 
you should be fine with string literals even in MSVC.

Btw, here's my variation of a header generator script:
https://github.com/earl/r3-zmq/blob/master/make-ext-header.r3


It has the nice advantage that it automatically generates a dispatch 
table for your commands, enabling a clean and concise RX_Call like:
https://github.com/earl/r3-zmq/blob/master/zmqext.c#L31-35

I vaguely recall that someone was working on ODBC support for R3. 
Did that ever become usable? Is there a link to this work?

Thanks! I've been using R2/Command to do the database access and 
R3 to process the data. All in the same script, using a trick I'll 
post to the Windows group.

There's a couple bugs in the REBOL portion of the ODBC extension 
mentioned above. I'll try to make a patching wrapper module.

This module loader for the odbc.dll extension handles the bug in 
its open handler that prevented it from using block specs with a 
target:

REBOL [
	title: "ODBC Extension Patch"
	type: module options: [private]
	needs: [%odbc.dll]
]

odbc: import 'odbc


system/schemes/odbc/actor/open: func [port [port!] /local result] 
bind [
	port/state: context [access: 'write commit: 'auto] 

 result: open-connection port/locals: make database-prototype [] case 
 [
		string? select port/spec 'host [ajoin ["dsn=" port/spec/host]]
		string? select port/spec 'target [port/spec/target]
		'else [cause-error 'access 'invalid-spec port/spec]
	]

 all [block? result lit-word? first result apply :cause-error result] 
	port
] odbc

Unfortunately, there's an error in its fallback behavior that makes 
it a bit difficult to use. In its docs, section 3.7:

 If there is no applicable REBOL datatype to contain a SQL value, 
 the value will be returned as a string.

The binary data is returned in a value of the string! type, rather 
than of the binary! type, which results in a corrupted string that 
you can't do anything with. If the value was returned as a binary 
then we could perform the conversions ourselves. I ran into this 
problem when returning SQL values of the tinyint and text types.

The method is pretty simple. Normally the functions return data in 
a particular format. Upon an error though they just return a block 
with 3 elements, the first two of which are lit-words, the last could 
be anything; this is passed directly to apply :cause-error. The error 
block can be recognized and handled easily with this code in the 
REBOL functions that wrap the natives:

 all [block? result lit-word? first result apply :cause-error result]


It's similar to the way error handling is done in the low-level sys 
load functions, particularly sys/load-header. Upon an error, instead 
of a block the function returns a word! value, which can be passed 
to cause-error by the calling function, using code like this:
	set [hdr: code:] load-header/required data
	if word? hdr [cause-error 'syntax hdr source]

Word values are easier to handle safely than error values. Note that 
the argument provided to cause-error in this case is different from 
that provided to load-header: This lets you trigger errors with information 
that load-header isn't provided, giving your errors more context 
that is useful to the programmer. Not exactly the same thing as the 
ODBC driver, but close.

I was using REBOL for datamining a few years ago and I can say it 
was easier to do string based parsing to get what I've needed.

It's always easier to do:  parse html [thru "<title>" copy title 
to "<"]  than parse complete html to something like a block structure 
and dig title in it.

Scraping a title is the simplest example. In reality, you get all 
sorts of tags with extra attributes that you need to skip, and values 
with extraneous newlines. He wouldn't understand how to normalise 
that, so his data would be left as a mess

We did a simple address list as an example, even preconverting it 
to plain text. It took all afternoon to discover all exceptions and 
fix them, so in most cases, it isn't worth it

I've done quite a few html analysers and with a bit of experience 
I have found that going with a brute force parse on pages ends-up 
being very effective.  I did a font downloading script for www.acidfonts.com 
a few years ago and it took more time to actually download all the 
pages than build the script.  :-)

some servers are anti-indexing and its in these cases where the brute 
parse is most effective.  I've even had to cater an oracle web server 
which didn't have ANY css, type, or id fields in all pages which 
are driven by form.  all urls can only be read once, and every page 
read is a redirect.   


only parse allowed me to cope in such a drastic anti-robot  environment. 
 it still took a day to build the robot.  and in the end, it even 
had an auto-user-creationg step at each 200 pages which created a 
google gmail account for the next batch.  :-)

in those cases, parse is king.

a fellow REBOL parse data miner tells me that some servers have pretty 
good algorithms which identify robots out of the thousands of requests 
they get, and you even have to put random-lenght pauses between reads 
which can go up to a few hours.

About examples: if you download the repository, or the source archive, 
you'll find a test directory which has the test suite for (almost) 
all the modules. the tests are often usable as "examples" for how 
to use the modules themselves.

About the HTML parser:


This started out because we had a need for a HTML filter. The mail 
client in Qtask allows users to view HTML emails, furthermore, the 
wiki editor submits HTML to Qtask. Whenever you are embedding HTML 
from an external (untrusted) source within your HTML, you have security 
problems. For this reason, we had to take the untrusted HTML and 
1) "filter" it so that it would be safe 2) make it embeddable (eg. 
only take what's inside <body>, and not the whole <html> etc.).

My approach was, instead of doing what many others do (try to remove 
things from the HTML that are known to be "bad", eg. use regexps 
to remove anything that starts with "javascript:" or anything between 
<script>...</script> etc.), was to only pass what was known to be 
good, and ignore everything else. This is a bit more limiting but 
I consider it to be safer (you don't have to play a game with attacker 
where every time they find a new vector, you have to add it to the 
"bad" list).

So the first HTML filter was done. It parsed the HTML (any version), 
and going through two finite state machines in a pipeline, rebuilt 
it as XHTML 1.0 strict.

This method avoided keeping any intermediate representations in memory. 
However, because of that there were a number of things it could not 
do (eg. no look ahead, and you get an explosion of the number of 
states if you want to "look behind" more).

So, as our needs became more complex (esp. because of the initial, 
never released version of the wiki editor), I had to change approach. 
Also, at that time Maarten was doing the S3 stuff and needed a XML 
parser as well.


So, first, the Filter was split up into three modules. The first 
is the parser, that takes a HTML or XML string and just sends "events" 
to a callback function. This can be used basically for anything. 
(Maarten never used it in the end.) The second part was the first 
FSM, the HTML normalizer. You'll still find it within the Power Mezz, 
but it's deprecated. The third part was the actual filter and regenerator 
(second FSM). You can find it in the repository history.

Then, the latter two modules were replaced by a different approach 
that builds a tree of blocks from the HTML and rewrites it as it 
is being built (to avoid doing too many passes). This is done by 
LOAD-HTML, that allows passing a set of rules used for filtering 
(so the actual filter is now a bunch of rules for LOAD-HTML). LOAD-HTML 
handles a lot of HTML weird cases, it's probably not at the level 
of a web browser, but it comes close.

The tree is being built with the Niwashi module, which was separated 
as a generic way to build trees incrementally following rules etc. 
(Niwashi means gardener in Japanese)

The HTML to text module has still not been rewritten to use LOAD-HTML 
instead of the older approach of the HTML normalizer followed by 
a FSM.

7.4 parses a string into a sequence of tags and text (etc.). (it 
also has a load-markup function that is similar to load/markup but 
also parses tag attributes and so on). 5.10 uses 7.4 and builds a 
tree from that sequence of tags and text.

the code is using a number of tricks to be "fast" (esp. expand-macros.r), 
so it's not as clean as it could be.

Second: your problem is that you are trying to mold the result, which 
is a tree where each node has a reference to the parent node. (much 
like faces in R2). That's why you see the "loop".

there is a mold-tree function in %mezz/trees.r if you want to mold 
the tree. Or, you could simply use form-html to pretty print the 
tree for you.

Also note that:


>> print form-html/with load-html "<p>A paragraph!" [pretty?: yes]
<html>
    <head>
        <title></title>
        </head>
    <body>
        <p>A paragraph!</p>
        </body>
    </html>

wow, thank you a lot! I knew this was to obvious "bug" to be real 
and I am probably doing something wrong. GREAT!


I initially imported only needed modules but got errors .. ( I will 
try and report ) the errors went away as I manually imported them. 
Just a second

there are a few things that it still can't do (the dialect i mean), 
but it's very powerful on the things it can do :) The documentation 
is here: http://www.rebol.it/power-mezz/mezz/niwashi.html

That was a109. There was another las-minute error that slipped into 
a110, but it wasn't a module system error, it was a startup error.

There has only been one issue in the modukle system itself (aside 
from the possible cURL thing) that has been discovered since a110, 
and that is not as much a bug as it is a gotcha, and hasn't been 
reported yet.