AltME groups: search

Filenames not ending in the generic suffix are left as-is; so event 
with extload present, if can use e.g. import %cURL-binding.so directly 
and bypass the suffix translation mechanism.

And for this to lead to eventually integrating an improved loading 
mechnism  directly into R3.

You can try them in the order the suffixes appear in system/options/file-types. 
That way your code doesn't have to special-case per platform; let 
the host code special-case it instead. Just in case .rx isn't supported 
on a platform, you might consider searching for 'extension and backtracking 
to get the file! suffixes.

Do we need a system/options/extension-paths setting? You could add 
it to system/options in your %extload.r3, and it would let you preconfigure 
your system with a directory of shared extensions, if such a thing 
exists. Or it could just be assigned the same block as system/options/module-paths 
by default.

An extension-paths setting and searching this path would be a good 
option (esp. on platforms with weaker loaders). But more about this 
later, gotta run :)

Search thru all your source files for RXARG_SERIES and replace it 
with RXA_SERIES

OMG... this is a good example of seeing RXARG_SERIES and understanding 
RXA_SERIES... I though I did this already.

But RXARG (the old one) was a shortcut to access the .series union 
member. The new RXA_SERIES one needs a frame pointer and a number. 
Which I don't need. So, the new approach is to access the union members 
directly?

I think that there are ways to call .NET APIs from C/C++ using COM 
interop and other methods. MSDN should have the details.

afaik, .net is a dll too.  so its probably just loading the .net 
libs.    


in fact, looking at some stuff related to C# and p/invoke I realized 
that the rebol .dll and  .net stuff are very similar in scope and 
function  :-)

the difference is that the type system in VM allows them to jit compile 
stuff more completely than can be done with REBOL's just-in-time 
evaluation and binding.

IIRC as soon as you load a .NET based DLL, this will trigger the 
startup of the .NET runtime. So, it's hidden. I don't know if a .NET 
DLL has a dynamic link to the runtime DLL and calls an init code 
sequence or if this is done inside the Windows loader.

And it is quite possible that this may not be doable at all (if the 
index is not stored in the "REBSER" structure and the REBSER has 
no back-pointer to the index-containing rebol value).

The workaround/solution is simple: don't pass REBSERs to function, 
but use the RXIARG instead and then access the index via .index and 
the REBSER via .series.

This is to allow changes to the internal structures to happen without 
breaking host or extension code, and for security reasons.

I personally think of the RXIARG (which is a neutered REBOL "value") 
as "the series", and of the REBSER as a subordniate helper structure 
(holding width/size/left and data).

Has anyone done some code using the RL_words_of_object, RL_Get_Field, 
RL_Set_Field for extensions? I would like to take a look at the code 
how to best use this stuff.


And, is it faster to seperate words & values on the C-side or to 
call an extension function just with two blocks: words, values?

For callbacks to set the callback function, your C code must have 
a pointer to the object where it resides (an object you made or some 
other system context) and the name of the function, as a word id.

Those need to be set in the CBI structure:
cbi->obj = obj;
cbi->word = word;


The word ID can be retrieved with RL_MAP_WORD. And a self created 
context in the same way. But how do I rerieve the ID of the global 
context (do we still have such a thing?)?

Most words won't be "global" unless they are exported from a module 
(at which point they will be in the lib context) or system internal 
(sys context). User script words are in a task-local user context 
(once we get that working), and module words are in module-local 
contexts. I'd be shocked if most callback functions weren't at least 
module-local, or often in hidden inner contexts.

There are no context IDs and there's no need for RL_MAP_WORD. RXA_WORD 
handles that

Just an object. Please refer to the cURL source and documentation

The DLL is used from two Rebol processes. One client and one server 
running from the same dir. I assume that the DLL doesn't share any 
memory between the processes. At least that's what I always understood 
how it works under windows.

I added a mutex around the callback and the string! creation. No 
effect, still crashes.

Thread synchronisation is also a complex topic. You can't just execute 
a piece of code in the context of another thread. The other thread 
has to execute it, and to make it do so you need a framework for 
that, such as a messaging infrastructure

Well, this gives me an idea... I create the CBI structs in thread 
A and use them from B... that sounds like a good candidate to fix.

And as said, I now moved all callback structur related code to a 
single thread (not the one the interpreter runs in) and it crashes 
too.

Everything is on the heap, and I even added mutex around the callback 
code section without any luck.

- R3 call init-lib

- init-lib installs a listener, setups a C-level callback handler 
(CBH) and returns
- the listener is sometimes triggered, calls the CBH (new thread)

- CBH prepares Rebol callback, calls Rebol callback (and here it 
crashes with string!, not with integer!), continues and ends

To complete the sequence from above:

- Rebol exectues the rebol side callback function, which make a synchronous 
call to the lib, getting all buffered messages and continues to process 
them

I have a very strange effect: The init_block gets a c filename attached. 
The .r file that is used to generate the .h header file of course 
doesn't inlcude it. And the generated init_block numbers don't include 
it too.

Can someone explain me, why is Carl using such an 'encryption' and 
why not just to use it as const char like I do here:

https://github.com/Oldes/R3-extension-FMOD/blob/master/main.c#L417

Is it because he wants to be compatible with some old compilers?

And compilers are affected?

And MSVC doesn't support C99.

So, if you don't do extremely long lines and a 65k+ init string, 
you should be fine with string literals even in MSVC.

Btw, here's my variation of a header generator script:
https://github.com/earl/r3-zmq/blob/master/make-ext-header.r3


It has the nice advantage that it automatically generates a dispatch 
table for your commands, enabling a clean and concise RX_Call like:
https://github.com/earl/r3-zmq/blob/master/zmqext.c#L31-35

Thanks! I've been using R2/Command to do the database access and 
R3 to process the data. All in the same script, using a trick I'll 
post to the Windows group.

Unfortunately, there's an error in its fallback behavior that makes 
it a bit difficult to use. In its docs, section 3.7:

 If there is no applicable REBOL datatype to contain a SQL value, 
 the value will be returned as a string.

The binary data is returned in a value of the string! type, rather 
than of the binary! type, which results in a corrupted string that 
you can't do anything with. If the value was returned as a binary 
then we could perform the conversions ourselves. I ran into this 
problem when returning SQL values of the tinyint and text types.

The method is pretty simple. Normally the functions return data in 
a particular format. Upon an error though they just return a block 
with 3 elements, the first two of which are lit-words, the last could 
be anything; this is passed directly to apply :cause-error. The error 
block can be recognized and handled easily with this code in the 
REBOL functions that wrap the natives:

 all [block? result lit-word? first result apply :cause-error result]


It's similar to the way error handling is done in the low-level sys 
load functions, particularly sys/load-header. Upon an error, instead 
of a block the function returns a word! value, which can be passed 
to cause-error by the calling function, using code like this:
	set [hdr: code:] load-header/required data
	if word? hdr [cause-error 'syntax hdr source]

Word values are easier to handle safely than error values. Note that 
the argument provided to cause-error in this case is different from 
that provided to load-header: This lets you trigger errors with information 
that load-header isn't provided, giving your errors more context 
that is useful to the programmer. Not exactly the same thing as the 
ODBC driver, but close.

I was using REBOL for datamining a few years ago and I can say it 
was easier to do string based parsing to get what I've needed.

It's always easier to do:  parse html [thru "<title>" copy title 
to "<"]  than parse complete html to something like a block structure 
and dig title in it.

For you, but my business partner wants to scrape web pages, and I 
don't think he would understand how to do it with parse

Scraping a title is the simplest example. In reality, you get all 
sorts of tags with extra attributes that you need to skip, and values 
with extraneous newlines. He wouldn't understand how to normalise 
that, so his data would be left as a mess

We did a simple address list as an example, even preconverting it 
to plain text. It took all afternoon to discover all exceptions and 
fix them, so in most cases, it isn't worth it

I've done quite a few html analysers and with a bit of experience 
I have found that going with a brute force parse on pages ends-up 
being very effective.  I did a font downloading script for www.acidfonts.com 
a few years ago and it took more time to actually download all the 
pages than build the script.  :-)

some servers are anti-indexing and its in these cases where the brute 
parse is most effective.  I've even had to cater an oracle web server 
which didn't have ANY css, type, or id fields in all pages which 
are driven by form.  all urls can only be read once, and every page 
read is a redirect.   


only parse allowed me to cope in such a drastic anti-robot  environment. 
 it still took a day to build the robot.  and in the end, it even 
had an auto-user-creationg step at each 200 pages which created a 
google gmail account for the next batch.  :-)

in those cases, parse is king.

a fellow REBOL parse data miner tells me that some servers have pretty 
good algorithms which identify robots out of the thousands of requests 
they get, and you even have to put random-lenght pauses between reads 
which can go up to a few hours.

About the HTML parser:


This started out because we had a need for a HTML filter. The mail 
client in Qtask allows users to view HTML emails, furthermore, the 
wiki editor submits HTML to Qtask. Whenever you are embedding HTML 
from an external (untrusted) source within your HTML, you have security 
problems. For this reason, we had to take the untrusted HTML and 
1) "filter" it so that it would be safe 2) make it embeddable (eg. 
only take what's inside <body>, and not the whole <html> etc.).

This *had* to work with *any* HTML (think about the stuff that comes 
out of Outlook, or the stuff you get as spam, or newsletters, and 
all that). You can't imagine how bad that can be. That had to be 
turned into something that would not break our own HTML pages.

My approach was, instead of doing what many others do (try to remove 
things from the HTML that are known to be "bad", eg. use regexps 
to remove anything that starts with "javascript:" or anything between 
<script>...</script> etc.), was to only pass what was known to be 
good, and ignore everything else. This is a bit more limiting but 
I consider it to be safer (you don't have to play a game with attacker 
where every time they find a new vector, you have to add it to the 
"bad" list).

So the first HTML filter was done. It parsed the HTML (any version), 
and going through two finite state machines in a pipeline, rebuilt 
it as XHTML 1.0 strict.

This method avoided keeping any intermediate representations in memory. 
However, because of that there were a number of things it could not 
do (eg. no look ahead, and you get an explosion of the number of 
states if you want to "look behind" more).

So, as our needs became more complex (esp. because of the initial, 
never released version of the wiki editor), I had to change approach. 
Also, at that time Maarten was doing the S3 stuff and needed a XML 
parser as well.


So, first, the Filter was split up into three modules. The first 
is the parser, that takes a HTML or XML string and just sends "events" 
to a callback function. This can be used basically for anything. 
(Maarten never used it in the end.) The second part was the first 
FSM, the HTML normalizer. You'll still find it within the Power Mezz, 
but it's deprecated. The third part was the actual filter and regenerator 
(second FSM). You can find it in the repository history.

Then, the latter two modules were replaced by a different approach 
that builds a tree of blocks from the HTML and rewrites it as it 
is being built (to avoid doing too many passes). This is done by 
LOAD-HTML, that allows passing a set of rules used for filtering 
(so the actual filter is now a bunch of rules for LOAD-HTML). LOAD-HTML 
handles a lot of HTML weird cases, it's probably not at the level 
of a web browser, but it comes close.

7.4 parses a string into a sequence of tags and text (etc.). (it 
also has a load-markup function that is similar to load/markup but 
also parses tag attributes and so on). 5.10 uses 7.4 and builds a 
tree from that sequence of tags and text.

Hi, first thanks for making and open sourcing power-mezz. 


I am trying to use load-html and am getting some strange results 
if sems it makes for example recursing [ html [ html [ html  ..... 
]]] on my simple html input (and on real one that I tried). I prepared 
two examples to make the point as clear as possible.


http://paste.factorcode.org/paste?id=2263(notice the stack owerflow 
error)

get-node and set-node are also from %mezz/trees.r ; most likely you 
don't want to mess around with %mezz/macros/trees.r , that is deep 
vodoo i use to make the html filter fast.

wow, thank you a lot! I knew this was to obvious "bug" to be real 
and I am probably doing something wrong. GREAT!


I initially imported only needed modules but got errors .. ( I will 
try and report ) the errors went away as I manually imported them. 
Just a second

It all works now according to your example.. and I tested and it 
handles improper html very well! Thanks!

I looked at html-rules in load-html and I am stunned by how well 
the code / dialect is

No, we had it post A110 and you said "whoops, last-minute errors 
slipped in".

There has only been one issue in the modukle system itself (aside 
from the possible cURL thing) that has been discovered since a110, 
and that is not as much a bug as it is a gotcha, and hasn't been 
reported yet.

Yeah. We have a pretty thorough test suite for modules, and all tests 
pass for a110. And I checked the tests for accuracy too :)

Most of a110 was dedicated to fixing those bugs, so I understand 
and appreciate the sentiment :)

Maybe using the same name for the "isolate" option and the "/no-share" 
refinement would be a good idea.

If import/no-share and options: [isolate] have the same effect, using 
the same name seems appropriate.

I only do the stuff that is inherent in the semantics of REBOL. And 
the implementing, of course.

Module path searching only works with non-file! module names. I.e.
import 'foo

will search the system/options/module-paths for a file name %foo.reb. 
This search is a _very_ useful feature which we shouldn't rid ourselves 
of by stupid wars over which suffix to use.


As a module author, I want to use a suffix which I know will make 
using the module easiest for my target audience.


As a module distributor, I also want to make it easiest for my target 
audience to use the modules _and_ I need to enforce consistency within 
my distribution.


The only one who does not really care about the suffix is the actual 
user. The user just wants to use third-party modules without having 
to homogenise their suffixes first. But the burden of choosing a 
sensible suffix is currently imposed on the user. This does not make 
much sense.

Having a default which _is_ intended to be used would be a strong 
incentive for both authors and distributors to stick with this default.

I can only say that Carl set the default, but I haven't seen any 
published scripts by him that use it. Good point, Andreas. Too bad 
we can't settle on .r, because R has taken that extension in most 
standard tools, and OSX uses .r for something else.

Here's another interesting trick: Make %rebol.r a unnamed module 
by putting type: module in its header and not giving it it a name. 
This isolates the words in it in its own context instead of putting 
them in the user context, controls the visibility of its code, and 
makes it easier to free up the memory it uses because its context 
is thrown away afterwards. It's the perfect occasion for an unnamed 
module: run once, and done for effect.


%rebol.r is a great place to set your preferred system/options/module-paths 
and system/options/default-suffix, to delay-load commonly used modules 
and extensions that you have checked out already, and to patch or 
wrap anything that you need to.

%rebol.r is run before any user code is run, even the --do and --import 
args. This makes it the best place to hack the module system for 
your own uses.

I would like to see some support for automatic logging of events.


Rebol currently has two "hooks" to allow "Front-end" code to execute 
automatically in the REBOL.r and USER.r files.

I would like to see another "hook" to allow "Back-end" code to execute 
automatically whenever Rebol stops its current evaluations for any 
reason such as encountering the HALT, QUIT, or any error which causes 
Rebol to stop evaluation.  


This could allow for some kind of automatic logging of  QUIT, HALT 
or ERROR as well as capturing the error messages.  It would be nice 
to allow capturing of any other information such as the script name, 
computer name, current timestamp, and where in the script the evaluation 
stopped, and how it stopped.


I would use this to log all of my scripts that is invoked (put the 
code in REBOL.r file) and then the termination conditions defined 
so they could be triggered automatically.  With such a function, 
I could even have certain selected scripts send an automatic SMS 
messages to my phone if certain scripts errored out etc.



I'd like to propose a new function called WHEN that can be used to 
trigger code at a terminating event.

WHEN  block  /all

For any specified terminating condition, evaluates what follows it.

Arguments:

block [block!] - Block of terminating events (conditions followed 
by values)
Refinements:

/all - Evaluate all terminating conditions (do not stop at first 
true terminating condition)


The WHEN command would be used to define code to be evaluated whenever 
a termination condition of any kind occurred in which continued evaluation 
will stop.

Examples:
WHEN/ALL [

 HALT?	[ print "This script will HALT now" ]  ; code to be evaluated 
 before the HALT is evaluated

 QUIT?	[ print "This script will QUIT now" ]   ; code to evaluate 
 befoe the QUIT is evaluated 

 ERR?	[ print "An error was discovered"  ]   ;code to evaluate after 
 an error is captured, but before the message is printed
	END?	[
		  print "Script about to end"
		  if  now/time  >  23:00:00  print "Stopped after 11 pm"

  ]                                                              ; 
   code to evaluate whenever the current evaluation has stopped for 
  any reason
	]


Note:  HALT?, QUIT?, ERR?, END? are all logic values which become 
true depending upon how evaluations will be stopped.

END? will always be true and one of the other terminating conditions 
will always be true.

What do you think?

Would TERMINATION be a good name as well.  


I would imagine that Rebol would have a common exit point (a place 
in the executable where all terminations go) before control is handed 
off to the OS.  If that is so, then the suggested code could be placed 
there, which should simplify intercepting QUIT or HALT.  To be more 
consistant, I'd like to amend my ERR? proposal to be "code to be 
evaluated after an error is captured and printed" instead of before 
being printed.

>> length? 4

** Script error: length? does not allow integer! for its series argument


I am proposing that this return NONE for any type which it can't 
deduce a length.


this is both usefull in practice and would make length? a truthy 
function.

and if your code doesn't handle the none case it will still raise 
an error, so you aren't even really breaking that capability.

Aha, interesting, and what if we just defined properly a dataset 
like:

    pickable!: make dataset! [series! ...]

we still need to add an extra if before the length? all the time.


its funny because the recent discussion on naming and *? words has 
made me realize that the length? and many others could in fact be 
altered to support this idiom if its done consistently.

we have two questions answered in one function.  using if/either/unless 
allows to filter based on capacity and the result allows us to act 
on the actual value.

it also prevents us from having to create sets of predefined filters 
which will:

    -be unknown and forgotten quickly (e.g. more things to remember, 
    bad)

    -might only really be usable by a very few functions to begin with. 
    (not very usefull)


(this relates to the alternative that ladislav pointed, using a dataset)

changing many of these functions would make them satisfy the *proposed* 
style convention that *? functions should be truthy, if possible. 
  


in many cases, this is the case, because there is no "possible" none 
return value and errors are returned instead.

we still need to add an extra if before the length? all the time.

 ... actually you would just move the IF to the native part and you 
 will use the IF also in cases where you are pretty sure it's a series 
 and want just its length.

On one hand, I like the idea that a none LENGTH? means "this value 
has no length", on the other the idea of a pickable! typset (maybe 
a different name though) gives us the ability to use it in func specs 
and such.

I'm not ready to say change LENGTH? without further thought.

note, that this proposal is strictly related to functions which have 
an '? at the end.  


if we go ahead with changes to the style convention which clearly 
define what a "truthy" function is and state that ****? is generally 
used to define such a function, I think that we could clean up a 
lot of the guess work in how to handle this often recurring case.

obviously not all cases will fit perfectly.  and the above was just 
one (simple) example, where you can get 95% of the work done in one 
line and then tailor the few special cases after.

The fact, that the code actually cannot be used means, that your 
idea is not supported by a real usage example, and is not completely 
thought-out.

give me your perceived disadvantages.  I've been pulling my own weight, 
but I'm not seeing alot of points related to LENGTH? and other such 
funcs...  I'm not talking about a specific function chain.

Generally, it looks, that REBOL is headed towards "less error triggering" 
approach, and I don't feel any of the changes as a problem, so, you 
may be right. But, there should be some limits, where to stop. One 
of my limits is the usage of NaNs in REBOL, I prefer clean errors 
triggered to NaN dissemination. So, in that case, I know where my 
preferred boundary is. In case of LENGTH? and INDEX? function, I 
am not totally sure, but I am sure, that any change to the existing 
state is expensive (postponing the release, taking resources needed 
to solve more important problems, ...) So, I would support such a 
change only when I would get really convincing arguments.

yes, in such a sense (expensive change), I understand.  


I will wait for brian to step up and get his opinion.  I know we've 
spoken about this somewhat before, but not as head-on, for myself... 
 With the recent talk about naming, IMHO it has put a new light on 
the possibility for this to be a bit more appreciated, adding ? at 
the end would now mean something concrete.


let it be known that I do like errors, I am not against errors, I 
have been putting less and less error recovery in my code to make 
sure it crashes and I fix bugs early.


its just that I can see many if not most ****? functions (my own 
included) being truthy and this is very usefull to facilitate  control 
flow.  This, as oposed to always putting error handlers in places 
where the error is only used as a negative reply, for which we can 
supply, IMHO, a reasonable value/standard.

the references would tell us how massively this string is being used 
within the application and allow us to track if we are leaking data 
in a way that it will never be able to GC it.

...and, the GC is able to collect a string no matter how many references 
refer to it

on objects info? could just return all the various reflective properties 
it has (words-of spec-of) and a few more like ram being used by object. 
   all types would benefit from this sorely lacking feature in REBOL.

in any case, the idea is to put as much information there as is possible. 
 


this would be VERY usefull for debugging, dialecting, and inspecting 
untrusted coded

no but traversing the "heap" (which is what I am guessing he is doing) 
he can count all references to a particular item.  and YES it wil 
be slow.  but for debugging it could be EXTREMELY usefull.  even 
more would be returning all the items which the GC has encountered 
which have references)

REBOL isn't heading towards less error triggering as a rule (though 
there may be less in practice); it is headed towards making errors 
trigger on more useful occasions. The principle is that errors are 
your friends, and if they aren't, we need to reevaluate them on a 
case by case basis. In some cases we have made things trigger an 
error in R3 that didn't in R2, with the hope of overall benefits 
in security, stability and debuggability. We even added ASSERT to 
explicitly trigger errors.

In the case of LENGTH? and INDEX?, we are only allowing them to return 
none when passed none, using none as the equivalent of SQL's unknown, 
not N/A. And even those are a bit iffy - we are only planing to allow 
those to have fewer intermediate none checks when doing none pass-through 
on FIND and SELECT return values. However, the disadvantage is that 
errors aren't triggered close enough to the source of the error. 
Hopefully they will be triggered later by ASSERT calls.

My preference would be to not lose the valuable information about 
which values have length and which shouldn't be passed to LENGTH?. 
We definitely don't want too many nones to be propagated to data 
that doesn't expect them; none leaks are tough to track down, which 
is the whole purpose of the unset! type.

We don't know the exact algorithm, but we know a lot about it, and 
that it's proprietary.

that's the kind of high level discussion that should conclude by 
a high level documentation about memory management and gc predictions 
... but probably better wasting the oportunity ... once again. I 
would not ask why?

GC stuff that we can figure out from experimentation and conversations:

- Mark and sweep
- Not generational or copying
- Probably zoned into block and binary/string zones.