World: r3wp

I still vote for foreign!  as a datatype.  Sure there will still 
be reasons for throwing errors on evaluation on foreign! but I don't 
like the fact that the errors occur on load.

Give linguists a tool that out-of-the-box can load a poem and let 
them count characters, words, and transitions.


Let REBOL pros worry about the hassles of evaluation.   Re: where 
the foreign data ends; Skip AI, just go with delimiters.  Whitespace 
or quotes or any of the braces etc.  And if a missing quote causes 
a script to fail ... well bad on the coder for not testing before 
release.  If the English professor gets wonky counts, well let them 
figure it out, but give them the wonky counts not a LOAD failure. 
 IMHO it really would open the door to so many more uses and users. 
 


And with Steeve's example ... build something like that into the 
product.  Don't make poor Professor Keating try and figure out why 
most of the Dead Poet's prose can't be analysed with REBOL's nifty 
cool block data functions.


And as long as REBOL had foreign! as a native datatype, writing a 
LINT would be dirt easy.

And I could LOAD %cobol.cob  and do all kinds of super cool preprocessing. 
 Or at least wait for Sunanda to write a super cool preprocessor 
and live vicariously through rebol of the year candidates.  ;)


REBOL could be the JavaDoc, EpyDoc, Doxygen, ..., ReST besting documentation 
super tool of tomorrow.

doc, the point is that I and others HAVE had ,many situations where 
this would have saved HOURS even given new reasons to use rebol at 
some jobs.  the point is not to allow new rebol dialects, its just 
that in MANY cases it would work..  we all know that this isn't a 
replacement for parse.  but allowing me to import external data through 
load is just faster than parsing.  You still have to tailor the extension 
to the dataset anyways.

We could load things like C code directly, and then parse it using 
block dialecting instead. way faster.

Ditto Maxim.   Adding ...  I don't look at this as being a overly 
good thing for rebols (although I do think we'd all benefit, save 
time and write a whole new breed of utilities), I think of it as 
a good thing for non-rebols (and us rebols with smaller brains and 
bigger laze).

ah, so we are at it once again ....

You guys write it, then we can talk about it.

it's incredible you think that "parse is not easy" and then you don't 
trust people telling you that what you ask for is silly at best.

parse *IS* easy. try to use regexps instead!

ok i give you load-foreign, don't kick me Gabriele :-)

context [
	stack: make block! 10
	foreign-rules: make block! 10
	out: end: value: pos: none
	push: func [type][
		stack: change/only stack out 
		out: make type 1 
	]
	pop:  to-paren [
		stack: back stack 
		out: append/only stack/1 out
	]
	set-&: [end: (misc/&: to string! copy/part pos end)]
	misc: context [
		&: none
		keep: func [&][append out &]
	]
	blanks: charset " ^-^M"
	rules: [
		  some [blanks | #"^/" (new-line out true)]	  
		| pos: foreign-rules
		| [#"]" | #")"] (print "missing [ or (" return stack/1)
		| #"[" (push block!) any [#"]" pop break | rules]
		| #"(" (push paren!) any [#")" pop break | rules]

  | (set [value pos] transcode/next/error pos append out value) :pos
	]
	
	set 'load-foreign func [
		text [string!] foreign [block!]
	][
		out: make block! 10
		stack: change/only clear head stack out
		clear foreign-rules
		foreach [rule code] foreign [

   append foreign-rules compose [(rule) set-& (to-paren bind code misc) 
   |]
		]
		append foreign-rules  [end skip]
		parse/all to-binary text [any rules]
		bind out 'system
		either 1 = length? out [first out][out]
	]
]

]

Steeve, you're just proving my and Gabriele's point. There's no way 
to implement a LOAD-FOREIGN that would work for any data. You have 
to provide parsing rules that are specific for the input data. Max 
and others are asking for a LOAD-FOREIGN that just works on any input 
data without providing any additionnal rules.

and if you have to provide the additional rules you might as well 
parse it from scratch.

i can generate strings by default each time there are errors by default, 
so that there is no need to provide rules.
It's a tiny modification

But strings are not good types for those  it would be better to simply 
add errors as objects in the loaded stream.

But after that you steel need to reparse the block to convert the 
errors in proper values.
SO, in the end, y

*steal

Ok, in the following version (much simpler) i just add foreign data 
as error objects in the stream.
Then you can reparse the result to manage errors as you want.

context [
	stack: make block! 10
	out: value: pos: none
	push: func [type][
		stack: change/only stack out 
		out: make type 1 
	]
	pop:  to-paren [
		stack: back stack 
		out: append/only stack/1 out
	]
	blanks: charset " ^-^M"
	rules: [
		  some [blanks | #"^/" (new-line out true)]	  
		| [#"]" | #")"] (print "missing [ or (" return stack/1)
		| #"[" (push block!) any [#"]" pop break | rules]
		| #"(" (push paren!) any [#")" pop break | rules]

  | pos: (set [value pos] transcode/next/error pos append out value) 
  :pos
	]
	set 'load-foreign func [
		text [string!] 
	][
		out: make block! 10
		stack: change/only clear head stack out
		parse/all to-binary text [any rules]
		bind out 'system
		either 1 = length? out [first out][out]
	]
]

probe load-foreign {
	either condition [
		f,o,r,e,i,g,n,0,1
	][
		false
	]
} 

>>[
    either condition [
        make error! [
            code: 200
            type: 'Syntax
            id: 'invalid
            arg1: "word"
            arg2: "f,o,r,e,i,g,n,0,1"
            arg3: none
            near: "(line 1) f,o,r,e,i,g,n,0,1"
            where: [transcode parse load-foreign ...]
        ]
    ] [
        false
    ]
]

>> load-foreign "either condition ]f,o,r,e,i,g,n,0,1[]false["
missing [ or (
== none


Not very usefull output. Remember that we are talking here of the 
general case where input can be *any* data.

i gave the design, it's simple to ehance

what is good to return in that case ?

It depends on the rules chosen to interpret the string content. That's 
the point, there's no simple way to implement a general case load-foreign 
that would return a block! of mixed REBOL values and "foreign" values 
as string. It's not possible to define a general set of rules because 
"foreign" can be just anything. LOAD just returns an error! when 
the syntax is incorrect and that's the best thing it can do.

doc, never said it should accept anything...

just that I have seen it would work so many times as to not make 
it a useless feature.

its also a question of speed.  load is blighthingly fast.

If you can define a set of rules for your specific needs, then you 
can just use parse or Steeve's load-foreign for that (or ask someone 
to write it for you if you don't want to implement it). Everyone 
can have its own specific needs, I can't see how to define a set 
of rules for "foreign" data that would be or could be useful to most 
users. That would probably end up defining a few additionnal datatypes 
(or alternative syntax for existing datatypes).

Max, did you ever consider WHY load is so fast?

What makes you think that it would be so fast if it was able to do 
what you ask?

Because.  ;)   If REBOL had a foreign! datatype! could the lexer 
not just ... about to throw syntax error, make a foreign! and continue?


I still don't see why this is deemed a bad extension to REBOL.   
Sure it "stinks" to guru's but it opens doors to fishermen, contractors 
and linguists, imho.  And yes it could look like a load of gunk if 
a quote, paren, brace or bracket was misaligned.  So?


For code purity, I'm sure Steeve could spend a few minutes to write 
4 lines to scan a loaded series! and recursively highlight the foreign! 
data, providing a semblance of a lint procedure for those writing 
apps.  My push for foreign! has little to do with apps, and just 
about all to do with data, human real-world data.   For people using 
data for sums, control, etc... just use the lint scan to reject data 
blocks that have ANY foreign! substances??

Why not just load the bad data as a string! ?

should be simple to parse out string!s when a numeric datatype is 
expected...

Are you guys listening?

yep

I'm just a bit amazed at the lack of comprehension.  I agree with 
DocKimbel and I advise to think on his words longer.

what !?

I have never met with anything I could not handle. Gee, we are complaining 
about REBOL parser not being able to handle ANY format, while even 
lamers as me are being able to use 'parse in opposition to regexp? 
Why this possession? What is exact deal breaker? Isn't it a bit naive? 
Brian, really - what fishermen are you talking about? For anything 
more complicated than one-liner, you have to come up with script 
anyway. And if you store code into script, you can write few parse 
rules, no? Well, maybe it is me, who never uses load (I don't like 
the fact it mysteriosly does decoding of few things here or there 
- libraries, jpg, which is fixed in R3), but with REBOL string parsing, 
you can do many things. I don't even agree to Max opinion, that parse 
will let you down. Well, if you want to parse streamed mp3 binary 
content, maybe so, but for some general data format, specifically 
delimited? Come on :-)

Parse is my beloved function in Rebol, i use it all the tuime.

*time

btw - there is now new blog posted - encoders/decoders. We should 
better care to get those things done right. You can write your decode-anything 
codec to input your mysterious data :-)

eh ?

(i don't understand)

Even my compromise has been rejected as too slow. Use TRANSCODE/error 
in a loop if necessary, but otherwise come to accept that LOAD loads 
REBOL data, and whatever DECODE can load (which is *anything* in 
theory).

Anton ... it's not (I don't think, but I guess it could be) lack 
of comprehension.  It's a desire to see REBOL usefulness expand beyond 
the borders of us 200 loyal fans.
>> book: load %somefile
>> sort book
>> length? unique book

etc...   Can you not see a linguist loving this?  And perhaps adding 
to the size of the REBOL user base?
>> account: [paid $10,000.00 to bob]
A general contractor would love to type that in.

It just seems so close to being a thing that could be done without 
too much fuss or bother if it was internal.

Everyone here in REBOL3 ... we could make REBOL "more" restrictive 
and magic would still happen.  Carl and we advocates don't have to 
worry about you/us lot.  We're hooked already.


For instance.   What if    ``\\ was made a special! datatype?  Would 
you complain?  Or just accept it as a new feature of REBOL?   I don't 
see what a catch-all foreign! would harm.   The lexical complexity 
of REBOL is wonderous, I don't see what could hurt if it was more 
human friendly to boot.

Pekr ... I grew up a commercial fisherman.   ;)

As someone who writes some of the internals, I can tell you that 
internalizing the fuss and bother doesn't make it less bothersome. 
It's not magic: The overhead would be the same if we added it to 
TRANSCODE - we might as well write the REBOL syntax in PARSE.

Flexibility is slow. If you want to load another syntax, try something 
like this:
    install-codec fishing-jargon
    load %blah.fish

Theoretical functions of course, but LOAD will call DECODE, which 
will look for an installed codec that it passes the data to.

Before I read this I was going to add ...

Feeling a little bit responsible for being this argument to the rebol 
community, in a small attempt to advocate for more users ...  I'd 
also like to say, that push come to shove; if the argument gets to 
a point of driving away the likes of Nenad, or Gabriele in some sense 
of disgust at the un-understanding of some of us versus the potential 
of a few thousand new faceless strangers that may be attracted to 
a "garbage loading dialect" language.  I'd vote to keep Doc and Gab 
happy and skip the faceless masses.   ;)

Check out TRANSCODE/error.

But ... I still don't see the performance hit of a lexical scanner 
that defaulted to "make foreign!" and continue whenever a position 
in the code would trigger a syntax error?  Or as Anton states ... 
is this an amazing lack of comprehension?


I've been working with Bison and Flex ... and from what I understand, 
this would not make the parse tree any more complex.  It would simply 
change the runtime semantics to "make foreign!" at every point where 
the scanner would fail.