AltME groups: search

Hm.  Version:   2.100.96.2.5  I quit and restarted.

And it stopped doing that.  I'll see if I can get it to do it again.

Yes. You can express a sequence of characters in a string as a string 
literal, but not a sequence of types in a block. You are going to 
need first sets and the other LL tricks for that.

Indeterminate, e.g. just ran it again and:

Ladislav: I didn't realize you could use "while" as the second argument 
to copy, I thought it only worked with to and thru...

kcollins: I'm using OS/X, I still haven't found a way to reproduce 
it.  Comes and goes.

What is the difference between BREAK and ACCEPT? Both "break" out 
of the rule, both with success (IMO).

I'm still running into some problems with PARSE... mainly from the 
expectation of what ANY and SOME should do.

For example:
>> parse "" [any [copy tmp to end]]
>> tmp
== ""

It's a small thing, and maybe too late to change. I wanted to point 
it out.

any [and skip copy tmp to end]
any [copy tmp [skip to end]]
etc...

I see your point, but what if the ANY block contains production rules 
?

parse "" [any [and skip copy tmp to end break | insert "1" and insert 
"2"]]

(i know, stupid example)

We have some cool new parse enhancements; really, really nice some 
of them. What I think will add the most value to PARSE--and maybe 
this is just me--are practical examples, idioms, and best practices.

For example


- Parsing an input that has nested structures, and how to collect 
the values you want.
- Showing the user where the parse failed.
- How to avoid infinite parse loops.
- How to safely modify the input stream.

More advanced examples would be great too of course.

Carl - first "error" in parse rewrite with some/any is the auto protection 
for non advancing input. It is like writting in BASIC

10 Print "Hello"
20 goto 10


... and not expecting it to run forever, because some magical internal 
mechanism kicks-in. If I write the code which could cause infinite 
loop, then be it. For me it causes the opposite reaction - some/any 
are not safe to use, let us use while instead ....


something like: parse str [some [to "abc"]] is so obvious and self 
explanatory, that actually not looping forever almost feels like 
parse error. But - even if I don't like it, maybe most such infinite 
loop hits are more difficult to notice, so that actually the prevention 
might be ok, I don't know. As for me though, I would probably prefer 
some internal capability to detect such case, and some debug option 
to show last rule/position, where it happens ...


I am not fluent enough with parse theory, but maybe it also relates 
to your loop vs matching note above ...

BenBran:
Not sure where to put this so asking here:


I downloaded a web script and it has a  snippet I don't understand:
buffer: make string! 1024         ;; contains the browser request
file: "index.html"
parse buffer ["get" ["http"  |   "/ "  |  copy file to " " ]]

what does:

copy file to " "

mean or do?
tia

The copy and to are parse operations. COPY copies the data covered 
by the next operation, the TO. TO covers the data from the current 
parse position until the first instance it can find of its argument.

The break being a parse match fail, and file being set to none for 
a zero-length match.

I get whats happening now.  If i compare buffer and file I see the 
clipped text:

>> probe file
== "index.html"

>> probe buffer
{GET /a.html HTTP/1.1
Host: localhost

User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/531.21.8 
(KHTML, like Gecko) Version/4.0.4 Safar
i/531.21.10

Accept: application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-US
Accept-Encoding: gzip, deflate
Connection: keep-alive
Address: 127.0.0.1}

>>probe parse buffer ["get" ["http" | "/ " | copy file to " "]]
== false

>> probe file
== "/a.html"
 
Should I have been able to see the results instead of  == false?

PARSE returns true if the rule matches and covers the entire input, 
or false otherwise. Your rule matched but there was input left over. 
PARSE's return value doesn't matter in this case, just whether file 
is set or not. If you are using R3 you can do this too:
parse buffer [ "get" [ "http" | "/" | return to " "]]

That would return the file instead of setting a variable and not 
return false because of leftover input.

There's a difference between COPY and SET in block parsing mode.

And there is a great likelihood of the bugs being fixed in R3. And 
there aren't many in PARSE, just that tag bug afaik.

Graham, I deleted bug #1449 since it was already reported as #682. 
See also #854 and #1160 (and #10, which was incorrectly "fixed").

Agreed (and the policy agrees too).

TO and THRU have limited argument syntax, and don't support full 
rules. Both R2 and R3 support literal value arguments (that don't 
count as rules). R3 also supports a block of literal values delimited 
by |, and those values are less limted.

Oh crap. Well, it was reported as a bug, and it's staying that way 
until Carl says otherwise :)

given that to and thru do "more" in R3, it probably is not bad to 
consider it a bug. (maybe it should be considered a bug in R2 as 
well, given that FIND does work with charsets...)

ahh... correction, it works under R3 and locks up in R2 :(

and finally a parse rule that works under r2 and r3

	parse/all txt [
		some [
			[ end | any nondigits ] [ date-rule | some digits  ] 
		]
	]

Yes, "it's faster than anything else, until it's not" is a perfect 
statement, and you got my agreement :-p

ladislav, Remark changes the input on the fly to implement function 
html unfolding, and using that improved speed by 50 times, when compared 
with traditional series manipulations.

so yes its seriously usable   ;-P

Now, I can make a bold statement: for any method distinct from the 
one using PARSE and CHANGE/PART combo holds, that it is faster than 
the above method, until it's not :-p

its not a single change/part which is the issue, its managing the 
stack, allocating all those blocks over and over... the sheer speed 
of the parse loop, blows away all the other looped/recursive algorythms 
in my usage so far.

And here I thought yesterday, wow I finally understood Parse and 
gosh it's awesome. And now I read change/part, which I used, is not 
the way to do things unless it is. I am confused! Generally, but 
also now specifically.

I think change/part is as fast as Rebol's change/part native, and 
hence usable, unless Ladislav proves such pov being somehow fundamentally 
wrong :-)

my take on "speed" is as follows - ppl sometimes object, that you 
use "interpreter". And my answer is - why should I care? The thing 
is either fast enough for me, or it is not fast enough for me. If 
you will try to edit video using REBOL level pixel manipulation, 
you surely will not be happy. But - if your app behaves real-time 
or generally time results are acceptable for you - why to worry at 
all?

I updated the above mentioned Rebol wikibook section - the speed 
discussion subsection added. So, read, it, please (the latest, i.e. 
yet unsighted version!) and see also the CureCode ticket #1570, which 
is related to this issue.

Of course that has other semantics than

>> head change/part "abc" take/part "123" 2 1
== "12bc"

and may lead to new confusion.

and extra speed consideration of having to allocate/copy/destroy 
a series

That's said too much; I think it's more that CHANGE/PART behaves 
as advertised and the /PART refinement just happens to have a different 
meaning for INSERT or APPEND. 

Neither one of /WITH, /TO, /SPAN and /RANGE communicate very well 
that they refer to the second argument though, and /TAKE has the 
drawback of suggesting that it's taking away from the second argument 
like TAKE instead of leaving the second argument untouched. 

CHANGE/FROM, however, seems to work:

>> head change/from #abcdef #123456 3
== #123def
>> head change/part/from #abcdef #12345 1 3
== #123bcdef 


All that under the assumption that for compatibility, /PART in it's 
current meaning will stay as it is.

It's funny, I always thought INSERT/part was the weird one, and CHANGE/part 
the normal one. Didn't stop me from adding /part to APPEND though, 
in the INSERT style.

/?  !?!!??!! 

and meaningless  ;-)

Gregg, I used to use append/part to avoid the memory overhead of 
copy/part in many case.
Instead of doing like in the Ladislav's example. 
>> change/part something copy/part something-else range part.
I used  to do.

>> change/part something append/part clear #{} something-else range 
part.
It's not faster, but saves memory.


So, I don't know if it's a good idea to discard this use case from 
append and insert.

Sometimes, I can't let the GC acts by himself because it's too late 
and tens of MB would be allocated for nothing.

(I just tested, and your example is much slower than the code allocating 
and GC-ing the new string)

Sometimes you don't want to put too much pressure on the GC, and 
sometimes you don't want to increase the total size of the pool too 
much, because that pool doesn't always get returned to the OS very 
quickly or at all. This is the motivation for additions like the 
/into option.

and the GC doesn't kick in too quick or it would be really slow  
(just try recycle/torture to see ;-)


so when you're doing serious work it REALLY grows... although it 
stabilizes


for example although stats often show 10MB... my OS tells me that 
its actually using 24 MB.  that will never shrink back down.

I've created my very first script. The script loops through a list 
of email (Kerio) log files, extracts the IP addresses, compiles them 
in a list and adds them to a (Peerblock) list in order to limit incoming 
spam. I find rebol perfect for this.

Improve the script by reading only the latest entries in the log, 
and I pare the date like this: parse/all txt [thru "[" copy found 
to "]" ]

I've used parse/all, and then used 'trim on the results.

Yes, that is exactly what I did and it works. However, for the sake 
of learning, how do I use the the space character as part of my rule?

This finds the IP in the log entry. What if I have two ip addresses 
and I want to pick them at the same time: ip: [some digits "." some 
digits "." some digits "." some digits __space__ some digits ...etc]

And the IP addresses are separatered by a space?

correct, and then, how do you place the space in the rule: {} ?

Yes, parse/all is great, and this is why I want to include the space 
not as a delimiter but as a character in the rule. As if, sometimes 
I want to find two strings separated by a character.

rebol and logs are like bread and butter

Then, I said, read only from the last read, and pare the date/time. 
I wanted to parse date AND time at the same time" [15/May/2010 17:59:56] 
But I hit a snag because of the space in between. I don't want date 
and time separater beause rebol can parse the string into a date-time 
easy. The space gave me trouble, and the brackets too.

Using the charset did not work. So I did this: parse/all txt [thru 
"[" copy found to "]" ]. And said to myself, time to learn more and 
could not find resources.

What is the problem in parsing date and time at the same time? I 
somehow don't understand, if the solution does what you need, or 
you need further help?

Processing text and rules seems so natural to rebol. I think I'm 
going to enjoy this.

Ok, continuing the discussion from "Performance" group, I'd like 
to ask for some help with parsing rebol format files.

Basically, I'd like to be able to extract a block near the beginning 
or end of a file, while minimizing disk access.

The files to be parsed could be large, so I don't want to load the 
entire contents, but chunks at a time.

So my parse rule should be able to detect when the input has been 
exhausted and ask for another chunk.

(When extracting a block near the end of a file, I'll have to parse 
in reverse, but I'll try to implement that later.)

And why you don't want to use the load/next which was advised?

Which is why, in that algorithm, I had to iteratively: load a chunk, 
append it and try LOAD/NEXT until it succeeded.
Which gives the algorithm O(n^2) performance.

My question for this O(n) parse algorithm is:
What rebol syntax do I need identify?
I suppose all I need is:
 - Comments (lines beginning with semi-colon ; )

 - Strings (single-line "" and multi-line {} ) watching out for escape 
 sequences eg. {^}}
 - Blocks

and you want to get just the first block? Skipping any content before 
it?

Well, I'd like the algorithm to be general enough to get any number 
of block in the file. So far I need just the first, second and last 
blocks.

And you need to parse the Altme's *.set files or something else as 
well?

I imagine it could be useful in other similar situations, so I'd 
like it to be pretty general.

I suppose a bonus functionality is to be able to get nested blocks.

(And a super bonus will be to get any datatype at any level, but 
I won't bother doing that until I need it.)

Must it ?

I think if I can parse single-line strings correctly, then a bracket 
inside won't cause a problem.

This means I'll be basically ignoring datatypes which allow strings 
in their syntax, and just jumping to the string part.

And then I tried issues, files, and I can't do it there, either.

Does anyone have any advice on how I should structure this algorithm?

I don't feel confident as I haven't studied parsing theory deeply.
http://en.wikipedia.org/wiki/Parsing

Should I do lexical analysis and syntactic analysis separately ?

I think I can do it all with just one parse, but it might not be 
a good idea.

I just found something interesting.

I remember Gabriele saying he thought PARSE would convert chars it 
encountered in its rule with strings before using, so these are equivalent:
	parse "a" [#"a"]
	parse "a" ["a"]

(Of course, the first one is a char and not a string, so consumes 
less memory.)

But I was just thinking it might be clearer to use strings instead 
of chars in the parse rule.
Then I discovered you can use issues:
	parse "a" [#a]

and the escape characters is interesting as you only need to type 
one of them in the issue:
	parse "^^" [#^]

Anton, the cost of disk reads dwarfs the cost of LOAD/next. And PARSE 
is much slower at loading REBOL data than LOAD. You might consider 
finding out the max size of the value you are loading, rounded up 
to multiples of 4096 (disk blocks), and just READ/part a bit more 
than that from the disk for each file. Then LOAD/next from the resulting 
string. There is no reason to do speculative reads once you have 
an upper bound on the size you will need to read. In a language like 
REBOL, minimizing disk reads sometimes means minimizing the number 
of calls to READ, not just the amount read.

And loading from the resulting string, not the file?

Your solution is similar to what I suggested, but is missing a couple 
speedups:

- Getting an estimate of how many characters the value would take 
on the high end, and using that as the initial read part.

- Chunk value reflecting hard disk sector value (the OS may load 
in 4096 byte blocks)

BrianH, finding out the size that is sure to encompass the desired 
block in all my input files requires prescanning the entirety of 
all of the files at least once. That's a good optimization for my 
specific case, but I want to make the function general enough that 
it can be used in other situations where the data may not be so consistent, 
and the desired block may not be always near the beginning or end.

I'm using REBOL to control a test by using the Parse dialect to check 
information returned from the test environment. From other looking 
around, it seems that the best approach would be to implement a Telnet 
scheme to handle the input/response give and take with the test environment, 
but I can't find an implementation I've been able to tweak. So.....my 
question is, has anyone had success with loading a Telnet client 
as a dll/shared library and getting Telnet functionality that way?

And come back here if you need help with the parsing part :)

Hi! Thanks for taking a look at the code. 


I went over it again, it seems that part of the problem was in the 
fact that the parsed objects weren't transliterated into strings 
as I had expected. I.e. if you look at the output of the code snippet 
above, it seems OK, but examination of the types of the data in the 
tokens array turn up things that don't convert to strings too well 
without help. I've puzzled over Carl's pretty printer, and I _think_ 
I understand why now... 


Either way, I was able to modify it to give me the kind of output 
I wanted. To repay you for your kind attention, I will post my code 
here, but in crushed form, so it doesn't take up too much space... 
; - )


REBOL [ Title: "REBOL Compressor" ] emit-space: func [ pos ] [ append 
out pick

[ #" " "" ] found? not any [ find "[(" last out find ")]" first pos 
] ] emit:

func [ from to ] [ emit-space from word: copy/part from to long: 
( length? out )

+ length? word if 80 < long [ append lines out out: copy "" ] append 
out
copy/part from to ] lines: copy [ ] clean-script: func [
Returns new script text with standard spacing.
 script "Original Script text"

/local str new ] [ out: append clear copy script newline parse script 
blk-rule:

[ some [ str: some [ newline ] ( ) | #";" [ thru newline | to end 
] new: ( ) | [

#"[" | #"(" ] ( emit str 1 ) blk-rule | [ #"]" | #")" ] ( emit str 
1 ) break |

skip ( set [ value new ] load/next str emit str new ) :new ] ] append 
lines out

remove lines/1 print [ length? lines "lines." ] lines ] write/lines 
%crushed.r
clean-script read %c.r print read %crushed.r

Thanks!

If you note that load/next is used when values are parsed, you can 
see why values aren't strings. MOLD can be your friend as FORM (and 
PRINT) will hide datatype details from you. e.g.

>> print first [x]
x
>> print first [x:]
x
>> print first ['x]
x
>> print first [:x]
x

JoshF, if this script stands alone then I would make these changes:

- Add as locals to EMIT: WORD and LONG.

- Add to CLEAN-SCRIPT's locals: LINES, OUT, EMIT-SPACE, EMIT, BLK-RULE, 
VALUE
- Move into CLEAN-SCRIPT's body:
	1	lines: copy []
	2	The EMIT-SPACE function
	3	The EMIT function
- Change this line:
	out: append clear copy script newline 
to:
	out: copy ""

(There's no point copying the string SCRIPT when the next thing you 
do is CLEAR it.)
- Remove this line:
	remove lines/1

(There seems no point in initializing OUT with a single newline char 
if it is only to be removed ultimately.)


After, that, you should have only one word, CLEAN-SCRIPT, defined 
globally, referring to a function with no side-effects.

Hrrrm.  I haven't messed with block parsing much, but it breaks any 
obvious intuition that >> parse [[2]] [do [(1 + 1)]] is true and 
>> parse [2] [do [(1 + 1)]] is also true.  :-{

Actually, DO is really easy to simulate, both in R2 and R3.
Just construct the rule on the fly.
>> parse [2][(rule: do [1 + 1]) 1 1 rule]
==true

and what if rule ic create dynamic form file in global words  and 
can not by  create in functions ?

I think he meant to ask "and what if RULE is created dynamically 
(ie. loaded from a file) and thus its words are global, and are not 
 (or cannot be) created by functions?"

ah ok thanks for the translation.
BIND and BIND? are the keys

Is there a foundational reason for DO not being available for string 
parsing, or is it just not implemented?

There are a lot of things that you can do in block parsing that you 
can't in string parsing. In this case, the result of DO is compared 
directly as a REBOL value. Strings don't directly contain REBOL values 
the way that blocks do. Even if you tried to limit the result types 
of the expression and trigger an error if they don't match, what 
you are left with isn't useful enough to justify adding it, imo. 
For instance, in your example it was a bad idea to use DO. We'll 
see though.

Micha, there was a direct solution proposed for this in the parse 
proposals, specifically to deal with local variables in recursive 
parse rules. However, it turns out that PARSE isn't really recursive: 
It fakes it. So there was no way to support this feature in a parse 
directive. The best way to do the local variables is to put the PARSE 
call and the rules in a function, and if you have to use recursive 
rules, recursively call that function in an IF (...) operation. It 
really works well, in a roundabout sort of way.

micha, you can also just push and pop values in a block you use like 
a stack.  you push before setting to a variable, you pop after the 
rule.

you just have to make sure to only push/pop once a complete rule 
is matched.  meaning you handle that in a paren at the END of the 
rule.

The best way to do the local variables is to put the PARSE call and 
the rules in a function, and if you have to use recursive rules, 
recursively call that function in an IF (...) operation. It really 
works well, in a roundabout sort of way.
 - this is too much of a roundabout for most cases, I have to add

Fork, that won't work on some return types, and that would lead to 
runtime errors. But in theory, yes.

Ladislav, true. But since PARSE doesn't really recurse, the only 
direct way to have local variables would be to BIND/copy the parse 
rules for each level of recursion. Doing the function recursion method 
is actually more efficient and easier than that.

I guess, that it is the time to propose a reasonable and efficient 
method

Thanks Ladislav. 

What do 'fni and 'fnii stand for?


I would certainly add a comment or doc string that USE-RULE is recursive/thread 
safe, which is why it's not much simpler.

And 'inner-inner-body. :-)

...and yet influencing what the CONTEXT-FN is actually doing

set 'use-rule func [

    "Create a recursion and thread-safe parse rule with local variables. 
    R2/R3 compatible."
    words [block!] "Local word(s) to the parse rule"
    rule  [block!] "Parse rule"
] [
    make object! [

        ; Create a new function context. 'Inner-body refers to a function 

        ; with access to CONTEXT-FN's context without being influenced 
        ; directly by the context.
        spec: copy [/local]
        append spec words
        inner-body: func ['word] [inner-inner-body word]
        context-fn: func spec reduce [:inner-body first words]
        

        ; Bind the rule the caller gave us to the new context we just created.
        inner-inner-body: func [word] [bind/copy rule word]
        bound-rule: context-fn
        

        ; Now define the use rule. Because this is an "active" rule,
        ; with state we need to include some state variables used
        ; by the internal PARSE call ('pos and 'success).
        pos: none
        success: none
        inner-inner-body: func [word] [

            ; If the parse of the rule succeeds, we set the parse position

            ; to the where the rule match ended, otherwise we don't change

            ; the parse position and use [end skip] to return a false 
            ; result (for R2 compatibility).
            success: either parse pos [bound-rule pos: to end] [
                [:pos]
            ] [
                [end skip]
            ]
        ]
        set 'rule copy/deep [pos: (context-fn) success]
    ]
    rule
]

Quite precise, except for the fact, that SUCCESS and POS are mainly 
used to "transfer" the inner parse state to the "outer parse"

set 'use-rule func [

    "Create a recursion and thread-safe parse rule with local variables. 
    R2/R3 compatible."
    words [block!] "Local word(s) to the parse rule"
    rule  [block!] "Parse rule"
] [
    make object! [

        ; Create a new function context. 'Inner-body refers to a function 

        ; with access to CONTEXT-FN's context without being influenced 
        ; directly by the context.
        spec: copy [/local]
        append spec words
        inner-body: func ['word] [inner-inner-body word]
        context-fn: func spec reduce [:inner-body first words]
        

        ; Bind the rule the caller gave us to the new context we just created.
        inner-inner-body: func [word] [bind/copy rule word]
        bound-rule: context-fn
        

        ; Now define the use rule. Because this is an "active" rule,
        ; with state we need to include some state variables used

        ; by the internal PARSE call ('pos and 'success). They are used to 
        ; "transfer" the inner parse state to the "outer parse".
        pos: none
        success: none
        inner-inner-body: func [word] [

            ; If the parse of the rule succeeds, we set the parse position

            ; to the where the rule match ended, otherwise we don't change

            ; the parse position and use [end skip] to return a false 
            ; result (for R2 compatibility).
            success: either parse pos [bound-rule pos: to end] [
                [:pos]
            ] [
                [end skip]
            ]
        ]
        set 'rule copy/deep [pos: (context-fn) success]
    ]
    rule
]

set 'use-rule func [

    "Create a recursion and thread-safe parse rule with local variables. 
    R2/R3 compatible."
    words [block!] "Local word(s) to the parse rule"
    rule  [block!] "Parse rule"
] [
    make object! [

        ; Create a new function context. 'Inner-body refers to a function 

        ; with access to CONTEXT-FN's context without being influenced 
        ; directly by the context.
        spec: copy [/local]
        append spec words
        inner-body: func ['word] [inner-inner-body word]
        context-fn: func spec reduce [:inner-body first words]
        

        ; Bind the rule the caller gave us to the new context we just created.
        inner-inner-body: func [word] [bind/copy rule word]
        bound-rule: context-fn
        

        ; Now define the use rule. Because this is an "active" rule,

        ; with state, we need to include some state variables used

        ; by the internal PARSE call ('pos and 'success). They are used to 
        ; "transfer" the inner parse state to the "outer parse".
        pos: none
        success: none
        inner-inner-body: func [word] [

            ; If the parse of the rule succeeds, we set the parse position

            ; to the point where the rule match ended, otherwise we don't 

            ; change the parse position and use [end skip] to return a false 
            ; result (for R2 compatibility).
            success: either parse pos [bound-rule pos: to end] [
                [:pos]
            ] [
                [end skip]
            ]
        ]
        set 'rule copy/deep [pos: (context-fn) success]
    ]
    rule
]