AltME groups: search

And do you want to avoid putting them into a block first ?

'parse' is the path to great explorations and inventions - and also 
to great confusion and maybe despair. ;-)


No really, it can be a bit confusing at times, but I guess, it can't 
be done otherwise to have such great functionality. There's no short 
cut with 'parse'. Learning by doing is the way to go. And it's a 
brilliant tool!

count-word-frequency: func[
	"Counts word frequency from the given text"
	text [string!] "text to analyse"
	/exclude ex [block!] "words which should not be counted"
	/local counts f wordchars nonwordchars
][
	counts: make hash! 100000

 wordchars: charset [#"a" - #"z" #"A" - #"Z" "�������������̊�؎����ύ�Ѫ�"]
	nonwordchars: complement wordchars
	parse/all text [
		any nonwordchars
		any [
			copy word some wordchars (
				;probe word
				if any [not exclude none? find ex word][
					either none? f: find/tail counts word [
						repend counts [ word 1 ]
					][
						change f (f/1 + 1)
					]
				]
			)
			any nonwordchars
		]
	]
	counts: to-block counts
	sort/skip/compare/reverse counts 2 2
	new-line/skip counts true 2
]

found missing czech chars->  wordchars: charset [#"a" - #"z" #"A" 
- #"Z" "���������������̊�؎����ύ�Ѫ����"]

Is this a bug?
parse/all {"some words"} {" }
;== ["some words"]
parse/all {and "some words"} {" }
;== ["and" "some words"]
parse {and "some words"} {" }
;== ["and" "some" "words"]
parse {"some words"} {" }
;== ["some words"]

Good question! It's in a tough corner of REBOL - parsing. REBOL is 
in many ways more like a human language, than a computer language. 
Strictly speaking, you can argue, that those examples have a bug 
or two, but can you live with it? The behaviour might make it difficult 
to parse input strings, written by humans, because people write all 
sorts of things. (If it can go wrong, it will.)


Try change the quotation marks to something else and see the results 
change, like:

>> parse/all {Xsome wordsX}�{X }
== ["" "some" "words"]

parse, without a rule, treats quotes specially. this is to allow 
parse to be used directly with things like csv data.

I think it's a bug! I was trying to use this to divide large string 
to words and found that I have all sentences inside , instead of 
just words. It's problem only if you have the divider on the edge.

this behavior is the one intended by Carl. so, it's so by design, 
and not a bug. but, you may try to convince Carl that you don't like 
it. ;)

I still think it's a bug - I cannot see the diference between parse 
and parse/all in this example. If Carl don't want to fix it, no problem 
for me, I used more complicated rule to do the same thing, just still 
think, it's a bug and it will confuse more people in the future as 
well.

and parse {,"a b, d"  ,d} {,} == ["" "a b, d" "d"]  (so probably 
Carl has true;-)

But it should be in documentation, that the quotes are very special 
characters for such a type of parsing!

Oldes a regex context will be a good addition. Where regex are the 
basic rules for numbers, white space, *words* and their negations.

anton: I think, that any parse rule which don have to be global variable, 
but you can still the name used in parse block. But probably it would 
be a security issue

I've thought about that as well. There are some base charsets we 
could probably standardize on, and that would be good (IMO). Beyond 
a few basics, though, consensus gets tough.

The singular/plural argument seems easy, but isn't (IMO); DIGITS 
could be done as SOME DIGIT, and you could argue that things like 
2 DIGITS reads better, though 1 DIGITS does not. You could double-define 
it, but that gets ugly too. So, what about DIG? That doesn't imply 
any singularity, though it's a bit terse, and not a full word (or, 
rather, the wrong full word).

II was sure I'd posted this just after Oldes' message.....But it 
ain't there now.....Maybe it's in the wrong group)
Andrew has a nice starter set:

http://www.rebol.org/cgi-bin/cgiwrap/rebol/view-script.r?script=common-parse-values.r

And I know he has extended that list extensively to include things 
like email address and URL

Maybe there should be no invalid datatypes .... everything can be 
converted to a datatype

if the parser thinks a datatype is invalid, well, let's call it an 
invalid! datatype!!

have a catchall for stuff it thinks is wrong

How about another way: integrate datatypes in string-parser. Basically 
a  load/next and check for type.
Then we can write (note i parse a string): 
parse "1 a , #2" [ integer! word! "," issue! ]

'invalite! has a problem: its easy to recognize where the wrong part 
starts, but harder to recognize where the wrong part ends.

Quick question for the parse experts. If I have a parse rule that 
looks like this:

parse spec [
	any [
		set arg string! (...) | set arg tuple! (...) | ...
	]
]

How would I add a rule like:

	set arg paren! (reduce arg)


that always occurred prior to the any rule and evaluated parenthesized 
expressions (i.e. I want parenthesized expressions to be reduced 
to a REBOL value that can be handled by the remainder of the parse 
rule).

Thanks both, works a treat.

My brain is still asleep.  How to go thru a document and add <strong> 
</strong> around every word that is in capitals and is more than 
a few characters long?

hmm, quite a challenge ...

I agree - a bit much to ask.  A more specific question would get 
a more specific answer :)

Something like:

file: read filename2parse
newfile: ""
Foreach word file [
   if Is-Capitals Word [
      newfile: join newfile ["<strong> " word " </strong> "]
]

The Is-Capitals function would have to be defined
Is-Capitals func [Word2Check] [
   some code here
]

that won't work because file is just text and not a block.

;thinking loud:
capitals: charset["#"A" - #"Z"]
capital: [5 capitals any capitals]

Yes, give me a minute...

capitalize-word: func [

    s [string!]
    /local len

][

    either 5 < len: length? s [

        s: rejoin ["<strong>" uppercase s/1 next s </strong>]

    ][

        s

    ]
 
]



capitalize-text: func [
    s [string!]

    /local result word-rule alpha non-alpha w c
][

    result: copy {}
    alpha: charset [#"A" - #"Z" #"a" - #"z"]

    non-alpha: complement alpha

     word-rule: [copy w [some alpha] (insert tail result capitalize-word 
     w)]
    other-rule: [copy c non-alpha (insert tail result c)]

    parse/all s [some [word-rule | other-rule] end]
    result

]

search for a series of capitalised words and strong them

bolden-word: func [
    s [string!]
    /local len
][
    either 5 < len: length? s [
        s: rejoin ["<strong>" s </strong>]
    ][
        s
    ]
 ]

enhance-text: func [
    s [string!]
    /local result word-rule alpha non-alpha w c
][
    result: copy {}
    alpha: charset [#"A" - #"Z"]
    non-alpha: complement alpha

    word-rule: [copy w [some alpha] (insert tail result bolden-word w)]
    other-rule: [copy c non-alpha (insert tail result c)]
    parse/all s [some [word-rule | other-rule] end]
    result
]

capitals: charset ["#"A" - #"Z"]
alpha: charset ["#"A" - #"Z" #"a" - #"z"]
non-alpha: complement alpha
parse/all/case [any non-alpha any [
    a: 5 capitals any capitals b: non-alpha (

        b: change/part a rejoin ["<strong>" copy/part a b "</strong>"] b
    ) :b |
    some alpha any non-alpha
] to end]

; A few fixes
capitals: charset ["#"A" - #"Z"]
alpha: charset ["#"A" - #"Z" #"a" - #"z"]
non-alpha: complement alpha
parse/all/case [any non-alpha any [
    a: 5 capitals any capitals b: [non-alpha | end] (

        b: change/part a rejoin ["<strong>" copy/part a b "</strong>"] b
    ) :b |
    some alpha any non-alpha
] to end]

capitals: charset ["#"A" - #"Z"] ... remove  leading "

Yeah ... it was a way to mark up text wherever a sequence of CAPS 
occurs

; Sorry, more fixes
capitals: charset ["#"A" - #"Z"]
alpha: charset ["#"A" - #"Z" #"a" - #"z"]
non-alpha: complement alpha
parse/all/case [any [

    any non-alpha a: 5 capitals any capitals b: [non-alpha | end] (

        b: change/part a rejoin ["<strong>" copy/part a b "</strong>"] b
    ) :b |
    some alpha
] to end]

Actually I would like to add a parse problem to the weeklyblog and 
get people to submit answers :)

I use parse quite a bit. It's funny, I've never needed the GUI of 
View, but I use parse daily.

And give a prize for the shortest answer

say a copy of Microsoft VB :)

Seriously though, three charsets and two temporary variables, there's 
got to be a more efficient way.

; Sorry, more fixes
capitals: charset [#"A" - #"Z"]
alpha: charset [#"A" - #"Z" #"a" - #"z"]
non-alpha: complement alpha
parse/all/case [any [to alpha [
    a: 5 capitals any capitals b: [non-alpha | end] (

        b: change/part a rejoin ["<strong>" copy/part a b "</strong>"] b
    ) :b |
    some alpha
]] to end]

because " a: 5 capitals any capitals b:" stops at "g" and friends.

The inserts are a nice touch though.

I think that punctuation is part of a word

A person is writing a text file.  It has headings which are denoted 
by caps, and terminating in ":".

Anyway, i have a working version now :)

Since you wrote one, do you know of a better one?  This is not a 
reflection on yours, but it is a great way to know what you considered 
the next best thing.

To use the simpler of the CS terms:


Parse is a rule-based, recursive-descent string and structure parser 
with backtracking. It is not a parser generator (like Lex/Yacc) or 
compiler (like most regex engines) - the engine follows the rules 
directly. Since Parse is recursive-descent it can handle patterns 
that regular expressions wouldn't be able to. Since Parse backtracks 
it can handle patterns that ordinary recursive-descent parsers can't.


Basically, it puts the text and structure processing abilities of 
Perl 5 to shame, let alone those of the lesser regex engines.


In theory, Perl 6 has caught up with REBOL, but Perl 6 only exists 
in theory for now. By the time it becomes actual REBOL should surpass 
it (especially if I have anything to say about it).

It's pretty easy to demonstrate patterns that regular expressions 
can't handle. It's only somewhat difficult to demonstrate patterns 
that can't be handled by a recursive descent parser without backtracking 
or unlimited lookahead.


I have never run into a pattern that can't be handled by Parse in 
theory - its only limits are in implementation (available memory 
and recursion depth). I am not qualified to describe its limits. 
Still, you have to be careful about how you write the rules or they 
will trip you up.

A little dry as explanations go, I suppose. You may get better luck 
by showing some magic parse code tricks :)

Somewhat buzzy: Its a simplified compiler-compiler. Could be used 
to build a java-compiler (eg such complex syntax), but its also as 
easy as regex for simpler things. But still readable. (less buzzy: 
not always that easy due to the poorer lockahead).

Volker, it's more like it can do what a compiler-compiler can do 
without needing to compile :)

And backtracking is about the same as unlimited lookahead, but more 
powerful.

Thanks Brian, but as is the theme with questions I ask, I don't ask 
for myself, but rather that the "world" can learn what "we" know. 
 So perhaps you should add your 2 cents to Henriks, and Tom's in 
a public forum of the Wikibook.

the compiling is no big argument  as compiler-compilers are for compiled 
languages anyway ;) the point is, you can mix a grammar and actions 
for semantics easy.

Volker, it still might be a good point that you can skip a step with 
parse, depending on the listener. Parse is more of a compiler-interpreter 
really. The real point I was making was about the lookahead.

aah. a compiler-compiler produces sourcecode to be compiled, but 
you can interpret data with it.

i guess that depends on the coco. the point is, a bnf by default, 
and code inside therules, instead of putting things in vars andprocess 
later. IMHO.

Jaimie, I meant that parse is itself an interpreter, not a compiler. 
It interprets compiler specs (or interpreter specs, etc.).

Volker, I've used a lot of compiler-compilers before and reviewed 
many more, and unlimited lookup or backtracking are rare.

then the advantages of parse are beeing like a compiler-compiler 
and habving unlimited lookup etc?

I'm not sure whether not having a seperate tokenizer is a plus or 
a minus, though.

I guess you could think of block parsing as using load as a tokenizer.

sounds good. if one finds a good tokenized representation. I am not 
an xml-guru :(

Still, "run away" is a common and sensible reaction to XML.

I'm a bit stuck because this parse stop after the first iteration. 
 Can anyone give me a hint as to why it stops after one line.

Here is some code:

data: read to-file Readfile

print length? data
224921


d: parse/all data [thru QuoteStr copy Note to QuoteStr thru QuoteStr 
thru quotestr

    copy Category to QuoteStr thru QuoteStr thru quotestr copy Flag to 
    QuoteStr
    thru newline (print index? data)]
1
== false


Data contains hundreds of "memos" in a csv file with three fields: 

 Memo, Category and Flag ("0"|"1")  all fileds are enclosed in quotes 
 and separated by commas.
  

It would be real simple if the Memo field didn't contain double quoted 
words; then 
parse data none
would even work; but alas many memos contain other "words".
It would even be simple if the memos didn't contain commas, then
parse data "," or parse/all data ","
would work; but alas many memos contain commas in the body.

Gordon, can you post a copy of short lines of the data?

if QuoteStr = "\"", then this looks like it to me:
Note
, "Category", "Flag" 
Note
, "Category", "Flag"

But you don't have a loop or anything - try this:
d: parse/all data [
   some [

      thru QuoteStr copy Note to QuoteStr thru QuoteStr thru quotestr

      copy Category to QuoteStr thru QuoteStr thru quotestr copy Flag to 
      QuoteStr
      thru newline (print index? data)
   ]
]

Okay, trying it now.  I see that the phrase: "print index? data" 
stays stuck on "1".  


I see that you have posted a new example.  I'll try that.  Be right 
back.

I'm pretty sure that you are right in that I have to loop throught 
the "Data".  That was my big stumbling block and the rest is just 
logic to figure out.  Thanks a bunch.

In the phrase.  "Print index :x", what does putting a colon before 
a variable do again?

This data was exported by PalmOS.  I like the Palm desktop for keeping 
track on notes/,memos addresses but the  search engine sucks badly. 
 Therefore I wanted to export the data to allow a nice Rebol search 
on it..  Therefore, the PalmOS export function does "escape" an embedded 
quote by quoting it again.  Ex:
Press the "Home" button
becomes
Press the 
Home
 button.

truth (as far as i know) is:   word  is is a shortcut for :word  
but there are a few places such as inside parse where the shortcut 
does not work so you need to make it explicit

I will get some troubleshooting data posted in a minute.

Tomc:  Do I understand that :word would be like "get word" except 
in a parse sentence?

Tomc:  Do I understand that :word would be like "get word" and needed 
in a parse sentence but you can just use the shortcut 'word' most 
everywhere else?

The colon before the word prevents the interpreter from evaluating 
active values like functions and parens. It's a safety thing.

Except when you want an active value assigned to the word to be evaluated, 
like when you are calling a function.

Thanks Tomc and BrianH.  I'll chew on it for a while.  Meanwhile 
I'm working on building some test data for the first problem.

okay so in the parse rules (except in a parenthesized code block) 
it means "be here now"?

That's interesting. Parens and paths used to be active - oh yeah, 
that was changeda while ago. Still, there are some value types that 
are active (function types, lit-path, lit-word) and if you think 
you will get one of these you should disable their evaluation by 
referencing them with a get-word, unless you intend them to be evaluated.

Gordon: I did not read this thread in a whole but as for converting 
CSV string to/from Rebol blocks, here is some fully functionnal functions 
:

;***** Conversion function from/to CSV format
csv-to-block: func [

 "Convert a string of CSV formated data to a Rebol block. First line 
 is header."
	csv-data [string!] "CSV data."

 /separator separ [char!] "Separator to use if different of comma 
 (,)."
	/without-header "Do not include header in the result."

 /local out line start end this-string header record value data chars 
 spaces chars-but-space

 ; CSV format information http://www.creativyst.com/Doc/Articles/CSV/CSV01.htm
] [
	out: copy []
	separ: any [separ #","]
	

 ; This function handle replacement of dual double-quote by quote 
 while copying substring
	this-string: func [s e] [replace/all copy/part s e {""} {"}]
	; CSV parsing rules

 header: [(line: copy []) value any [separ value] (if not without-header 
 [append/only out line])]

 record: [(line: copy []) value any [separ value] (append/only out 
 line)]

 value: [any spaces data any spaces (append line this-string start 
 end)]

 data: [start: some chars-but-space end: | #"^"" start: any [some 
 chars | {""} | #"," | newline] end: #"^""]
	chars: complement charset rejoin [ {"} separ newline]
	spaces: charset exclude { ^-} form separ
	chars-but-space: exclude chars spaces
	
	parse/all csv-data [header any [newline record] any newline end]
	out
]

block-to-csv: func [
	"Convert a block of blocks to a CSV formated string." 
	blk-data [block!] "block of data to convert"
	/separator separ "Separator to use if different of comma (,)."
	/local out csv-string record value v
] [
	out: copy ""
	separ: any [separ #","]
	; This function convert a string to a CSV formated one

 csv-string: func [val] [head insert next copy {""} replace/all copy 
 val {"} {""} ]
	record: [into [some [value (append out #",")]]]

 value: [set v string! (append out csv-string v) | set v any-type! 
 (append out form v)]
	

 parse/all blk-data [any [record (remove back tail out append out 
 newline)]]
	out
]

DideC: Thanks.  I've copied and pasted it for review and added it 
to my local public library.  This script should be useful especially 
with the html help page.   Documentation on a script is very rare 
and much appreciated.


Graham: Did a search using  "librarian" and search term of  "sql 
cvs" and didn't come up with anything.  Although, I think we've got 
it covered now anyhow.

What I was trying to do above is to look for the macro text preceded 
by a space or newline, and ending in a space or newline.

Heart: Heart regular rate and rhythm, no rubs, murmurs, or gallops 
noted. 

A: Abdomen:  soft, nontender, no mass, no hernia, no guarding, no 
rebound tenderness, no ascites, non obese 
Hbp Hypertension (high blood pressure) #401.9.  
Ii Type II Diabetes #250.00

So, someone might type

heart:
A: with striae

That A: isn't delimited by whitespace.

it is .. it's preceded by a newline character

so, a: is a macro, whereas "a" is not.

isn't there a controled vocabulary for this sort of thing

Is a macro always the first word in a line?

no, it can be anywhere in a line.

Is there a seperate syntax for defining macros?

no, it's just a text file which is read in at start up.

So in use, a macro a: will always be a: in the text. Will it be A: 
sometimes, or "a:"?

so, personal shorthand should expand into a controlled vocab ideally

Ii Type II Diabetes #250.00
here the macro expansion includes a code (CPT) from the AMA.