World: r3wp

its happened often yes.  less lately, since I'm dealing more with 
XML and less with raw data.

more like:

parse load/all "hello, world!" [2 word!]

I have wondered sometimes, what effects it would have, if such commas 
was just ignored. We need commas in numbers, but maybe commas could 
just be ignored beside that.

So do you suggest, load/all "hello, world!" should return [hello 
world!] ? (Notice no comma.)

And without space, comma should maybe split the text? Like:
>> load/all "hello,world!"
== [hello world!]

yes, I always thought that commas should be removed of decimals, 
and simply ignored when loaded.


in mechanical data, commas are never used for decimals.  because 
apps need to load it back and all software accept that dots are for 
decimals and commas for separating lists.   why should REBOL try 
to be different, its just alienating itself from all the data it 
could gobble up effortlessly.

so a comma would be an exact alias for a space, when its not within 
a string.

I almost agree. Here we use comma as decimal point. A few countries 
does that. So all data with money amounts have numbers with comma 
as decimal point here.

But it should be possible to take care of those numbers with commas, 
and ignore all other commas, I think. As we don't ever write
42,
but always something like
42,00

if it's a decimal. So if 42, is seen, it can just be read as integer 
42 and ignore the comma (if using load/all for example).

this is exactly the reason why CSV was it a really fucked up idea. 
comas are there in sentences and multivalued fields, not just numbers.
i always use TSV.

it would make sense to settle w some CSV parser, but not as a default 
behaviour. i was already surprised that parse handles double quotes 
too...

>> parse/all {"asd qwe" zxc} none
== ["asd qwe" " zxc"]

>> parse/all {"asd qwe" zxc} " " 
== ["asd qwe" "zxc"]


it's nice, but it also means there is no plain "split-by-a-character" 
function in rebol, which is just as annoying as missing a join-by-a-character

Although gerneral happy with the default parse seperators find it 
neglegent to not permit overriding them.  and like Max finds, block 
parsing ia a rarity when working with real world data streams.

parse/all string none actually is a CSV loader.  its not a split 
functions.   I always found this dumb, but its the way Carl implemented 
it.

rule, when given as a string is used to specify the CSV separator.

it should also honor line breaks within strings then

eh, didn't know it didn't ! yeah that sucks.

Question on string and block parsing:
   http://stackoverflow.com/questions/6392533

only the second string is checked.
Should be:
['apple some [and string! into ["a" some "b" ]]]

can't post the response

Want me to post it for you?

yep ;-)

Done, thanks.

Parse (YC S11): A Heroku For Mobile Apps.
Great name for a startup...

http://techcrunch.com/2011/08/04/yc-funded-parse-a-heroku-for-mobile-apps/

Can anyone gift me an effecient R2 'parse solution for this problem 
(I am assuming 'parse will out-perform any other approach):

SET UP

I have a huge list of HTML named character entities, eg (a very short 
example):

       named-entities: ["nbsp" "cent" "agrave" "larr" "rarr" "crarr" ] ;; 
       etc
   
And I have some text that may contain some named entities, eg:

       text: "To send, press the ← arrow & then press ↵."
   
PROBLEM

I want to escape every "&" in the text, unless it is part of a named 
entity, eg (assuming a function called escape-amps):
        probe escape-amps text entities

         == "To send, press the ← arrow & then press ↵."
  
TO MAKE IT EASY....

You can can assume a different set up for the named-entities block 
if you want; eg, this may be better for you:

       named-entities: [" " "¢" "à" "←" "→" "↵" 
       ] ;; etc 
   
Any help on this would be much appreciated!

ne: ["←" | "↵"]	; and the rest of the named entities
s: "To send, press the ← arrow & then press ↵."
parse s [
	any [
		to #"&" [ne | skip mark: (insert mark "amp;")]
	]
]
s

== {To send, press the ← arrow & then press ↵.}

It may be faster to drop the & from the entities and change the rule 
to:

any [thru #"&" [ne | mark: (insert mark "amp;")]

That's strange. My 2nd suggestion gives a different result:

ne: ["larr;" | "crarr;"]
s: "To send, press the ← arrow & then press ↵."
parse s [
	any [
		thru #"&" [ne | mark: (insert mark "amp;")]
	]
]
s

== {To send, press the ← arrow & amp;then press ↵.}

Seems like a bug, or am I just tired?

Thanks for the quick contributions, geomol.

I see a different result too -- a space between the "&" and the "amp"

not fluent with html escaping, what's the aim? To replace stand-alone 
#"&" with "&amp"?

also remember - parse does not count spaces in. You are better in 
using parse/all

'I want to escape every "&" in the text, unless it is part of a named 
entity' - just to make sure: if the entity is not in the ENTITIES 
list, like e.g. " and it is encountered in the given TEXT, what 
exactly should happen?

The aim --- Basically, yes, Petr.

Ladislav -- if it is not in the list, then I'd like it escaped, please.

Think of it as a whitelist of ecceptable named entities. All others 
are suspect :)

ecceptable ==> acceptable

Yes, OK, I just wanted to know

Geomol - your code basically works, no? Just use parse/all:


>> parse/all s [any [thru #"&" [ne | mark: (insert mark "amp;")]]]
== false
>> s
== {To send, press the ← arrow & then press ↵.}

I guess, that this should be efficient:

alpha: make bitset! [#"a" - #"z" #"A" - #"Z"]
escape-amps: func [
	text [string!]
	entities [hash!]
	/local result pos1 pos2
][
	result: copy ""
	parse/all text [
		pos1:
		any [
			; find the next amp
			thru #"&"
			pos2:
			[
				; entity check
				some alpha pos3: #";" (
					; entity candidate
					unless find entities copy/part pos2 pos3 [
						; not an entity
						insert insert tail result copy/part pos1 pos2 "amp;"
						pos1: pos2
					]
				)
				| (
					; not an entity
					insert insert tail result copy/part pos1 pos2 "amp;"
					pos1: pos2
				)
			]
			| (insert tail result pos1) end skip ; no amp found
		]
	]
	result
]

(in place inserts are too slow)

(= inefficient)

Err: pos3 should be added as a local

This is how it works:

>> probe escape-amps text named-entities

{To send, press the ← arrow & then press ↵.&susp;123}

== {To send, press the ← arrow & then press ↵.&susp;123}

With TEXT defined:


>> text: "To send, press the ← arrow & then press ↵.&susp;123"

Pekr, yeah, probably because I left out the /all refinement. Makes 
sense.

Thanks Ladislav and Geomol.

Both your solutions work with my test data -- that's always a good 
sign :)


I'll do some timing tests with large entity lists ..... But I won't 
be able to do that for 24 hours.

Other approaches still welcome!

Two suggestions:


- store your named entities as a hash! (order of magnitude speedup 
for FIND)


- if you have loooong "words", restrict Ladislav's `some alpha` to 
the maximum length of a valid entity

This alternative does not use the COPY call, so, it has to be faster:

alpha: make bitset! [#"a" - #"z" #"A" - #"Z"]
escape-amps: func [
	text [string!]
	entities [hash!]
	/local result pos1 pos2 pos3
][
	result: copy ""
	parse/all text [
		pos1:
		any [
			; find the next amp
			thru #"&"
			pos2:
			[
				; entity check
				some alpha pos3: #";" (
					; entity candidate
					unless find entities copy/part pos2 pos3 [
						; not an entity
						insert insert/part tail result pos1 pos2 "amp;"
						pos1: pos2
					]
				)
				| (
					; not an entity
					insert insert/part tail result pos1 pos2 "amp;"
					pos1: pos2
				)
			]
			| (insert tail result pos1) end skip ; no amp found
		]
	]
	result
]

Perhaps building a parse rule from the list of entities may be faster 
if there is a lot of text to process:

This assumes the entities are provided as strings in a block.

escape-amps: func [

  text [string!]

  entities [block!]

][
  
  skip-it: complement charset [#"&"]

  entity: copy []

  foreach ent entities [ insert entity compose [(ent) |]]

  head remove back tail entity

  parse/all text 
[
    any [

      entity |

      "&" pos: (insert pos "amp;" pos: skip pos 4) :pos |

      some skip-it
     ]

  ]

  head tex
t
]

That should read head text at the end of the function.

Also I feel using skip could be very slow if the text contains a 
lot of "non-matching text". The "skip-it" technique could also be 
applied to Ladislav's code.