r3wp [groups: 83 posts: 189283]
  • Home
  • Script library
  • AltME Archive
  • Mailing list
  • Articles Index
  • Site search

World: r3wp

[Power Mezz] Discussions of the Power Mezz

Again... it's easier to do manual changes to parse malformed pages 
if you do string parsing where you don't care about 99% of the page 
We did a simple address list as an example, even preconverting it 
to plain text. It took all afternoon to discover all exceptions and 
fix them, so in most cases, it isn't worth it
I've done quite a few html analysers and with a bit of experience 
I have found that going with a brute force parse on pages ends-up 
being very effective.  I did a font downloading script for www.acidfonts.com 
a few years ago and it took more time to actually download all the 
pages than build the script.  :-)
some servers are anti-indexing and its in these cases where the brute 
parse is most effective.  I've even had to cater an oracle web server 
which didn't have ANY css, type, or id fields in all pages which 
are driven by form.  all urls can only be read once, and every page 
read is a redirect.   

only parse allowed me to cope in such a drastic anti-robot  environment. 
 it still took a day to build the robot.  and in the end, it even 
had an auto-user-creationg step at each 200 pages which created a 
google gmail account for the next batch.  :-)

in those cases, parse is king.
a fellow REBOL parse data miner tells me that some servers have pretty 
good algorithms which identify robots out of the thousands of requests 
they get, and you even have to put random-lenght pauses between reads 
which can go up to a few hours.
I don't see my partner doing that :-)
that why he has you  ;-)
I'm not going to do it for him, either, thank you very much
So if he can get by with an HTML parser, that would be great for 
About examples: if you download the repository, or the source archive, 
you'll find a test directory which has the test suite for (almost) 
all the modules. the tests are often usable as "examples" for how 
to use the modules themselves.
About the HTML parser:

This started out because we had a need for a HTML filter. The mail 
client in Qtask allows users to view HTML emails, furthermore, the 
wiki editor submits HTML to Qtask. Whenever you are embedding HTML 
from an external (untrusted) source within your HTML, you have security 
problems. For this reason, we had to take the untrusted HTML and 
1) "filter" it so that it would be safe 2) make it embeddable (eg. 
only take what's inside <body>, and not the whole <html> etc.).
This *had* to work with *any* HTML (think about the stuff that comes 
out of Outlook, or the stuff you get as spam, or newsletters, and 
all that). You can't imagine how bad that can be. That had to be 
turned into something that would not break our own HTML pages.
My approach was, instead of doing what many others do (try to remove 
things from the HTML that are known to be "bad", eg. use regexps 
to remove anything that starts with "javascript:" or anything between 
<script>...</script> etc.), was to only pass what was known to be 
good, and ignore everything else. This is a bit more limiting but 
I consider it to be safer (you don't have to play a game with attacker 
where every time they find a new vector, you have to add it to the 
"bad" list).
So the first HTML filter was done. It parsed the HTML (any version), 
and going through two finite state machines in a pipeline, rebuilt 
it as XHTML 1.0 strict.
This method avoided keeping any intermediate representations in memory. 
However, because of that there were a number of things it could not 
do (eg. no look ahead, and you get an explosion of the number of 
states if you want to "look behind" more).
So, as our needs became more complex (esp. because of the initial, 
never released version of the wiki editor), I had to change approach. 
Also, at that time Maarten was doing the S3 stuff and needed a XML 
parser as well.

So, first, the Filter was split up into three modules. The first 
is the parser, that takes a HTML or XML string and just sends "events" 
to a callback function. This can be used basically for anything. 
(Maarten never used it in the end.) The second part was the first 
FSM, the HTML normalizer. You'll still find it within the Power Mezz, 
but it's deprecated. The third part was the actual filter and regenerator 
(second FSM). You can find it in the repository history.
Then, the latter two modules were replaced by a different approach 
that builds a tree of blocks from the HTML and rewrites it as it 
is being built (to avoid doing too many passes). This is done by 
LOAD-HTML, that allows passing a set of rules used for filtering 
(so the actual filter is now a bunch of rules for LOAD-HTML). LOAD-HTML 
handles a lot of HTML weird cases, it's probably not at the level 
of a web browser, but it comes close.
The tree is being built with the Niwashi module, which was separated 
as a generic way to build trees incrementally following rules etc. 
(Niwashi means gardener in Japanese)
The HTML to text module has still not been rewritten to use LOAD-HTML 
instead of the older approach of the HTML normalizer followed by 
a FSM.
Thanks for the clarification
So the parser in 5.10 is the newest one? But where does the parser 
in 7.4 fit in?
7.4 parses a string into a sequence of tags and text (etc.). (it 
also has a load-markup function that is similar to load/markup but 
also parses tag attributes and so on). 5.10 uses 7.4 and builds a 
tree from that sequence of tags and text.
(i never got around to change wetan to show module dependencies. 
if you look at the script header though, you'll see that load-html.r 
depends on ml-parser.r)
the code is using a number of tricks to be "fast" (esp. expand-macros.r), 
so it's not as clean as it could be.
Hi, first thanks for making and open sourcing power-mezz. 

I am trying to use load-html and am getting some strange results 
if sems it makes for example recursing [ html [ html [ html  ..... 
]]] on my simple html input (and on real one that I tried). I prepared 
two examples to make the point as clear as possible.

http://paste.factorcode.org/paste?id=2263(notice the stack owerflow 
I added 2 more cases to the paste (2 annotaitons). load-html seems 
quite complex since it uses many other modules (that I don't understant 
either).. so I rather see if you find something obvious in my approach 
or the bug in power-mezz
First: you only need to import %mezz/load-html.r in your examples. 
You're not using the other modules; they will be loaded automatically 
by load-html.r - you never need to worry about dependencies.
Second: your problem is that you are trying to mold the result, which 
is a tree where each node has a reference to the parent node. (much 
like faces in R2). That's why you see the "loop".
there is a mold-tree function in %mezz/trees.r if you want to mold 
the tree. Or, you could simply use form-html to pretty print the 
tree for you.
Eg. for your first example:

t: load-html p
print mold-tree t

[root [] [html [] [head [] [title [] [text [value "t"]]]] [body [] 
[h2 [] [text [value "HEADING"]]] [p [] [text [value "first para"]]] 
[p [] [text [value "second para"]]]]]]

print form-html/with t [pretty?: yes]

        <p>first para</p>
        <p>second para</p>
(the pretty? option to form-html is something i only use for debugging, 
so it's not as pretty as it should be i guess)
You can also do things like:

>> mold-tree get-node t/childs/html/childs/head/childs/title
== {[title [] [text [value "t"]]]}
get-node and set-node are also from %mezz/trees.r ; most likely you 
don't want to mess around with %mezz/macros/trees.r , that is deep 
vodoo i use to make the html filter fast.
(if you have performance problems, we'll talk about it :)
other examples:

>> get-node t/childs/html/childs/head/childs/title/childs/text/prop/value 
== "t"

>> get-node t/childs/html/childs/body/childs/h2/childs/text/prop/value 
Also note that:

>> print form-html/with load-html "<p>A paragraph!" [pretty?: yes]
        <p>A paragraph!</p>
ie. load-html tries to cope with malformed input as much as possible.
wow, thank you a lot! I knew this was to obvious "bug" to be real 
and I am probably doing something wrong. GREAT!

I initially imported only needed modules but got errors .. ( I will 
try and report ) the errors went away as I manually imported them. 
Just a second
very good that you cope with bad html .. I will need that functionality 
because no html is perfect.
I was planing to use beaurtifullsoup if you didn't but since you 
do that is even much better
I tried now, the problem with import was that I didn't set the absolute 
path to load-module/from before.
It all works now according to your example.. and I tested and it 
handles improper html very well! Thanks!
I looked at html-rules in load-html and I am stunned by how well 
the code / dialect is
there are a few things that it still can't do (the dialect i mean), 
but it's very powerful on the things it can do :) The documentation 
is here: http://www.rebol.it/power-mezz/mezz/niwashi.html
it's one of the parts that i think it's best documented, so it's 
worth reading.
Gabriele - what is that? New templating system? Reminds me of Temple 
:-) (sorry, not following the discussion)
No, though it could easily be used to reimplement Temple, this time 
with the ability to load any html.