Mailing List Archive: 49091 messages
  • Home
  • Script library
  • AltME Archive
  • Mailing list
  • Articles Index
  • Site search
 

How to properly parse HTML and XHTML Meta Tags

 [1/7] from: vonja:sbcglobal at: 11-Sep-2008 20:00


Hello Rebol Group, I'm a bit new, I have a couple of the Rebol books and have gone over the different tutorial a few times but I'm having trouble with the following code of mine. For example: I'm attempting to parse the meta tags but the tag can end in either
or "/>" I've tried to write the below script a different way, over 50 times, but to no avail. I don't know how to properly code it where it will check for either ending tag ">" or "/>" sample meta tag: <meta name="description" content="Having trouble with this below script" /> The end result should look like: Having trouble with this below script -not- Having trouble with this below script / If I change the script from ">" to "/>" and the meta tag is <meta name="description" content="Having trouble with this below script"> Then the script will not catch the ">" since it's looking for "/>" REBOL CODE: page: read http://www.rebol.com ; webpage to be parsed title: [] description: [] keywords: [] parse page [ thru <title> copy title to </title>] parse page [ thru "<meta name=^"keywords^" content=" copy keywords to
] parse page [ thru "<meta name=^"description^" content=" copy description to ">" ] print title print description print keywords Thank you in advance for your assistance. Regards, Von

 [2/7] from: Tom:Conlin::gmail at: 11-Sep-2008 23:32


vonja-sbcglobal.net wrote:
> Hello Rebol Group, > I'm a bit new, I have a couple of the Rebol books and have gone
<<quoted lines omitted: 22>>
> ">" ] > title: copy ""
description: copy [] keywords: copy []
> print title > print description > print keywords > > Thank you in advance for your assistance. > > Regards, > Von >
Hi Von welcome, note 1: when you initialize words with empty strings or blocks you *do* want to copy the empty string or block. \ (otherwise they can be the *same* empty block or string) title: copy "" description: copy [] keywords: copy [] note 2: when using parse for more than simple string splitting get use to using the /all refinement and handling white space yourself. you could define a class of chars that are not "/>" then copy some of them. downside is you would have to check if a "/" you ran into was followed by ">" and if not concatenate and continue. this code untested and un-run tag-end: charset "/>" content: complement tag-end ... parse page [ ... thru "<meta name=^"keywords^" content=" some[ copy token some content here: ;;; make a pointer to where parse is (append keywords token all[#"/" == first :here #">" != second :here append keywords "/" here: next :here ;;; move parse pointer over "/" ]) :here ;;; set where pars will resume ] thru ">" ... ] ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; you could detect closing angle and see of the proceeding char is a slash and if so remove it from the copied string. note: this is running parse once not multiple times using braces for string that contain double quotes and taking the destination for the content copied from the meta name=<dest> i.e keyword or description block... parse page [ thru <head> some[ thru {<META NAME="} copy dest to {"} {"} thru {content=} copy token to ">" here: thru ">" (if #"/" = first back :here [trim/with token "/"] append get to-word dest token ) ] <title> copy title to </title> tag! ] print title print description print keywords ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; but ultimately I would probably start with blk: load/markup <source> which would return a block of string! and tag! then process the tags; if I used parse I would end with the rule like [{<META NAME="} ... ["/>" | ">"]] note: this won't work with the page: read <source> because there may be a "/>" beyond the first ">" that closes the meta tag but with load/markup each tag and string element is isolated hope that helps

 [3/7] from: vonja:sbcglobal at: 12-Sep-2008 0:06


Thanks Tom, I kept on plugging away and came up with I believe a working script. It's going to take some time for me to digest what you've written me. I'll play around with yours tomorrow; I really appreciate your help! I've updated note 1 that you had provided me :-) Here's what I came up with right before you sent your reply. page: read http://www.rebol.com ; webpage to be parsed title: copy "" description: copy [] keywords: copy [] parse page [ thru <title> copy title to </title>] print title parse page [ thru "<meta name=^"keywords^" content=" copy keywords to
] either not none? (find/last keywords "/") [ keywords: tail keywords keywords-tail: skip keywords -1 if keywords-tail = "/" [keywords: remove keywords-tail] print head keywords ][if/else empty? keywords [print "blank"][print keywords]] parse page [ thru "<meta name=^"description^" content=" copy description to ">" ] either not none? (find/last description "/") [ description: tail description description-tail: skip description -1 if description-tail = "/" [description: remove description-tail] print head description ][if/else empty? description [print "blank"][print description]]

 [4/7] from: christian::ensel::gmx::de at: 12-Sep-2008 9:42


Hi Von, in your special case, it doesn't seem to be necessary to go thru the > or /> hassle, if you rely on " as a delimiter. But keep in mind that in many, many cases the solution below as well as yours will fail. E.g. in cases where the content and name attributes are given in reverse order, which is valid HTML, too. However, have a look at the following PARSE-METATAGS. HTH, Christian ------------------------------------------------------------------------ parse-metatags: func [page [url!] /local title keywords description] [ page: read http://www.rebol.com parse page [thru <title> copy title to </title>] parse/all page [thru {<meta name="keywords" content="} copy keywords to {"}] parse/all page [thru {<meta name="description" content="} copy description to {"}] foreach keyword keywords: parse/all any [keywords ""] "," [trim keyword] reduce [ 'title title 'keywords keywords 'description description ] ]
>> parse-metatags http://www.rebol.com
== [ title "REBOL Technologies" keywords ["REBOL" "Web 3.0" "Web 2.0" "programming" "Internet" software "domain specific language" "di stributed computing" "collaboration" "operating systems" "development" rebel ] description {REBOL: a Web 3.0 language and system based on new lightweight computing methods. Site inclu des products, downloads, documentation, and support.} ] vonja-sbcglobal.net schrieb:

 [5/7] from: vonja::sbcglobal::net at: 12-Sep-2008 10:17


Hi Christian, Hmmm, both are very good points. Is PARSE-METATAGS in a different scripting language? Unable to find it in the Rebol dictionary or Rebol.org library. Thank you for your response. --Von

 [6/7] from: christian:ensel:gmx at: 12-Sep-2008 19:32


The source should have been right there below the signature. Anyway, I'll cite it again (and it's definitely REBOL ;-) ----- parse-metatags: func [page [url!] /local title keywords description] [ page: read http://www.rebol.com parse page [thru <title> copy title to </title>] parse/all page [thru {<meta name="keywords" content="} copy keywords to {"}] parse/all page [thru {<meta name="description" content="} copy description to {"}] foreach keyword keywords: parse/all any [keywords ""] "," [trim keyword] reduce [ 'title title 'keywords keywords 'description description ] ] ----- Beware of unintentional line breaks in the code above due to e-mail transportation. HTH, Christian vonja-sbcglobal.net schrieb:

 [7/7] from: vonja::sbcglobal::net at: 12-Sep-2008 11:43


Thank you Christian, much more elegant and I like the use of the double quote rather than looking for the "/>" or the ">" You got me thinking about valid HTML and thought I should also check for a single quote too. Hopefully, I'll be smart enough to figure it out ;-)

Notes
  • Quoted lines have been omitted from some messages.
    View the message alone to see the lines that have been omitted