Asp Forum - Stripping unwanted html

Wild Al

10/6/2006 2:14:00 AM

Hi everyone:

I'm trying to strip html with the exception of a few html tags.

I have found the following code:

def strip_tags(html)
if html.index("<")
text = ""
tokenizer = HTML::Tokenizer.new(html)

while token = tokenizer.next
node = HTML::Node.parse(nil, 0, 0, token, false)
# result is only the content of any Text nodes
text << node.to_s if node.class == HTML::Text
end
# strip any comments, and if they have a newline at the end (ie.
line with
# only a comment) strip that too
text.gsub(/[\n]?/m, "")
else
html # already plain text
end
end

I'm trying to understand what is going on in this code but cannot find
documenation for HTML::Tokenizer or HTML::Node.parse. Does anyone know
the use of the parameters in the parse method?

In the while loop, how do you access the html tag. If I could access
the html tags, I could then decide if I wanted to keep the tag or not.

Thanks for reading,
Wild Al

--
Posted via http://www.ruby-....

6 Answers

Michael Moen

10/6/2006 3:39:00 AM

Wild Al wrote:
> Hi everyone:
>
> I'm trying to strip html with the exception of a few html tags.
>
> I have found the following code:
>
> def strip_tags(html)
> if html.index("<")
> text = ""
> tokenizer = HTML::Tokenizer.new(html)
>
> while token = tokenizer.next
> node = HTML::Node.parse(nil, 0, 0, token, false)
> # result is only the content of any Text nodes
> text << node.to_s if node.class == HTML::Text
> end
> # strip any comments, and if they have a newline at the end (ie.
> line with
> # only a comment) strip that too
> text.gsub(/[\n]?/m, "")
> else
> html # already plain text
> end
> end
>
> I'm trying to understand what is going on in this code but cannot find
> documenation for HTML::Tokenizer or HTML::Node.parse. Does anyone know
> the use of the parameters in the parse method?
>
> In the while loop, how do you access the html tag. If I could access
> the html tags, I could then decide if I wanted to keep the tag or not.
>
> Thanks for reading,
> Wild Al

Al- I recently needed a parser similar to Perl's HTML::Scrubber. <a
href="http://www.underpantsgnome.com/2006/09/09/using-hpricot-to-scrub-html/">This...
is what I came up with , you may find it useful.

Michael

--
Posted via http://www.ruby-....

Paul Lutus

10/6/2006 5:49:00 AM

Wild Al wrote:

/ ...

> In the while loop, how do you access the html tag. If I could access
> the html tags, I could then decide if I wanted to keep the tag or not.

Why not create your own method? This should give you some ideas:

#!/usr/bin/ruby -w

data = File.read("/path/page.html")

data.scan(/<(\w+?)>/) { |tag|
puts tag
}

This puts out the tag name extracted from each HTML tag in the page.

--
Paul Lutus
http://www.ara...

eden li

10/6/2006 9:20:00 AM

Hey, that's RoR's strip_tags method :)

I ran into the same issue so went down your path and instead of
fretting about no docs, just hacked my way through using IRB.

Here's a modified version of what I came up with, maybe you'll find it
useful?

def strip_tags_except(html, exceptions = [])
if html.index("<")
text = ""
tokenizer = HTML::Tokenizer.new(html)
while token = tokenizer.next
case node = HTML::Node.parse(nil, 0, 0, token, false)
when HTML::Tag
text << node.to_s if exceptions.include?(node.name)
when HTML::Text
text << node.to_s
end
end
text
else
html
end
end

The one I had also stripped attributes and closed up dangling tags if
it found any. Have a look at RoR's strip_links for more examples of
HTML::Node/HTML::Tokenizer usage.

Wild Al wrote:
> Hi everyone:
>
> I'm trying to strip html with the exception of a few html tags.
>
> I have found the following code:
>
> def strip_tags(html)
> if html.index("<")
> text = ""
> tokenizer = HTML::Tokenizer.new(html)
>
> while token = tokenizer.next
> node = HTML::Node.parse(nil, 0, 0, token, false)
> # result is only the content of any Text nodes
> text << node.to_s if node.class == HTML::Text
> end
> # strip any comments, and if they have a newline at the end (ie.
> line with
> # only a comment) strip that too
> text.gsub(/[\n]?/m, "")
> else
> html # already plain text
> end
> end
>
> I'm trying to understand what is going on in this code but cannot find
> documenation for HTML::Tokenizer or HTML::Node.parse. Does anyone know
> the use of the parameters in the parse method?
>
> In the while loop, how do you access the html tag. If I could access
> the html tags, I could then decide if I wanted to keep the tag or not.
>
> Thanks for reading,
> Wild Al
>
> --
> Posted via http://www.ruby-....

eden li

10/6/2006 9:24:00 AM

eden wrote:
> Have a look at RoR's strip_links for more examples of
> HTML::Node/HTML::Tokenizer usage.

Sorry, I meant sanitize:
http://api.rubyonrails.com/classes/ActionView/Helpers/TextHelper.ht...

Wild Al

10/8/2006 7:54:00 PM

eden wrote:
>
> Here's a modified version of what I came up with, maybe you'll find it
> useful?
>
> def strip_tags_except(html, exceptions = [])
> if html.index("<")
> text = ""
> tokenizer = HTML::Tokenizer.new(html)
> while token = tokenizer.next
> case node = HTML::Node.parse(nil, 0, 0, token, false)
> when HTML::Tag
> text << node.to_s if exceptions.include?(node.name)
> when HTML::Text
> text << node.to_s
> end
> end
> text
> else
> html
> end
> end

I found this method very useful; it is exactly what I needed. Thanks.
To all others: your suggestions helped too, especially in understanding
ruby. Thanks again...

--
Posted via http://www.ruby-....

eden li

10/9/2006 5:36:00 AM

Glad I could help. One security-related caveat.

The method I posted doesn't strip attributes, so it may be possible for
someone to "hack" your site by putting javascript onto one of the
allowed tags.

You can fix that by changing the last line of the first branch of the
if statement from "text" to "sanitize(text)", eg:

if html.index("<")
...
sanitize(text)
else
...

Wild Al wrote:
> I found this method very useful; it is exactly what I needed. Thanks.
> To all others: your suggestions helped too, especially in understanding
> ruby. Thanks again...

comp.lang.ruby

Stripping unwanted html

Wild Al

Michael Moen

Paul Lutus

eden li

eden li

Wild Al

eden li

x Login to ForumsZone