[lnkForumImage]
TotalShareware - Download Free Software
Confronta i prezzi di migliaia di prodotti.
Asp Forum
 Home | Login | Register | Search 


 

Forums >

comp.lang.ruby

Re: Help with HTML parsing

Dan Diebolt

1/1/2009 10:38:00 PM

[Note: parts of this message were removed to make it a legal post.]

open-uri gives you access to http/https/ftp while Hpricot is a html parser. So for your intended application you would potentially use both - open-uri would fetch the html document and whereas Hpricot would do the parsing using xpath/css selectors etc. open-uri is used when you are loading your html from the web url. The open-uri part happens in the blink of an eye and then you are off manipulating your hpricot doc:
require 'open-uri'
doc = open("http://qwantz....) { |f| Hpricot(f) }
9 Answers

Vivek Netha

1/2/2009 10:26:00 PM

0

ok, i've started to work with hpricot. i'm debating if i should use
open-uri or watir. but thats another discussion. my current issue is,
though, with xpath. this is what i have so far...

#!ruby
require 'watir'
require 'open-uri'
require 'rubygems'
require 'hpricot'


Watir.options_file = 'c:/ruby/options.yml'
Watir::Browser.default = 'ie'

test_site = "http://www.google...

br = Watir::Browser.new

br.goto test_site
br.text_field(:name, 'q').set("Dungeons & Dragons")
br.button(:name, 'btnG').click

doc = Hpricot(br.html)

#after the above, i'm trying to store all result elements in an array

x = Array.new
x << doc.search("//li[@class='g w0']/h3/a")

the XPath checker in firefox shows that I have the right path. but the
command doesn't work in irb. how do i drill down to specific elements so
that i can extract the title text and url? and wats wrong with my
xpath??

plz help!


Viv.
--
Posted via http://www.ruby-....

Phlip

1/2/2009 11:16:00 PM

0

Vivek Netha wrote:

> x = Array.new
> x << doc.search("//li[@class='g w0']/h3/a")
>
> the XPath checker in firefox shows that I have the right path. but the
> command doesn't work in irb. how do i drill down to specific elements so
> that i can extract the title text and url? and wats wrong with my
> xpath??

Don't use Hpricot - its XPath support only covers a very few predicate and path
types. Use libxml-ruby, if you can install it, or REXML, if you don't mind the
slow speed (the RE stands for Regular Expressions!), or nokogiri, which I don't
know how to recommend yet, but I will soon.

BTW Google for my street-name and XPath for all kinds of fun in Ruby with them.

--
Phlip

Vivek Netha

1/7/2009 10:30:00 PM

0

Hi Philip,

could you give me more pointers on how exactly you would do it using
REXML. with actual code, if possible.

thx.

Phlip wrote:
> Vivek Netha wrote:
>
>> x = Array.new
>> x << doc.search("//li[@class='g w0']/h3/a")
>>
>> the XPath checker in firefox shows that I have the right path. but the
>> command doesn't work in irb. how do i drill down to specific elements so
>> that i can extract the title text and url? and wats wrong with my
>> xpath??
>
> Don't use Hpricot - its XPath support only covers a very few predicate
> and path
> types. Use libxml-ruby, if you can install it, or REXML, if you don't
> mind the
> slow speed (the RE stands for Regular Expressions!), or nokogiri, which
> I don't
> know how to recommend yet, but I will soon.
>
> BTW Google for my street-name and XPath for all kinds of fun in Ruby
> with them.

--
Posted via http://www.ruby-....

Phlip

1/8/2009 4:44:00 AM

0

Vivek Netha wrote:

> could you give me more pointers on how exactly you would do it using
> REXML. with actual code, if possible.

>>> x = Array.new
>>> x << doc.search("//li[@class='g w0']/h3/a")

Sure, but this is just tutorial-level REXML (hint hint), and uncompiled:

require 'rexml/document'
doc = REXML::Document.new(my_xml_string)
x << REXML::XPath.first(doc, '//li[ @class = "g w0" ]/h3/a').text

Now we come to the very nub of the gist. A @class of "g w0" is the same as "w0
g", or any other permutation, yet XPath is literal-minded, and unaware of CSS,
so it will only match one of those permutations, when any other could have been
just as significant.

You could try contains(@class, "w0"), but that would match "w0p0p", which is a
different class.

This is why nokogiri is interesting - it might do CSS Selector notation, which
is less exact than XPath, and more aware of CSS rules. (Look up assert_select()
to see what I mean.)

For further REXML abuse, try this...

http://www.google.com/codesearch?q=REXML%3A%3AX...

....but be warned (again, I think), it's _almost_ as slow as Windows Vista with
more than one program running, so /caveat emptor/.

--
Phlip

Tom Morris

1/10/2009 4:55:00 AM

0

On 2009-01-08, Phlip <phlip2005@gmail.com> wrote:
> Vivek Netha wrote:
>>>> x << doc.search("//li[@class='g w0']/h3/a")
>
> You could try contains(@class, "w0"), but that would match "w0p0p", which is a
> different class.
>

XPath that solves this:
contains(concat(' ', @class, ' '), ' w0 ')

You can join multiple classes together like this:
contains(concat(' ', @class, ' '), ' w0 ') && contains(concat(' ',
@class, ' '), ' g ')

It's not pretty. I've written it too much in my life.
I do wish XPath had a class selector.

--
Tom Morris
<http://tommorr...

Phlip

1/10/2009 9:47:00 AM

0

Tom Morris wrote:

> XPath that solves this:
> contains(concat(' ', @class, ' '), ' w0 ')
>
> You can join multiple classes together like this:
> contains(concat(' ', @class, ' '), ' w0 ') && contains(concat(' ',
> @class, ' '), ' g ')
>
> It's not pretty. I've written it too much in my life.
> I do wish XPath had a class selector.

Noted. I'm writing an XPath DSL, above the level of a raw XML library, and I
just added to its pending feature list these line-items:

# TODO :class => :symbol should do the trick contains(concat(' ', @class, '
'), ' w0 ')
# TODO :class => [] should do the trick contains(concat(' ', @class, ' '), '
w0 ') && contains(concat(' ',@class, ' '), ' g ')
# TODO :class => a string should be raw.

Those won't fix low-level XPath, but they will be useful when my DSL targets
XHTML...

Aaron Patterson

1/10/2009 6:22:00 PM

0

On Sat, Jan 10, 2009 at 06:49:32PM +0900, Phlip wrote:
> Tom Morris wrote:
>
>> XPath that solves this:
>> contains(concat(' ', @class, ' '), ' w0 ')
>>
>> You can join multiple classes together like this:
>> contains(concat(' ', @class, ' '), ' w0 ') && contains(concat(' ',
>> @class, ' '), ' g ')
>>
>> It's not pretty. I've written it too much in my life.
>> I do wish XPath had a class selector.
>
> Noted. I'm writing an XPath DSL, above the level of a raw XML library,
> and I just added to its pending feature list these line-items:
>
> # TODO :class => :symbol should do the trick contains(concat(' ',
> @class, ' '), ' w0 ')
> # TODO :class => [] should do the trick contains(concat(' ', @class, '
> '), ' w0 ') && contains(concat(' ',@class, ' '), ' g ')
> # TODO :class => a string should be raw.
>
> Those won't fix low-level XPath, but they will be useful when my DSL
> targets XHTML...

Couldn't you use CSS selectors when targeting XHTML (or XML even)? In
Nokogiri, I have a CSS to XPath converter and just push the XPath down
to libxml. For example:

require 'nokogiri'

doc = Nokogiri::HTML(<<-eoxml)
<html>
<body>
<div class="one two">
Hello
</div>
</body>
</html>
eoxml

puts doc.at('.one.two').content

It also doesn't care if you're using XML or HTML. The CSS selectors will get
converted to XPath either way.

--
Aaron Patterson
http://tenderlovem...

Phlip

1/10/2009 7:07:00 PM

0

> Couldn't you use CSS selectors when targeting XHTML (or XML even)?

> puts doc.at('.one.two').content

I'm going to. As soon as I get off my lazy butt, I will make nokogiri into a
pluggable back-end to the DSL. Then the CSS selection notation is free.

But I can't make the DSL itself into a CSS hybrid because that would
disadvantage the other pluggable XML parsers. So if you want pure XML
compliance, use REXML, if you want some XHTML awareness use LibXML, and if you
want CSS notation use Nokogiri. (After I'm done!)

Aaron Patterson

1/10/2009 8:48:00 PM

0

On Sun, Jan 11, 2009 at 04:09:54AM +0900, Phlip wrote:
>> Couldn't you use CSS selectors when targeting XHTML (or XML even)?
>
>> puts doc.at('.one.two').content
>
> I'm going to. As soon as I get off my lazy butt, I will make nokogiri
> into a pluggable back-end to the DSL. Then the CSS selection notation is
> free.
>
> But I can't make the DSL itself into a CSS hybrid because that would
> disadvantage the other pluggable XML parsers. So if you want pure XML
> compliance, use REXML, if you want some XHTML awareness use LibXML, and
> if you want CSS notation use Nokogiri. (After I'm done!)

The CSS to XPath conversion can stand alone as a pure ruby library.
Feel free to take it from my source if you like. Then you can give CSS
notation to anything that supports XPath. :-)

--
Aaron Patterson
http://tenderlovem...