Asp Forum - WAtir usage - comp.lang.ruby

junkone

7/8/2006 5:23:00 PM

Hi
I found Watir and am excited to use it. I cannot find much examples in
terms of data scraping. I have gotne to a point wherei can use it to
submit form details.
I need to scrape data from the tables etc. How do i do it.
For eg my if ie is my browser object, how do i get the tables in the
page and extract data from 3 table in the page into a array?

Pl help.

Seede

6 Answers

Xavier Noria

7/8/2006 8:04:00 PM

On Jul 8, 2006, at 19:25, junkone@rogers.com wrote:

> Hi
> I found Watir and am excited to use it. I cannot find much examples in
> terms of data scraping. I have gotne to a point wherei can use it to
> submit form details.
> I need to scrape data from the tables etc. How do i do it.
> For eg my if ie is my browser object, how do i get the tables in the
> page and extract data from 3 table in the page into a array?

Watir is a wrapper around MSIE. In my limited experience crawling
with Watir, besides reading its documentation I think it's good to
study the underlying API, available here:

http://msdn.microsoft.com/library/default.asp?url=...
browser/mshtml/reference/reference.asp

In particular the interfaces IWebBrowser2, IHTMLDocument2, and
IHTMLDocument3:

http://msdn.microsoft.com/workshop/browser/webbrowser/...
ifaces/iwebbrowser2/iwebbrowser2.asp
http://msdn.microsoft.com/workshop/browser/mshtml/referen...
document2/document2.asp
http://msdn.microsoft.com/workshop/browser/mshtml/referen...
document3/document3.asp

There you see what can be done grabbing the ie attribute of the
Watir::IE object, in case the wrapping interface does not address
some particular need. Additionally, I have found reading watir.rb
very instructive, you have a better knowledge of what you are doing
and may give context and idioms.

-- fxn

James Britt

7/8/2006 8:33:00 PM

Xavier Noria wrote:
> On Jul 8, 2006, at 19:25, junkone@rogers.com wrote:

> There you see what can be done grabbing the ie attribute of the
> Watir::IE object, in case the wrapping interface does not address some
> particular need. Additionally, I have found reading watir.rb very
> instructive, you have a better knowledge of what you are doing and may
> give context and idioms.

About a year or so ago I poked around trying to get HTML from IE in
Watir. The problem (and maybe this has changed) was that the IE object
did not have a way to get the literal text currently in the browser;
there was no COM 'view|source', so to speak.

What it exposed was access to IE's internal node tree, which is *not*
what is in the browser. IE adds all sorts of things (for DHTML and such,
as well as, I think, the various implied attributes defined in the HTML
DTD. Something akin to the post-schema infoset, I think.) so it is
useless for verifying server output. Still, it is handy in other ways;
you can locate, say, table elements and td content and so on.

I wrote a method that did a recursive walk down the internal IE tree and
created an XML string from it. Contact me off-list if you would like me
to send it to you. It's old and may not work with the current Watir,
but might give you some ideas.

james DOT britt AT gmail DOT com

--
James Britt

http://www.ru... - Ruby Help & Documentation
http://www.artima.c... - The Journal By & For Rubyists
http://www.rub... - The Ruby Store for Ruby Stuff
http://refreshing... - Design, technology, usability

Xavier Noria

7/8/2006 11:58:00 PM

On Jul 8, 2006, at 22:04, Xavier Noria wrote:

> On Jul 8, 2006, at 19:25, junkone@rogers.com wrote:
>
>> Hi
>> I found Watir and am excited to use it. I cannot find much
>> examples in
>> terms of data scraping. I have gotne to a point wherei can use it to
>> submit form details.
>> I need to scrape data from the tables etc. How do i do it.
>> For eg my if ie is my browser object, how do i get the tables in the
>> page and extract data from 3 table in the page into a array?

> There you see what can be done grabbing the ie attribute of the
> Watir::IE object, in case the wrapping interface does not address
> some particular need. Additionally, I have found reading watir.rb
> very instructive, you have a better knowledge of what you are doing
> and may give context and idioms.

By the way, regarding the question about tables you can do this:

table = ie.table(:index, 3)

That index is 1-based and returns an object of type Watir::Table or
nil. As per the definition of order between tables, it is delegated
to IE this way:

@ieController.document.getElementsByTagName("TABLE")

-- fxn

Alan Ark

7/10/2006 9:13:00 PM

You should be able to do something like

Browser.html()

To grab the original source.

There is also a Watir specific newsgroup somewhere around rubyforge.

-----Original Message-----
From: James Britt [mailto:james.britt@gmail.com]
Sent: Saturday, July 08, 2006 1:33 PM
To: ruby-talk ML
Subject: Re: WAtir usage

Xavier Noria wrote:
> On Jul 8, 2006, at 19:25, junkone@rogers.com wrote:

> There you see what can be done grabbing the ie attribute of the
> Watir::IE object, in case the wrapping interface does not address some
> particular need. Additionally, I have found reading watir.rb very
> instructive, you have a better knowledge of what you are doing and may
> give context and idioms.

About a year or so ago I poked around trying to get HTML from IE in
Watir. The problem (and maybe this has changed) was that the IE object
did not have a way to get the literal text currently in the browser;
there was no COM 'view|source', so to speak.

What it exposed was access to IE's internal node tree, which is *not*
what is in the browser. IE adds all sorts of things (for DHTML and such,
as well as, I think, the various implied attributes defined in the HTML
DTD. Something akin to the post-schema infoset, I think.) so it is
useless for verifying server output. Still, it is handy in other ways;
you can locate, say, table elements and td content and so on.

I wrote a method that did a recursive walk down the internal IE tree and
created an XML string from it. Contact me off-list if you would like me
to send it to you. It's old and may not work with the current Watir,
but might give you some ideas.

james DOT britt AT gmail DOT com

--
James Britt

http://www.ru... - Ruby Help & Documentation
http://www.artima.c... - The Journal By & For Rubyists
http://www.rub... - The Ruby Store for Ruby Stuff
http://refreshing... - Design, technology, usability

--
No virus found in this incoming message.
Checked by AVG Free Edition.
Version: 7.1.394 / Virus Database: 268.9.10/383 - Release Date: 7/7/2006

--
No virus found in this outgoing message.
Checked by AVG Free Edition.
Version: 7.1.394 / Virus Database: 268.9.10/383 - Release Date: 7/7/2006

James Britt

7/10/2006 9:35:00 PM

Alan Ark wrote:
> You should be able to do something like
>
> Browser.html()
>
> To grab the original source.

Well, that *would* save a good deal of trouble.

Thanks!

--
James Britt

"You harmonize; then you customize."
- Wilson Pickett

Xavier Noria

7/10/2006 10:31:00 PM

On Jul 10, 2006, at 23:13, Alan Ark wrote:

> You should be able to do something like
>
> Browser.html()
>
> To grab the original source.

What do you mean? Watir::IE#html gives a dump of the internal MSIE
representation, tags are in uppercase, quotes are removed from some
attributes, etc. What James mentioned a few messages above.

-- fxn

comp.lang.ruby

WAtir usage

junkone

Xavier Noria

James Britt

Xavier Noria

Alan Ark

James Britt

Xavier Noria

x Login to ForumsZone