Asp Forum - looping through & downloading web page - WATIR codes

michael

12/5/2006 6:55:00 AM

What I am trying to realize through code below is to read a line by
line in items.txt, and download each ensuing pages by having WATIR
click 'Next' link.

Now what I wish this code to do is when WATIR reaches the last page,
and there is no more "Next" link on the page, then go to the second
line in items.txt, searching again, and download ensuing pages again.

But unfortunately, the WATIR code close down when there is no more
"Next" link on the first item search.. and it does not go to searching
with the second line in items.txt.

Could anybody help me how I should adjust the code below so that when
there is no more "Next" link on the page, then the second line in
items.txt goes into searching, download each page by clicking "Next"
link.. on and on??

any suggestion will be deeply appreciated.

Thanks.

=====================================================================
require 'watir'

ie = Watir::IE.start("http://www.test...)

file = File.open("c:/items.txt", "r")
lines = file.readlines()

PageNo = 1

0.upto(lines.length - 1) {|j|

$item = lines[j]

ie.text_field(:name, "item").value = $item

ie.button(:name, "Search").click

a = ie.contains_text("Next")

while a

File.open("#{PageNo}.html", "wb") { |f|

f << '<html>'
f << ie.html
f << '</html>'

PageNo = PageNo + 1

ie.link(:text, "Next").click

}

end

}

file.close()
lines.clear()

3 Answers

Paul Lutus

12/5/2006 8:50:00 AM

curious wrote:

> What I am trying to realize through code below is to read a line by
> line in items.txt, and download each ensuing pages by having WATIR
> click 'Next' link.
>
> Now what I wish this code to do is when WATIR reaches the last page,
> and there is no more "Next" link on the page, then go to the second
> line in items.txt, searching again, and download ensuing pages again.
>
> But unfortunately, the WATIR code close down when there is no more
> "Next" link on the first item search.. and it does not go to searching
> with the second line in items.txt.
>
> Could anybody help me how I should adjust the code below so that when
> there is no more "Next" link on the page, then the second line in
> items.txt goes into searching, download each page by clicking "Next"
> link.. on and on??
>
> any suggestion will be deeply appreciated.

Rather than simply rewriting your code for you, I offer these suggestions:

1. Always close the opened file handle when the write is completed. You are
not closing your file handles until the last file is read, which in the
worst case leaves many file handles open and abandoned.

2. Instead of doing this:

0.upto(lines.length - 1) {|j|
$item = lines[j]

Do it this way:

lines.each do |item|
# ... your code here
end

3. Avoid use of global variables when possible.

I would offer more suggestions but I don't actually understand what your
program is meant to accomplish.

--
Paul Lutus
http://www.ara...

michael

12/5/2006 8:56:00 AM

I found out what the problem is..

it is the last part..

ie.link(:text, "Next").click.. in the last web page, although "Next"
text is contained, link is not present since it is the last page..

Is there anyway, I can control "ie.link(:text, "Next").click" by
checking the presence of link beforehand?? so if link is present, then
proceed, otherwise.. halt.. I think my code will do as I wish once
this gets done..

thanks.

Paul Lutus wrote:
> curious wrote:
>
> > What I am trying to realize through code below is to read a line by
> > line in items.txt, and download each ensuing pages by having WATIR
> > click 'Next' link.
> >
> > Now what I wish this code to do is when WATIR reaches the last page,
> > and there is no more "Next" link on the page, then go to the second
> > line in items.txt, searching again, and download ensuing pages again.
> >
> > But unfortunately, the WATIR code close down when there is no more
> > "Next" link on the first item search.. and it does not go to searching
> > with the second line in items.txt.
> >
> > Could anybody help me how I should adjust the code below so that when
> > there is no more "Next" link on the page, then the second line in
> > items.txt goes into searching, download each page by clicking "Next"
> > link.. on and on??
> >
> > any suggestion will be deeply appreciated.
>
> Rather than simply rewriting your code for you, I offer these suggestions:
>
> 1. Always close the opened file handle when the write is completed. You are
> not closing your file handles until the last file is read, which in the
> worst case leaves many file handles open and abandoned.
>
> 2. Instead of doing this:
>
> 0.upto(lines.length - 1) {|j|
> $item = lines[j]
>
> Do it this way:
>
> lines.each do |item|
> # ... your code here
> end
>
> 3. Avoid use of global variables when possible.
>
> I would offer more suggestions but I don't actually understand what your
> program is meant to accomplish.
>
> --
> Paul Lutus
> http://www.ara...

Paul Lutus

12/5/2006 9:19:00 AM

curious wrote:

> I found out what the problem is..
>
> it is the last part..
>
> ie.link(:text, "Next").click.. in the last web page, although "Next"
> text is contained, link is not present since it is the last page..
>
> Is there anyway, I can control "ie.link(:text, "Next").click" by
> checking the presence of link beforehand??

Sorry, I don't know enough about Watir to answer this question.

> so if link is present, then
> proceed, otherwise.. halt.. I think my code will do as I wish once
> this gets done..

No, it won't. It won't because you are not closing your file handles. If you
don't close your file handles, the writes will not be completed and this
could lead to file corruption.

At the end of the process, after writing any number files, you close the
last of the file handles, but you must close all of them.

I'm sorry I can't answer more of your questions.

--
Paul Lutus
http://www.ara...

comp.lang.ruby

looping through & downloading web page - WATIR codes

michael

Paul Lutus

michael

Paul Lutus

x Login to ForumsZone