[lnkForumImage]
TotalShareware - Download Free Software
Confronta i prezzi di migliaia di prodotti.
Asp Forum
 Home | Login | Register | Search 


 

Forums >

comp.lang.ruby

Need advice for a script to scrap my Verizon account

Steve

5/4/2007 9:49:00 PM

I am new to Ruby. I would like to write a script that will login to
my Verizon account and grab my minute details and calculate my
remaining unused minutes for the month. This would involve opening an
HTTPS connection, then scraping and calculating the results. I have
found one Perl and one Java/Firefox solution that does the same, but I
thought it would be a good project to learn more about Ruby. I would
rather not use external shells (to wget, for example) and rely on the
internal Ruby libraries instead. I'm looking for advice on what
libraries to use to get me started down the right path. Thank you.
7 Answers

jgbailey

5/4/2007 9:55:00 PM

0

On 5/4/07, Steve <zflyer@gmail.com> wrote:
> I am new to Ruby. I would like to write a script that will login to
> my Verizon account and grab my minute details and calculate my
> remaining unused minutes for the month. This would involve opening an
> HTTPS connection, then scraping and calculating the results. I have

I haven't used it myself, but if you might want to look at
WWW::Mechanize, which handles things like cookies for you. I found a
good looking blog post about it here:

http://neurogami.com/caf...

Another excellent library is open-uri, which comes standard with Ruby.
If Verizon does not have complex cookie requirements, you might be
able to handle the details yourself and use open-uri.

Good luck!

Justin

Philip Hallstrom

5/4/2007 9:57:00 PM

0

Bil Kleb

5/5/2007 1:33:00 AM

0

Steve wrote:
> I am new to Ruby. I would like to write a script that will login to
> my Verizon account and grab my minute details and calculate my
> remaining unused minutes for the month.

I just did something similar for our electronic
time sheet -- I had to negotiate a frameset, viz,

URL = 'https://site.somewhere...
require 'rubygems'; require 'mechanize'
agent = WWW::Mechanize.new
form = agent.get("#{URL}").forms.name('loginForm').first
form.userid = ARGV[0] # 1st commandline argument
form.password = ARGV[1] # 2nd " "
frame = agent.submit(form).frames.name('Main').href
form = agent.get("#{URL}/#{frame}").forms.name('timeCard').first
form.ENTRY_24378_1_3 = '2.25' # set hours for 1st week, 3rd day
agent.submit(form)

The next step was to use Hpricot to scrape the relations
of table inputs to time codes, but I haven't gotten there
yet...

Regards,
--
Bil Kleb
http://fun3d.lar...

Marcin Raczkowski

5/5/2007 9:36:00 AM

0

On Friday 04 May 2007 21:49, Steve wrote:
> I am new to Ruby. I would like to write a script that will login to
> my Verizon account and grab my minute details and calculate my
> remaining unused minutes for the month. This would involve opening an
> HTTPS connection, then scraping and calculating the results. I have
> found one Perl and one Java/Firefox solution that does the same, but I
> thought it would be a good project to learn more about Ruby. I would
> rather not use external shells (to wget, for example) and rely on the
> internal Ruby libraries instead. I'm looking for advice on what
> libraries to use to get me started down the right path. Thank you.

try http://www.s... - great ruby toolkit for webscraping
and Net:Mechanize

--
Marcin Raczkowski
---
Friends teach what you should know
Enemies Teach what you have to know

brenton.leanhardt@gmail.com

5/6/2007 4:27:00 PM

0

On May 4, 5:49 pm, Steve <zfl...@gmail.com> wrote:
> I am new to Ruby. I would like to write a script that will login to
> my Verizon account and grab my minute details and calculate my
> remaining unused minutes for the month. This would involve opening an
> HTTPS connection, then scraping and calculating the results. I have
> found one Perl and one Java/Firefox solution that does the same, but I
> thought it would be a good project to learn more about Ruby. I would
> rather not use external shells (to wget, for example) and rely on the
> internal Ruby libraries instead. I'm looking for advice on what
> libraries to use to get me started down the right path. Thank you.

It's not fancy, but I did pretty much the same thing you are talking
about with curb (a wrapper for lib curl) and hpricot. You can see a
quick solution I can up with here: http://exawkuser.blogspot.com/2007/02/stickittodama....
Verizon has probably changed their site in a way that now breaks my
script but you should be able to see the basics of web scraping in
ruby.

-Brenton

Peter Szinek

5/6/2007 5:48:00 PM

0

> try http://www.s... - great ruby toolkit for webscraping

Yeah, scRUBYt! is really great and all, but currently it has one major
shortcoming from the viewpoint of verizon scraping: it is built on
WWW::Mechanize, which can not handle javascript/AJAX and it seems the
verizon page has plenty of those.

FireWatir is just being integrated into scRUBYt! and scraping such a
site won't be a problem once it will be finished - however, until then I
would suggest to use either Watir or FireWatir to handle the AJAXy stuff.

Cheers,
Peter
__
http://www.rubyra... :: Ruby and Web2.0 blog
http://s... :: Ruby web scraping framework
http://rubykitch... :: The indexed archive of all things Ruby

Marcin Raczkowski

5/7/2007 11:37:00 AM

0

On Sunday 06 May 2007 17:47, Peter Szinek wrote:
> > try http://www.s... - great ruby toolkit for webscraping
>
> Yeah, scRUBYt! is really great and all, but currently it has one major
> shortcoming from the viewpoint of verizon scraping: it is built on
> WWW::Mechanize, which can not handle javascript/AJAX and it seems the
> verizon page has plenty of those.
>
> FireWatir is just being integrated into scRUBYt! and scraping such a
> site won't be a problem once it will be finished - however, until then I
> would suggest to use either Watir or FireWatir to handle the AJAXy stuff.
>
> Cheers,
> Peter
> __
> http://www.rubyra... :: Ruby and Web2.0 blog
> http://s... :: Ruby web scraping framework
> http://rubykitch... :: The indexed archive of all things Ruby

i had problem with site with javascripts and i must say most of it is easly
fixable - using firewatir is overkill in my opinion

--
Marcin Raczkowski
---
Friends teach what you should know
Enemies Teach what you have to know