[lnkForumImage]
TotalShareware - Download Free Software
Confronta i prezzi di migliaia di prodotti.
Asp Forum
 Home | Login | Register | Search 


 

Forums >

comp.lang.ruby

Reading Data from a Website

anon1m0us

1/18/2007 8:22:00 PM

Hi;
No clue how to do this. My program to go to a website and read data and
process it. Don't kow where to even begin! How do I go to a website in
RUBY? How to I start reading the data?
10 Answers

andy

1/18/2007 8:32:00 PM

0


On Jan 18, 2007, at 2:25 PM, anon1m0us wrote:

> Hi;
> No clue how to do this. My program to go to a website and read data
> and
> process it. Don't kow where to even begin! How do I go to a website in
> RUBY? How to I start reading the data?

Look at WWW::Mechanize.

--
Andy Lester => andy@petdance.com => www.petdance.com => AIM:petdance

anon1m0us

1/18/2007 8:44:00 PM

0

Is that a website? Where do I see that stuff?
In addition;
i need to view the Source of the website since the information are
contained in tables on the website.

Andy Lester wrote:
> On Jan 18, 2007, at 2:25 PM, anon1m0us wrote:
>
> > Hi;
> > No clue how to do this. My program to go to a website and read data
> > and
> > process it. Don't kow where to even begin! How do I go to a website in
> > RUBY? How to I start reading the data?
>
> Look at WWW::Mechanize.
>
> --
> Andy Lester => andy@petdance.com => www.petdance.com => AIM:petdance

pat eyler

1/18/2007 8:53:00 PM

0

On 1/18/07, Andy Lester <andy@petdance.com> wrote:
>
> On Jan 18, 2007, at 2:25 PM, anon1m0us wrote:
>
> > Hi;
> > No clue how to do this. My program to go to a website and read data
> > and
> > process it. Don't kow where to even begin! How do I go to a website in
> > RUBY? How to I start reading the data?
>
> Look at WWW::Mechanize.

Or Hpricot ...

>
> --
> Andy Lester => andy@petdance.com => www.petdance.com => AIM:petdance
>
>
>
>
>
>


--
thanks,
-pate
-------------------------
http://on-ruby.bl...

andy

1/18/2007 8:55:00 PM

0

>>
>> Look at WWW::Mechanize.
>
> Or Hpricot ...

WWW::Mechanize is a wrapper around Hpricot, just as the Perl
WWW::Mechanize is a wrapper around LWP. It handles lots of the
drudgery.

--
Andy Lester => andy@petdance.com => www.petdance.com => AIM:petdance

Peter Szinek

1/18/2007 9:35:00 PM

0

anon1m0us wrote:
> Hi;
> No clue how to do this. My program to go to a website and read data and
> process it. Don't kow where to even begin! How do I go to a website in
> RUBY? How to I start reading the data?

You could check out my older (but still fine I guess) article on this:

http://www.rubyra.../data-extraction-for-web-20-screen-scraping-in...

It would need some polishing and adding HPricot there (working on it
actually), but even like this it could provide some help.

btw. I am just releasing (in 2-3-4 something days) a powerful web
extraction language written in Ruby. It is based on Mechanize and
Hpricot and it really does a lot of heavy lifting (although I may be a
little bit biased for obvious reasons :-) - well you will see it
yourself next week)

Peter
__
http://www.rubyra...

Martin Boese

1/19/2007 1:02:00 PM

0

It's very easy, just do:

> require 'net/http'
> website = Net::HTTP.get 'www.yahoo.com', '/'

Now you have the yahoo.com startpage sourcode in website. To see it:

> puts website

The Net::HTTP documentation has more examples:

http://www.ruby-doc.org/stdlib/libdoc/net/http/rdoc/...


Martin



On Thursday 18 January 2007 20:25, anon1m0us wrote:
> Hi;
> No clue how to do this. My program to go to a website and read data and
> process it. Don't kow where to even begin! How do I go to a website in
> RUBY? How to I start reading the data?

Gavin Baker

1/19/2007 2:57:00 PM

0


On 18 Jan 2007, at 21:34, Peter Szinek wrote:

> anon1m0us wrote:
>> Hi;
>> No clue how to do this. My program to go to a website and read
>> data and
>> process it. Don't kow where to even begin! How do I go to a
>> website in
>> RUBY? How to I start reading the data?

<snip>

> btw. I am just releasing (in 2-3-4 something days) a powerful web
> extraction language written in Ruby. It is based on Mechanize and
> Hpricot and it really does a lot of heavy lifting (although I may
> be a little bit biased for obvious reasons :-) - well you will see
> it yourself next week)

After finding your article on screen scraping *very* useful, I'm
really looking forward to this!

Gav

Peter Szinek

1/19/2007 3:08:00 PM

0

Gavin Baker wrote:
>
> On 18 Jan 2007, at 21:34, Peter Szinek wrote:

> After finding your article on screen scraping *very* useful, I'm really
> looking forward to this!

I am happy to hear this... Web scraping can be very-very-very tedious,
(even with a superb tool like scRUBYt! :-)) so I will need a lot of
users to try it on a lot of pages to help find and report the problems
and come out with a really stable system. On the pages I am testing it
works perfectly (and it already has a decent feature set), however, so
far nearly always when I went to a previously unknown page there were
some problems...

However, as you will see it will worth the time to report problems etc.
because in the case of complex scenarios the solution will be much-much
faster and robust than with a hand-coded stuff...

Back to coding :)

Cheers,
Peter

__
http://www.rubyra...

Sam Smoot

1/19/2007 3:33:00 PM

0


Martin Boese wrote:
> It's very easy, just do:
>
> > require 'net/http'
> > website = Net::HTTP.get 'www.yahoo.com', '/'
>
> Now you have the yahoo.com startpage sourcode in website. To see it:
>
> > puts website
>
> The Net::HTTP documentation has more examples:
>
> http://www.ruby-doc.org/stdlib/libdoc/net/http/rdoc/...
>
>
> Martin

Great example, but because I'm lazy, I prefer open-uri:

> require 'open-uri'
> puts open('http://www.yaho...).read

Probably better to get familiar with Net::HTTP, but when that gets
old... :-)

(Alex Furman)

1/21/2007 1:38:00 AM

0

You can try SWExplorerAutomation SWEA (http://webi...)

anon1m0us wrote:
> Hi;
> No clue how to do this. My program to go to a website and read data and
> process it. Don't kow where to even begin! How do I go to a website in
> RUBY? How to I start reading the data?