Asp Forum - Looking for web crawler written in Ruby

Tuwewe

1/31/2006 7:12:00 AM

Hi, I am looking for web crawler(spider) written in Ruby. I googled but
couldn't find much information. Can anyone give me some links?

Thanks in advance.

Kevin

6 Answers

Gene Tani

1/31/2006 1:10:00 PM

Kev wrote:
> Hi, I am looking for web crawler(spider) written in Ruby. I googled but
> couldn't find much information. Can anyone give me some links?
>

http://www.jamesbritt.com/articles/RubyAndVbaForWebDictionarySea...
http://www.linux-magazine.com/issue/51/Ruby_Web_S...
http://www.acc.umu.se/~r2d2/programming/ruby/w...

http://snippets.textdrive.com/pos...
(look at cached page)

http://rubyforge.org/pro...
look at Mechanize

Justin Collins

1/31/2006 7:45:00 PM

Kev wrote:
> Hi, I am looking for web crawler(spider) written in Ruby. I googled but
> couldn't find much information. Can anyone give me some links?
>
> Thanks in advance.
>
> Kevin
>

If you are _really_ desperate, the first Ruby program I ever wrote was
kind of a web crawler:
http://students.seattleu.edu/collinsj/programs_netcr...

But it's really terrible. It isn't hard to write your own, if you need
to, using Net::HTTP
(http://ruby-doc.org/stdlib/libdoc/net/http/rdoc/...) and URI
(http://ruby-doc.org/stdlib/libdoc/uri/rdoc/...).

-Justin

James Gray

1/31/2006 9:45:00 PM

On Jan 31, 2006, at 1:11 AM, Kev wrote:

> Hi, I am looking for web crawler(spider) written in Ruby. I googled
> but
> couldn't find much information. Can anyone give me some links?

The code I translated from Higher-Order Perl at the tail end of the
following blog article may be of use to you.

http://blog.grayproductions.net/articles/2006/01/31/...
chapters-4-and-5

James Edward Gray II

Tuwewe

2/1/2006 12:02:00 PM

Many thanks, I appreciate it very much.

Bill Guindon

2/1/2006 10:57:00 PM

On 1/31/06, Kev <tuweiwen@gmail.com> wrote:
> Hi, I am looking for web crawler(spider) written in Ruby. I googled but
> couldn't find much information. Can anyone give me some links?

Some more offerings here:
http://blade.nagaokaut.ac.jp/cgi-bin/vframe.rb/ruby/ruby-talk/146169?146...

now you're practically crawling with spiders :)

> Thanks in advance.
>
> Kevin
>
>

--
Bill Guindon (aka aGorilla)
The best answer to most questions is "it depends".

james_b

2/2/2006 12:16:00 AM

Gene Tani wrote:
> Kev wrote:
>
>>Hi, I am looking for web crawler(spider) written in Ruby. I googled but
>>couldn't find much information. Can anyone give me some links?
>>
>
>
> http://www.jame.../articles/RubyAndVbaForWebDictionarySea...

There's not much spidering going on there, just some page scraping

>
> http://rubyforge.org/pro...
> look at Mechanize
>

My Mechanize example may be helpful, though.

http://neurogami.com/caf...

--
James Britt

http://www.ru... - Ruby Help & Documentation
http://www.artima.c... - The Journal By & For Rubyists
http://www.rub... - The Ruby Store for Ruby Stuff
http://www.jame... - Playing with Better Toys
http://www.30seco... - Building Better Tools

comp.lang.ruby

Looking for web crawler written in Ruby

Tuwewe

Gene Tani

Justin Collins

James Gray

Tuwewe

Bill Guindon

james_b

x Login to ForumsZone