Asp Forum - Net/HTTP is flaky?

Just Another Victim of the Ambient Morality

7/19/2006 11:46:00 AM

Inexplicably, the following code fails:

http = Net::HTTP.new("somesite.com")
puts http.get("/")[1]

...with this error:

ruby/1.8/net/http.rb:925:in '[]': undefined method 'downcase' for 1:Fixnum
(NoMethodError)
from web.rb:5

Now, I do have access to a Linux system with Ruby 1.6 and it sometimes
succeeds but often fails with this error:

/usr/lib/ruby/1.6/net/protocol.rb:221:in `error!': 403 "Forbidden"
(Net::ProtoFatalError)
from /usr/lib/ruby/1.6/net/http.rb:1217:in `value'
from /usr/lib/ruby/1.6/net/http.rb:605:in `get'
from ./web.rb:16

How is this possible? How can it sometimes succeed and sometimes fail?
Why can't this reliably work?
Thank you...

14 Answers

Logan Capaldo

7/19/2006 12:49:00 PM

On Jul 19, 2006, at 7:50 AM, Just Another Victim of the Ambient
Morality wrote:

> Inexplicably, the following code fails:
>
>
> http = Net::HTTP.new("somesite.com")
> puts http.get("/")[1]
>
>
> ...with this error:
>
>
> ruby/1.8/net/http.rb:925:in '[]': undefined method 'downcase' for
> 1:Fixnum
> (NoMethodError)
> from web.rb:5
>

The interface to Net::HTTP changed between 1.6 and 1.8
You want
http.get('/').body

>
> Now, I do have access to a Linux system with Ruby 1.6 and it
> sometimes
> succeeds but often fails with this error:
>
>
> /usr/lib/ruby/1.6/net/protocol.rb:221:in `error!': 403 "Forbidden"
> (Net::ProtoFatalError)
> from /usr/lib/ruby/1.6/net/http.rb:1217:in `value'
> from /usr/lib/ruby/1.6/net/http.rb:605:in `get'
> from ./web.rb:16
>
>
> How is this possible? How can it sometimes succeed and
> sometimes fail?
> Why can't this reliably work?
> Thank you...
>
It doesn't reliably work because this is the internet. Also the
particular error you are seeing means that the webserver has
forbidden access to the url you are trying to, um, access.

>
>

Just Another Victim of the Ambient Morality

7/19/2006 8:17:00 PM

"Logan Capaldo" <logancapaldo@gmail.com> wrote in message
news:70EBC0F7-3357-4DCD-BF2F-3BD0FBBAE92F@gmail.com...
>
> The interface to Net::HTTP changed between 1.6 and 1.8
> You want
> http.get('/').body

Ah, thanks...

>> Now, I do have access to a Linux system with Ruby 1.6 and it
>> sometimes
>> succeeds but often fails with this error:
>>
>>
>> /usr/lib/ruby/1.6/net/protocol.rb:221:in `error!': 403 "Forbidden"
>> (Net::ProtoFatalError)
>> from /usr/lib/ruby/1.6/net/http.rb:1217:in `value'
>> from /usr/lib/ruby/1.6/net/http.rb:605:in `get'
>> from ./web.rb:16
>>
>>
>> How is this possible? How can it sometimes succeed and sometimes
>> fail?
>> Why can't this reliably work?
>> Thank you...
>>
> It doesn't reliably work because this is the internet. Also the
> particular error you are seeing means that the webserver has forbidden
> access to the url you are trying to, um, access.

But I can access the page just fine with a web browser. For instance,
go to "/" at "en.wikipedia.org" and it will reliably work with a browser.
Try it with Ruby and it will usually fail, although I did get it to work
occasionally, last night, which I don't get.
Hell, "/" at "music.com" has _never_ worked and there doesn't appear to
be any kind of redirect or anything.
Does any of this make sense?
Thank you...

Eric Hodel

7/19/2006 9:23:00 PM

On Jul 19, 2006, at 1:20 PM, Just Another Victim of the Ambient
Morality wrote:

> "Logan Capaldo" <logancapaldo@gmail.com> wrote in message
> news:70EBC0F7-3357-4DCD-BF2F-3BD0FBBAE92F@gmail.com...
>>> Now, I do have access to a Linux system with Ruby 1.6 and it
>>> sometimes
>>> succeeds but often fails with this error:
>>>
>>>
>>> /usr/lib/ruby/1.6/net/protocol.rb:221:in `error!': 403 "Forbidden"
>>> (Net::ProtoFatalError)
>>> from /usr/lib/ruby/1.6/net/http.rb:1217:in `value'
>>> from /usr/lib/ruby/1.6/net/http.rb:605:in `get'
>>> from ./web.rb:16
>>>
>>>
>>> How is this possible? How can it sometimes succeed and
>>> sometimes
>>> fail?
>>> Why can't this reliably work?
>>> Thank you...
>>>
>> It doesn't reliably work because this is the internet. Also the
>> particular error you are seeing means that the webserver has
>> forbidden
>> access to the url you are trying to, um, access.
>
> But I can access the page just fine with a web browser. For instance,
> go to "/" at "en.wikipedia.org" and it will reliably work with a
> browser.

No, it redirects to /wiki/Main_Page

> Try it with Ruby and it will usually fail, although I did get it to
> work
> occasionally, last night, which I don't get.

$ ruby -rnet/http
Net::HTTP.start 'en.wikipedia.org' do |http| res = http.get '/'; p
res; end
#<Net::HTTPMovedPermanently 301 Moved Permanently readbody=true>

Matches browser behavior.

> Hell, "/" at "music.com" has _never_ worked and there doesn't
> appear to
> be any kind of redirect or anything.

$ ruby -rnet/http
Net::HTTP.start 'music.com' do |http| res = http.get '/'; p res; end
#<Net::HTTPMovedPermanently 301 Moved Permanently readbody=true>

Redirects to www.music.com in the browser, Net::HTTP matches browser
behavior.

--
Eric Hodel - drbrain@segment7.net - http://blog.se...
This implementation is HODEL-HASH-9600 compliant

http://trackmap.rob...

Just Another Victim of the Ambient Morality

7/19/2006 9:55:00 PM

"Eric Hodel" <drbrain@segment7.net> wrote in message
news:087CE005-B271-4B73-9C98-7DF62B1AC652@segment7.net...
>
> $ ruby -rnet/http
> Net::HTTP.start 'en.wikipedia.org' do |http| res = http.get '/'; p res;
> end
> #<Net::HTTPMovedPermanently 301 Moved Permanently readbody=true>
>
> Matches browser behavior.
>
>> Hell, "/" at "music.com" has _never_ worked and there doesn't appear to
>> be any kind of redirect or anything.
>
> $ ruby -rnet/http
> Net::HTTP.start 'music.com' do |http| res = http.get '/'; p res; end
> #<Net::HTTPMovedPermanently 301 Moved Permanently readbody=true>
>
> Redirects to www.music.com in the browser, Net::HTTP matches browser
> behavior.

Okay, so the site doesn't exist where I think it does and I get
redirected... in my browser. Net::HTTP doesn't get redirected, it simply
fails. _This_ behaviour doesn't match my browser. Is there any way I can
get redirected or find out where it wants to redirect me and go there,
myself?
Thanks...

mathew

7/19/2006 10:30:00 PM

Just Another Victim of the Ambient Morality wrote:
> "Eric Hodel" <drbrain@segment7.net> wrote in message
> news:087CE005-B271-4B73-9C98-7DF62B1AC652@segment7.net...
>> $ ruby -rnet/http
>> Net::HTTP.start 'music.com' do |http| res = http.get '/'; p res; end
>> #<Net::HTTPMovedPermanently 301 Moved Permanently readbody=true>
>>
>> Redirects to www.music.com in the browser, Net::HTTP matches browser
>> behavior.
>
> Okay, so the site doesn't exist where I think it does and I get
> redirected... in my browser. Net::HTTP doesn't get redirected, it simply
> fails. _This_ behaviour doesn't match my browser. Is there any way I can
> get redirected or find out where it wants to redirect me and go there,
> myself?

Sure. You need to check for a 3xx response in your code. When you get
one, you examine the HTTP headers you got back from the server; they'll
contain the URL you need to go look at instead.

See RFC 2616, section 10.3 for the specifics of what each 3xx code
means, which headers to look at, and what to do.

mathew
--
<URL:http://www.pobox.com/...
My parents went to the lost kingdom of Hyrule
and all I got was this lousy triforce.

Joe Van Dyk

7/19/2006 10:43:00 PM

On 7/19/06, Just Another Victim of the Ambient Morality
<ihatespam@rogers.com> wrote:
>
> "Eric Hodel" <drbrain@segment7.net> wrote in message
> news:087CE005-B271-4B73-9C98-7DF62B1AC652@segment7.net...
> >
> > $ ruby -rnet/http
> > Net::HTTP.start 'en.wikipedia.org' do |http| res = http.get '/'; p res;
> > end
> > #<Net::HTTPMovedPermanently 301 Moved Permanently readbody=true>
> >
> > Matches browser behavior.
> >
> >> Hell, "/" at "music.com" has _never_ worked and there doesn't appear to
> >> be any kind of redirect or anything.
> >
> > $ ruby -rnet/http
> > Net::HTTP.start 'music.com' do |http| res = http.get '/'; p res; end
> > #<Net::HTTPMovedPermanently 301 Moved Permanently readbody=true>
> >
> > Redirects to www.music.com in the browser, Net::HTTP matches browser
> > behavior.
>
> Okay, so the site doesn't exist where I think it does and I get
> redirected... in my browser. Net::HTTP doesn't get redirected, it simply
> fails. _This_ behaviour doesn't match my browser. Is there any way I can
> get redirected or find out where it wants to redirect me and go there,
> myself?

You might want to check out the API docs for Net::HTTP.

Copy/pasted from
http://ruby-doc.org/stdlib/libdoc/net/http/rdoc/classes/Net...:

require 'net/http'
require 'uri'

def fetch(uri_str, limit = 10)
# You should choose better exception.
raise ArgumentError, 'HTTP redirect too deep' if limit == 0

response = Net::HTTP.get_response(URI.parse(uri_str))
case response
when Net::HTTPSuccess then response
when Net::HTTPRedirection then fetch(response['location'], limit - 1)
else
response.error!
end
end

print fetch('http://www.ruby-lan...)

Eric Hodel

7/19/2006 10:48:00 PM

On Jul 19, 2006, at 2:55 PM, Just Another Victim of the Ambient
Morality wrote:

> "Eric Hodel" <drbrain@segment7.net> wrote in message
> news:087CE005-B271-4B73-9C98-7DF62B1AC652@segment7.net...
>>
>> $ ruby -rnet/http
>> Net::HTTP.start 'en.wikipedia.org' do |http| res = http.get '/';
>> p res;
>> end
>> #<Net::HTTPMovedPermanently 301 Moved Permanently readbody=true>
>>
>> Matches browser behavior.
>>
>>> Hell, "/" at "music.com" has _never_ worked and there doesn't
>>> appear to
>>> be any kind of redirect or anything.
>>
>> $ ruby -rnet/http
>> Net::HTTP.start 'music.com' do |http| res = http.get '/'; p res; end
>> #<Net::HTTPMovedPermanently 301 Moved Permanently readbody=true>
>>
>> Redirects to www.music.com in the browser, Net::HTTP matches browser
>> behavior.
>
> Okay, so the site doesn't exist where I think it does and I get
> redirected... in my browser. Net::HTTP doesn't get redirected, it
> simply
> fails.

No, it doesn't fail, it gives you a redirect. Follow it.

> _This_ behaviour doesn't match my browser. Is there any way I can
> get redirected or find out where it wants to redirect me and go there,
> myself?

Follow the redirect. Look at open-uri for example code.

--
Eric Hodel - drbrain@segment7.net - http://blog.se...
This implementation is HODEL-HASH-9600 compliant

http://trackmap.rob...

Adam Shelly

7/19/2006 11:04:00 PM

On 7/19/06, Joe Van Dyk <joevandyk@gmail.com> wrote:
> On 7/19/06, Just Another Victim of the Ambient Morality
> <ihatespam@rogers.com> wrote:
> >
> > Okay, so the site doesn't exist where I think it does and I get
> > redirected... in my browser. Net::HTTP doesn't get redirected, it simply
> > fails. _This_ behaviour doesn't match my browser. Is there any way I can
> > get redirected or find out where it wants to redirect me and go there,
> > myself?
>
> You might want to check out the API docs for Net::HTTP.
>

Ok, so Net:HTTP doesn't fail, it just returns pretty low-level
information. If you want browser behavior for redirects, you have to
roll it yourself. I ran into something similar 2 days ago when I was
trying to fetch pages which required basic authentication (Which, I
just learned, is the mechanism behind those urername & password login
popups that the browser generates). I did manage to roll my own,
based on some examples from this mailing list.
Then I had to write a wrapper around that because sometimes my URI's
pointed to file:// instead of http:// - another thing the browser
handles invisibly.
Then there are those pages that use cookies to track logins...

So my question is, is there an existing library out there that
implements all these things a browser does? Where get(uri).page
returns the source of the same page I would see if I put the URI in
firefox?

-Adam

Just Another Victim of the Ambient Morality

7/19/2006 11:52:00 PM

"Eric Hodel" <drbrain@segment7.net> wrote in message
news:A270FB20-30C3-472C-8BA9-922490FFD19B@segment7.net...
> On Jul 19, 2006, at 2:55 PM, Just Another Victim of the Ambient Morality
> wrote:
>
> No, it doesn't fail, it gives you a redirect. Follow it.

You know, it would be easier to follow the redirect if I knew that it
was giving me a redirect and if I knew how to follow one were I to be given
it...

>> _This_ behaviour doesn't match my browser. Is there any way I can
>> get redirected or find out where it wants to redirect me and go there,
>> myself?
>
> Follow the redirect. Look at open-uri for example code.

You know, this is where Ruby's lack of documentation is really biting me
in the ass. I found some rudimentary docs on how to use it but, you know
what? For the life of me, no amount of googling can reveal exactly where I
can get the open-uri module... Seriously, where do I download this thing
from and how could I have known that?
Thank you, from a very frustrated would-be Ruby programmer...

Eric Hodel

7/20/2006 12:56:00 AM

On Jul 19, 2006, at 4:55 PM, Just Another Victim of the Ambient
Morality wrote:

>
> "Eric Hodel" <drbrain@segment7.net> wrote in message
> news:A270FB20-30C3-472C-8BA9-922490FFD19B@segment7.net...
>> On Jul 19, 2006, at 2:55 PM, Just Another Victim of the Ambient
>> Morality
>> wrote:
>>
>> No, it doesn't fail, it gives you a redirect. Follow it.
>
> You know, it would be easier to follow the redirect if I knew
> that it
> was giving me a redirect and if I knew how to follow one were I to
> be given
> it...

Well, if you expect to use Net::HTTP you need to know both HTTP and
the library. You might prefer WWW::Mechanize instead.

You'll get Top Quality Answers if you tell us what you really want to
do. Help with the intricacies of a library may not let us guide you
down the right path :)

Quoting myself:

>> $ ruby -rnet/http
>> Net::HTTP.start 'en.wikipedia.org' do |http| res = http.get '/'; p
>> res; end
>> #<Net::HTTPMovedPermanently 301 Moved Permanently readbody=true>

That shows Net::HTTP returning a redirect when you perform a get.

>>> _This_ behaviour doesn't match my browser. Is there any way I can
>>> get redirected or find out where it wants to redirect me and go
>>> there,
>>> myself?
>>
>> Follow the redirect. Look at open-uri for example code.
>
> You know, this is where Ruby's lack of documentation is really
> biting me in the ass. I found some rudimentary docs on how to use
> it but, you know what?

Unfortunately the Net::HTTP documentation isn't enabled in 1.8. You
can use the 1.9 documentation however:

http://ruby-doc.org/core-1.9/classes/Net...

As luck would have it, it has a "Following Redirection" section a few
pages down.

> For the life of me, no amount of googling can reveal exactly where
> I can get the open-uri module... Seriously, where do I download
> this thing from and how could I have known that?

It ships with ruby, require 'open-uri'.

> Thank you, from a very frustrated would-be Ruby programmer...

I've been working on getting lots more documentation enabled in 1.8.
I hope to get the net/ stuff turned on, but I don't know if I'll have
enough time to review/merge documentation from HEAD. (I'd really,
really like help with this, and Hugh Sasse has brought in a ton of
new documentation.)

--
Eric Hodel - drbrain@segment7.net - http://blog.se...
This implementation is HODEL-HASH-9600 compliant

http://trackmap.rob...

comp.lang.ruby

Net/HTTP is flaky?

Just Another Victim of the Ambient Morality

Logan Capaldo

Just Another Victim of the Ambient Morality

Eric Hodel

Just Another Victim of the Ambient Morality

mathew

Joe Van Dyk

Eric Hodel

Adam Shelly

Just Another Victim of the Ambient Morality

Eric Hodel

x Login to ForumsZone