Asp Forum - What's the Best Way to Mimic an HTTP Request?

Daniel Miessler

11/5/2008 4:25:00 PM

I'm trying to write a tool that will take a domain as an argument and
make a request to http://onsa... and then capture the list of
domains that share that same IP. I want to parse out those IPs and put
them into an array that I can print to a file later.

Here's the code I'm trying to use:

--
require 'net/http'
require 'uri'

PATH = '/query.jsp'
USERAGENT = 'Opera'
HOST = 'onsamehost.com'

@http = Net::HTTP.new(HOST, 80)

resp, data = @http.get2(PATH, {'User-Agent' => USERAGENT})

puts resp
puts data
--

The problem is that I keep getting a redirect
(#<Net::HTTPMovedPermanently:0xb7c35ffc>), which doesn't happen when I
make the request from a regular browser.

So I sniffed the regular request with wireshark, and a browser sends a
bunch of additional headers when it makes the request. Cookies,
referrer, etc.

Are any of these headers more necessary than others, and is there a
preferred way to send the headers using Ruby?

Thanks for any thoughts...
--
Posted via http://www.ruby-....

13 Answers

James Herdman

11/5/2008 4:29:00 PM

[Note: parts of this message were removed to make it a legal post.]

Is there a Ruby front end for Curl?

James

On Wed, Nov 5, 2008 at 11:24 AM, Daniel Miessler <daniel@dmiessler.com>wrote:

> I'm trying to write a tool that will take a domain as an argument and
> make a request to http://onsa... and then capture the list of
> domains that share that same IP. I want to parse out those IPs and put
> them into an array that I can print to a file later.
>
> Here's the code I'm trying to use:
>
> --
> require 'net/http'
> require 'uri'
>
> PATH = '/query.jsp'
> USERAGENT = 'Opera'
> HOST = 'onsamehost.com'
>
> @http = Net::HTTP.new(HOST, 80)
>
> resp, data = @http.get2(PATH, {'User-Agent' => USERAGENT})
>
> puts resp
> puts data
> --
>
> The problem is that I keep getting a redirect
> (#<Net::HTTPMovedPermanently:0xb7c35ffc>), which doesn't happen when I
> make the request from a regular browser.
>
> So I sniffed the regular request with wireshark, and a browser sends a
> bunch of additional headers when it makes the request. Cookies,
> referrer, etc.
>
> Are any of these headers more necessary than others, and is there a
> preferred way to send the headers using Ruby?
>
> Thanks for any thoughts...
> --
> Posted via http://www.ruby-....
>
>

Hassan Schroeder

11/5/2008 4:47:00 PM

On Wed, Nov 5, 2008 at 8:24 AM, Daniel Miessler <daniel@dmiessler.com> wrote:

> The problem is that I keep getting a redirect
> (#<Net::HTTPMovedPermanently:0xb7c35ffc>), which doesn't happen when I
> make the request from a regular browser.

Actually, it does -- you just don't see it.

When you request e.g. `http::/example.com` most servers will send
a redirect to the default page, e.g. `http://example.com/...`.

You need to either handle it or pass the default page's full URL.

HTH,
--
Hassan Schroeder ------------------------ hassan.schroeder@gmail.com

Michael Libby

11/5/2008 4:57:00 PM

On Wed, Nov 5, 2008 at 10:24 AM, Daniel Miessler <daniel@dmiessler.com> wrote:

> The problem is that I keep getting a redirect
> (#<Net::HTTPMovedPermanently:0xb7c35ffc>), which doesn't happen when I
> make the request from a regular browser.

That site makes heavy use of redirects. Watch closely while running
queries or check your browser history.

> So I sniffed the regular request with wireshark, and a browser sends a
> bunch of additional headers when it makes the request. Cookies,
> referrer, etc.
>
> Are any of these headers more necessary than others, and is there a
> preferred way to send the headers using Ruby?

Headers probably have no effect here.

What you probably want is code like this:

require 'net/http'
require 'uri'

def fetch(uri_str, limit = 10)
# You should choose better exception.
raise ArgumentError, 'HTTP redirect too deep' if limit == 0

response = Net::HTTP.get_response(URI.parse(uri_str))
case response
when Net::HTTPSuccess then response
when Net::HTTPRedirection then fetch(response['location'], limit - 1)
else
response.error!
end
end

resp = fetch('http://www.ruby-lan...)
puts resp.body

(from http://ruby-doc.org/stdlib/libdoc/net/http/rdoc/... --
"Following Redirection")

regards,
Michael Libby

Daniel Miessler

11/5/2008 5:08:00 PM

Thanks, much, Michael. Unfortunately I'm not quite tracking on why that
was necessary. It just seems a bit elaborate given what I thought was a
simple problem.

But I totally appreciate it...I just wish it were something simpler.
--
Posted via http://www.ruby-....

Michael Libby

11/5/2008 5:55:00 PM

On Wed, Nov 5, 2008 at 11:08 AM, Daniel Miessler <daniel@dmiessler.com> wrote:
> Thanks, much, Michael. Unfortunately I'm not quite tracking on why that
> was necessary. It just seems a bit elaborate given what I thought was a
> simple problem.
>
> But I totally appreciate it...I just wish it were something simpler.

The site you're hitting makes heavy use of redirects (and not really
for their intended purpose). What this means is that you submit your
request for a given URL and the server responds with a redirect and a
new URL. If you are working in a browser, your browser automatically
requests that URL, and the server again responds with a redirect and a
new URL. Again, a web browser handles requesting that next URL
automatically. This URL is the actual results page with the data you
want. It's the web site making you jump through hoops to get where you
want to go.

Net::HTTP does not have a built in facility for following redirects
the way your browser does. So you have to write code to follow
redirects by submitting new requests until you get to one that is not
a redirect, which is what the fetch() method from the Net::HTTP
example does.

-Michael

Avdi Grimm

11/5/2008 6:01:00 PM

You may want to look into using Mechanize rather than straight-up Net::HTTP.

--
Avdi

Home: http:...
Developer Blog: http:.../devblog/
Twitter: http://twitte...
Journal: http://avdi.livej...

Daniel Miessler

11/5/2008 6:01:00 PM

Michael Libby wrote:

> The site you're hitting makes heavy use of redirects (and not really
> for their intended purpose). What this means is that you submit your
> request for a given URL and the server responds with a redirect and a
> new URL. If you are working in a browser, your browser automatically
> requests that URL, and the server again responds with a redirect and a
> new URL. Again, a web browser handles requesting that next URL
> automatically. This URL is the actual results page with the data you
> want. It's the web site making you jump through hoops to get where you
> want to go.

Ah, I see.

You appear, by my estimation, to rock.

: Daniel :
--
Posted via http://www.ruby-....

Daniel Miessler

11/5/2008 6:12:00 PM

Avdi Grimm wrote:
> You may want to look into using Mechanize rather than straight-up
> Net::HTTP.

Mechanize for Ruby? Interesting. I didn't know Ruby had an
implementation. Thanks, Avdi.
--
Posted via http://www.ruby-....

Uwe Petschke

11/21/2008 9:26:00 AM

Daniel Miessler wrote:
> The problem is that I keep getting a redirect
> (#<Net::HTTPMovedPermanently:0xb7c35ffc>), which doesn't happen when I
> make the request from a regular browser.
>
> So I sniffed the regular request with wireshark, and a browser sends a
> bunch of additional headers when it makes the request. Cookies,
> referrer, etc.
>
> Are any of these headers more necessary than others, and is there a
> preferred way to send the headers using Ruby?
>

We have had similar issues where we didn't see a redirect when sniffing
the browser but it happened for our code. The reason was HTTP/1.1.
With HTTP/1.1 it is required to specify the host you expect to be
talking with (as more than one virtual host may be serviced by one
server):
GET / HTTP/1.1
Host: www.apache.org
(see http://www.apacheweek.com/featu... for reference)

Hope that helps in avoiding the redirect ;-)

Uwe
--
Posted via http://www.ruby-....

Ray O'Hara

8/1/2010 10:51:00 PM

"Sweetbac" <sweetbacz@scbglobal.net> wrote in message
news:i34tgt$onp$1@news.eternal-september.org...
>
> "Ray OHara" <raymond-ohara@hotmail.com> wrote in message
>
>> and Mamie Van Doren the original "sweater girl".
>
> I thought you were the original "sweater girl"?
>

you're the one who lives in SF and is known as sweetbuns or bac or whatever.

comp.lang.ruby

What's the Best Way to Mimic an HTTP Request?

Daniel Miessler

James Herdman

Hassan Schroeder

Michael Libby

Daniel Miessler

Michael Libby

Avdi Grimm

Daniel Miessler

Daniel Miessler

Uwe Petschke

Ray O'Hara

x Login to ForumsZone