[lnkForumImage]
TotalShareware - Download Free Software
Confronta i prezzi di migliaia di prodotti.
Asp Forum
 Home | Login | Register | Search 


 

Forums >

comp.lang.ruby

Faking the refering page with Mechanize

Ehud Rosenberg

11/6/2007 12:02:00 AM

Hi,
I am trying to access a certain web page from my server, but I want the
referring page to be a different one.
It seems the the get method accepts both a url and a "ref" parameter
that is is supposed to be the referring page, but when I try to send
that url as follows:
agent.get(url_to_get, "http://www.mininova...)
I get the following error:
ndefined method `uri' for "http://www.mininova...:String.

Anyone knows what is the correct way to do this?

Thanks!
Ehud
--
Posted via http://www.ruby-....
14 Answers

Ehud Rosenberg

11/6/2007 12:24:00 AM

0

Konrad Meyer wrote:
> Quoth Ehud Rosenberg:
>> Anyone knows what is the correct way to do this?
>>
>> Thanks!
>> Ehud
>
> What you need to do is create a Mechanize::URI object (or something like
> that)
> from the string "http://www.mininova..., and call #get() with that.
>
> HTH,

Hi Konrad,
Thanks for the quick reply :)
I managed to get around that error by doing the following:
ref = WWW::Mechanize::Page.new(URI.parse("http://www.mininova...),
{'content-type'=>'text/html'})
doc = agent.get(url_to_get, ref)

Problem is, when I sniff the package it seems that the referring url is
ignored and it is still set to localhost:3000. I've also tried
specifying a nil page (as I've seen in the mechanize get code itself)
but to no avail...

Any help would be appreciated.
BTW - removing the referrer altogether would also work for, albeit it's
a less preferred option.

Thanks!

--
Posted via http://www.ruby-....

Ehud Rosenberg

11/7/2007 1:38:00 PM

0

Anyone? I'm really stumped here :)
Thanks...
--
Posted via http://www.ruby-....

Cédric Finance

11/7/2007 2:36:00 PM

0

Note: parts of this message were removed by the gateway to make it a legal Usenet post.

I tried it and it seems that the header is correct.

require 'mechanize'
require 'logger'
logger = Logger.new $stdout
agent = WWW::Mechanize.new
agent.log = logger
ref = WWW::Mechanize::Page.new("http://www.mininov...,{ 'content-type' =>
'text/html' })
agent.get url, ref

output from logger ==>
I, [2007-11-07T14:34:06.401219 #23290] INFO -- : Net::HTTP::Get: /
D, [2007-11-07T14:34:06.401585 #23290] DEBUG -- : request-header:
accept-languag
e => en-us,en;q0.5
D, [2007-11-07T14:34:06.401664 #23290] DEBUG -- : request-header: accept =>
*/*
D, [2007-11-07T14:34:06.401733 #23290] DEBUG -- : request-header:
accept-encoding => gzip,identity
D, [2007-11-07T14:34:06.401801 #23290] DEBUG -- : request-header: user-agent
=> WWW-Mechanize/0.6.5 (http://rubyforge.org/projects/...)
D, [2007-11-07T14:34:06.401869 #23290] DEBUG -- : request-header: referer =>
http://www.mi...


On Nov 7, 2007 2:37 PM, Ehud Rosenberg <ehudros@gmail.com> wrote:

> Anyone? I'm really stumped here :)
> Thanks...
> --
> Posted via http://www.ruby-....
>
>

Ehud Rosenberg

11/7/2007 5:22:00 PM

0

Wow, that's strange.
Thanks for taking the time to check this, I'll try and run it again to
see if I can figure out what im doing wrong.

Thanks :)
--
Posted via http://www.ruby-....

Ehud Rosenberg

11/8/2007 4:44:00 PM

0

In the following line:
ref =
WWW::Mechanize::Page.new("http://www.mininov...,{'content-type'=>'text/html'
})
the second apostrophe after mininova.org does not match the first one ("
vs. ').

When I try to run it with both sides contained by a double quote, i get
a message saying:
NoMethodError: undefined method `path' for
"http://www.mininova.org&qu...

That's why I used the URI.parse method before.
this is really strange, are you sure that's the exact syntax you used?

Thanks!
Ehud
--
Posted via http://www.ruby-....

Ehud Rosenberg

11/8/2007 6:24:00 PM

0

OK, I'm stupid :)
I don't need mechanize to fake the referring page at all, but rails
itself in a redirect_to statement.
The flow is as such:
a. mechanize retrieves a page and scrapes it for a url
b. a redirect_to to that url is sent back to the cient
c. he goes and fetches the url (only right now he gets redirected since
his referrer is wrong.

I am trying to do the following to set the referrer:
request.env['HTTP_REFERER'] = 'http://www.som...
redirect_to "www.site.com"

but when I sniff the headers, it's still set to my own site...

Any ideas?
Thanks!
--
Posted via http://www.ruby-....

Cédric Finance

11/8/2007 6:27:00 PM

0

Note: parts of this message were removed by the gateway to make it a legal Usenet post.

#irb
irb(main):001:0> require 'mechanize'
=> true
irb(main):002:0> require 'logger'
=> true
irb(main):003:0> logger = Logger.new $stdout ; ""
=> ""
irb(main):004:0> agent = WWW::Mechanize.new ; ""
=> ""
irb(main):005:0> agent.log = logger ; ""
=> ""
irb(main):006:0> ref = WWW::Mechanize:: Page.new("http://www.mininova...,{
'co
ntent-type' => 'text/html' }) ; ""
=> ""
irb(main):007:0> agent.get "http://google..., ref ; ""
I, [2007-11-08T19:25:06.144484 #21091] INFO -- : Net::HTTP::Get: /
D, [2007-11-08T19:25:06.144937 #21091] DEBUG -- : request-header:
accept-language => en-us,en;q0.5
D, [2007-11-08T19:25:06.145032 #21091] DEBUG -- : request-header: accept =>
*/*
D, [2007-11-08T19:25:06.145106 #21091] DEBUG -- : request-header:
accept-encoding => gzip,identity
D, [2007-11-08T19:25:06.145178 #21091] DEBUG -- : request-header: user-agent
=> WWW-Mechanize/0.6.5 (http://rubyforge.org/projects/...)
D, [2007-11-08T19:25:06.145253 #21091] DEBUG -- : request-header: referer =>
http://www.mi...


Which version of mechanize do you use?

On Nov 8, 2007 5:44 PM, Ehud Rosenberg <ehudros@gmail.com> wrote:

> In the following line:
> ref =
> WWW::Mechanize::Page.new("http://www.mi...',{'content-type'=<http://www.mi...%27,%7B%27content-type%27=>
> >'text/html'
> })
> the second apostrophe after mininova.org does not match the first one ("
> vs. ').
>
> When I try to run it with both sides contained by a double quote, i get
> a message saying:
> NoMethodError: undefined method `path' for
> "http://www.mininova...:String
>
> That's why I used the URI.parse method before.
> this is really strange, are you sure that's the exact syntax you used?
>
> Thanks!
> Ehud
> --
> Posted via http://www.ruby-....
>
>

Ehud Rosenberg

11/8/2007 6:31:00 PM

0

> Which version of mechanize do you use?
0.6.10.

But as it seems, I don't need to set the referring page with mechanize
but with rails itself...
--
Posted via http://www.ruby-....

Cédric Finance

11/8/2007 6:41:00 PM

0

Note: parts of this message were removed by the gateway to make it a legal Usenet post.

I don't think that you can do what you want this way, the referer is set by
your browser not by the website you are browsing.
(Moreover, you are changing the referer from the request, not from the
response). And I don't see why a server response
could have a referer header.
If you want to load a page with another referer from your rails app, try to
use some javascript to change the referer of the
browser or make an ajax page request with the right referer (I think it
should be possible to set the header from the ajax
request).

On Nov 8, 2007 7:23 PM, Ehud Rosenberg <ehudros@gmail.com> wrote:

> OK, I'm stupid :)
> I don't need mechanize to fake the referring page at all, but rails
> itself in a redirect_to statement.
> The flow is as such:
> a. mechanize retrieves a page and scrapes it for a url
> b. a redirect_to to that url is sent back to the cient
> c. he goes and fetches the url (only right now he gets redirected since
> his referrer is wrong.
>
> I am trying to do the following to set the referrer:
> request.env['HTTP_REFERER'] = 'http://www.som...
> redirect_to "www.site.com"
>
> but when I sniff the headers, it's still set to my own site...
>
> Any ideas?
> Thanks!
> --
> Posted via http://www.ruby-....
>
>

Ehud Rosenberg

11/8/2007 6:51:00 PM

0

Cédric Finance wrote:
> I don't think that you can do what you want this way, the referer is set
> by
> your browser not by the website you are browsing.
> (Moreover, you are changing the referer from the request, not from the
> response). And I don't see why a server response
> could have a referer header.
> If you want to load a page with another referer from your rails app, try
> to
> use some javascript to change the referer of the
> browser or make an ajax page request with the right referer (I think it
> should be possible to set the header from the ajax
> request).

Yes, it has finally dawned on me that what I'm trying to do is not
really possible this way. I doubt it can be done using ajax, though it's
work a bit of research.
Anyway, thanks for putting me on the right track.



--
Posted via http://www.ruby-....