[lnkForumImage]
TotalShareware - Download Free Software
Confronta i prezzi di migliaia di prodotti.
Asp Forum
 Home | Login | Register | Search 


 

Forums >

comp.lang.ruby

grabbing html from pages that require login

Adam Akhtar

8/19/2008 12:21:00 PM

Hi ive searched the forums already but couldnt come up with any concrete
information.

Ive written a script that accesses certain pages on ebay and scapes
information. Its using open-uri to grab html and just standard regexps
to grab the data i want. When building the script I used ebay webpages I
saved to my harddisk. All worked fine.

However when the script attmepts to access the actual online webpages I
get an "file or directory not found ..." error.
What I think is happening is that ebay is automatically redirecting the
script to the user login. Even if im already logged in via firefox i
guess the fact that the request is coming from something else outside of
firefox makes ebay suspect its from another location and thus for
security reasons requests the user to reenter their detials. Im assuming
this is to do with cookies???

So how do get round this issue? Is there a way to get avoid this login
requirement or somehow providing my script with the info it needs to
login and continue from there?
--
Posted via http://www.ruby-....
4 Answers

Rob Biedenharn

8/19/2008 4:19:00 PM

0

You might be able to use httpclient for this. It manages a session
much like a browser might and can track cookies that the site might
send. I was able to use this for doing some pretty ugly manipulation
of some pop-up windows that was clearly intended to be used only by a
human with a browser.

gem install httpclient

-Rob

On Aug 19, 2008, at 8:20 AM, Adam Akhtar wrote:

> Hi ive searched the forums already but couldnt come up with any
> concrete
> information.
>
> Ive written a script that accesses certain pages on ebay and scapes
> information. Its using open-uri to grab html and just standard regexps
> to grab the data i want. When building the script I used ebay
> webpages I
> saved to my harddisk. All worked fine.
>
> However when the script attmepts to access the actual online
> webpages I
> get an "file or directory not found ..." error.
> What I think is happening is that ebay is automatically redirecting
> the
> script to the user login. Even if im already logged in via firefox i
> guess the fact that the request is coming from something else
> outside of
> firefox makes ebay suspect its from another location and thus for
> security reasons requests the user to reenter their detials. Im
> assuming
> this is to do with cookies???
>
> So how do get round this issue? Is there a way to get avoid this login
> requirement or somehow providing my script with the info it needs to
> login and continue from there?
> --

Rob Biedenharn http://agileconsult...
Rob@AgileConsultingLLC.com

Adam Akhtar

8/20/2008 3:07:00 AM

0


Thank you for that rob, ill look into it now.

--
Posted via http://www.ruby-....

brabuhr

8/20/2008 3:19:00 AM

0

On Tue, Aug 19, 2008 at 12:19 PM, Rob Biedenharn
<Rob@agileconsultingllc.com> wrote:
> You might be able to use httpclient for this. It manages a session much
> like a browser might and can track cookies that the site might send. I was
> able to use this for doing some pretty ugly manipulation of some pop-up
> windows that was clearly intended to be used only by a human with a browser.
>
> gem install httpclient

You may also wish to look at: mechanize, scrubyt, firewatir, etc.

reuben doetsch

8/20/2008 1:31:00 PM

0

[Note: parts of this message were removed to make it a legal post.]

Use mechanize, it makes logins very easy

On Tue, Aug 19, 2008 at 10:19 PM, <brabuhr@gmail.com> wrote:

> On Tue, Aug 19, 2008 at 12:19 PM, Rob Biedenharn
> <Rob@agileconsultingllc.com> wrote:
> > You might be able to use httpclient for this. It manages a session much
> > like a browser might and can track cookies that the site might send. I
> was
> > able to use this for doing some pretty ugly manipulation of some pop-up
> > windows that was clearly intended to be used only by a human with a
> browser.
> >
> > gem install httpclient
>
> You may also wish to look at: mechanize, scrubyt, firewatir, etc.
>
>