[lnkForumImage]
TotalShareware - Download Free Software
Confronta i prezzi di migliaia di prodotti.
Asp Forum
 Home | Login | Register | Search 


 

Forums >

comp.lang.ruby

Bounty: $250 - Successfully log in to AOL Webmail with Mechanize

dave.myron@contentfree.com

10/30/2007 7:58:00 PM

We're about to release a gem called Blackbook that will automate
scraping contacts from various services. We have Gmail, Hotmail and
YahooMail all done - they were easy - but AOL Webmail (http://
webmail.aol.com) has stumped us.

After countless hours comparing LiveHTTPHeader captures of successful
logins using Firefox (with meta redirects and javascript disabled)
with logs of unsuccessful sessions using Mechanize, we're finally
putting a bounty on AOL's head.

Comparing the Firefox session with Mechanize's shows they're nearly
identical (seems like all but the Accepts header matches) until the
very last request: Firefox is authenticated and Mechanize is diverted
to an error page.

We're offering a $250 bounty to the first person to successfully log
in to AOL Webmail with Mechanize.

If you'd like a starting point (which might not help much since it
ends up at an error page), here's our code:

=============

require 'rubygems'
require 'mechanize'

agent = WWW::Mechanize.new
agent.user_agent_alias = 'Windows IE 6'

page = agent.get( 'http://webmail.ao... )

page = agent.get( page.body.scan(/snsRedir\("(https:[^"]
+)/).first.first )

form = page.forms.name('AOLLoginForm').first
form.loginId = '<your username>'
form.password = '<your password>'
page = agent.submit(form, form.buttons.first)

page = agent.get( page.body.scan(/onLoad="checkError[^\)]
+/).first.scan(/'([^']+)'/).last.first )

=============

That mimics what we can do in Firefox (cutting and pasting where we
use #scan in the code) and the last request should authenticate us
(with several cookies including one named Auth) but we get redirected
to an error page.

Good luck. In addition to your bounty, your work will also net the
community another gem.

Dave
11 Answers

Shanti Braford

10/31/2007 1:11:00 AM

0

Hi Dave,

I don't think I can help you re: Mechanize and AOL Webmail (at the
moment at least).

Just wanted to make sure you've seen this:
http://rubyforge.org/projects...

... in case any pooling of efforts would help. Looking forward to
seeing your results!

Cheers,

- Shanti

--
Posted via http://www.ruby-....

Peter Szinek

10/31/2007 8:35:00 AM

0

dave.myron@contentfree.com wrote:
> We're about to release a gem called Blackbook that will automate
> scraping contacts from various services. We have Gmail, Hotmail and
> YahooMail all done - they were easy - but AOL Webmail (http://
> webmail.aol.com) has stumped us.

Do you *absolutely* need mechanize?

Cheers,
Peter
___
http://www.rubyra...
http://s...

dave.myron@contentfree.com

10/31/2007 6:57:00 PM

0

> Just wanted to make sure you've seen this:http://rubyforge.org/projects...

We hadn't - and looking now I'm not sure how we missed it. We're doing
things a little different (we plan on having different exporters too
like csv, xml, etc) so I'm not sure how much merging could happen (but
I'll check in with them). Unfortunately, they don't have AOL figured
out either.

It's weird that these PHP scripts (using curl) seem to be able to log
in fine but we can't (and it seems MOG couldn't either).

dave.myron@contentfree.com

10/31/2007 7:28:00 PM

0

> Do you *absolutely* need mechanize?

It's helpful and the other completed importers (Gmail, Yahoo, Hotmail)
use it already. To keep dependencies down I'd *like* it to use
Mechanize, but if you've got something using scRUBYt then I'm sure it
could be translated (or at least help highlight where Mechanize is
falling down).

Dave

Konrad Meyer

10/31/2007 11:40:00 PM

0

Quoth dave.myron@contentfree.com:
> > Just wanted to make sure you've seen
this:http://rubyforge.org/projects...
>
> We hadn't - and looking now I'm not sure how we missed it. We're doing
> things a little different (we plan on having different exporters too
> like csv, xml, etc) so I'm not sure how much merging could happen (but
> I'll check in with them). Unfortunately, they don't have AOL figured
> out either.
>
> It's weird that these PHP scripts (using curl) seem to be able to log
> in fine but we can't (and it seems MOG couldn't either).

Could AOL be rejecting you based on UserAgent?

--
Konrad Meyer <konrad@tylerc.org> http://konrad.sobertil...

yermej

11/1/2007 1:54:00 AM

0

On Oct 31, 6:39 pm, Konrad Meyer <kon...@tylerc.org> wrote:
> Quoth dave.my...@contentfree.com:> > Just wanted to make sure you've seen
>
> this:http://rubyforge.org/projects...
>
>
>
> > We hadn't - and looking now I'm not sure how we missed it. We're doing
> > things a little different (we plan on having different exporters too
> > like csv, xml, etc) so I'm not sure how much merging could happen (but
> > I'll check in with them). Unfortunately, they don't have AOL figured
> > out either.
>
> > It's weird that these PHP scripts (using curl) seem to be able to log
> > in fine but we can't (and it seems MOG couldn't either).
>
> Could AOL be rejecting you based on UserAgent?
>
> --
> Konrad Meyer <kon...@tylerc.org>http://konrad.sobertil...
>
> signature.asc
> 1KDownload

This line

agent.user_agent_alias = 'Windows IE 6'

in the sample code results in a UserAgent that matches IE 6.

mortee

11/1/2007 2:24:00 AM

0

dave.myron@contentfree.com

11/1/2007 5:30:00 AM

0

> AOL webmail using Mechanize, to the point where it greets you with "Hi,
> <username>". If that's what you needed, then I have the solution for you.
>
> mortee

If you want to send it directly to me, that's fine. Also feel free to
paste it here.

Dave

mortee

11/1/2007 7:13:00 PM

0

dave.myron@contentfree.com wrote:
>> AOL webmail using Mechanize, to the point where it greets you with "Hi,
>> <username>". If that's what you needed, then I have the solution for you.
>>
>> mortee
>
> If you want to send it directly to me, that's fine. Also feel free to
> paste it here.

I've sent a private mail to you; have you received it?...
Sorry for the noise.

mortee

dave.myron@contentfree.com

11/2/2007 12:57:00 AM

0

You're the man, Marton! Consider the bounty won!

Marton found a bug in Mechanize's #to_absolute_uri method caused by
the use of #zip which caused already-escaped characters beyond the
first to be dropped. He replaced it with some SyncEnumerator hackery.
I'll let the Aaron Patterson know about the fix to Mechanize.

Good job Marton. We (and Blackbook and probably Mechanize, too)
appreciate your genius (and tolerance level for reading through that
nasty bit of Javascript to highlight the problem).

Dave