[lnkForumImage]
TotalShareware - Download Free Software
Confronta i prezzi di migliaia di prodotti.
Asp Forum
 Home | Login | Register | Search 


 

Forums >

comp.lang.ruby

Problems with mechanize and fields embedded in tables

Todd A. Jacobs

10/30/2007 7:00:00 PM

I'm working with the following versions:

ruby 1.8.2
libwww-mechanize-ruby 0.6.10

and have run across an odd problem. One site that I'm trying to scrape
has started embedding form fields inside of tables, and mechanize no
longer recognizes them as fields.

The fields are there in the HTML code, but aren't accessible to
mechanize. I've tried a couple of work-arounds, but field_add! doesn't
seem to support adding check boxes or file upload fields (is there
another way to add them explicitly?), and I can't see any other way to
find those embedded fields.

If this is a bug in mechanize, how do I report it? If it's a bug in the
coder, what can I do to resolve the problem?

--
"Oh, look: rocks!"
-- Doctor Who, "Destiny of the Daleks"
6 Answers

Todd A. Jacobs

10/30/2007 8:32:00 PM

0

In the course of debugging, I tried this:

require 'mechanize'
agent = WWW::Mechanize.new
selection='http://seeker.dice.com/jobsearch/servlet/JobSearch?op=302&dockey=xml/4/6/46ab274a1ab667a09cd9aac11c6bef37@endeca...
page = agent.get(selection)
page = agent.click page.links.text('Click Here to Apply')
reply_form = page.forms.with.name('APPLICATION_FORM').first
pp reply_form

As you can see, the SEEKER_CC checkbox and RESUME_FILE filename fields
aren't showing up, but they ARE in the HTML. I suppose it helps if you
have access to the data sources and the methodology of the (error-prone)
programmer that's accessing them. :)

--
"Oh, look: rocks!"
-- Doctor Who, "Destiny of the Daleks"

Todd A. Jacobs

10/30/2007 8:47:00 PM

0

On Tue, Oct 30, 2007 at 12:00:03PM -0700, Todd A. Jacobs wrote:

> mechanize. I've tried a couple of work-arounds, but field_add! doesn't
> seem to support adding check boxes or file upload fields (is there

I've managed to add the fields explicitly:

carbon = WWW::Mechanize::RadioButton.new('SEEKER_CC', nil, true, reply_form)
upload_field = WWW::Mechanize::FileUpload.new('RESUME_FILE', 'foo')

reply_form.checkboxes.push(carbon)
reply_form.file_uploads.push(upload_field)

but this seems kind of kludgy. I'm still looking for a better way.

--
"Oh, look: rocks!"
-- Doctor Who, "Destiny of the Daleks"

7stud --

10/30/2007 9:14:00 PM

0

Todd A. Jacobs wrote:
>
> aren't showing up, but they ARE in the HTML. I suppose it helps if you
> have access to the data sources and the methodology of the (error-prone)
> programmer that's accessing them. :)

It sounds like javascript may be adding the fields you want. When you
load the page in a browser, the browser's javascript software kicks in
and can add html to the page. However, when you grab a page with
mechanize, you get the pre-javascript page, and as far as I know,
mechanize does not have the ability to interpret the javascript and make
changes to the html based on what the javascript says to do.

Well designed websites design their pages so that users without
javascript enabled are served simpler pages that have all the required
html for forms and the necessary html to navigate around the website.
The trick is getting the server to send you those pages. You have to be
good with html and js and dig around a bit to figure it out. Or, if the
site has a lot of traffic, there might be an article on how to do it.

--
Posted via http://www.ruby-....

Todd A. Jacobs

10/31/2007 4:24:00 AM

0

On Wed, Oct 31, 2007 at 06:14:01AM +0900, 7stud -- wrote:

> It sounds like javascript may be adding the fields you want. When you

Nope. You can see the fields in lynx, so it's definitely not
client-side.

--
"Oh, look: rocks!"
-- Doctor Who, "Destiny of the Daleks"

Daniel Brumbaugh Keeney

10/31/2007 7:17:00 AM

0

On 10/30/07, 7stud -- <bbxx789_05ss@yahoo.com> wrote:
> Well designed websites design their pages so that users without
> javascript enabled are served simpler pages that have all the required
> html for forms and the necessary html to navigate around the website.
> The trick is getting the server to send you those pages. You have to be
> good with html and js and dig around a bit to figure it out.

Every browser will let you turn off JavaScript, that's the easy way to
get served a simple version, which is probably what you want.

-------------------------------------------
Daniel Brumbaugh Keeney
Devi Web Development
Devi.WebMaster@gMail.com
-------------------------------------------

Stewart

2/25/2012 3:34:00 PM

0


"The Revd" <peeling@degenerate.Grik> wrote in message
news:m6nhk79dl9jcovhtce30ete00vbd2eb8e8@4ax.com...
> On Sat, 25 Feb 2012 07:22:30 -0500, "Stewfart" <got_anus@gmail.com>
> wrote:
>
>>
>>"The Revd" <peeling@degenerate.Grik> wrote in message
>>news:0nrek7dvuo5u3gmqi6ddp0saaaeq6e2lt6@4ax.com...
>>> On Thu, 23 Feb 2012 19:41:21 -0500, "Stewfart"
>>> <got_anus@gmail.com>
>>> wrote:
>>>
>>>>
>>>>"The Revd" <peeling@degenerate.Grik> wrote in message
>>>>news:6497k7520oe1kohd6hull3l06ro8l78v9t@4ax.com...
>>>>> On Mon, 20 Feb 2012 19:32:39 -0500, "Stewfart"
>>>>> <got_anus@gmail.com>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>>"The Revd" <peeling@degenerate.Grik> wrote in message
>>>>>>news:k1a4k79l9beq1aefmerujviahbedeocuk7@4ax.com...
>>>>>>> On Sun, 19 Feb 2012 16:31:12 -0500, "Stewfart"
>>>>>>> <got_anus@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>"The Revd" <peeling@degenerate.Grik> wrote in message
>>>>>>>>news:50p2k7phh35hq0cbv0hmkaoji2brsun1do@4ax.com...
>>>>>>>>> On Sun, 19 Feb 2012 14:07:30 -0500, "Stewfart"
>>>>>>>>> <got_anus@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>"The Revd" <peeling@degenerate.Grik> wrote in message
>>>>>>>>>>news:sqd2k79mvgl8ri9mf7capeg9eksot4q2hf@4ax.com...
>>>>>>>>>>> On Sun, 19 Feb 2012 08:04:04 -0500, "Stewfart"
>>>>>>>>>>> <got_anus@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>"The Revd" <peeling@degenerate.Grik> wrote in message
>>>>>>>>>>>>news:no50k71k4d6i82ildlm84rpmi4ucj135ta@4ax.com...
>>>>>>>>>>>>> On Sat, 18 Feb 2012 12:50:46 -0500, "Stewfart"
>>>>>>>>>>>>> <got_anus@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>"The Revd" <peeling@degenerate.Grik> wrote in message
>>>>>>>>>>>>>>news:ukmvj7lrs4tei54hljblbu95sjnv7lmovu@4ax.com...
>>>>>>>>>>>>>>> On Sat, 18 Feb 2012 11:35:35 -0500, "Stewfart"
>>>>>>>>>>>>>>> <got_anus@gmail.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>"The Revd" <peeling@degenerate.Grik> wrote in message
>>>>>>>>>>>>>>>>news:l8gvj75an1ojiucgrtanlpnvp8f91odvsf@4ax.com...
>>>>>>>>>>>>>>>>> On Sat, 18 Feb 2012 09:29:23 -0500, "Stewfart"
>>>>>>>>>>>>>>>>> <got_anus@gmail.com>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>"Student" <student@yahoo.com> wrote in message
>>>>>>>>>>>>>>>>>>news:dsednSMc-893N6LSnZ2dnUVZ_tmdnZ2d@giganews.com...
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Jews are mentally screwed up from birth when their
>>>>>>>>>>>>>>>>>>> parents
>>>>>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>>> teaching them
>>>>>>>>>>>>>>>>>>> they are "chosen people" Recent survery in Israel
>>>>>>>>>>>>>>>>>>> reveals
>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>> about 70% of
>>>>>>>>>>>>>>>>>>> Israeli believe they are "chosen"
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> http://www.haaretz.com/jewish-world/survey-record-number-of-israeli-jews-believe-in-go...
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> When the rest of the world does not treat them as
>>>>>>>>>>>>>>>>>>> gods
>>>>>>>>>>>>>>>>>>> gift
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>> planet earth
>>>>>>>>>>>>>>>>>>> they cry foul. It is this attitude of "chosen"
>>>>>>>>>>>>>>>>>>> belief
>>>>>>>>>>>>>>>>>>> system
>>>>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>>>> has made Jews
>>>>>>>>>>>>>>>>>>> so arrogant and contributed to the Holocaust when
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> Christians
>>>>>>>>>>>>>>>>>>> killed 50% of
>>>>>>>>>>>>>>>>>>> them
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Norman Finklestein is no exception to this tribal
>>>>>>>>>>>>>>>>>>> racist
>>>>>>>>>>>>>>>>>>> belief
>>>>>>>>>>>>>>>>>>> system of "chosen people"
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I cannot stomach the constant whining of the Jews.
>>>>>>>>>>>>>>>>>>> They
>>>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>>> suffer
>>>>>>>>>>>>>>>>>>> from "mental
>>>>>>>>>>>>>>>>>>> fibromyalgia" .
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Student
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>Then quit whining like the little bitch that you are
>>>>>>>>>>>>>>>>>>and
>>>>>>>>>>>>>>>>>>do
>>>>>>>>>>>>>>>>>>something
>>>>>>>>>>>>>>>>>>about it, spanky.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> How about another 'holocaust'?T, jew towelhead?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>Go for it, bro rebby. Make my day.....though I know
>>>>>>>>>>>>>>>>you
>>>>>>>>>>>>>>>>won't.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> You won't like it when it happens, jew towelhead.
>>>>>>>>>>>>>>> But,
>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>> survive, you can always hope for reparations.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>Your words are big, but your actions are small.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Reparations, you want already?
>>>>>>>>>>>>
>>>>>>>>>>>>Just looking to see if you would act on your beliefs or
>>>>>>>>>>>>continue
>>>>>>>>>>>>to
>>>>>>>>>>>>bellow hot air, bro rebby. Hot air it is.
>>>>>>>>>>>
>>>>>>>>>>> You're looking in the wrong place, jew towelhead.
>>>>>>>>>>
>>>>>>>>>>No, bro reb.....there is no place to look, hot air
>>>>>>>>>>dissipates
>>>>>>>>>>far
>>>>>>>>>>too
>>>>>>>>>>quickly.
>>>>>>>>>
>>>>>>>>> There is no hot air, jew towelhead.
>>>>>>>>
>>>>>>>>There is a glut of hot air, as it is all you have to offer.
>>>>>>>
>>>>>>> No, it isn't.
>>>>>>
>>>>>>Yes, it is.
>>>>>
>>>>> No way.
>>>>
>>>>Yes way.
>>>
>>> No fucking way.
>>
>>Yes way.
>
> No way, Jos?.

Yes way.

>
>>>
>>>>>
>>>>>>> You have absolutely nothing to offer: but if you at
>>>>>>> least had hot air to offer, you might be of some use to the
>>>>>>> human
>>>>>>> race.
>>>>>>
>>>>>>Hot air is what you live for, bro reb. It reverberates through
>>>>>>your
>>>>>>anal pores.
>>>>>
>>>>> You are full of shit. It comes out of all you orifices.
>>>>
>>>>Nope, I said it first...you need to be more creative. Of course,
>>>>you
>>>>could just give up and drop out of a thread. Again.
>>>
>>> YOU are the one who constantly drops out of threads with your jew
>>> tail
>>> between your jew legs.
>>
>>Google tells a much different story, bro reb.
>
> No, it doesn't. It confirms what I've said.

No, it confirms quite the opposite of what you say. You haven't even
looked to find out, as that would be a cite (and a site), and you
don't do them. But at least now you know what they are.

>
>> I'd give you a cite for
>>it, but you don't do cites. At least you now know what they are
>>since
>>you've been educated.
>
> WE do the educating, jew towelhead.

Far from it, bro reb. You are not qualified to educate anyone...there
are cites for this, but you don't do them.