Asp Forum - Posting over https

Andrew Stewart

10/9/2006 3:04:00 PM

Hello Rubyists,

I am struggling to post data over https and would be grateful for any
help.

I know the server works because I can hit it with curl and get the
response I expect:

curl -d <url-encoded form data> url

When I try to do the same from my Ruby code the server doesn't like it.

The URL to hit looks like:

https://some.host.com/foo/bar.asp?service=s...

My code looks like:

require 'net/https'
require 'uri'

url = <as above>
uri = URI.parse(url)
request = Net::HTTP.new(uri.host, uri.port)
request.use_ssl = true
request.verify_mode = OpenSSL::SSL::VERIFY_NONE
data = { 'key1' => 'value1',
'key2' => 'value2',
'keyn' => 'valuen' }.map {|k,v| "#{URI.escape
(k.to_s)}=#{URI.escape(v.to_s)}"}.join('&')
response = request.post("#{uri.path}?#{uri.query}", data)

When I tried this with curl, I used the value of data as my url-
encoded form data and everything was fine.

Any ideas?

Thanks in advance,
Andy Stewart

5 Answers

Aaron Patterson

10/9/2006 3:53:00 PM

On Tue, Oct 10, 2006 at 12:04:26AM +0900, Andrew Stewart wrote:
> Hello Rubyists,
>
> I am struggling to post data over https and would be grateful for any
> help.
>
> I know the server works because I can hit it with curl and get the
> response I expect:
>
> curl -d <url-encoded form data> url
>
> When I try to do the same from my Ruby code the server doesn't like it.
>
> The URL to hit looks like:
>
> https://some.host.com/foo/bar.asp?service=s...
>
> My code looks like:
>
> require 'net/https'
> require 'uri'
>
> url = <as above>
> uri = URI.parse(url)
> request = Net::HTTP.new(uri.host, uri.port)
> request.use_ssl = true
> request.verify_mode = OpenSSL::SSL::VERIFY_NONE
> data = { 'key1' => 'value1',
> 'key2' => 'value2',
> 'keyn' => 'valuen' }.map {|k,v| "#{URI.escape
> (k.to_s)}=#{URI.escape(v.to_s)}"}.join('&')
> response = request.post("#{uri.path}?#{uri.query}", data)
>
> When I tried this with curl, I used the value of data as my url-
> encoded form data and everything was fine.
>
> Any ideas?

Try using Mechanize. You can post forms over https with it, and you
won't have to deal with encoding query parameters or setting up
Net::HTTP.

http://mechanize.ruby...

--Aaron

--
Aaron Patterson
http://tenderlovem...

Andrew Stewart

10/9/2006 4:54:00 PM

Hi,

>> I am struggling to post data over https and would be grateful for any
>> help.

<snip/>

> Try using Mechanize. You can post forms over https with it, and you
> won't have to deal with encoding query parameters or setting up
> Net::HTTP.

OK, but I would still like to understand what's wrong with my code so
I can learn and improve.

Thanks anyway for the pointer,

Andy Stewart

Richard Conroy

10/9/2006 5:02:00 PM

On 10/9/06, Andrew Stewart <boss@airbladesoftware.com> wrote:
> Hi,
>
> >> I am struggling to post data over https and would be grateful for any
> >> help.
>
> <snip/>
>
> > Try using Mechanize. You can post forms over https with it, and you
> > won't have to deal with encoding query parameters or setting up
> > Net::HTTP.
>
> OK, but I would still like to understand what's wrong with my code so
> I can learn and improve.

Its admirable, but I think Net::HTTP is a bit too low level, and requires a
bit too much HTTP knowledge for many applications. Mechanize seems
to be the business, and possibly Hpricot might also have similar
functionality.

I found Net::HTTP a bit too frustrating to use and the lack of good examples
bothered me a lot. It seems that the real work along these lines (programmatic
HTTP/HTML processing) is done with Mechanize & Hpricot.

Ara.T.Howard

10/9/2006 5:52:00 PM

Andrew Stewart

10/10/2006 1:17:00 PM

Hello again,

> fortytwo :/var/www/html > ruby post.rb https://fortytwo.me...
> form.cgi
> ---
> key1:
> - value1
> key2:
> - value2
> keyn:
> - valuen

Many thanks for the pointers to Mechanize, Hpricot and this fully-
fledged code sample. I do appreciate your taking the time to
respond, and so quickly.

Adapting my code in the light of Ara's code above, with
'self.set_debug_output $stderr' added to the HTTPS initialize method,
I compared the client-server conversation with curl's, which I knew
worked. This led me to discover that the server expected the content-
type header to be set to 'application/x-www-form-urlencoded'.

For the record, here's the code which works with the server I am
talking to (Protx's payment gateway). Now that I have it working,
I'll take Aaron and Richard's advice and look into replacing it with
higher-level code using Mechanize.

require 'net/https'
require 'uri'

class HTTPS < Net::HTTP
def initialize *a, &b
super
self.use_ssl = true
self.verify_mode = OpenSSL::SSL::VERIFY_NONE
#self.set_debug_output $stderr
end
end

url = 'https://some.host.com/foo/bar.asp?service=somese...
uri = URI.parse url
data = {
'key1' => 'value1',
'key2' => 'value2',
'keyn' => 'valuen'
}
e = lambda {|x| URI.escape x}
q = lambda {|h| h.map {|k,v| [e[k], e[v]].join '='}.join('&')}
form = q[data]
HTTPS.start(uri.host, uri.port) do |https|
response = https.post("#{uri.path}?#{uri.query}", form, {'content-
type' => 'application/x-www-form-urlencoded'})
puts response.body
end

Regards,
Andy Stewart

comp.lang.ruby

Posting over https

Andrew Stewart

Aaron Patterson

Andrew Stewart

Richard Conroy

Ara.T.Howard

Andrew Stewart

x Login to ForumsZone