Florian Frank
7/16/2006 2:15:00 AM
Luke Burton wrote:
>I just think that net/http's abysmal speed is somewhat anomalous. I am
>confident there is a simple explanation - maybe some tight loop in there
>doing something not particularly clever - but I haven't had the time as
>yet to dive right into net/http and find the reason. And I haven't had
>much success with Ruby profilers either. More suggestions welcome here!
>
>
I think you're right. In the GET case (for both open-uri and net/http) I
have found a possibility to speed up the download. The method
BufferedIO#rbuf_fill in net/protocol.rb uses a fixed buffer size of
1024. I changed this to 8192:
def rbuf_fill
timeout(@read_timeout) {
@rbuf << @io.sysread(8192)
}
end
This lead to almost the same speed I could achieve with the use of
direct sockets.
>I have gone ahead and changed the critical section of my code to use
>TCPSocket instead. That solved the HTTP GET problem, but I still
>struggle with PUTs:
>
>file = File.open(resultFile, "r")
>http = Net::HTTP.new(@uri.host, @uri.port)
>http.put("/put/" + URI.escape(File.basename(resultFile)), file.read)
>
>Now that's not real pleasant because it relies on snarfing the whole
>file into memory first. I would have liked to do something like:
>
>http.put("/put/" + URI.escape(File.basename(resultFile))) do
>|datasocket|
> while file.eof? == false
> datasocket.write(file.read(4096))
> end
>end
>
>
You can do this by setting Put#body_stream= to your IO object. The only
problem is, that the related method also sets a very small buffer size:
def send_request_with_body_stream(sock, ver, path, f)
raise ArgumentError, "Content-Length not given and
Transfer-Encoding is not `chunked'" unless content_length() or chunked?
unless content_type()
warn 'net/http: warning: Content-Type did not set; using
application/x-www-form-urlencoded' if $VERBOSE
set_content_type 'application/x-www-form-urlencoded'
end
write_header sock, ver, path
if chunked?
while s = f.read(1024)
sock.write(sprintf("%x\r\n", s.length) << s << "\r\n")
end
sock.write "0\r\n\r\n"
else
while s = f.read(1024)
sock.write s
end
end
end
I think this is rather unfortunate. It would be better, if those methods
would use higher buffer values and/or make them tweakable if necessary.