[lnkForumImage]
TotalShareware - Download Free Software
Confronta i prezzi di migliaia di prodotti.
Asp Forum
 Home | Login | Register | Search 


 

Forums >

comp.lang.python

HTTP POST uploading large files

Wolfgang Draxinger

1/19/2008 11:19:00 PM

I'm thinking about writing a script to upload videos to sites
like YouTube or Google Video, which is usually done by a HTTP
POST.

The problem is, that videos, by nature are rather big files,
however urllib2 wants it's Request objects being prepared
beforehand, which would mean to first load the whole file to
memory.

I looked into pycURL, knowing that cURL can POST send files
directily from the file system, however pycURL doesn't expose
the neccesary functions yet.

Am I just blind for some urllib2/httplib feature, or some other
library? Or do I really have to fiddle around with sockets
myself (I hope not...).

Thanks in advance

Wolfgang Draxinger
--
E-Mail address works, Jabber: hexarith@jabber.org, ICQ: 134682867
4 Answers

Gabriel Genellina

1/20/2008 2:39:00 AM

0

En Sat, 19 Jan 2008 21:19:24 -0200, Wolfgang Draxinger
<wdraxinger@darkstargames.de> escribi�:

> I'm thinking about writing a script to upload videos to sites
> like YouTube or Google Video, which is usually done by a HTTP
> POST.
>
> The problem is, that videos, by nature are rather big files,
> however urllib2 wants it's Request objects being prepared
> beforehand, which would mean to first load the whole file to
> memory.
>
> I looked into pycURL, knowing that cURL can POST send files
> directily from the file system, however pycURL doesn't expose
> the neccesary functions yet.
>
> Am I just blind for some urllib2/httplib feature, or some other
> library? Or do I really have to fiddle around with sockets
> myself (I hope not...).

I'm afraid urllib2 currently doesn't handle this. Neither the lower layer,
httplib. HTTPConnection should be upgraded to handle 'Transfer-Encoding:
chunked', by example. (Chunked responses are handled correctly, but a
request cannot be chunked)

A Q&D approach would be to patch httplib.HTTPConnection.send, to accept a
file or file-like argument. Around line 707, instead of
self.sock.sendall(str):

if hasattr(str, 'read'):
BUFSIZE = 4*1024
while True:
block = str.read(BUFSIZE)
if not block: break
self.sock.sendall(block)
else:
self.sock.sendall(str)

and ensure the Content-Length header is already set, so no attempt is made
to compute len(str)

--
Gabriel Genellina

Brian Smith

1/20/2008 4:09:00 PM

0

Wolfgang Draxinger wrote:
> The problem is, that videos, by nature are rather big files,
> however urllib2 wants it's Request objects being prepared
> beforehand, which would mean to first load the whole file to memory.

Try using mmap. Here is some untested code:

map = mmap(file.fileno(), len(file), access=ACCESS_READ)
try:
data = mmap.read()
request = Request(url, data, headers)
...
finally:
map.close()


- Brian

Paul Rubin

1/20/2008 5:13:00 PM

0

Wolfgang Draxinger <wdraxinger@darkstargames.de> writes:
> Am I just blind for some urllib2/httplib feature, or some other
> library? Or do I really have to fiddle around with sockets
> myself (I hope not...).

I did something like that by just opening a socket and writing the
stuff with socket.sendall. It's only about 5 lines of code and it's
pretty straightforward.

Wolfgang Draxinger

1/20/2008 6:10:00 PM

0

Paul Rubin wrote:

> Wolfgang Draxinger <wdraxinger@darkstargames.de> writes:
>> Am I just blind for some urllib2/httplib feature, or some
>> other library? Or do I really have to fiddle around with
>> sockets myself (I hope not...).
>
> I did something like that by just opening a socket and writing
> the
> stuff with socket.sendall. It's only about 5 lines of code and
> it's pretty straightforward.

Well, for YouTube you've to fiddle around with cookies,
form/multipart data and stuff like that. It's a bit more than
just opening a socket, there's some serious HTTP going on.

However I found a solution: The curl program, that comes with
libcurl and can be found on most *nix systems allows to do
pretty sophisticated HTTP requests, among them also sending
files by POST. So instead of using urllib2 or sockets from
Python, now my program just generates the appropriate calls to
curl, provides the in memory storage of cookies and does the
neccesary HTML parsing*.

Wolfgang Draxinger

*) YouTube uploads videos in a two part process: First you set
the various video options, in return you get a form with some
hidden input fields, some of them providing a handle to the
already sent video information. That data has to be extracted
from the form and be put into the POST that also transfers the
video file.

--
E-Mail address works, Jabber: hexarith@jabber.org, ICQ: 134682867