[lnkForumImage]
TotalShareware - Download Free Software
Confronta i prezzi di migliaia di prodotti.
Asp Forum
 Home | Login | Register | Search 


 

Forums >

comp.lang.python

help on file storage for split multi part download

binaryj

3/6/2008 4:34:00 PM

HI everyone on this group!

i am want to write a split part downloading software for use in some
projects. (something like a download accelerator)

what i think i need is ( my brain is totally exausted at this moment
so pls ignore any typos i make)

storage class which can write the file splits that are currently being
downloaded to the disk. this is exactly what other download
accelerators do, i guess.

can this be done using the python file class?? i am pretty good at
handling file uploads (at server end) this is the first time i have to
think the other way round.

any code samples would be greatly appreciated.

thanks
binary-j
django web developer
5 Answers

Gabriel Genellina

3/6/2008 8:39:00 PM

0

En Thu, 06 Mar 2008 14:34:27 -0200, <coolman.guron@gmail.com> escribi�:

> storage class which can write the file splits that are currently being
> downloaded to the disk. this is exactly what other download
> accelerators do, i guess.
>
> can this be done using the python file class?? i am pretty good at
> handling file uploads (at server end) this is the first time i have to
> think the other way round.

Uh, unless I misundersand you, a standard file object is enough. First
create a file with the required size (open(...,'wb'), seek(n-1),
write(chr(0))). For each downloaded chunk you have to know its position in
the file; then just seek() and write() it.

--
Gabriel Genellina

binaryj

3/7/2008 6:17:00 AM

0

On Mar 7, 1:38 am, "Gabriel Genellina" <gagsl-...@yahoo.com.ar> wrote:
> En Thu, 06 Mar 2008 14:34:27 -0200, <coolman.gu...@gmail.com> escribi?:
>
> > storage class which can write the file splits that are currently being
> > downloaded to the disk. this is exactly what other download
> > accelerators do, i guess.
>
> > can this be done using the python file class?? i am pretty good at
> > handling file uploads (at server end) this is the first time i have to
> > think the other way round.
>
> Uh, unless I misundersand you, a standard file object is enough. First
> create a file with the required size (open(...,'wb'), seek(n-1),
> write(chr(0))). For each downloaded chunk you have to know its position in
> the file; then just seek() and write() it.
>
> --
> Gabriel Genellina

well yes its possible to use the file class that way.

BUT the thing thats going in my mind is thread safety. i plan to start
each part of the file download in a different thread. and then when
each thread had downloaded more than 100kb (or eof or boundary
reached) write the buffer to the disk. can this be achieved using
mutex ? i have never shared objects between threads.

is there a way to write this without using threads at all ???

Gabriel Genellina

3/7/2008 9:15:00 AM

0

En Fri, 07 Mar 2008 04:16:42 -0200, <coolman.guron@gmail.com> escribi�:
> On Mar 7, 1:38 am, "Gabriel Genellina" <gagsl-...@yahoo.com.ar> wrote:
>> En Thu, 06 Mar 2008 14:34:27 -0200, <coolman.gu...@gmail.com> escribi�:
>>
>> > storage class which can write the file splits that are currently being
>> > downloaded to the disk. this is exactly what other download
>> > accelerators do, i guess.
>>
>> Uh, unless I misundersand you, a standard file object is enough. First
>> create a file with the required size (open(...,'wb'), seek(n-1),
>> write(chr(0))). For each downloaded chunk you have to know its position
>> in
>> the file; then just seek() and write() it.
>
> BUT the thing thats going in my mind is thread safety. i plan to start
> each part of the file download in a different thread. and then when
> each thread had downloaded more than 100kb (or eof or boundary
> reached) write the buffer to the disk. can this be achieved using
> mutex ? i have never shared objects between threads.

Use a different (single) thread to write the file; the others put write
requests on a Queue.queue object, and the writer just gets the requests
and processes them.

> is there a way to write this without using threads at all ???

Using asyncore, and perhaps the Twisted framework.

--
Gabriel Genellina

binaryj

3/8/2008 10:27:00 AM

0

On Mar 7, 2:14 pm, "Gabriel Genellina" <gagsl-...@yahoo.com.ar> wrote:
> En Fri, 07 Mar 2008 04:16:42 -0200, <coolman.gu...@gmail.com> escribi?:
>
>
>
> > On Mar 7, 1:38 am, "Gabriel Genellina" <gagsl-...@yahoo.com.ar> wrote:
> >> En Thu, 06 Mar 2008 14:34:27 -0200, <coolman.gu...@gmail.com> escribi?:
>
> >> > storage class which can write the file splits that are currently being
> >> > downloaded to the disk. this is exactly what otherdownload
> >> > accelerators do, i guess.
>
> >> Uh, unless I misundersand you, a standard file object is enough. First
> >> create a file with the required size (open(...,'wb'), seek(n-1),
> >> write(chr(0))). For each downloaded chunk you have to know its position
> >> in
> >> the file; then just seek() and write() it.
>
> > BUT the thing thats going in my mind is thread safety. i plan to start
> > each part of the filedownloadin a different thread. and then when
> > each thread had downloaded more than 100kb (or eof or boundary
> > reached) write the buffer to the disk. can this be achieved using
> > mutex ? i have never shared objects between threads.
>
> Use a different (single) thread to write the file; the others put write
> requests on a Queue.queue object, and the writer just gets the requests
> and processes them.
>
> > is there a way to write this without using threads at all ???
>
> Using asyncore, and perhaps the Twisted framework.
>
> --
> Gabriel Genellina

asyncore is basically a server thing right?

Gabriel Genellina

3/9/2008 4:55:00 PM

0

En Sat, 08 Mar 2008 08:27:12 -0200, <coolman.guron@gmail.com> escribió:
> On Mar 7, 2:14 pm, "Gabriel Genellina" <gagsl-...@yahoo.com.ar> wrote:
>> En Fri, 07 Mar 2008 04:16:42 -0200, <coolman.gu...@gmail.com> escribi�:
>>
>> > BUT the thing thats going in my mind is thread safety. i plan to start
>> > each part of the filedownloadin a different thread. and then when
>> > each thread had downloaded more than 100kb (or eof or boundary
>> > reached) write the buffer to the disk. can this be achieved using
>> > mutex ? i have never shared objects between threads.
>>
>> Use a different (single) thread to write the file; the others put write
>> requests on a Queue.queue object, and the writer just gets the requests
>> and processes them.
>>
>> > is there a way to write this without using threads at all ???
>>
>> Using asyncore, and perhaps the Twisted framework.
>
> asyncore is basically a server thing right?

asyncore is usually used to build servers, because in a server you want to
handle many requests with few resources, but you can use it to write a
client too.
Here is an example: http://effbot.org/zone/asyncore-ftp-...

How many files and how many simultaneous connections do you plan to
handle? Using multiple threads to download and a single thread to write,
connected thru a queue, looks like the "simplest thing that probably
works" to me unless you have other constraints.

--
Gabriel Genellina