Asp Forum - Efficient file downloading

Kyle Hunter

2/22/2008 1:55:00 AM

Hello,

I'm using open-uri to download files using a buffer. It seems very
inefficient in terms of resource usage (CPU is ~10-20% in usage).

If possible, I'd like some suggestions for downloading a file which
names the outputted file the same as the URL, and does not actually
write if the file comes out to a 404 (or some other exception hits).

Current code:
BUFFER_SIZE=4096
def download(url)
from = open(url)
if (buffer = from.read(BUFFER_SIZE))
puts "Downloading #{url}"
File.open(url.split('/').last, 'wb') do |file|
begin
file.write(buffer)
end while (buffer = from.read(BUFFER_SIZE))
end
end
end
--
Posted via http://www.ruby-....

5 Answers

James Tucker

2/22/2008 2:25:00 AM

On 22 Feb 2008, at 01:54, Kyle Hunter wrote:

> Hello,
>
> I'm using open-uri to download files using a buffer. It seems very
> inefficient in terms of resource usage (CPU is ~10-20% in usage).
>
> If possible, I'd like some suggestions for downloading a file which
> names the outputted file the same as the URL, and does not actually
> write if the file comes out to a 404 (or some other exception hits).
>
> Current code:
> BUFFER_SIZE=4096

Try making that a lot lot bigger.

>
> def download(url)
> from = open(url)
> if (buffer = from.read(BUFFER_SIZE))
> puts "Downloading #{url}"
> File.open(url.split('/').last, 'wb') do |file|
> begin
> file.write(buffer)
> end while (buffer = from.read(BUFFER_SIZE))
> end
> end
> end
> --
> Posted via http://www.ruby-....
>

Kyle Hunter

2/22/2008 2:34:00 AM

James Tucker wrote:
> On 22 Feb 2008, at 01:54, Kyle Hunter wrote:
>
>> BUFFER_SIZE=4096
> Try making that a lot lot bigger.

Doh! Thanks James. Brings it down to much more reasonable usage. I
totally overlooked that very small buffer size that was set - thanks.
--
Posted via http://www.ruby-....

fedzor

2/22/2008 10:06:00 PM

On Feb 21, 2008, at 8:54 PM, Kyle Hunter wrote:

> Hello,
>
> I'm using open-uri to download files using a buffer. It seems very
> inefficient in terms of resource usage (CPU is ~10-20% in usage).
>
> If possible, I'd like some suggestions for downloading a file which
> names the outputted file the same as the URL, and does not actually
> write if the file comes out to a 404 (or some other exception hits).
>
> Current code:
> BUFFER_SIZE=4096
> def download(url)
> from = open(url)
> if (buffer = from.read(BUFFER_SIZE))
> puts "Downloading #{url}"
> File.open(url.split('/').last, 'wb') do |file|
> begin
> file.write(buffer)
> end while (buffer = from.read(BUFFER_SIZE))
> end
> end
> end

$ sudo gem install snoopy
$ snoopy http://en.wikipedia.org/wiki...
=> file Main_Page

Ta dah! there's a lot of magic behind it right now, and torrentz
don't work (fixed on my machine, need to release it). It does
segmented downloading, ideal for large files. For smaller ones, it
still works fine.

The problem with open-uri is this: it downloads the whole thing to
your tmp directory first, so using the BUFFER_SIZE thing won't
actually help.

snoopy won't not write the file if there's an error.

-------------------------------------------------------|
~ Ari
Some people want love
Others want money
Me... Well...
I just want this code to compile

Matt

4/12/2012 12:29:00 AM

On Apr 11, 6:18 pm, MattB <trdell1...@Nomorespamgmail.com> wrote:
> On Wed, 11 Apr 2012 17:05:33 -0700 (PDT), Matt
>
>
>
>
>
> <matttel...@sprynet.com> wrote:
> >On Apr 11, 2:23 pm, MattB <trdell1...@Nomorespamgmail.com> wrote:
> >> Prosecutor's quandary: Zimmerman may be indicted, then acquitted
>
> >>http://www.cnn.com/2012/04/10/opinion/dershowitz-trayvon-pr......
>
> >> On the basis of the evidence currently in the public record, one
> >> likely outcome of the case against George Zimmerman is a mixed one:
> >> There may be sufficient evidence for a reasonable prosecutor to indict
> >> him for manslaughter, but there may also be doubt sufficient for a
> >> reasonable jury to acquit him.
>
> >Absolutely. That's how our justice system works. You are accused, you
> >get a trial, you may be found not guilty. If, when all the evidence is
> >presented,
> >the jury finds him not guilty, I will have no problem with it.
>
> Me neither.
>
>
>
> >> Any such predictions should be accepted with an abundance of caution,
> >> however, because the evidence known to the special prosecutor, but not
> >> to the public, may paint a different picture. It may be stronger or
> >> weaker.
>
> >Could be. We won't know until it is presented. What is important is
> >that
> >he is being tried, not that he MUST be convicted.
>
> That is true. I do hope they rework that law no matter the
> verdict. Personally don't like how it was used in this case.

If he is found guilty and the law is used as a defense, it will
effectively
rework the way the law is interpreted. Works for me. I understand the
original intent. Someone comes at you, you defend yourself, you are
not
guilty of any crime. The problem is, that's been true in the law for
thousands
of years. Why is this law even necessary?

Matt

.MattB.

4/12/2012 1:11:00 AM

On Wed, 11 Apr 2012 17:29:15 -0700 (PDT), Matt
<matttelles@sprynet.com> wrote:

>On Apr 11, 6:18?pm, MattB <trdell1...@Nomorespamgmail.com> wrote:
>> On Wed, 11 Apr 2012 17:05:33 -0700 (PDT), Matt
>>
>>
>>
>>
>>
>> <matttel...@sprynet.com> wrote:
>> >On Apr 11, 2:23?pm, MattB <trdell1...@Nomorespamgmail.com> wrote:
>> >> Prosecutor's quandary: Zimmerman may be indicted, then acquitted
>>
>> >>http://www.cnn.com/2012/04/10/opinion/dershowitz-trayvon-pr......
>>
>> >> On the basis of the evidence currently in the public record, one
>> >> likely outcome of the case against George Zimmerman is a mixed one:
>> >> There may be sufficient evidence for a reasonable prosecutor to indict
>> >> him for manslaughter, but there may also be doubt sufficient for a
>> >> reasonable jury to acquit him.
>>
>> >Absolutely. That's how our justice system works. You are accused, you
>> >get a trial, you may be found not guilty. If, when all the evidence is
>> >presented,
>> >the jury finds him not guilty, I will have no problem with it.
>>
>> ? ? Me neither.
>>
>>
>>
>> >> Any such predictions should be accepted with an abundance of caution,
>> >> however, because the evidence known to the special prosecutor, but not
>> >> to the public, may paint a different picture. It may be stronger or
>> >> weaker.
>>
>> >Could be. We won't know until it is presented. What is important is
>> >that
>> >he is being tried, not that he MUST be convicted.
>>
>> ? ?That is true. ?I do hope they rework that law no matter the
>> verdict. ?Personally don't like how it was used in this case.
>
>If he is found guilty and the law is used as a defense, it will
>effectively
>rework the way the law is interpreted. Works for me. I understand the
>original intent. Someone comes at you, you defend yourself, you are
>not
>guilty of any crime. The problem is, that's been true in the law for
>thousands
>of years. Why is this law even necessary?

I live now in Washington State. At times it feels the criminals
have more rights than the law abiding people.

Someone on here posted some common sense rules on guns in his State.
Wish all States has such.
Training might have prevented this.

>
>Matt

comp.lang.ruby

Efficient file downloading

Kyle Hunter

James Tucker

Kyle Hunter

fedzor

Matt

.MattB.

x Login to ForumsZone