[lnkForumImage]
TotalShareware - Download Free Software

Confronta i prezzi di migliaia di prodotti.
Asp Forum
 Home | Login | Register | Search 


 

Forums >

comp.lang.ruby

How do I reduce the memory usage of a script?

Scott Ellsworth

7/13/2005 9:28:00 PM

Hi, all.

Please find attached a simple Ruby script that rummages through my
ITunes files, reads the first megabyte or so, finds the encoder, and
then prints the encoder and filename. This lets me know which tracks
need re-ripping.

This script blows through half a gig of RAM while running, and I really
do not see why. It should only have perhaps a few megabytes at max in
RAM.

FWIW, the output looks like:
iTunes v4.9, QuickTime 7.0.1 /Users/work/Music/iTunes/iTunes
Music/Yellowcard/Ocean Avenue Song1.m4a
iTunes v4.9, QuickTime 7.0.1 /Users/work/Music/iTunes/iTunes
Music/Yellowcard/Ocean Avenue Song2.m4a
iTunes v4.9, QuickTime 7.0.1 /Users/work/Music/iTunes/iTunes
Music/Yellowcard/Ocean Avenue Song3.m4a

Style and speed optimizations are accepted, but the runtime is under a
minute now for the 5500 files I have in my library, so memory usage is
my real problem.

Help?

#!/usr/bin/env ruby
require 'find'
def procpath(f)
if File.file?(f) then
if File.fnmatch("*.m4a",f) then
found = false
data = IO.read(f, 65536*8)
re = /[[:alnum:]_., ]{9,}/
data.scan(re) do |string|
if (string =~ /QuickTime/) then
filename = File.basename(f)
dirname = File.dirname(f)
# puts "#{string} #{dirname}"
puts "#{string} #{dirname} #{filename}"
found = true
break
end
end
if (!found) then
puts "Unknown #{f}"
end
end
elsif File.directory?(f) && !File.fnmatch(".", f) &&
!File.fnmatch("..", f) then
Dir.foreach(f) { |subf| procpath(subf) }
end
end

Find.find("/Users/work/Music/iTunes/iTunes Music/") do |f|
procpath(f)
end

Scott
--
scott@alodar.nospam.com
Java, Cocoa, and Database consulting for the life sciences

--
Scott Ellsworth
scott@alodar.nospam.com
Java and database consulting for the life sciences
6 Answers

John Carter

7/13/2005 9:39:00 PM

0

Logan Capaldo

7/13/2005 9:50:00 PM

0


On Jul 13, 2005, at 5:30 PM, Scott Ellsworth wrote:

> Hi, all.
>
> Please find attached a simple Ruby script that rummages through my
> ITunes files, reads the first megabyte or so, finds the encoder, and
> then prints the encoder and filename. This lets me know which tracks
> need re-ripping.
>
> This script blows through half a gig of RAM while running, and I
> really
> do not see why. It should only have perhaps a few megabytes at max in
> RAM.
>
> FWIW, the output looks like:
> iTunes v4.9, QuickTime 7.0.1 /Users/work/Music/iTunes/iTunes
> Music/Yellowcard/Ocean Avenue Song1.m4a
> iTunes v4.9, QuickTime 7.0.1 /Users/work/Music/iTunes/iTunes
> Music/Yellowcard/Ocean Avenue Song2.m4a
> iTunes v4.9, QuickTime 7.0.1 /Users/work/Music/iTunes/iTunes
> Music/Yellowcard/Ocean Avenue Song3.m4a
>
> Style and speed optimizations are accepted, but the runtime is under a
> minute now for the 5500 files I have in my library, so memory usage is
> my real problem.
>
> Help?
>
> #!/usr/bin/env ruby
> require 'find'
> def procpath(f)
> if File.file?(f) then
> if File.fnmatch("*.m4a",f) then
> found = false
> data = IO.read(f, 65536*8)
> re = /[[:alnum:]_., ]{9,}/
> data.scan(re) do |string|
> if (string =~ /QuickTime/) then
> filename = File.basename(f)
> dirname = File.dirname(f)
> # puts "#{string} #{dirname}"
> puts "#{string} #{dirname} #{filename}"
> found = true
> break
> end
> end
> if (!found) then
> puts "Unknown #{f}"
> end
> end
> elsif File.directory?(f) && !File.fnmatch(".", f) &&
> !File.fnmatch("..", f) then
> Dir.foreach(f) { |subf| procpath(subf) }
> end
> end
>
> Find.find("/Users/work/Music/iTunes/iTunes Music/") do |f|
> procpath(f)
> end
>
> Scott
> --
> scott@alodar.nospam.com
> Java, Cocoa, and Database consulting for the life sciences
>
> --
> Scott Ellsworth
> scott@alodar.nospam.com
> Java and database consulting for the life sciences
>
>

Well mileage may vary and all that jazz, but on my box it took up
like ~30M virtual according to top and like 1.5MB ~ 2MB physical.
Have you tried explicity invoking the GC?


daz

7/13/2005 9:55:00 PM

0


Scott Ellsworth wrote:
> Hi, all.
>
[...]
>
> This script blows through half a gig of RAM while running, and I really
> do not see why. It should only have perhaps a few megabytes at max in
> RAM.
>
[...]

> if (!found) then
> puts "Unknown #{f}"
else
data = nil
GC.start # garbage collect
> end


Any better with that addition ?

daz



daz

7/13/2005 10:06:00 PM

0


(Called away from keyboard)

Compare last with:

if (!found) then
puts "Unknown #{f}"
end
data = nil
GC.start # garbage collect

.... which will garbage collect more often.

Best,

daz


Scott Ellsworth

7/14/2005 12:30:00 AM

0

In article <8HKdneMVh9MZDUjfSa8jmA@karoo.co.uk>,
"daz" <dooby@d10.karoo.co.uk> wrote:

> if (!found) then
> puts "Unknown #{f}"
> end
> data = nil
> GC.start # garbage collect

This did seem to drop the memory usage on my MacOS X 10.4.2 system.

I will investigate the Find.find command next to see if I can get rid of
some recursion. An array of 5500 paths should not be _that_ big, at
least in comparison with four or five levels of directory depth.

Scott

--
Scott Ellsworth
scott@alodar.nospam.com
Java and database consulting for the life sciences

Robert Klemme

7/14/2005 8:31:00 AM

0

Scott Ellsworth wrote:
> In article <8HKdneMVh9MZDUjfSa8jmA@karoo.co.uk>,
> "daz" <dooby@d10.karoo.co.uk> wrote:
>
>> if (!found) then
>> puts "Unknown #{f}"
>> end
>> data = nil
>> GC.start # garbage collect
>
> This did seem to drop the memory usage on my MacOS X 10.4.2 system.
>
> I will investigate the Find.find command next to see if I can get rid
> of some recursion. An array of 5500 paths should not be _that_ big,
> at least in comparison with four or five levels of directory depth.

The problem might be that the data is still around while you enter the
recursion. If you want to verify that this is the case you can simply do
data = nil after processing. But: You definitely need to throw out the
recursion from propath() - otherwise you'll be processing directories over
and over again (I smell something like O(n*n) here)!

Kind regards

robert