Asp Forum - Memory Usage - comp.lang.ruby

Horacio Sanson

12/12/2005 10:10:00 AM

I have a ruby script that reads data from a database, processes the data (long
texts) and saves the results back to the database. I works great and at good
speed.

The problem is that I ran the script on friday and left it until monday when I
saw some memory related errors in the console (Insufficient Memory
something...). I ran the script again and fired up the windows task manager
and then I saw that for every 10~15 records in the DB the script processes
the memory goes up 1MB and there are no signs of it going down.

I started the script this morning and the PC was at 400MB ram usage, now is
night and the PC is at 1.05GB ram usage.

Is this a memory leak?? is it caused by my scritpt? or is a Ruby thing??

I use Ruby 1.8.2 and ActiveRecord 1.13.1 in a WinXP machine.

any comments on how to find out where the memory is going are welcome.

regards,
Horacio

6 Answers

Chad Perrin

12/12/2005 12:08:00 PM

On Mon, Dec 12, 2005 at 07:10:27PM +0900, Horacio Sanson wrote:
>
> I started the script this morning and the PC was at 400MB ram usage, now is
> night and the PC is at 1.05GB ram usage.
>
> Is this a memory leak?? is it caused by my scritpt? or is a Ruby thing??
>
> I use Ruby 1.8.2 and ActiveRecord 1.13.1 in a WinXP machine.
>
> any comments on how to find out where the memory is going are welcome.

Are you making heavy use of symbols? They don't relinquish RAM
unless/until the script exits, as far as I'm aware.

--
Chad Perrin [ CCD CopyWrite | http://ccd.ap... ]

unix virus: If you're using a unixlike OS, please forward
this to 20 others and erase your system partition.

Peter Hickman

12/12/2005 12:16:00 PM

I've had a problem like this using the flickr api and net/http. All it
did was download images from Flickr, it ran pretty much 24/7 but some
days it would just eat massive amounts of virtual ram, 900Mb plus.

I've wrapped the whole thing in a shell script now to get around the
problem. Any idea how to track where the memory is going on a live system?

Gene Tani

12/12/2005 4:14:00 PM

Horacio Sanson wrote:
> I have a ruby script that reads data from a database, processes the data (long
> texts) and saves the results back to the database. I works great and at good
> speed.
>
> The problem is that I ran the script on friday and left it until monday when I
> saw some memory related errors in the console (Insufficient Memory
> something...). I ran the script again and fired up the windows task manager
> and then I saw that for every 10~15 records in the DB the script processes
> the memory goes up 1MB and there are no signs of it going down.
>
> I started the script this morning and the PC was at 400MB ram usage, now is
> night and the PC is at 1.05GB ram usage.
>
> Is this a memory leak?? is it caused by my scritpt? or is a Ruby thing??
>
> I use Ruby 1.8.2 and ActiveRecord 1.13.1 in a WinXP machine.
>
> any comments on how to find out where the memory is going are welcome.
>
> regards,
> Horacio

a couple recent threads on digging into ruby'c GC:
http://groups.google.com/group/comp.lang.ruby/browse_thread/thread/4e7792...

and John Carter's thing to dump out your objects and (linux) processes
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-t...

Horacio Sanson

12/13/2005 12:58:00 AM

Thanks,

I already ran the Robert Klemme's code to see the object count of each class.
I ran the code every time my script finished processing each record of the
database and the numbers are like this:

String: 42812
Array: 16651
WIN32OLE: 3184
Hash: 2065

These are the biggest entries but they do not increase all the time, they
fluctuate up and down. Strings goes up to 60,000+ objects and then drops
down back to 20,000+ objects. Then starts growing again. This happens with
all objects. Other classes are below 100 or are even 0.

I do not use symbols apart from those needed by ActiveRecord.

going to test now the MemoryProfiler of John Carter.

BTW, are these numbers normal??

thanks for the tips.
Horacio

Tuesday 13 December 2005 01:17?Gene Tani ????????:
> Horacio Sanson wrote:
> > I have a ruby script that reads data from a database, processes the data
> > (long texts) and saves the results back to the database. I works great
> > and at good speed.
> >
> > The problem is that I ran the script on friday and left it until monday
> > when I saw some memory related errors in the console (Insufficient Memory
> > something...). I ran the script again and fired up the windows task
> > manager and then I saw that for every 10~15 records in the DB the script
> > processes the memory goes up 1MB and there are no signs of it going down.
> >
> > I started the script this morning and the PC was at 400MB ram usage, now
> > is night and the PC is at 1.05GB ram usage.
> >
> > Is this a memory leak?? is it caused by my scritpt? or is a Ruby thing??
> >
> > I use Ruby 1.8.2 and ActiveRecord 1.13.1 in a WinXP machine.
> >
> > any comments on how to find out where the memory is going are welcome.
> >
> > regards,
> > Horacio
>
> a couple recent threads on digging into ruby'c GC:
> http://groups.google.com/group/comp.lang.ruby/browse_thread/threa...
>75949460
>
> and John Carter's thing to dump out your objects and (linux) processes
> http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-t...

Gene Tani

12/13/2005 3:02:00 AM

Horacio Sanson wrote:
> Thanks,
>
> I already ran the Robert Klemme's code to see the object count of each class.
> I ran the code every time my script finished processing each record of the
> database and the numbers are like this:
>
> String: 42812
> Array: 16651
> WIN32OLE: 3184
> Hash: 2065
>
>
> These are the biggest entries but they do not increase all the time, they
> fluctuate up and down. Strings goes up to 60,000+ objects and then drops
> down back to 20,000+ objects. Then starts growing again. This happens with
> all objects. Other classes are below 100 or are even 0.
>
> I do not use symbols apart from those needed by ActiveRecord.
>
> going to test now the MemoryProfiler of John Carter.
>
> BTW, are these numbers normal??
>
>
> thanks for the tips.
> Horacio
>
> Tuesday 13 December 2005 01:17?Gene Tani ????????:
> > Horacio Sanson wrote:
> > > I have a ruby script that reads data from a database, processes the data
> > > (long texts) and saves the results back to the database. I works great
> > > and at good speed.
> > >
> > > The problem is that I ran the script on friday and left it until monday
> > > when I saw some memory related errors in the console (Insufficient Memory
> > > something...). I ran the script again and fired up the windows task
> > > manager and then I saw that for every 10~15 records in the DB the script
> > > processes the memory goes up 1MB and there are no signs of it going down.
> > >
> > > I started the script this morning and the PC was at 400MB ram usage, now
> > > is night and the PC is at 1.05GB ram usage.
> > >
> > > Is this a memory leak?? is it caused by my scritpt? or is a Ruby thing??
> > >
> > > I use Ruby 1.8.2 and ActiveRecord 1.13.1 in a WinXP machine.
> > >
> > > any comments on how to find out where the memory is going are welcome.
> > >
> > > regards,
> > > Horacio
> >
> > a couple recent threads on digging into ruby'c GC:
> > http://groups.google.com/group/comp.lang.ruby/browse_thread/threa...
> >75949460
> >
> > and John Carter's thing to dump out your objects and (linux) processes
> > http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-t...

i just found this, tagged by 1 del.icio.us user (thx whoeve you are!),
but it really talks about the same things.

http://theexciter.com/articles/finding-leaking-ruby-objects?commented...

Gene Tani

12/13/2005 3:02:00 AM

comp.lang.ruby

Memory Usage

Horacio Sanson

Chad Perrin

Peter Hickman

Gene Tani

Horacio Sanson

Gene Tani

Gene Tani

x Login to ForumsZone