[lnkForumImage]
TotalShareware - Download Free Software

Confronta i prezzi di migliaia di prodotti.
Asp Forum
 Home | Login | Register | Search 


 

Forums >

comp.lang.ruby

Memory Leak Madness

Brandon Casci

1/3/2007 2:46:00 PM

I'm having one hell of a time trying to find and stop a memory leak in a
ruby daemon. It starts off using a tiny 14 MB of RAM, and 150+MB after a
day. I've read through many forum and blog posts, and tried several
profilers. I don't see anything unusual.

I even tried this guy
http://scottstuff.net/blog/articles/2006/08/17/memory-leak-profiling-...

That showed my overall object count going up and down as normal, but
memory was not being released as objects came and went.

I've gone through my code made sure I'm squashing any unused objects.
Well...I set them to nil, I just guessed that would help. I'm also
calling .clear on any arrays and hashes. I noticed this helps with
garbage collection. I also added a thread that runs does nothing but
sleep and run garbage collection every 5 minutes, although I'm pretty
sure this isn't helping much.

All I've managed to greatly slow the memory leak. Before the daemon
would consume 100+ MB in a few hours.

There are two things my daemon makes heavy use of that is out of
control, hpricot and dbi (talking to ms sql server). I suspect one or
both of these guys may be my problem. I'll write into both projects for
advice, but is there anything else I can do so ruby will let go of
unused objects?

--
Posted via http://www.ruby-....

18 Answers

Wilson Bilkovich

1/3/2007 3:45:00 PM

0

On 1/3/07, Brandon Casci <brandon@loudcity.net> wrote:
> I'm having one hell of a time trying to find and stop a memory leak in a
> ruby daemon. It starts off using a tiny 14 MB of RAM, and 150+MB after a
> day. I've read through many forum and blog posts, and tried several
> profilers. I don't see anything unusual.
>
> I even tried this guy
> http://scottstuff.net/blog/articles/2006/08/17/memory-leak-profiling-...
>
> That showed my overall object count going up and down as normal, but
> memory was not being released as objects came and went.
>
> I've gone through my code made sure I'm squashing any unused objects.
> Well...I set them to nil, I just guessed that would help. I'm also
> calling .clear on any arrays and hashes. I noticed this helps with
> garbage collection. I also added a thread that runs does nothing but
> sleep and run garbage collection every 5 minutes, although I'm pretty
> sure this isn't helping much.
>
> All I've managed to greatly slow the memory leak. Before the daemon
> would consume 100+ MB in a few hours.
>
> There are two things my daemon makes heavy use of that is out of
> control, hpricot and dbi (talking to ms sql server). I suspect one or
> both of these guys may be my problem. I'll write into both projects for
> advice, but is there anything else I can do so ruby will let go of
> unused objects?

Try loading this library when you daemon starts up:
http://moonbase.rydia.net/mental/blog/programming/fastthread-...

DBI almost certainly makes use of locking, and fastthread cleans that
up significantly.

Just a thought.

Robert Klemme

1/3/2007 3:50:00 PM

0

On 03.01.2007 15:46, Brandon Casci wrote:
> I'm having one hell of a time trying to find and stop a memory leak in a
> ruby daemon. It starts off using a tiny 14 MB of RAM, and 150+MB after a
> day. I've read through many forum and blog posts, and tried several
> profilers. I don't see anything unusual.
>
> I even tried this guy
> http://scottstuff.net/blog/articles/2006/08/17/memory-leak-profiling-...
>
> That showed my overall object count going up and down as normal, but
> memory was not being released as objects came and went.
>
> I've gone through my code made sure I'm squashing any unused objects.
> Well...I set them to nil, I just guessed that would help. I'm also
> calling .clear on any arrays and hashes. I noticed this helps with
> garbage collection. I also added a thread that runs does nothing but
> sleep and run garbage collection every 5 minutes, although I'm pretty
> sure this isn't helping much.
>
> All I've managed to greatly slow the memory leak. Before the daemon
> would consume 100+ MB in a few hours.

Is there some point where it stabilizes after a day or so?

> There are two things my daemon makes heavy use of that is out of
> control, hpricot and dbi (talking to ms sql server). I suspect one or
> both of these guys may be my problem. I'll write into both projects for
> advice, but is there anything else I can do so ruby will let go of
> unused objects?

It may actually be doing it. The memory footprint you see on OS side
need not be directly related to the number of objects currently in
memory. I am not sure whether Ruby ever gives back memory to the OS but
if not the effect you see might actually be from a situation where there
were needed many objects at once.

I definitively would not add a thread that does GC. The interpreter
will take care of this.

I'd probably do some object statistics (per class) and also summarize
String, Array and Hash sizes. If you encounter a significant increase
in volume somewhere then this might indicate where the problem lies.
You could try something like this:

require 'pp'

def stats_2
counts = Hash.new(0)
sizes = Hash.new(0)

ObjectSpace.each_object(Object) do |obj|
counts[obj.class] += 1

if obj.respond_to?(:size) && obj.method(:size).arity == 0
sizes[obj.class] += obj.size
end
end

pp "counts", counts, "sizes", sizes
end

Signal.trap :INT do
stats_2
end


Kind regards

robert

Brandon Casci

1/3/2007 4:03:00 PM

0

Do you mean this by locking?

@mutex.synchronize do
@myarray << station
end

Becuase I do that in two spots, although I do .clear out that
frequently.

--
Posted via http://www.ruby-....

Ara.T.Howard

1/3/2007 4:18:00 PM

0

Robert Klemme

1/3/2007 4:28:00 PM

0

On 03.01.2007 17:17, ara.t.howard@noaa.gov wrote:
> On Thu, 4 Jan 2007, Robert Klemme wrote:
>
>> It may actually be doing it. The memory footprint you see on OS side
>> need not be directly related to the number of objects currently in
>> memory. I am not sure whether Ruby ever gives back memory to the OS
>> but if not the effect you see might actually be from a situation where
>> there were needed many objects at once.
>>
>
> silly robert - don't you know that free always returns memory to the OS!
>
> http://groups-beta.google.com/group/comp.lang.ruby/browse_frm/thread/f6dda5896cbba030/adf86eb9e1fda322?lnk=gst&q=zed+shaw++free&rnum=1#adf86e...
>
> ;-)

That says it all... :-)

sillybert

Brandon Casci

1/3/2007 4:32:00 PM

0

Robert Klemme wrote:
>
> Is there some point where it stabilizes after a day or so?
>

Nope..it keeps going like...like pacman. *wocka* *wocka* *wocka*.
Although it does hover at various intervals before it grows, and before
I made my changes it would grow pretty steady. When you watch the memory
count for the process, it takes 3 steps forward, and two steps back but
the end result is always an increase.

Good times!!

> I definitively would not add a thread that does GC. The interpreter
> will take care of this.

Ya...I took that out. It felt wrong to do, but I wanted to see what
would happen.


>
> I'd probably do some object statistics (per class) and also summarize
> String, Array and Hash sizes.

I'm pretty sure whats what the MemoryProfiler here does:
http://scottstuff.net/blog/articles/2006/08/17/memory-leak-profiling-...

It takes that information, and writes the changes to log files at
specified intervals.

Here is what my daemon does....
It has a couple threads, Thread1 one gets work, and Thread2 performs
work

Thread 1:

-Thread1 gets rows from a database every N seconds
-it builds a series of objects based on the database rows, and appends
them to an array. They are called "work items". The array is protected
by a mutex-sync because Thread2 picks items off of it.
-it sleeps for a while then repeats the process

Thread 2:

- Thread two gets a "work item" from the above array. An item is
"popped" off the array by calling .shift from within a mutex-sync block
- Hpricot pulls information down from url's inside the "work item"
- the information gets put put into hashes, which get dumped to disk via
YAML::dump

And that's it. It's a pretty small daemon.











--
Posted via http://www.ruby-....

Jano Svitok

1/3/2007 4:55:00 PM

0

On 1/3/07, Brandon Casci <brandon@loudcity.net> wrote:
> Here is what my daemon does....
> It has a couple threads, Thread1 one gets work, and Thread2 performs
> work
>
> Thread 1:
>
> -Thread1 gets rows from a database every N seconds
> -it builds a series of objects based on the database rows, and appends
> them to an array. They are called "work items". The array is protected
> by a mutex-sync because Thread2 picks items off of it.
> -it sleeps for a while then repeats the process
>
> Thread 2:
>
> - Thread two gets a "work item" from the above array. An item is
> "popped" off the array by calling .shift from within a mutex-sync block
> - Hpricot pulls information down from url's inside the "work item"
> - the information gets put put into hashes, which get dumped to disk via
> YAML::dump

Sometime ago there was a problem with Array#shift leaking memory.
The proposed solution that time was to replace push/shift pairs with
unshift/pop (i.e. entering the data in the opposite direction). I
think it should be fixed now, but I'm not sure (1.8.5-p0 had this bug,
maybe 1.8.5-p2 doesn't).

khaines

1/3/2007 5:09:00 PM

0

Paul Rogers

1/3/2007 5:38:00 PM

0

I think you said you are using Windows/Ms SQL Server.
The ADO driver leaks memory like you wouldnt beleive.
In my rails app I switched to odbc and it runs much better.


khaines@enigo.com wrote:
> On Thu, 4 Jan 2007, Brandon Casci wrote:
>
> > Do you mean this by locking?
> >
> > @mutex.synchronize do
> > @myarray << station
> > end
> >
> > Becuase I do that in two spots, although I do .clear out that
> > frequently.
>
> Do you have situations where you end up with a lot of threads? Do you
> then have a lot of threads all using the same Mutex? There's an issue
> with the way Ruby handles the memory allocated to an array when values are
> shifted off of it. In short, Mutex uses an array to manage the queue of
> waiting threads. It pushes onto it and shifts them off of it, and if you
> have a lot of threads, you will see what seems to be inexplicable memory
> usage as a consequence. Also be aware that if you use shift() on arrays
> elsewhere, your arrays are using more memory than you think.
>
> The current best fix is to use the fastthread library if you can. It
> replaces the Ruby threading support items like Mutex with much faster C
> versions.
>
> If, for whatever reason, you can't do this, you can override the
> definitions of the Thread lock() and unlock() methods to use unshift and
> pop instead of push and shift for placing items into the queue and taking
> them off. This doesn't hold a candle to all that's being done with
> fasthread, but it does eliminate a bad RAM usage issue if you really can't
> use fasthread for whatever reason.
>
> Something like this:
>
> class Thread
>
> def lock
> while (Thread.critical = true; @locked)
> @waiting.unshift Thread.current
> Thread.stop
> end
> @locked = true
> Thread.critical = false
> self
> end
>
> def unlock
> return unless @locked
> Thread.critical = true
> @locked = false
> begin
> t = @waiting.pop
> t.wakeup if t
> rescue ThreadError
> retry
> end
> Thread.critical = false
> begin
> t.run if t
> rescue ThreadError
> end
> self
> end
>
> end
>
>
> Kirk Haines

Brandon Casci

1/3/2007 6:11:00 PM

0

unknown wrote:
>
> Do you have situations where you end up with a lot of threads? Do you
> then have a lot of threads all using the same Mutex?

Up to 15 threads work the queue. One threads add work items, the others
pull items out of the queue. All threads access the queue via this
object. Here is the latest code (the work queue is initialized once and
is accessible to all threads):

class StationWorkQueue


def initialize
@mutex = Mutex.new
@stations = []
end

def add_station(station)
@mutex.synchronize do
@stations.clear if @stations.size == 0
@stations.unshift(station)
end
station = nil
return nil
end

def get_station
return @stations.pop
end

def size
@mutex.synchronize do
return @stations.size
end
end
end

By the way, my worker threads don't die. Is there any kind of memory
issue with that? They run in a loop. They keep calling
StationWorkQueue.get_station, perform work, then sleep for a short
while so the CPU doesn't spike, and then go looking for more work. They
drain the queue pretty fast too.

unshift/pop doesn't seem to stop the memory consumption, but it's been
slowed down a great deal. It's only gained a few MB over the last 30
minutes. I'll let this run for a few hours and see what happens. I'll
also try fastthread.

I may also try replacing my simple work queue with reliable-msg. That
was my original plan, so I could scale this across multiple processes
and/or machines but I just wanted the satisfaction of seeing something
done fast so I took at stab at it this way.

I also created a second daemon to load the YAML files, look at the data
and decide which database it should get plopped into. That one leaks
memory like crazy, and it's simpler than the first daemon. So ya....I'll
take that other suggestion and try ODBC over ADO.

Fun fun!


--
Posted via http://www.ruby-....