Robert Klemme
3/9/2007 10:08:00 AM
On 08.03.2007 23:18, ara.t.howard@noaa.gov wrote:
> On Fri, 9 Mar 2007, dkmd_nielsen wrote:
>
>> The process that initiated my message earlier (about deleting array
>> elements) is a rather long running process of rebuilding and
>> reconfiguring parameter files. There hundreds of files, each with as
>> many as 22,000 parameters to processed. For example, four small test
>> files ran in about two minutes. There is a ton of string manipulation
>> going on, which probably translated into lots of trailing string parts
>> and pointer laying around RAM...clogging it up. I was thinking of
>> manually initiating garbage collection after every five or ten files
>> processed. Is that a smart thing?
To OP: generally "manual" GC is considered bad since it interferes with
the automatic mechanism.
>> What are yours thoughts on manually initiated garbage collection?
>> What kinds of practices result in bits and pieces of objects and
>> pointers being left laying around in the ether of RAM? Are there
>> tools that help see what happens to RAM while a process runs, like a
>> debugger does with variables?
>>
>> Thanks for everything
>> dvn
>
> if you can fork - that's the best - then you just let each child's death
> clean
> up that sub-segment of work's memory.
Also, forking has the added advantage of better utilizing multi core CPU's.
If you do encounter excessive memory usage then you should
a) make sure you do not hold onto stuff longer than needed
b) check your algorithms for inefficient dealing with objects; since you
mention string processing, this is a typical gotcha:
s += "foo" # creates a new string
s << "foo" # just appends to s
Another one
a=[]
a += ["foo", "bar"] # creates another array
a << "foo" << "bar" # just appends
a.concat ["foo", "bar"] # just appends
c) If files you are processing are large then you might also try to do
some kind of stream processing where you do not have to keep the whole
file's content in memory (if that's applicable to your problem domain).
Kind regards
robert