[lnkForumImage]
TotalShareware - Download Free Software
Confronta i prezzi di migliaia di prodotti.
Asp Forum
 Home | Login | Register | Search 


 

Forums >

comp.lang.ruby

How to reclaim memory without GC.start

dtuttle1@gmail.com

7/16/2007 9:30:00 PM

Hi - I have a memory and cpu-constrained embedded app. There's a long-
running loop with a local object (buffer) which eats up memory. A
GC.start on every iteration will reclaim it, but that eats the cpu.

Is there a way to deallocate the memory used by the temporary objects?

file = File.new file_path
while buffer = file.read(512)
stream_host_connection.write buffer
stream_host_connection.flush
# GC.start - too expensive
end

Thanks, Dave
25 Answers

Ari Brown

7/16/2007 10:29:00 PM

0

On Jul 16, 2007, at 5:30 PM, dtuttle1@gmail.com wrote:

> Hi - I have a memory and cpu-constrained embedded app. There's a long-
> running loop with a local object (buffer) which eats up memory. A
> GC.start on every iteration will reclaim it, but that eats the cpu.
>
> Is there a way to deallocate the memory used by the temporary objects?
>
> file = File.new file_path
> while buffer = file.read(512)
> stream_host_connection.write buffer
> stream_host_connection.flush
> # GC.start - too expensive
> end

Unfortunately, I don't know how to do it. If I did, I would include
it in every single one of my programs. But you should have an extra
'=' after buffer. It should read...
...
while buffer == file.read(512)
...

Note the double '='.

G'luck, and PLEASE pop an email to the list if you figure out how to
accomplish it.
-------------------------------------------|
Nietzsche is my copilot

Todd Benson

7/16/2007 11:22:00 PM

0

On 7/16/07, Ari Brown <ari@aribrown.com> wrote:
> On Jul 16, 2007, at 5:30 PM, dtuttle1@gmail.com wrote:
>
> > Hi - I have a memory and cpu-constrained embedded app. There's a long-
> > running loop with a local object (buffer) which eats up memory. A
> > GC.start on every iteration will reclaim it, but that eats the cpu.
> >
> > Is there a way to deallocate the memory used by the temporary objects?
> >
> > file = File.new file_path
> > while buffer = file.read(512)
> > stream_host_connection.write buffer
> > stream_host_connection.flush
> > # GC.start - too expensive
> > end
>
> Unfortunately, I don't know how to do it. If I did, I would include
> it in every single one of my programs. But you should have an extra
> '=' after buffer. It should read...
> ...
> while buffer == file.read(512)

No, Ari. I think you misread the program. There is an assignment to
buffer, not a comparison. The while loop will exit when
file.read(512) returns nil. I know why you thought it was a
comparison, though.

Todd

Travis D Warlick Jr

7/16/2007 11:31:00 PM

0

dtuttle1@gmail.com wrote:
> Hi - I have a memory and cpu-constrained embedded app. There's a long-
> running loop with a local object (buffer) which eats up memory. A
> GC.start on every iteration will reclaim it, but that eats the cpu.
>
> Is there a way to deallocate the memory used by the temporary objects?
>
> file = File.new file_path
> while buffer = file.read(512)
> stream_host_connection.write buffer
> stream_host_connection.flush
> # GC.start - too expensive
> end

Someone please correct me if I'm wrong, but...

You don't need to deallocate buffer every time because it is not getting
duplicated at the beginning of every iteration -- it is being
overwritten, so the memory usage for buffer will always stay at 512
bytes (plus the static overhead for the object).

Deallocating it every time would actually increase CPU usage because you
would be deleting and recreating the buffer object at the beginning of
every iteration.

If you're really constrained on memory, you might want to consider
decreasing the amount that you read at every iteration. However,
remember that you increase your CPU usage by the same factor that you
decrease your memory usage in this case.

--
Travis Warlick

"Programming in Java is like dealing with your mom --
it's kind, forgiving, and gently chastising.
Programming in C++ is like dealing with a disgruntled
girlfriend -- it's cold, unforgiving, and doesn't tell
you what you've done wrong."

Lionel Bouton

7/16/2007 11:56:00 PM

0

Travis D Warlick Jr wrote the following on 17.07.2007 01:31 :
> dtuttle1@gmail.com wrote:
>
>> Hi - I have a memory and cpu-constrained embedded app. There's a long-
>> running loop with a local object (buffer) which eats up memory. A
>> GC.start on every iteration will reclaim it, but that eats the cpu.
>>
>> Is there a way to deallocate the memory used by the temporary objects?
>>
>> file = File.new file_path
>> while buffer = file.read(512)
>> stream_host_connection.write buffer
>> stream_host_connection.flush
>> # GC.start - too expensive
>> end
>>
>
> Someone please correct me if I'm wrong, but...
>
> You don't need to deallocate buffer every time because it is not getting
> duplicated at the beginning of every iteration -- it is being
> overwritten, so the memory usage for buffer will always stay at 512
> bytes (plus the static overhead for the object).
>
> Deallocating it every time would actually increase CPU usage because you
> would be deleting and recreating the buffer object at the beginning of
> every iteration.
>
> If you're really constrained on memory, you might want to consider
> decreasing the amount that you read at every iteration. However,
> remember that you increase your CPU usage by the same factor that you
> decrease your memory usage in this case.
>

Actually, as variable affectation is done by reference and the *content*
of buffer is originally duplicated by File#read memory usage is not
likely to change linearly with the value passed to File#read...

You can still play with different values passed to it and expect some
change of behaviour (ie: depending on the actual amount of memory
allocated for each read, you may or may not hit a sweat spot in the GC
heuristics).

My advise would be to do larger reads and GC.start less often:

read_amount = 65536
read_count = 0
loops_between_gcs = 100
while buffer = file.read(read_amount)
stream_host_connection.write buffer
stream_host_connection.flush
GC.start if ((read_count += 1) % loops_between_gcs) == 0
end


But note that if stream_host_connection.flush doesn't actually wait
until the buffer isn't referenced anymore anywhere in your code (or in
some other process sitting on the same system as a matter of fact),
calling GC.start probably won't have any effect.

Lionel

dtuttle1@gmail.com

7/17/2007 12:05:00 AM

0

On Jul 16, 4:31 pm, Travis D Warlick Jr <warli...@operissystems.com>
wrote:
> dtutt...@gmail.com wrote:
> > Hi - I have a memory and cpu-constrained embedded app. There's a long-
> > running loop with a local object (buffer) which eats up memory. A
> > GC.start on every iteration will reclaim it, but that eats the cpu.
>
> > Is there a way to deallocate the memory used by the temporary objects?
>
> > file = File.new file_path
> > while buffer = file.read(512)
> > stream_host_connection.write buffer
> > stream_host_connection.flush
> > # GC.start - too expensive
> > end
>
> Someone please correct me if I'm wrong, but...
>
> You don't need to deallocate buffer every time because it is not getting
> duplicated at the beginning of every iteration -- it is being
> overwritten, so the memory usage for buffer will always stay at 512
> bytes (plus the static overhead for the object).
>
> Deallocating it every time would actually increase CPU usage because you
> would be deleting and recreating the buffer object at the beginning of
> every iteration.
>
> If you're really constrained on memory, you might want to consider
> decreasing the amount that you read at every iteration. However,
> remember that you increase your CPU usage by the same factor that you
> decrease your memory usage in this case.
>
> --
> Travis Warlick
>
> "Programming in Java is like dealing with your mom --
> it's kind, forgiving, and gently chastising.
> Programming in C++ is like dealing with a disgruntled
> girlfriend -- it's cold, unforgiving, and doesn't tell
> you what you've done wrong."

The object is being overwritten, but the old copies aren't being
garbage collected. I know this because the memory does stay constant
if I call GC.start on every iteration.
I think I'll go with a c extension for this small part of the code.

Thanks, Dave

Travis D Warlick Jr

7/17/2007 12:54:00 AM

0

Lionel Bouton wrote:
> Actually, as variable affectation is done by reference

I knew that... woops... **beats head on keyboard**

However, A chance to redeem myself: I remembered that IO#read returns a
string, so change the while to:

while buffer[0...512] = file.read(512)

That should overwrite the contents of the buffer

--
Travis Warlick

"Programming in Java is like dealing with your mom --
it's kind, forgiving, and gently chastising.
Programming in C++ is like dealing with a disgruntled
girlfriend -- it's cold, unforgiving, and doesn't tell
you what you've done wrong."

Joel VanderWerf

7/17/2007 1:03:00 AM

0

Travis D Warlick Jr wrote:
> Lionel Bouton wrote:
>> Actually, as variable affectation is done by reference
>
> I knew that... woops... **beats head on keyboard**
>
> However, A chance to redeem myself: I remembered that IO#read returns a
> string, so change the while to:
>
> while buffer[0...512] = file.read(512)
>
> That should overwrite the contents of the buffer

Do you mean file.read(512, buffer) ?


---------------------------------------------------------------- IO#read
ios.read([length [, buffer]]) => string, buffer, or nil
------------------------------------------------------------------------
Reads at most length bytes from the I/O stream, or to the end of
file if length is omitted or is nil. length must be a non-negative
integer or nil. If the optional buffer argument is present, it
must reference a String, which will receive the data.

At end of file, it returns nil or "" depend on length. ios.read()
and ios.read(nil) returns "". ios.read(positive-integer) returns
nil.

f = File.new("testfile")
f.read(16) #=> "This is line one"


--
vjoel : Joel VanderWerf : path berkeley edu : 510 665 3407

Travis D Warlick Jr

7/17/2007 1:35:00 AM

0

Joel VanderWerf wrote:
> Travis D Warlick Jr wrote:
>> Lionel Bouton wrote:
>>
>> while buffer[0...512] = file.read(512)
>>
>
> Do you mean file.read(512, buffer) ?

Ahh, yes. That would be an much prettier way to do it.

while file.read(512, buffer)

--
Travis Warlick

"Programming in Java is like dealing with your mom --
it's kind, forgiving, and gently chastising.
Programming in C++ is like dealing with a disgruntled
girlfriend -- it's cold, unforgiving, and doesn't tell
you what you've done wrong."

Florian Groß

7/17/2007 1:35:00 AM

0



On Jul 16, 11:30 pm, "dtutt...@gmail.com" <dtutt...@gmail.com> wrote:
> file = File.new file_path
> while buffer = file.read(512)
> stream_host_connection.write buffer
> stream_host_connection.flush
> # GC.start - too expensive
> end

Note that IO#read supports a buffer argument:

buffer = ""
while file.read(512, buffer)
...
end

In that case it won't create a new string object for each call. It
will just modify buffer instead.

Ari Brown

7/17/2007 1:41:00 AM

0


On Jul 16, 2007, at 7:22 PM, Todd Benson wrote:
> No, Ari. I think you misread the program. There is an assignment to
> buffer, not a comparison. The while loop will exit when
> file.read(512) returns nil. I know why you thought it was a
> comparison, though.

Yups. That is correct. My mistake, and thanks for pointing it out! At
least now I know I can do assignment in my 'while' statements. Right?

~ Ari
English is like a pseudo-random number generator - there are a
bajillion rules to it, but nobody cares.