[lnkForumImage]
TotalShareware - Download Free Software
Confronta i prezzi di migliaia di prodotti.
Asp Forum
 Home | Login | Register | Search 


 

Forums >

comp.lang.ruby

memory allocation failure: heavy files

Michael Linfield

4/5/2008 7:59:00 PM

Seems like any combo I try, eventually it lands me in a NoMemory error.
I've tried:

####
file = File.open(datafile)

file.each{|data| @store << data}
file.close()

@large = @store.to_s.split(" ")
####
File.open(datafile) do |file|
data = file.gets.to_s.split(" ")
end
####
(this appears to be the most inefficient one)
file = File.readlines(datafile)
file.each{|data| @store << data}
@large = @store.to_s.split(" ")
####

And this is with only a 24mb datafile, to which effect I had to split
the file.
The total size of the file is around 250-300mb.

Code inefficiency issue?

Thanks,

- Mac
--
Posted via http://www.ruby-....
4 Answers

James Tucker

4/5/2008 8:08:00 PM

0


On 5 Apr 2008, at 20:58, Michael Linfield wrote:
> Seems like any combo I try, eventually it lands me in a NoMemory
> error.
> I've tried:
>
> ####
> file = File.open(datafile)
>
> file.each{|data| @store << data}
> file.close()
>
> @large = @store.to_s.split(" ")
> ####
> File.open(datafile) do |file|
> data = file.gets.to_s.split(" ")
> end
> ####
> (this appears to be the most inefficient one)
> file = File.readlines(datafile)
> file.each{|data| @store << data}
> @large = @store.to_s.split(" ")
> ####
>
> And this is with only a 24mb datafile, to which effect I had to split
> the file.
> The total size of the file is around 250-300mb.
>
> Code inefficiency issue?
>
> Thanks,
>
> - Mac
> --
> Posted via http://www.ruby-....
>



You'll want to buffer data reads, and do the splits on data that's
buffered properly and completely. Many of the operations there slurp
or are never dropping data. If you have a *very* long line, there may
be more work required, too, also dependent on ram.

Michael Linfield

4/5/2008 10:53:00 PM

0

Effect achieved via:

@store = []
IO.foreach("testpart.txt"){|x| @store << x.split(" ").grep(/#{format}/)}

#format is a Regexp

- Mac

--
Posted via http://www.ruby-....

Robert Klemme

4/6/2008 9:03:00 AM

0

On 06.04.2008 00:52, Michael Linfield wrote:
> Effect achieved via:
>
> @store = []
> IO.foreach("testpart.txt"){|x| @store << x.split(" ").grep(/#{format}/)}
>
> #format is a Regexp

If format is a regular expression why then don't you do "...grep(format)"?

Kind regards

robert

Michael Linfield

4/6/2008 7:13:00 PM

0

Robert Klemme wrote:
> On 06.04.2008 00:52, Michael Linfield wrote:
>> Effect achieved via:
>>
>> @store = []
>> IO.foreach("testpart.txt"){|x| @store << x.split(" ").grep(/#{format}/)}
>>
>> #format is a Regexp
>
> If format is a regular expression why then don't you do
> "...grep(format)"?
>
> Kind regards
>
> robert

Just a habit, both ways work.
--
Posted via http://www.ruby-....