Robert Klemme
12/14/2006 12:19:00 PM
On 14.12.2006 12:14, Mark Van de vyver wrote:
> Hi,
> Thanks for the efforts put into making Ruby available!
> While new to Ruby it seems worth investing some time in. Eventually I'd
> like to unpack some data from a binary file and insert it into a MySQL
> table.
> At the moment I'm having trouble looping over the source data file, and
> just viewing the data to make sure things are proceeding correctly.
>
> FYI: eventually I need this to run quickly (many GB's to be processed),
> but for the moment I'll settle for getting anything to work :)
>
> The ruby details are:
> $ ruby -v
> ruby 1.8.5 (2006-08-25) [i386-cygwin]
>
> -----------
> Attempt 1 ( an error )
> -----------
> File.foreach("./INPUT/T200510A.BIN", "rb") do |f|
> print f.read(19).unpack('I4 I4 I4 I2 I2 a2 a1')
> end
> NoMethodError: [FATAL] failed to allocate memory
This does not work because the second argument to foreach is the /line
separator/ - not the file mode.
> -----------
> Attempt 2 ( an error )
> -----------
> File.open("./INPUT/T200510A.BIN", "rb").readlines do |f|
> data = f.read(19).unpack('I4 I4 I4 I2 I2 a2 a1')
> print data
> end
> /usr/lib/ruby/1.8/irb.rb:298:in `inspect': failed to allocate memory
> (NoMemoryError)
This fails because you use readlines - this will try to read the whole
file into memory as /lines/ - and since there are no lines in a binary
file this chokes while attempting to store the whole file or a large
portion of it in a single line (i.e. String).
You should do this
File.open("./INPUT/T200510A.BIN", "rb") do |io|
while ( data = io.read(19) )
p data.unpack('I4I4I4I2I2a2a1')
end
end
Regards
robert