[lnkForumImage]
TotalShareware - Download Free Software
Confronta i prezzi di migliaia di prodotti.
Asp Forum
 Home | Login | Register | Search 


 

Forums >

comp.lang.ruby

SparseFile

Erik Veenstra

1/6/2007 12:27:00 AM

I had to send huge files over a network to another machine.
Most of these files were image files for QEMU: typically 4 GB,
of which only a small portion (~ 400 MB) was used. Both client
and server were Ruby programs on Linux boxes, communicating via
FTP.

I thought it was a good idea to use sparse files [1], so I
searched for a SparseFile class, couldn't find one and wrote
one myself:

http://www.erikve...rubycodesnippets/index....

It seems to work pretty well... ;]

Any thoughts? Ideas? Comments? Do you want it to be available
as a library/gem? Should it become part of Ruby Facets? Trans?

gegroet,
Erik V. - http://www.erikve...

[1] http://en.wikipedia.org/wiki/S...
15 Answers

Erik Veenstra

1/6/2007 1:02:00 AM

0

> I had to send huge files over a network to another machine.

(That's a bit misleading. SparseFile is about saving disk space
on the server; not about saving bandwidth....)

A quick test on Windows2000/NTFS: Cygwin works as expected,
including the savings. On plain Windows, the full size of the
file is allocated, although the checksums of the files are
correct.

For now, it looks safe to use SparseFile in your
platform-agnostic application. What about OS/X? BSD?

gegroet,
Erik V. - http://www.erikve...

Ara.T.Howard

1/6/2007 1:03:00 AM

0

Eric Hodel

1/6/2007 1:10:00 AM

0

On Jan 5, 2007, at 16:26, Erik Veenstra wrote:

> I had to send huge files over a network to another machine.
> Most of these files were image files for QEMU: typically 4 GB,
> of which only a small portion (~ 400 MB) was used. Both client
> and server were Ruby programs on Linux boxes, communicating via
> FTP.
>
> I thought it was a good idea to use sparse files [1], so I
> searched for a SparseFile class, couldn't find one and wrote
> one myself:
>
> http://www.erikveen.dds.nl/rubycodesnippets/index....
>
> It seems to work pretty well... ;]
>
> Any thoughts? Ideas? Comments? Do you want it to be available
> as a library/gem? Should it become part of Ruby Facets? Trans?

You can have a gem hosted on Rubyforge in about 20 minutes with Hoe.
(Provided you have a project to store it in.)

--
Eric Hodel - drbrain@segment7.net - http://blog.se...

I LIT YOUR GEM ON FIRE!

Erik Veenstra

1/6/2007 1:47:00 AM

0

> why not simply zip them? of course one can also read/write
> directly to zipped files. the reason i ask is that we do this
> alot.

* Availability: A zipped file is not readily available (for
QEMU).

* Speed: It takes an awful lot of time to zip and unzip
(typically 4 GB per file!)

* Size: The sparse file is smaller than the zipped file (using
gzip). (Although I didn't expect that... ;])

* Code: Better readable code:

io = TCPServer.new("0.0.0.0", 4444).accept
SparseFile.open("file") do |f|
while (not io.closed? and data = io.gets)
f.write(data)
end
end

> what would you consider trade-offs to be with SparseFile?

* It might me just a little bit slower than writing to ordinary
files, because of the extra check and the housekeeping. (But
it is much, much faster than using ZIP-files...)

* Over time, after thousands or millions of random-access
writes to the same file, the holes are gone. ZIP-files might
stay small. (But ZIP-files are very, very slow...) Notice
this is not a disadvantage compared to ordinary files.

gegroet,
Erik V. - http://www.erikve...

----------------------------------------------------------------

$ ls -lh winxp.img
-rw-r--r-- 1 erik erik 4.0G 2007-01-05 17:29 winxp.img

$ du -h winxp.img
997M winxp.img

$ gzip < winxp.img | wc -c
1467534047 # That's more than the sparse file!

$ time gzip < winxp.img > /dev/null
real 5m55.910s # 5 Minutes!
user 4m51.922s
sys 0m6.816s

----------------------------------------------------------------

Robert Klemme

1/6/2007 2:02:00 PM

0

On 06.01.2007 01:26, Erik Veenstra wrote:
> I had to send huge files over a network to another machine.
> Most of these files were image files for QEMU: typically 4 GB,
> of which only a small portion (~ 400 MB) was used. Both client
> and server were Ruby programs on Linux boxes, communicating via
> FTP.
>
> I thought it was a good idea to use sparse files [1], so I
> searched for a SparseFile class, couldn't find one and wrote
> one myself:
>
> http://www.erikveen.dds.nl/rubycodesnippets/index....
>
> It seems to work pretty well... ;]
>
> Any thoughts? Ideas? Comments? Do you want it to be available
> as a library/gem? Should it become part of Ruby Facets? Trans?

Looks good! Do you think it's feasible and reasonable to include it
into the std lib's File class? Could be a flag in open mode, i.e.

File.open("foo", "wsb") do |io|
....

Maybe functionality can go into a Module and which will be included if
the flag is present?

Kind regards

robert

Erik Veenstra

1/7/2007 5:34:00 PM

0

> Looks good! Do you think it's feasible and reasonable to
> include it into the std lib's File class? Could be a flag in
> open mode, i.e.

Reasonable? Well, I think so. It's just a special kind of file
handling. However, it's not too special.

Feasible? I don't know. Somebody who understandes the code in
file.c could answer that. That's not me... ;]

Is somebody able and willing to patch file.c? I would do it
myself, if I was a C programmer...

> File.open("foo", "wsb") do |io|

That sounds reasonable.

gegroet,
Erik V. - http://www.erikve...

Olivier

1/7/2007 6:10:00 PM

0

Hi,

I didn't know about this kind of files, so I googled around for some
explanations. Then, I looked at your code, and I have a question :

What's the purpose of this strange String in the following line ?

SPARSE_BLOCKS[length] ||= "%SCRIPT_SPARSEFILE%00" * length

Is it juste a fake string, or does it have a particular meaning ?

Thanks for any answer !

--
Olivier Renaud

Erik Veenstra

1/7/2007 9:08:00 PM

0

> What's the purpose of this strange String in the following
> line ?
>
> SPARSE_BLOCKS[length] ||= "%SCRIPT_SPARSEFILE%00" * length

Oops... Should have been:

SPARSE_BLOCKS[length] ||= "\000" * length

The Ruby code which generates my site, does a gsub with a
string as the replacement. That means that "\0" has a special
meaning... ;]

Using gsub in the block form seems to work..

Thanks.

gegroet,
Erik V. - http://www.erikve...

Olivier

1/7/2007 11:06:00 PM

0

Le dimanche 07 janvier 2007 22:07, Erik Veenstra a écrit :
> > What's the purpose of this strange String in the following
> > line ?
> >
> > SPARSE_BLOCKS[length] ||= "%SCRIPT_SPARSEFILE%00" * length
>
> Oops... Should have been:
>
> SPARSE_BLOCKS[length] ||= "\000" * length
>
> The Ruby code which generates my site, does a gsub with a
> string as the replacement. That means that "\0" has a special
> meaning... ;]
>
> Using gsub in the block form seems to work..
>
> Thanks.
>
> gegroet,
> Erik V. - http://www.erikve...

Oh yes, I really thought I was missing something :)
Thanks !

--
Olivier Renaud

Erik Veenstra

1/8/2007 12:45:00 AM

0

I updated the implementation [1] of SparseFile [2] and added
some unit tests [3].

Any thoughts? Ideas? Comments? I welcome any feedback.

gegroet,
Erik V. - http://www.erikve...

[1] http://localhost:1234/rubycodesnippets/index.html#4.1.0
[2] http://localhost:1234/rubycodesnippets/index.html#4.0.0
[3] http://localhost:1234/rubycodesnippets/index.html#4.3.0