[lnkForumImage]
TotalShareware - Download Free Software
Confronta i prezzi di migliaia di prodotti.
Asp Forum
 Home | Login | Register | Search 


 

Forums >

comp.lang.ruby

Beyond YAML? (scaling

Bil Kleb

5/3/2007 12:47:00 PM

Hi,

I've been using YAML files to store hashes of numbers, e.g.,

{ ["O_Kc_01"] => [ 0.01232, 0.01212, 0.03222, ... ], ... }

This has worked wonderfully for portability and visibility
into the system as I've been creating it.

Recently, however, I've increased my problem size by orders
of magnitude in both the number of variables and the number
of associated values. The resulting YAML files are prohibitive:
10s of MBs big and requiring 10s of minutes to dump/load.

Where should I go from here?

Thanks,
--
Bil Kleb
http://fun3d.lar...
26 Answers

James Gray

5/3/2007 2:04:00 PM

0

On May 3, 2007, at 8:50 AM, Bil Kleb wrote:

> Hi,
>
> I've been using YAML files to store hashes of numbers, e.g.,
>
> { ["O_Kc_01"] => [ 0.01232, 0.01212, 0.03222, ... ], ... }
>
> This has worked wonderfully for portability and visibility
> into the system as I've been creating it.
>
> Recently, however, I've increased my problem size by orders
> of magnitude in both the number of variables and the number
> of associated values. The resulting YAML files are prohibitive:
> 10s of MBs big and requiring 10s of minutes to dump/load.
>
> Where should I go from here?

Some random thoughts:

* If they are just super straightforward lists of numbers like this a
trivial flat file scheme, say with one number per line, might get the
job done.
* XML can be pretty darn easy to output manually and if you use
REXML's stream parser (not slurping everything into a DOM) you should
be able to read it reasonably quick.
* If you are willing to sacrifice a little visibility, you can always
take the step up to a real database, even if it's just sqlite. These
have varying degrees of portability as well.
* You might want to look at KirbyBase. (It has a younger brother
Mongoose, but that uses binary output.)

Hope something in there helps.

James Edward Gray II

Brian Candler

5/3/2007 2:05:00 PM

0

On Thu, May 03, 2007 at 10:50:06PM +0900, Bil Kleb wrote:
> I've been using YAML files to store hashes of numbers, e.g.,
>
> { ["O_Kc_01"] => [ 0.01232, 0.01212, 0.03222, ... ], ... }
>
> This has worked wonderfully for portability and visibility
> into the system as I've been creating it.
>
> Recently, however, I've increased my problem size by orders
> of magnitude in both the number of variables and the number
> of associated values. The resulting YAML files are prohibitive:
> 10s of MBs big and requiring 10s of minutes to dump/load.
>
> Where should I go from here?

Use a SQL database?

It all depends what sort of processing you're doing. If you're adding to a
dataset (rather than starting with an entirely fresh data set each time),
having a database makes sense. If you're doing searches across the data,
and/or if the data is larger than the available amount of RAM, then a
database makes sense. If you're only touching small subsets of the data at
any one time, then a database makes sense.

Put it another way, does your processing really require you to read the
entire collection of objects into RAM before you can perform any processing?

If it does, and your serialisation needs are as simple as it appears above,
then maybe something like CSV would be better.

O_Kc_01,0.01232,0.01212,0.03222,...

If the source of the data is another Ruby program, then Marshal will be much
faster than YAML (but unfortunately binary).

You could consider using something like Madeleine:
http://madeleine.ruby...
This snapshots your object tree to disk (using Marshal by default I think,
but can also use YAML). You can then make incremental changes and
occasionally rewrite the snapshot.

B.

Jamey Cribbs

5/3/2007 2:05:00 PM

0

Bil Kleb wrote:
> Hi,
>
> I've been using YAML files to store hashes of numbers, e.g.,
>
> { ["O_Kc_01"] => [ 0.01232, 0.01212, 0.03222, ... ], ... }
>
> This has worked wonderfully for portability and visibility
> into the system as I've been creating it.
>
> Recently, however, I've increased my problem size by orders
> of magnitude in both the number of variables and the number
> of associated values. The resulting YAML files are prohibitive:
> 10s of MBs big and requiring 10s of minutes to dump/load.
>
> Where should I go from here?

Hey, Bil. If you don't mind a couple of shameless plugs, you might want
to try KirbyBase or Mongoose.

KirbyBase should be faster than YAML and it still stores the data in
plain text files, if that is important to you.

Mongoose is faster than KirbyBase, at the expense of the data not being
stored as plain text.

I don't know if either will be fast enough for you.

HTH,

Jamey Cribbs

khaines

5/3/2007 2:12:00 PM

0

Bil Kleb

5/3/2007 2:42:00 PM

0

khaines@enigo.com wrote:
>
> I guess that depends on whether you need the files to be easily readable
> or not. If you don't, Marshal will be faster than YAML.

At this point, I'm looking for an easy out that will
reduce size and increase speed, and I'm willing to
go binary if necessary.

Of the answers I've seen so far (thanks everyone!),
migrating to Marshal seems to be the Simplest Thing
That Could Possibly Work.

Thanks,
--
Bil Kleb
http://fun3d.lar...

Bil Kleb

5/3/2007 3:11:00 PM

0

Bil Kleb wrote:
> khaines@enigo.com wrote:
>>
>> I guess that depends on whether you need the files to be easily
>> readable or not. If you don't, Marshal will be faster than YAML.
>
> At this point, I'm looking for an easy out that will
> reduce size and increase speed, and I'm willing to
> go binary if necessary.
>
> Of the answers I've seen so far (thanks everyone!),
> migrating to Marshal seems to be the Simplest Thing
> That Could Possibly Work.

Well, maybe not so simple...

`dump': can't dump hash with default proc (TypeError)

which seems to be due to the trick I learned from zenspider
and drbrain to quickly setup a hash of arrays:,

Hash.new{ |hash,key| hash[key]=[] }

Later,
--
Bil Kleb
http://fun3d.lar...

Bil Kleb

5/3/2007 3:17:00 PM

0

Jamey Cribbs wrote:
>
> Hey, Bil.

Hi.

> KirbyBase should be faster than YAML and it still stores the data in
> plain text files, if that is important to you.

Plain text will be too big -- I've got an n^2 problem.

> Mongoose is faster than KirbyBase, at the expense of the data not being
> stored as plain text.

Sounds intriguing, but where can I find some docs? So far, I'm
coming up empty...

Regards,
--
Bil Kleb
http://fun3d.lar...

Jamey Cribbs

5/3/2007 3:29:00 PM

0

Bil Kleb wrote:
> Jamey Cribbs wrote:
>>
>> Hey, Bil.
>
> Hi.
>
>> KirbyBase should be faster than YAML and it still stores the data in
>> plain text files, if that is important to you.
>
> Plain text will be too big -- I've got an n^2 problem.
>
>> Mongoose is faster than KirbyBase, at the expense of the data not
>> being stored as plain text.
>
> Sounds intriguing, but where can I find some docs? So far, I'm
> coming up empty...

Docs are light compared to KirbyBase. If you download the
distribution, there is the README file, some pretty good examples in the
aptly named "examples" directory, and unit tests in the "tests" directory.

HTH,

Jamey

Bil Kleb

5/3/2007 3:32:00 PM

0

Brian Candler wrote:
>
> Use a SQL database?

I always suspect that I should be doing that more often,
but as my experience with databases is rather limited
and infrequent, I always shy away from those as James
already knows. Regardless, I should probably overcome
my aggressive incompetence one day!

> It all depends what sort of processing you're doing. If you're adding to a
> dataset (rather than starting with an entirely fresh data set each time),
> having a database makes sense.

In this point, I'm generating an entirely fresh data set
each time, but I can foresee a point where that will change
to an incremental model...

> Put it another way, does your processing really require you to read the
> entire collection of objects into RAM before you can perform any processing?

Yes, AFAIK, but I suppose there are algorithms that could
compute statistical correlations incrementally.

> You could consider using something like Madeleine:
> http://madeleine.ruby...
> This snapshots your object tree to disk (using Marshal by default I think,
> but can also use YAML). You can then make incremental changes and
> occasionally rewrite the snapshot.

Probably not a good fit as I won't change existing data,
only add new...

Thanks,
--
Bil Kleb
http://fun3d.lar...

Bil Kleb

5/3/2007 3:34:00 PM

0

Jamey Cribbs wrote:
>
> Docs are light compared to KirbyBase. If you download the
> distribution, there is the README file, some pretty good examples in the
> aptly named "examples" directory, and unit tests in the "tests" directory.

Roger, I was afraid you'd say that. :)

Please throw those up on your Rubyforge webpage at some point?

Later,
--
Bil Kleb
http://fun3d.lar...