[lnkForumImage]
TotalShareware - Download Free Software
Confronta i prezzi di migliaia di prodotti.
Asp Forum
 Home | Login | Register | Search 


 

Forums >

comp.lang.ruby

Looking for a Fast Persistent Store

Bob Hutchison

8/9/2006 1:30:00 PM

Hi,

I'm looking for a persistent store, where:
* I can use it from Ruby
* it is fast
* it is transactional
* it can update more than one store in a single transaction
* simple String -> String mappings in the store are sufficient
* ideally uses files in the filesystem

These kinds of stores are normally implemented as persistent hashes
or BTrees (or both). I know that Sleepycat's Berkeley DB <http://
www.sleepycat.com/> does this, and I've used Java based systems that
do this. I also know of some C based things but they don't have Ruby
wrappers. I can't find anything in the Ruby world, and I don't care
too much if it isn't pure Ruby.

I know about Purple <http://purple.rubyforg... and the QDMB Ruby
wrapper <http://qdbm.sourceforg.... Neither do the multiple hash/
BTree in a transaction thing.

The trick appears to be with transactions.

I know this can be done easily enough in a Relational DB, but I know
that, for example, JDBC/MySQL combination is significantly slower at
what I need to do than Perst (a pure Java thing that's startlingly
fast). I've taken a look at sqlite, which satisfies all requirements
but I can't shake this feeling that things can be better.

Does anyone have any ideas?

Thanks,
Bob

----
Bob Hutchison -- blogs at <http://www.rec...
hutch/>
Recursive Design Inc. -- <http://www.rec...>
Raconteur -- <http://www.raconteur...
xampl for Ruby -- <http://rubyforge.org/projects/...
46 Answers

Harold Hausman

8/9/2006 1:56:00 PM

0

On 8/9/06, Bob Hutchison <hutch@recursive.ca> wrote:
> Hi,
>
> I'm looking for a persistent store, where:
> * I can use it from Ruby
> * it is fast
> * it is transactional
> * it can update more than one store in a single transaction
> * simple String -> String mappings in the store are sufficient
> * ideally uses files in the filesystem
>
...
>
> Does anyone have any ideas?
>
> Thanks,
> Bob
>

Hi Bob,

If you're the fun and adventurous type you might want to look into
Mongoose. ( http://rubyforge.org/projects... ) It's new, and
claims to be fast. In the past I wrote some small apps that used the
same authors other system called KirbyBase which was very nice, I
haven't tried Mongoose yet, but I'm happily awaiting a problem with
which to use it on. Though now that I'm thinking about it I don't
know if it has built-in "transaction" support. I suppose it depends on
what you want it to do.

Have you tried SQLite and found it to be too slow? It sounds to me
like you might be prematurely optimizing. We have several medium to
medium-large sized apps that make heavy use of SQLite and speed has
been not a problem at all.

What domain are you working in that requires this kind of speed
anyway? Just curious.

Hope that helps,
-Harold

Robert Klemme

8/9/2006 2:05:00 PM

0

On 09.08.2006 15:30, Bob Hutchison wrote:
> Hi,
>
> I'm looking for a persistent store, where:
> * I can use it from Ruby
> * it is fast
> * it is transactional
> * it can update more than one store in a single transaction
> * simple String -> String mappings in the store are sufficient
> * ideally uses files in the filesystem

What about PStore?

robert

Ara.T.Howard

8/9/2006 2:08:00 PM

0

M. Edward (Ed) Borasky

8/9/2006 2:15:00 PM

0

Harold Hausman wrote:
> Have you tried SQLite and found it to be too slow? It sounds to me
> like you might be prematurely optimizing. We have several medium to
> medium-large sized apps that make heavy use of SQLite and speed has
> been not a problem at all.
On Linux, at any rate, one ought to be able to make SQLite extremely
fast by adding RAM and tuning the I/O subsystem and the kernel, and
using tricks like memory mapped files, assuming the database in question
isn't too large. I'm assuming it's small, otherwise he'd pretty much
need a humongous database server and an industrial strength database
like Oracle or PostgreSQL.

Bob Hutchison

8/9/2006 2:39:00 PM

0


On Aug 9, 2006, at 9:56 AM, Harold Hausman wrote:
> Hi Bob,
>
> If you're the fun and adventurous type you might want to look into
> Mongoose. ( http://rubyforge.org/projects... ) It's new, and
> claims to be fast. In the past I wrote some small apps that used the
> same authors other system called KirbyBase which was very nice, I
> haven't tried Mongoose yet, but I'm happily awaiting a problem with
> which to use it on. Though now that I'm thinking about it I don't
> know if it has built-in "transaction" support. I suppose it depends on
> what you want it to do.

I have had a look at Mongoose, and quite like it. I'm sure to find a
use for it. However, for this particular situation Mongoose's
transactions are too week.

>
> Have you tried SQLite and found it to be too slow? It sounds to me
> like you might be prematurely optimizing. We have several medium to
> medium-large sized apps that make heavy use of SQLite and speed has
> been not a problem at all.

In the Java world things like Perst and JDBM were very fast. SQLite
was problematic for some reason that is a bit foggy just now (I'll
think of it).

>
> What domain are you working in that requires this kind of speed
> anyway? Just curious.

I write web based applications. The situations I'm thinking of are
either financial or CMS-like applications. In the financial cases
I've got situations where there might be a few million (probably not
more than ten million, well maybe 20 million) small things tucked
away. In the case of CMS-like stuff there are far fewer things, lets
say 25k at most and usually more like a thousand or so, but they are
bigger things.

The key factor (pardon the pun) is that the *vast* majority of the
time (and I know how to make it 100% of the time) queries are as
you'd have in a hash table (i.e. a single key where the key is
usually a String).

This is for the persistent store associated with xampl (see URL in my
signature). Xampl already has a few persistent stores but I now need
one a little fancier.


>
> Hope that helps,
> -Harold
>

Thanks Harold.

Cheers,
Bob

----
Bob Hutchison -- blogs at <http://www.rec...
hutch/>
Recursive Design Inc. -- <http://www.rec...>
Raconteur -- <http://www.raconteur...
xampl for Ruby -- <http://rubyforge.org/projects/...

Bob Hutchison

8/9/2006 2:42:00 PM

0


On Aug 9, 2006, at 10:05 AM, Robert Klemme wrote:

> On 09.08.2006 15:30, Bob Hutchison wrote:
>> Hi,
>> I'm looking for a persistent store, where:
>> * I can use it from Ruby
>> * it is fast
>> * it is transactional
>> * it can update more than one store in a single transaction
>> * simple String -> String mappings in the store are sufficient
>> * ideally uses files in the filesystem
>
> What about PStore?

Oh. Right. PStore never registered with me before.

I just had a look at it and it seems to be along the lines of QDBM
and Purple, transactional on a single store and each store being one
hash/tree.

Thanks for pointing that out.

Cheers,
Bob

>
> robert
>

----
Bob Hutchison -- blogs at <http://www.rec...
hutch/>
Recursive Design Inc. -- <http://www.rec...>
Raconteur -- <http://www.raconteur...
xampl for Ruby -- <http://rubyforge.org/projects/...

Bob Hutchison

8/9/2006 3:08:00 PM

0

Hi Ara,

On Aug 9, 2006, at 10:08 AM, ara.t.howard@noaa.gov wrote:

> On Wed, 9 Aug 2006, Bob Hutchison wrote:
>
>> Hi,
>>
>> I'm looking for a persistent store, where:
>> * I can use it from Ruby
>> * it is fast
>> * it is transactional
>> * it can update more than one store in a single transaction
>> * simple String -> String mappings in the store are sufficient
>> * ideally uses files in the filesystem
>>
>> These kinds of stores are normally implemented as persistent
>> hashes or
>> BTrees (or both). I know that Sleepycat's Berkeley DB
>> <http://www.sleepyca... does this, and I've used Java based
>> systems that
>> do this. I also know of some C based things but they don't have Ruby
>> wrappers. I can't find anything in the Ruby world, and I don't
>> care too much
>> if it isn't pure Ruby.
>>
>> I know about Purple <http://purple.rubyforg... and the QDMB
>> Ruby wrapper <http://qdbm.sourceforg.... Neither do the
>> multiple hash/BTree in a transaction thing.
>>
>> The trick appears to be with transactions.
>
> check out joel's fsdb - it's very nice if you want to go pure
> ruby. i've used
> it on several projects.

I'm already using that (version 0.5 -- I can't get to RAA right now
for some reason and there are no files on Rubyforge for fsdb so I
don't know if there is a more recent version). In version 0.5 the
transactions were not sufficient I think (it would be nice if I was
wrong).

>
>> I know this can be done easily enough in a Relational DB, but I
>> know that,
>> for example, JDBC/MySQL combination is significantly slower at
>> what I need
>> to do than Perst (a pure Java thing that's startlingly fast). I've
>> taken a
>> look at sqlite, which satisfies all requirements but I can't shake
>> this
>> feeling that things can be better.
>>
>> Does anyone have any ideas?
>
> sqlite is hard to beat - i use it in at least 20 production systems
> and have
> never had a single error. ruby queue uses it under the hood for
> the cluster
> job store and this is used over nfs - we've been through disk
> failures and
> power failures and come through unscathed.

That's good to hear. Don't misunderstand, I *like* SQLite but I don't
know that it is suitable. If I can't find anything else I'll use it.

The thing that makes me nervous is that I can do what I need with
just two tables. One that has (key, value-kind, value), and another
that has (index-name, index-value, key, value-kind). Each column is a
String. The vast majority of queries would be based on key and value-
kind on the first table. The remaining queries would be a select/join
kind of thing. And I can very easily jam the key and value-kind into
the same field (but that would be an optimisation that may not be
necessary). The trouble is that the first table might get to be very
very large. I don't know how SQLite behaves with a single huge table.
I suppose I'm going to have to find out.

>
> it's also very fast, esp if you use in memory tables, but on linux
> it's just
> as easy to copy the db to /dev/shm and then go...

That's interesting.

>
> if you end up using it try with my arrayfields package - james'
> sqlite binding
> detects it automatically and the tuples with come back as arrays
> with named
> field access - it's faster and requires less memory than a hash,
> and is also
> really convenient to code with.
>

I noticed that the other day. Nice.

Cheers,
Bob

> cheers.
>
> -a
> --
> to foster inner awareness, introspection, and reasoning is more
> efficient than
> meditation and prayer.
> - h.h. the 14th dali lama
>

----
Bob Hutchison -- blogs at <http://www.rec...
hutch/>
Recursive Design Inc. -- <http://www.rec...>
Raconteur -- <http://www.raconteur...
xampl for Ruby -- <http://rubyforge.org/projects/...

Bob Hutchison

8/9/2006 3:11:00 PM

0


On Aug 9, 2006, at 10:15 AM, M. Edward (Ed) Borasky wrote:

> Harold Hausman wrote:
>> Have you tried SQLite and found it to be too slow? It sounds to me
>> like you might be prematurely optimizing. We have several medium to
>> medium-large sized apps that make heavy use of SQLite and speed has
>> been not a problem at all.
> On Linux, at any rate, one ought to be able to make SQLite
> extremely fast by adding RAM and tuning the I/O subsystem and the
> kernel, and using tricks like memory mapped files, assuming the
> database in question isn't too large. I'm assuming it's small,
> otherwise he'd pretty much need a humongous database server and an
> industrial strength database like Oracle or PostgreSQL.
>

A few GB max for one set of applications, far far less for others.

Cheers,
Bob

----
Bob Hutchison -- blogs at <http://www.rec...
hutch/>
Recursive Design Inc. -- <http://www.rec...>
Raconteur -- <http://www.raconteur...
xampl for Ruby -- <http://rubyforge.org/projects/...

Ara.T.Howard

8/9/2006 3:16:00 PM

0

Kroeger, Simon (ext)

8/9/2006 3:17:00 PM

0



> From: Bob Hutchison [mailto:hutch@recursive.ca]
> Sent: Wednesday, August 09, 2006 4:39 PM
>
> > Have you tried SQLite and found it to be too slow? It sounds to me
> > like you might be prematurely optimizing. We have several medium to
> > medium-large sized apps that make heavy use of SQLite and speed has
> > been not a problem at all.
>
> In the Java world things like Perst and JDBM were very fast. SQLite
> was problematic for some reason that is a bit foggy just now (I'll
> think of it).

All I can say is SQLite3 was amazingly fast at solving every problem
I threw at it (I don't know for the ruby bindings, but they appear
to be relativ thin).

I have to admit though that I never had more than a few 100 thousand
items in a single table.

Being blessed with the power and speed of complex SQL statements is
invaluable especialy in a scripting language.

cheers

Simon