[lnkForumImage]
TotalShareware - Download Free Software
Confronta i prezzi di migliaia di prodotti.
Asp Forum
 Home | Login | Register | Search 


 

Forums >

comp.lang.ruby

[ANN] Mongoose 0.2.5 - The "Two Steps Forward, One Step Back" release

Jamey Cribbs

7/25/2006 3:27:00 PM

You can download it from: http://rubyforge.org/projects...

*What's New*

Well, there's a lot of new stuff in this release, and some old stuff put
back in, as well. John Long pointed out that the use of #instance_eval
in the 0.2.0 release would be problematic because the query block would
no longer have access to instance variables from the calling object.
After looking at this all weekend and getting feedback from a number of
people, I have decided to go back to the previous query syntax whereby
you specify the table class as a block parameter and qualify each column
name with the table class name. So, it's back to:

Dog.find { |dog| dog.breed == "German Shepard" }

instead of:

Dog.find { breed == "German Shepard" }

Besides this one step back, there have been a lot of steps forward. The
query engine code is totally re-written, thanks to input and code ideas
from Logan Capaldo. I have added a bunch of methods from ActiveRecord,
including dynamic finder methods like Table.find_by_user_name.
Additionally, I have added Table.import and Table.export methods that
allow you to get data in and out of a table via CSV. So, grab the
latest release and let me know what you think.

Documentation is still light, so the best way to learn is to look in the
"example" directory and at the unit tests.

*What is Mongoose*

Mongoose is a database management system written in Ruby. It has an
ActiveRecord-like interface, uses Skiplists for its indexing, and
Marshal for its data serialization. I named it Mongoose, because, like
Rudyard Kipling's Rikki-Tikki-Tavi, my aim is for it to be small, quick,
and friendly.

You can find rudimentary documentation in the README file and some
sample scripts in the example directory.

Jamey Cribbs
jcribbs@netpromi.com
19 Answers

GFunk913

7/26/2006 4:47:00 AM

0


Jamey Cribbs wrote:
> You can download it from: http://rubyforge.org/projects...
>
> *What's New*
>
> Well, there's a lot of new stuff in this release, and some old stuff put
> back in, as well. John Long pointed out that the use of #instance_eval
> in the 0.2.0 release would be problematic because the query block would
> no longer have access to instance variables from the calling object.
> After looking at this all weekend and getting feedback from a number of
> people, I have decided to go back to the previous query syntax whereby
> you specify the table class as a block parameter and qualify each column
> name with the table class name. So, it's back to:
>
> Dog.find { |dog| dog.breed == "German Shepard" }
>
> instead of:
>
> Dog.find { breed == "German Shepard" }
>
> Besides this one step back, there have been a lot of steps forward. The
> query engine code is totally re-written, thanks to input and code ideas
> from Logan Capaldo. I have added a bunch of methods from ActiveRecord,
> including dynamic finder methods like Table.find_by_user_name.
> Additionally, I have added Table.import and Table.export methods that
> allow you to get data in and out of a table via CSV. So, grab the
> latest release and let me know what you think.
>
> Documentation is still light, so the best way to learn is to look in the
> "example" directory and at the unit tests.
>
> *What is Mongoose*
>
> Mongoose is a database management system written in Ruby. It has an
> ActiveRecord-like interface, uses Skiplists for its indexing, and
> Marshal for its data serialization. I named it Mongoose, because, like
> Rudyard Kipling's Rikki-Tikki-Tavi, my aim is for it to be small, quick,
> and friendly.
>
> You can find rudimentary documentation in the README file and some
> sample scripts in the example directory.
>
> Jamey Cribbs
> jcribbs@netpromi.com

>

First off, great job on Mongoose, it looks very promising.

I had a quick/vague question, wondering if you could shed any light on
it. Off the top of your head, how do you feel Mongoose would perform
as the backend storage for an object persistance system, where the
objects in question are very large (i.e. marshalling to disk creates a
40MB file)?

Jamey Cribbs

7/26/2006 12:47:00 PM

0

GFunk913@gmail.com wrote:
> I had a quick/vague question, wondering if you could shed any light on
> it. Off the top of your head, how do you feel Mongoose would perform
> as the backend storage for an object persistance system, where the
> objects in question are very large (i.e. marshalling to disk creates a
> 40MB file)?
>
>
Is each object 40MB? (God, I hope not!) Or are you saying that, the
total size of all the objects saved to disk is 40MB?

If it's the latter, Mongoose might work just fine. In a test run I had
an 8MB file that had about 80,000 records (obviously, lots of small
records). Mongoose did a #find for a particular record, grabbed it,
instantiated an object from the Marshaled data, and returned it to me,
in something like 0.003 seconds.

A little more info on your data would help me give you a more accurate
answer.

Jamey

Confidentiality Notice: This email message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and/or privileged information. If you are not the intended recipient(s), you are hereby notified that any dissemination, unauthorized review, use, disclosure or distribution of this email and any materials contained in any attachments is prohibited. If you receive this message in error, or are not the intended recipient(s), please immediately notify the sender by email and destroy all copies of the original message, including attachments.

Mike Harris

7/26/2006 2:55:00 PM

0

Jamey Cribbs wrote:

> GFunk913@gmail.com wrote:
>
>> I had a quick/vague question, wondering if you could shed any light on
>> it. Off the top of your head, how do you feel Mongoose would perform
>> as the backend storage for an object persistance system, where the
>> objects in question are very large (i.e. marshalling to disk creates a
>> 40MB file)?
>>
>>
>
> Is each object 40MB? (God, I hope not!) Or are you saying that, the
> total size of all the objects saved to disk is 40MB?
>
> If it's the latter, Mongoose might work just fine. In a test run I
> had an 8MB file that had about 80,000 records (obviously, lots of
> small records). Mongoose did a #find for a particular record, grabbed
> it, instantiated an object from the Marshaled data, and returned it to
> me, in something like 0.003 seconds.
>
> A little more info on your data would help me give you a more accurate
> answer.
>
> Jamey
>
> Confidentiality Notice: This email message, including any attachments,
> is for the sole use of the intended recipient(s) and may contain
> confidential and/or privileged information. If you are not the
> intended recipient(s), you are hereby notified that any dissemination,
> unauthorized review, use, disclosure or distribution of this email and
> any materials contained in any attachments is prohibited. If you
> receive this message in error, or are not the intended recipient(s),
> please immediately notify the sender by email and destroy all copies
> of the original message, including attachments.
>
>
Sure, I certainly appreciate your help. I have an array that is 40MB
when marshalled to disk. The array contains 200 objects of approx.
equal size. These objects could certainly be further broken up, but I
would prefer not to.

Ideally, the entire array would be marshalled as one piece, with a
transparent object-persistance mechanism. However, I'm open to
marshalling its elements individually, which would probably be the best
way to go, for a number of reasons.

I'm currently using a home-grown "persistant array," where each array
element is marshalled to a seperate file and stored that way.
Obviously, this is non-ideal, which is why I'm looking at other
options. Mongoose looks like a great compromise between the flexibility
of marshalling and storing in the file system and the helpfulness of
storing in a DB.

I guess I don't really have a question anymore, it all seems pretty
straightforward at this point. Thanks for your help, and thanks for
taking the time to write and release Mongoose.

GFunk913

7/28/2006 1:44:00 AM

0

Jamey Cribbs wrote:
> You can download it from: http://rubyforge.org/projects...
>
> *What's New*
>
> Well, there's a lot of new stuff in this release, and some old stuff put
> back in, as well. John Long pointed out that the use of #instance_eval
> in the 0.2.0 release would be problematic because the query block would
> no longer have access to instance variables from the calling object.
> After looking at this all weekend and getting feedback from a number of
> people, I have decided to go back to the previous query syntax whereby
> you specify the table class as a block parameter and qualify each column
> name with the table class name. So, it's back to:
>
> Dog.find { |dog| dog.breed == "German Shepard" }
>
> instead of:
>
> Dog.find { breed == "German Shepard" }
>
> Besides this one step back, there have been a lot of steps forward. The
> query engine code is totally re-written, thanks to input and code ideas
> from Logan Capaldo. I have added a bunch of methods from ActiveRecord,
> including dynamic finder methods like Table.find_by_user_name.
> Additionally, I have added Table.import and Table.export methods that
> allow you to get data in and out of a table via CSV. So, grab the
> latest release and let me know what you think.
>
> Documentation is still light, so the best way to learn is to look in the
> "example" directory and at the unit tests.
>
> *What is Mongoose*
>
> Mongoose is a database management system written in Ruby. It has an
> ActiveRecord-like interface, uses Skiplists for its indexing, and
> Marshal for its data serialization. I named it Mongoose, because, like
> Rudyard Kipling's Rikki-Tikki-Tavi, my aim is for it to be small, quick,
> and friendly.
>
> You can find rudimentary documentation in the README file and some
> sample scripts in the example directory.
>
> Jamey Cribbs
> jcribbs@netpromi.com

Got another question for you, if you don't mind sparing a couple
minutes. Thanks in advance for taking the time.

What would be the best/easiest way to accomplish the intent of the
following code?

require 'mongoose'

# Create a class for your table.
class Thing < Mongoose::Table
end

# Create a database instance.
db = Mongoose::Database.new

# Create new table. Notice how you specify whether a column is indexed
or not.
db.create_table(:thing) do |tbl|
tbl.add_column(:foo,:dunno_what_to_put)
end

# Add a record. You can also use #create.
rec = Thing.new
rec.foo = (1..100).to_a
rec.save

puts Thing.find.first.foo.size #100

# Close database. This will write the indexes out to disk so they can
be
# initialized quickly next time.
db.close

Logan Capaldo

7/28/2006 2:07:00 AM

0


On Jul 27, 2006, at 9:45 PM, GFunk913@gmail.com wrote:

> Jamey Cribbs wrote:
>> You can download it from: http://rubyforge.org/projects...
>>
>> *What's New*
>>
>> Well, there's a lot of new stuff in this release, and some old
>> stuff put
>> back in, as well. John Long pointed out that the use of
>> #instance_eval
>> in the 0.2.0 release would be problematic because the query block
>> would
>> no longer have access to instance variables from the calling object.
>> After looking at this all weekend and getting feedback from a
>> number of
>> people, I have decided to go back to the previous query syntax
>> whereby
>> you specify the table class as a block parameter and qualify each
>> column
>> name with the table class name. So, it's back to:
>>
>> Dog.find { |dog| dog.breed == "German Shepard" }
>>
>> instead of:
>>
>> Dog.find { breed == "German Shepard" }
>>
>> Besides this one step back, there have been a lot of steps
>> forward. The
>> query engine code is totally re-written, thanks to input and code
>> ideas
>> from Logan Capaldo. I have added a bunch of methods from
>> ActiveRecord,
>> including dynamic finder methods like Table.find_by_user_name.
>> Additionally, I have added Table.import and Table.export methods that
>> allow you to get data in and out of a table via CSV. So, grab the
>> latest release and let me know what you think.
>>
>> Documentation is still light, so the best way to learn is to look
>> in the
>> "example" directory and at the unit tests.
>>
>> *What is Mongoose*
>>
>> Mongoose is a database management system written in Ruby. It has an
>> ActiveRecord-like interface, uses Skiplists for its indexing, and
>> Marshal for its data serialization. I named it Mongoose, because,
>> like
>> Rudyard Kipling's Rikki-Tikki-Tavi, my aim is for it to be small,
>> quick,
>> and friendly.
>>
>> You can find rudimentary documentation in the README file and some
>> sample scripts in the example directory.
>>
>> Jamey Cribbs
>> jcribbs@netpromi.com
>
> Got another question for you, if you don't mind sparing a couple
> minutes. Thanks in advance for taking the time.
>
> What would be the best/easiest way to accomplish the intent of the
> following code?
>
> require 'mongoose'
>
> # Create a class for your table.
> class Thing < Mongoose::Table
> end
>
> # Create a database instance.
> db = Mongoose::Database.new
>
> # Create new table. Notice how you specify whether a column is
> indexed
> or not.
> db.create_table(:thing) do |tbl|
> tbl.add_column(:foo,:dunno_what_to_put)
> end
>
> # Add a record. You can also use #create.
> rec = Thing.new
> rec.foo = (1..100).to_a
> rec.save
>
> puts Thing.find.first.foo.size #100
>
> # Close database. This will write the indexes out to disk so they can
> be
> # initialized quickly next time.
> db.close
>
>

It's a (semi-)relation db. Store the array as a set of rows.

Something like:

class ArrayTable < Mongoose::Table
end

db.create_table(:array_table) do |tbl|
tbl.add_column(:array_id, :array_position, :array_value)
end

array_id = 1

(1..100).each_with_index do |value, position|
ArrayTable.create :array_id => array_id, :array_position => position,
:array_value => value
end

Jamey Cribbs

7/28/2006 2:27:00 AM

0

GFunk913@gmail.com wrote:
> # Create new table. Notice how you specify whether a column is indexed
> or not.
> db.create_table(:thing) do |tbl|
> tbl.add_column(:foo,:dunno_what_to_put)
> end
>
> # Add a record. You can also use #create.
> rec = Thing.new
> rec.foo = (1..100).to_a
> rec.save
>
> puts Thing.find.first.foo.size #100
>
>
Logan already gave you one way to do this, and, probably from a purely
relational perspective, his suggestion is the right way to do it.

Additionally, I know that ActiveRecord has the serialize class method
that you can use to have the object specified Marshaled into and out of
the db. I am thinking of supporting that method, just because I want to
stay close to ActiveRecord's api. The funny thing is, Mongoose
*already* Marhsals all of the data (that's how it stores the table
records). Right now, I don't think I am doing much data checking in the
#save method, so you could probably get away with declaring the data
type as :string and then just going ahead and saving the array to
rec.foo like you are doing in the example. It should work.

Of course, in an upcoming version, I will make all of this work the way
it should, i.e. do more data type checking in #save and also give you
the ability to specify which columns you want to serialize.

HTH,

Jamey

GFunk913

7/28/2006 3:07:00 AM

0


Jamey Cribbs wrote:
> GFunk913@gmail.com wrote:
> > # Create new table. Notice how you specify whether a column is indexed
> > or not.
> > db.create_table(:thing) do |tbl|
> > tbl.add_column(:foo,:dunno_what_to_put)
> > end
> >
> > # Add a record. You can also use #create.
> > rec = Thing.new
> > rec.foo = (1..100).to_a
> > rec.save
> >
> > puts Thing.find.first.foo.size #100
> >
> >
> Logan already gave you one way to do this, and, probably from a purely
> relational perspective, his suggestion is the right way to do it.
>
> Additionally, I know that ActiveRecord has the serialize class method
> that you can use to have the object specified Marshaled into and out of
> the db. I am thinking of supporting that method, just because I want to
> stay close to ActiveRecord's api. The funny thing is, Mongoose
> *already* Marhsals all of the data (that's how it stores the table
> records). Right now, I don't think I am doing much data checking in the
> #save method, so you could probably get away with declaring the data
> type as :string and then just going ahead and saving the array to
> rec.foo like you are doing in the example. It should work.
>
> Of course, in an upcoming version, I will make all of this work the way
> it should, i.e. do more data type checking in #save and also give you
> the ability to specify which columns you want to serialize.
>
> HTH,
>
> Jamey

Yup, I understand i could relational-ize the array, it was a bad
example. Pretend i said an instance of some arbitrary Foo class
instead of an array. the point is I want to serialize arbitrary data,
taking advantage of marshaling.

I tried the :string method. I get a dump format error when trying to
read it back out of the found record.

I'm familiar with the serialize functionality in activerecord, that's
the basic idea. I think the problem is basically that the proper
escaping for storing dump strings isn't being done. I'll try and see
if I can hack something in to make this work.

GFunk913

7/28/2006 3:10:00 AM

0


GFunk...@gmail.com wrote:
> Jamey Cribbs wrote:
> > GFunk913@gmail.com wrote:
> > > # Create new table. Notice how you specify whether a column is indexed
> > > or not.
> > > db.create_table(:thing) do |tbl|
> > > tbl.add_column(:foo,:dunno_what_to_put)
> > > end
> > >
> > > # Add a record. You can also use #create.
> > > rec = Thing.new
> > > rec.foo = (1..100).to_a
> > > rec.save
> > >
> > > puts Thing.find.first.foo.size #100
> > >
> > >
> > Logan already gave you one way to do this, and, probably from a purely
> > relational perspective, his suggestion is the right way to do it.
> >
> > Additionally, I know that ActiveRecord has the serialize class method
> > that you can use to have the object specified Marshaled into and out of
> > the db. I am thinking of supporting that method, just because I want to
> > stay close to ActiveRecord's api. The funny thing is, Mongoose
> > *already* Marhsals all of the data (that's how it stores the table
> > records). Right now, I don't think I am doing much data checking in the
> > #save method, so you could probably get away with declaring the data
> > type as :string and then just going ahead and saving the array to
> > rec.foo like you are doing in the example. It should work.
> >
> > Of course, in an upcoming version, I will make all of this work the way
> > it should, i.e. do more data type checking in #save and also give you
> > the ability to specify which columns you want to serialize.
> >
> > HTH,
> >
> > Jamey
>
> Yup, I understand i could relational-ize the array, it was a bad
> example. Pretend i said an instance of some arbitrary Foo class
> instead of an array. the point is I want to serialize arbitrary data,
> taking advantage of marshaling.
>
> I tried the :string method. I get a dump format error when trying to
> read it back out of the found record.
>
> I'm familiar with the serialize functionality in activerecord, that's
> the basic idea. I think the problem is basically that the proper
> escaping for storing dump strings isn't being done. I'll try and see
> if I can hack something in to make this work.

The AR serialize method does some serializing via YAML. From the docs:

Specifies that the attribute by the name of attr_name should be
serialized before saving to the database and unserialized after loading
from the database. The serialization is done through YAML.

I don't know if that would be my prefered way of handling it. I'm
using this functionality in an app presently and it works fine, but it
seems uneccesary to me (unless I'm missing something, which is
certainly possible).

GFunk913

7/28/2006 5:33:00 AM

0


GFunk913@gmail.com wrote:
> Jamey Cribbs wrote:
> > GFunk913@gmail.com wrote:
> > > # Create new table. Notice how you specify whether a column is indexed
> > > or not.
> > > db.create_table(:thing) do |tbl|
> > > tbl.add_column(:foo,:dunno_what_to_put)
> > > end
> > >
> > > # Add a record. You can also use #create.
> > > rec = Thing.new
> > > rec.foo = (1..100).to_a
> > > rec.save
> > >
> > > puts Thing.find.first.foo.size #100
> > >
> > >
> > Logan already gave you one way to do this, and, probably from a purely
> > relational perspective, his suggestion is the right way to do it.
> >
> > Additionally, I know that ActiveRecord has the serialize class method
> > that you can use to have the object specified Marshaled into and out of
> > the db. I am thinking of supporting that method, just because I want to
> > stay close to ActiveRecord's api. The funny thing is, Mongoose
> > *already* Marhsals all of the data (that's how it stores the table
> > records). Right now, I don't think I am doing much data checking in the
> > #save method, so you could probably get away with declaring the data
> > type as :string and then just going ahead and saving the array to
> > rec.foo like you are doing in the example. It should work.
> >
> > Of course, in an upcoming version, I will make all of this work the way
> > it should, i.e. do more data type checking in #save and also give you
> > the ability to specify which columns you want to serialize.
> >
> > HTH,
> >
> > Jamey
>
> Yup, I understand i could relational-ize the array, it was a bad
> example. Pretend i said an instance of some arbitrary Foo class
> instead of an array. the point is I want to serialize arbitrary data,
> taking advantage of marshaling.
>
> I tried the :string method. I get a dump format error when trying to
> read it back out of the found record.
>
> I'm familiar with the serialize functionality in activerecord, that's
> the basic idea. I think the problem is basically that the proper
> escaping for storing dump strings isn't being done. I'll try and see
> if I can hack something in to make this work.

FYI, from what I can tell, at least for the array (1..9).to_a, writing
the marshal dump to mongoose and them reading it back makes 2 changes

1. adds a \r where the string contains a \n
2. removes a trailer \016 (shift out)

This is what causes the dump format error

Jamey Cribbs

7/28/2006 12:15:00 PM

0

GFunk913@gmail.com wrote:
> FYI, from what I can tell, at least for the array (1..9).to_a, writing
> the marshal dump to mongoose and them reading it back makes 2 changes
>
> 1. adds a \r where the string contains a \n
> 2. removes a trailer \016 (shift out)
>
> This is what causes the dump format error
>
>

I'll take a look at this over the weekend and see if I can get it working.

Jamey

Confidentiality Notice: This email message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and/or privileged information. If you are not the intended recipient(s), you are hereby notified that any dissemination, unauthorized review, use, disclosure or distribution of this email and any materials contained in any attachments is prohibited. If you receive this message in error, or are not the intended recipient(s), please immediately notify the sender by email and destroy all copies of the original message, including attachments.