Asp Forum - Dictionary or Database—Please advise

Jeremy

2/26/2010 3:58:00 PM

I have lots of data that I currently store in dictionaries. However,
the memory requirements are becoming a problem. I am considering
using a database of some sorts instead, but I have never used them
before. Would a database be more memory efficient than a dictionary?
I also need platform independence without having to install a database
and Python interface on all the platforms I'll be using. Is there
something built-in to Python that will allow me to do this?

Thanks,
Jeremy

17 Answers

Chris Rebert

2/26/2010 4:30:00 PM

On Fri, Feb 26, 2010 at 7:58 AM, Jeremy <jlconlin@gmail.com> wrote:
> I have lots of data that I currently store in dictionaries. However,
> the memory requirements are becoming a problem. I am considering
> using a database of some sorts instead, but I have never used them
> before. Would a database be more memory efficient than a dictionary?
> I also need platform independence without having to install a database
> and Python interface on all the platforms I'll be using. Is there
> something built-in to Python that will allow me to do this?

If you won't be using the SQL features of the database, `shelve` might
be another option; from what I can grok, I sounds like a dictionary
stored mostly on disk rather than entirely in RAM (not 100% sure
though):
http://docs.python.org/library/s...

It's in the std lib and supports several native dbm libraries for its
backend; one of them should almost always be present.

Cheers,
Chris
--
http://blog.re...

lbolla

2/26/2010 4:30:00 PM

On Feb 26, 3:58 pm, Jeremy <jlcon...@gmail.com> wrote:
> I have lots of data that I currently store in dictionaries. However,
> the memory requirements are becoming a problem. I am considering
> using a database of some sorts instead, but I have never used them
> before. Would a database be more memory efficient than a dictionary?
> I also need platform independence without having to install a database
> and Python interface on all the platforms I'll be using. Is there
> something built-in to Python that will allow me to do this?
>
> Thanks,
> Jeremy

Maybe shelve would be enough for your needs?
http://docs.python.org/library/s...

Roy Smith

2/26/2010 4:40:00 PM

In article
<891a98fa-c398-455a-981f-bf72af77256c@s36g2000prh.googlegroups.com>,
Jeremy <jlconlin@gmail.com> wrote:

> I have lots of data that I currently store in dictionaries. However,
> the memory requirements are becoming a problem. I am considering
> using a database of some sorts instead, but I have never used them
> before. Would a database be more memory efficient than a dictionary?
> I also need platform independence without having to install a database
> and Python interface on all the platforms I'll be using. Is there
> something built-in to Python that will allow me to do this?
>
> Thanks,
> Jeremy

This is a very vague question, so it'll get a vague answer :-)

If you have so much data that you're running into memory problems, then
yes, storing the data externally in an disk-resident database seems like a
reasonable idea.

Once you get into databases, platform independence will be an issue. There
are many databases out there to pick from. If you want something which
will work on a lot of platforms, a reasonable place to start looking is
MySQL. It's free, runs on lots of platforms, has good Python support, and
there's lots of people on the net who know it and are willing to give help
and advice.

Databases have a bit of a learning curve. If you've never done any
database work, don't expect to download MySql (or any other database) this
afternoon and be up and running by tomorrow.

Whatever database you pick, you're almost certainly going to end up having
to install it wherever you install your application. There's no such thing
as a universally available database that you can expect to be available
everywhere.

Have fun!

Jeremy

2/26/2010 5:31:00 PM

On Feb 26, 9:29 am, Chris Rebert <c...@rebertia.com> wrote:
> On Fri, Feb 26, 2010 at 7:58 AM, Jeremy <jlcon...@gmail.com> wrote:
> > I have lots of data that I currently store in dictionaries. However,
> > the memory requirements are becoming a problem. I am considering
> > using a database of some sorts instead, but I have never used them
> > before. Would a database be more memory efficient than a dictionary?
> > I also need platform independence without having to install a database
> > and Python interface on all the platforms I'll be using. Is there
> > something built-in to Python that will allow me to do this?
>
> If you won't be using the SQL features of the database, `shelve` might
> be another option; from what I can grok, I sounds like a dictionary
> stored mostly on disk rather than entirely in RAM (not 100% sure
> though):http://docs.python.org/library/s...
>
> It's in the std lib and supports several native dbm libraries for its
> backend; one of them should almost always be present.
>
> Cheers,
> Chris
> --http://blog.re...

Shelve looks like an interesting option, but what might pose an issue
is that I'm reading the data from a disk instead of memory. I didn't
mention this in my original post, but I was hoping that by using a
database it would be more memory efficient in storing data in RAM so I
wouldn't have to read from (or swap to/from) disk. Would using the
shelve package make reading/writing data from disk faster since it is
in a binary format?

Jeremy

D'Arcy J.M. Cain

2/26/2010 5:58:00 PM

On Fri, 26 Feb 2010 11:39:51 -0500
Roy Smith <roy@panix.com> wrote:
> Once you get into databases, platform independence will be an issue. There
> are many databases out there to pick from. If you want something which
> will work on a lot of platforms, a reasonable place to start looking is
> MySQL. It's free, runs on lots of platforms, has good Python support, and
> there's lots of people on the net who know it and are willing to give help
> and advice.

Or PostgreSQL. It's free, runs on lots of platforms, has good Python
support, and there's lots of people on the net who know it and are
willing to give help and advice. In addition, it is a truly enterprise
level, SQL standard, fully transactional database. Don't start with
MySQL and uprade to PostgreSQL later when you get big. Start with the
best one now and be ready.

> Databases have a bit of a learning curve. If you've never done any
> database work, don't expect to download MySql (or any other database) this
> afternoon and be up and running by tomorrow.

Whatever database you get, there will probably be plenty of tutorials.
See http://www.postgresql.org/docs/current/interactive/tut...
for the PostgreSQL one.

--
D'Arcy J.M. Cain <darcy@druid.net> | Democracy is three wolves
http://www.druid.... | and a sheep voting on
+1 416 425 1212 (DoD#0082) (eNTP) | what's for dinner.

mk

2/26/2010 6:04:00 PM

Jeremy wrote:
> I have lots of data that I currently store in dictionaries. However,
> the memory requirements are becoming a problem. I am considering
> using a database of some sorts instead, but I have never used them
> before. Would a database be more memory efficient than a dictionary?
> I also need platform independence without having to install a database
> and Python interface on all the platforms I'll be using. Is there
> something built-in to Python that will allow me to do this?

Since you use dictionaries, I guess that simple store saving key:value
will do?

If so, bsddb support built into Python will do just nicely.

bsddb is multiplatform, although I have not personally tested if a
binary db created on one platform will be usable on another. You'd have
to check this.

Caveat: from what some people say I gather that binary format between
bsddb versions tends to change.

There's also ultra-cool-and-modern Tokyo Cabinet key:value store with
Python bindings:

http://pypi.python.org/...

I didn't test it, though.

Regards,
mk

mk

2/26/2010 6:27:00 PM

D'Arcy J.M. Cain wrote:

> Or PostgreSQL. It's free, runs on lots of platforms, has good Python
> support, and there's lots of people on the net who know it and are
> willing to give help and advice. In addition, it is a truly enterprise
> level, SQL standard, fully transactional database. Don't start with
> MySQL and uprade to PostgreSQL later when you get big. Start with the
> best one now and be ready.

I second that: I burned my fingers on MySQL quite a few times and don't
want to have anything to do with it anymore. Eventually you hit the wall
with MySQL (although I haven't tested latest and best, perhaps they
improved).

E.g. don't even get me started on replication that tends to randomly
fizzle out quietly without telling you anything about it. Or that
FOREIGN KEY is accepted but referential integrity is not enforced. Or
that if you want real transactions, you have to choose InnoDB table type
but then you lose much of the performance. Etc.

No, if you have a choice, avoid MySQL and go for PGSQL. It's fantastic,
if (necessarily) complex.

Regards,
mk

CM

2/26/2010 8:27:00 PM

On Feb 26, 10:58 am, Jeremy <jlcon...@gmail.com> wrote:
> I have lots of data

How much is "lots"?

> that I currently store in dictionaries. However,
> the memory requirements are becoming a problem. I am considering
> using a database of some sorts instead, but I have never used them
> before. Would a database be more memory efficient than a dictionary?

What do you mean by more efficient?

> I also need platform independence without having to install a database
> and Python interface on all the platforms I'll be using. Is there
> something built-in to Python that will allow me to do this?

The SQLite datbase engine is built into Python 2.5 and up. I have
heard on this list that there may be problems with it with Python 2.6
and up on Linux, but I've stayed with 2.5 and it works fine for me on
WinXP, Vista, and Linux.

You can use it as a disk-stored single database file, or an in-memory-
only database. The SQLite website (http://www.s...) claims it
is the "most widely deployed SQL database engine in the world.", for
what that's worth.

Have a look at this:
http://docs.python.org/library/sq...

Che

>
> Thanks,
> Jeremy

Patrick Sabin

2/26/2010 8:57:00 PM

> Shelve looks like an interesting option, but what might pose an issue
> is that I'm reading the data from a disk instead of memory. I didn't
> mention this in my original post, but I was hoping that by using a
> database it would be more memory efficient in storing data in RAM so I
> wouldn't have to read from (or swap to/from) disk.

A database usually stores data on disk and not in RAM. However you could
use sqlite with :memory:, so that it runs in RAM.

> Would using the
> shelve package make reading/writing data from disk faster since it is
> in a binary format?
>
Faster than what? Shelve uses caching, so it is likely to be faster than
a self-made solution. However, accessing disk is much slower than
accessing RAM.

> Jeremy
- Patrick

aahz

2/26/2010 10:39:00 PM

In article <mailman.296.1267217819.4577.python-list@python.org>,
Patrick Sabin <patrick.just4fun@gmail.com> wrote:
>
>A database usually stores data on disk and not in RAM. However you could
>use sqlite with :memory:, so that it runs in RAM.

The OP wants transparent caching, so :memory: wouldn't work.
--
Aahz (aahz@pythoncraft.com) <*> http://www.python...

"Many customs in this life persist because they ease friction and promote
productivity as a result of universal agreement, and whether they are
precisely the optimal choices is much less important." --Henry Spencer

comp.lang.python

Dictionary or Database—Please advise

Jeremy

Chris Rebert

lbolla

Roy Smith

Jeremy

D'Arcy J.M. Cain

mk

mk

CM

Patrick Sabin

aahz

x Login to ForumsZone