Asp Forum - anydbm safe for simultaneous writes?

chris

2/27/2008 9:17:00 PM

I need simple data persistence for a cgi application that will be used
potentially by multiple clients simultaneously. So I need something
that can handle locking among writes. Sqlite probably does this, but
I am using Python 2.4.4, which does not include sqlite. The dbm-style
modules would probably be fine, but I have no idea if they are "write
safe" (I have no experience with the underlying unix stuff). Any tips
appreciated.

Thanks,
Chris

4 Answers

Dennis Lee Bieber

2/28/2008 8:30:00 AM

On Wed, 27 Feb 2008 13:17:05 -0800 (PST), chris
<chris.stromberger@gmail.com> declaimed the following in
comp.lang.python:

> that can handle locking among writes. Sqlite probably does this, but
> I am using Python 2.4.4, which does not include sqlite. The dbm-style

And does something prevent you from installing an sqlite3 binary and
pysqlite2 DB-API?
--
Wulfraed Dennis Lee Bieber KD6MOG
wlfraed@ix.netcom.com wulfraed@bestiaria.com
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: web-asst@bestiaria.com)
HTTP://www.bestiaria.com/

Brian Smith

2/28/2008 1:47:00 PM

Chris wrote:
> I need simple data persistence for a cgi application that
> will be used potentially by multiple clients simultaneously.
> So I need something that can handle locking among writes.
> Sqlite probably does this, but I am using Python 2.4.4, which
> does not include sqlite. The dbm-style modules would
> probably be fine, but I have no idea if they are "write safe"
> (I have no experience with the underlying unix stuff). Any
> tips appreciated.

No, you cannot assume that this will work without locking. Locking is
not trivial to do in Python. And, even with a working locking mechanism,
you still have to invalidate the in-memory caches any time a write to
the database is done. Futher, most dbm modules do not have ACID
properties.

I suggest intalling the pysqlite module and using it, regardless of your
version of CPython. According to pysqlite developer, the version of
pysqlite included in CPython 2.5 is old.

- Brian

chris

2/28/2008 5:06:00 PM

On Feb 28, 7:47 am, "Brian Smith" <br...@briansmith.org> wrote:
> Chris wrote:
> > I need simple data persistence for a cgi application that
> > will be used potentially by multiple clients simultaneously.
> > So I need something that can handle locking among writes.
> > Sqlite probably does this, but I am using Python 2.4.4, which
> > does not include sqlite. The dbm-style modules would
> > probably be fine, but I have no idea if they are "write safe"
> > (I have no experience with the underlying unix stuff). Any
> > tips appreciated.
>
> No, you cannot assume that this will work without locking. Locking is
> not trivial to do in Python. And, even with a working locking mechanism,
> you still have to invalidate the in-memory caches any time a write to
> the database is done. Futher, most dbm modules do not have ACID
> properties.
>
> I suggest intalling the pysqlite module and using it, regardless of your
> version of CPython. According to pysqlite developer, the version of
> pysqlite included in CPython 2.5 is old.
>
> - Brian

Thanks for the info. In response to Dennis's question--the system is
controlled by a tyrannical system administrator, so I was hoping to
not have to ask him to install anything extra. But it sounds like
sqlite (or mysql) is the way to go, so I will pursue that path.

Thanks,
Chris

Eric S. Johansson

3/1/2008 5:29:00 AM

chris wrote:
> I need simple data persistence for a cgi application that will be used
> potentially by multiple clients simultaneously. So I need something
> that can handle locking among writes. Sqlite probably does this, but
> I am using Python 2.4.4, which does not include sqlite. The dbm-style
> modules would probably be fine, but I have no idea if they are "write
> safe" (I have no experience with the underlying unix stuff). Any tips
> appreciated.

the often repeated answer that you need locking is correct but an incomplete
answer. it really depends on which DBM you are using. If you are using a
fairly recent bsdbm (a.k.a. sleepy cat) it does have the kind of lucky needs to
fairly complex transactions. Unfortunately, the API is a sufficiently
unintelligible that it will take more than an afternoon to figure out how to
even start to use it.

gdbm is a nice DBM that permits single writer/multiple readers. If you open a
DBM for read, any writer blocks. You open it for read and some times multiple
readers can get in but not always (or at least that's the way it seems here in
practice). when the DBM is busy, you will get an exception with an error value
of: (11, 'Resource temporarily unavailable'). Just busy wait until this
exception goes away and you'll get access to the DBM file. Yes, this officially
sucks but at least it's a workaround for the problem.

another way to solve this particular problem with DBM files is to stick inside a
Pyro daemon. Performance won't be too bad and you should be able to get it
working relatively easily. I will warn you that the RPC model for Pyro does
take some getting used to if you're familiar with more traditional RPC
environments. Once you wrap your head around the Pyro model, it's pretty nice.
If you want, I can send you a copy of my Pyro daemon I use to wrap a DBM so I
don't have to worry about multiple processes accessing the same DBM.

the one thing that really bothers me about the DBM interfaces is that the two
main DBM's are really quite full-featured but the documentation presents a very
sketchy description of what they support and how. As a result, I suspect that
DBMS don't get used as often as they could and people are pushed into more
complex databases because they don't understand what DBM's are capable of.

Other folks have recommended some form of SQL and while SQL light is a very nice
small database, personally, I find SQL unintelligible and I have lost more days
than I care to think about trying to figure out how to do something in SQL. As
result, I tend to go more towards databases such as metakit and buzhug
(http://buzhug.source...). the former is like gdbm and only handles a
single writer. It's really intended for single process use but I don't know if
you can put it in a Pyro controlled deamon. The latter looks pretty interesting
because the documentation implies that it supports concurrent access on a per
record level (menu item: concurrency control).

Given that I'm currently replacing a DBM for very much the same reason you are,
I'm going to try using buzhug rather than facing SQL again. I would be glad to
compare notes with you if you care to go the same route. Just let me know off list.

I wish you the best of luck in your project.

---eric

--
Speech-recognition in use. It makes mistakes, I correct some.

comp.lang.python

anydbm safe for simultaneous writes?

chris

Dennis Lee Bieber

Brian Smith

chris

Eric S. Johansson

x Login to ForumsZone