[lnkForumImage]
TotalShareware - Download Free Software
Confronta i prezzi di migliaia di prodotti.
Asp Forum
 Home | Login | Register | Search 


 

Forums >

comp.lang.python

Re: parallel processing in standard library

Robert Kern

12/27/2007 9:13:00 PM

Emin.shopper Martinian.shopper wrote:
> Dear Experts,
>
> Is there any hope of a parallel processing toolkit being incorporated
> into the python standard library? I've seen a wide variety of toolkits
> each with various features and limitations. Unfortunately, each has its
> own API. For coarse-grained parallelism, I suspect I'd be pretty happy
> with many of the existing toolkits, but if I'm going to pick one API to
> learn and program to, I'd rather pick one that I'm confident is going to
> be supported for a while.
>
> So is there any hope of adoption of a parallel processing system into
> the python standard library? If not, is there any hope of something like
> the db-api for coarse grained parallelism (i.e, a common API that
> different toolkits can support)?

The problem is that for SQL databases, there is a substantial API that they can
all share. The implementations are primarily differentiated by other factors
like speed, in-memory or on-disk, embedded or server, the flavor of SQL, etc.
and only secondarily differentiated by their extensions to the DB-API. With
parallel processing, the API itself is a key differentiator between toolkits and
approaches. Different problems require different APIs, not just different
implementations.

I suspect that one of the smaller implementations like processing.py might get
adopted into the standard library if the author decides to push for it. The ones
I am thinking of are relatively new, so I imagine that it might take a couple of
years of vigorous use by the community before it gets into the standard library.

My recommendation to you is to pick one of the smaller implementations that
solves the problems in front of you. Read and understand that module so you
could maintain it yourself if you had to. Post to this list about how you use
it. Blog about it if you blog. Write some Python Cookbook recipes to show how
you solve problems with it. If there is a lively community around it, that will
help it get into the standard library. Things get into the standard library
*because* they are supported, not the other way around.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco
4 Answers

Stefan Behnel

12/28/2007 9:08:00 AM

0

Robert Kern wrote:
> The problem is that for SQL databases, there is a substantial API that they can
> all share. The implementations are primarily differentiated by other factors
> like speed, in-memory or on-disk, embedded or server, the flavor of SQL, etc.
> and only secondarily differentiated by their extensions to the DB-API. With
> parallel processing, the API itself is a key differentiator between toolkits and
> approaches. Different problems require different APIs, not just different
> implementations.

Well, there is one parallel processing API that already *is* part of stdlib:
the threading module. So the processing module would fit just nicely into the
idea of a "standard" library.

Stefan

Christian Heimes

12/28/2007 1:46:00 PM

0

Stefan Behnel wrote:
> Well, there is one parallel processing API that already *is* part of
stdlib:
> the threading module. So the processing module would fit just nicely into the
> idea of a "standard" library.

Don't you forget the select module and its siblings for I/O bound
concurrency?

Christian

Stefan Behnel

12/28/2007 2:16:00 PM

0

Christian Heimes wrote:
> Stefan Behnel wrote:
> > Well, there is one parallel processing API that already *is* part of
> stdlib:
>> the threading module. So the processing module would fit just nicely into the
>> idea of a "standard" library.
>
> Don't you forget the select module and its siblings for I/O bound
> concurrency?

Hmm, when I think of "parallel processing", it's usually about processing, not
about I/O. If it starts getting I/O bound, it's rather worth considering
single-threaded processing instead.

Stefan

Robert Kern

12/29/2007 6:16:00 AM

0

Stefan Behnel wrote:
> Robert Kern wrote:
>> The problem is that for SQL databases, there is a substantial API that they can
>> all share. The implementations are primarily differentiated by other factors
>> like speed, in-memory or on-disk, embedded or server, the flavor of SQL, etc.
>> and only secondarily differentiated by their extensions to the DB-API. With
>> parallel processing, the API itself is a key differentiator between toolkits and
>> approaches. Different problems require different APIs, not just different
>> implementations.
>
> Well, there is one parallel processing API that already *is* part of stdlib:
> the threading module. So the processing module would fit just nicely into the
> idea of a "standard" library.

True. I suspect that if any of them get into the standard library, it will be
that one.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco