Asp Forum - Re: Multiprocessing problem

MRAB

3/3/2010 12:53:00 AM

Matt Chaput wrote:
> Hi,
>
> I'm having a problem with the multiprocessing package.
>
> I'm trying to use a simple pattern where a supervisor object starts a
> bunch of worker processes, instantiating them with two queues (a job
> queue for tasks to complete and an results queue for the results). The
> supervisor puts all the jobs in the "job" queue, then join()s the
> workers, and then pulls all the completed results off the "results" queue.
>
> (I don't think I can just use something like Pool.imap_unordered for
> this because the workers need to be objects with state.)
>
> Here's a simplified example:
>
> http://pastie....
>
> The problem is that seemingly randomly, but almost always, the worker
> processes will deadlock at some point and stop working before they
> complete. This will leave the whole program stalled forever. This seems
> more likely the more work each worker does (to the point where adding
> the time.sleep(0.01) as seen in the example code above guarantees it).
> The problem seems to occur on both Windows and Mac OS X.
>
> I've tried many random variations of the code (e.g. using JoinableQueue,
> calling cancel_join_thread() on one or both queues even though I have no
> idea what it does, etc.) but keep having the problem.
>
> Am I just using multiprocessing wrong? Is this a bug? Any advice?
>
There's a difference between multithreading and multiprocessing.

In multithreading the threads share the same address space, so objects
can be passed between the threads simply by passing references to those
objects.

In multiprocessing, however, the process don't share an address space,
so the objects themselves need to be transferred between the processes
via pipes, but the pipes have a limited capacity.

If the main process doesn't get the results from the queue until the
worker processes terminate, and the worker processes don't terminate
until they've put their results in the queue, and the pipe consequently
fills up, then deadlock can result.

comp.lang.python

Re: Multiprocessing problem

MRAB

x Login to ForumsZone