[lnkForumImage]
TotalShareware - Download Free Software
Confronta i prezzi di migliaia di prodotti.
Asp Forum
 Home | Login | Register | Search 


 

Forums >

comp.lang.python

future multi-threading for-loops

Aaron Brady

2/5/2008 3:22:00 AM

Some iterables and control loops can be multithreaded. Worries that
it takes a syntax change.

for X in A:
def f( x ):
normal suite( x )
start_new_thread( target= f, args= ( X, ) )

Perhaps a control-flow wrapper, or method on iterable.

@parallel
for X in A:
normal suite( X )

for X in parallel( A ):
normal suite( X )

Discussion presued about multi-core systems. Allow user certain
control over what runs on multi-core. Clearly, not generally
applicable. -- But, from __future__ import does change syntax.
14 Answers

Aaron Brady

2/5/2008 5:46:00 AM

0

On Feb 4, 9:22 pm, castiro...@gmail.com wrote:
> Some iterables and control loops can be multithreaded.  Worries that
> it takes a syntax change.
>
> for X in A:
>     def f( x ):
>         normal suite( x )
>     start_new_thread( target= f, args= ( X, ) )
>
> Perhaps a control-flow wrapper, or method on iterable.
>
> @parallel
> for X in A:
>     normal suite( X )
>
> for X in parallel( A ):
>     normal suite( X )
>
> Discussion presued about multi-core systems.  Allow user certain
> control over what runs on multi-core.  Clearly, not generally
> applicable.  -- But, from __future__ import does change syntax.

Some timing stats: On Windows XP, Python 3.0a2.

[timing code, 10,000 calls]
[ f( X ) ]: 0.0210021106034
[ start_new_thread( f, X ) ]: 1.15759908033
[ Thread( f, X ).start() ]: 1.85400099733
[ Thread( f, X ).start and .join() ]: 1.93716743329

Are threads an OS bottleneck?

Gabriel Genellina

2/5/2008 6:27:00 AM

0

On 5 feb, 03:46, castiro...@gmail.com wrote:

> Some timing stats: On Windows XP, Python 3.0a2.
>
> [timing code, 10,000 calls]
> [ f( X ) ]: 0.0210021106034
> [ start_new_thread( f, X ) ]: 1.15759908033
> [ Thread( f, X ).start() ]: 1.85400099733
> [ Thread( f, X ).start and .join() ]: 1.93716743329
>
> Are threads an OS bottleneck?

I don't understand your threading issues, but I would not use 3.0a2
for benchmarking anything (except benchmarking Python 3.0 itself).
AFAIK the developers are first trying to get it right and stable;
speed issues will be addressed later.

--
Gabriel Genellina

Aaron Brady

2/5/2008 6:37:00 AM

0

On Feb 5, 12:26 am, Gabriel Genellina <gagsl-...@yahoo.com.ar> wrote:
> On 5 feb, 03:46, castiro...@gmail.com wrote:
>
> > Some timing stats: On Windows XP, Python 3.0a2.
>
> > [timing code, 10,000 calls]
> > [ f( X ) ]: 0.0210021106034
> > [ start_new_thread( f, X ) ]: 1.15759908033
> > [ Thread( f, X ).start() ]: 1.85400099733
> > [ Thread( f, X ).start and .join() ]: 1.93716743329
>
> > Are threads an OS bottleneck?
>
> I don't understand your threading issues, but I would not use 3.0a2
> for benchmarking anything (except benchmarking Python 3.0 itself).
> AFAIK the developers are first trying to get it right and stable;
> speed issues will be addressed later.
>
> --
> Gabriel Genellina

Multi-threaded control flow is a worthwhile priority.

Marc 'BlackJack' Rintsch

2/5/2008 7:21:00 AM

0

On Mon, 04 Feb 2008 19:22:29 -0800, castironpi wrote:

> Some iterables and control loops can be multithreaded. Worries that
> it takes a syntax change.
>
> for X in A:
> def f( x ):
> normal suite( x )
> start_new_thread( target= f, args= ( X, ) )
>
> Perhaps a control-flow wrapper, or method on iterable.
>
> @parallel
> for X in A:
> normal suite( X )
>
> for X in parallel( A ):
> normal suite( X )
>
> Discussion presued about multi-core systems. Allow user certain
> control over what runs on multi-core. Clearly, not generally
> applicable. -- But, from __future__ import does change syntax.

Why not simply writing a function?

def execute_parallel(f, A):
for args in A:
start_new_thread(target=f, args=args)


def f(x):
normal_suit(x)

parallel(f, A)

Ciao,
Marc 'BlackJack' Rintsch

Boris Borcic

2/5/2008 9:47:00 AM

0

castironpi@gmail.com wrote:
> On Feb 5, 12:26 am, Gabriel Genellina <gagsl-...@yahoo.com.ar> wrote:
>> On 5 feb, 03:46, castiro...@gmail.com wrote:
>>
>>> Some timing stats: On Windows XP, Python 3.0a2.
(...)
>>> Are threads an OS bottleneck?
>> I don't understand your threading issues, but I would not use 3.0a2
>> for benchmarking anything (except benchmarking Python 3.0 itself).
>> AFAIK the developers are first trying to get it right and stable;
>> speed issues will be addressed later.
>>
>> --
>> Gabriel Genellina
>
> Multi-threaded control flow is a worthwhile priority.

What he said is that you should apply the idea to the mixing of your thread of
benchmarking with the thread of python development.

Christian Heimes

2/5/2008 4:11:00 PM

0

castironpi@gmail.com wrote:
> Multi-threaded control flow is a worthwhile priority.

It is? That's totally new to me. Given the fact that threads don't scale
I highly doubt your claim, too.

dmitrey

2/5/2008 5:42:00 PM

0

On Feb 5, 5:22 am, castiro...@gmail.com wrote:
> Some iterables and control loops can be multithreaded. Worries that
> it takes a syntax change.
>
> for X in A:
> def f( x ):
> normal suite( x )
> start_new_thread( target= f, args= ( X, ) )
>
> Perhaps a control-flow wrapper, or method on iterable.
>
> @parallel
> for X in A:
> normal suite( X )
>
> for X in parallel( A ):
> normal suite( X )
>
I would propose "for X IN A" for parallel and remain "for X in A" for
sequential.
BTW for fortress lang I had proposed "for X <- A" and "for X <= A" for
sequential/parallel instead of current "for X <- seq(A)", "for X <-
A", mb they will implement my way instead.

dmitrey

2/5/2008 5:44:00 PM

0

On Feb 5, 6:11 pm, Christian Heimes <li...@cheimes.de> wrote:
> castiro...@gmail.com wrote:
> > Multi-threaded control flow is a worthwhile priority.
>
> It is? That's totally new to me. Given the fact that threads don't scale
> I highly doubt your claim, too.

I would propose "for X IN A" for parallel and remain "for X in A" for
sequential.
BTW for fortress lang I had proposed "for X <- A" and "for X <= A" for
sequential/parallel instead of current "for X <- seq(A)", "for X <-
A", mb they will implement my way instead.

John Nagle

2/5/2008 6:12:00 PM

0

Christian Heimes wrote:
> castironpi@gmail.com wrote:
>> Multi-threaded control flow is a worthwhile priority.
>
> It is? That's totally new to me. Given the fact that threads don't scale
> I highly doubt your claim, too.

There's plenty that can be done to automatically extract parallelism
from programs, but given the architecture of CPython, with the "global
interpreter lock" and the inability to detect dynamism at compile time,
it's probably not going to be feasible with that implementation.

I've been to a recent talk at Stanford (in EE380) in which
someone was describing an optimizing compiler for Matlab intended
to generate production DSP code. I've heard two other talks
on how to automatically parallelize loops for execution in a GPU.
It's quite possible to do extreme optimizations like that.
But those are things you do after solving the problems of being
10x-30x slower than C.

The real optimization trick for Python is figuring out at compile time what
might change at run time, and what won't be. Then all the things that
can't change can be hard-bound during compilation. Shed Skin does
some of that.

John Nagle

Aaron Brady

2/6/2008 12:15:00 AM

0

On Feb 5, 1:21 am, Marc 'BlackJack' Rintsch <bj_...@gmx.net> wrote:
> On Mon, 04 Feb 2008 19:22:29 -0800, castironpi wrote:
> > Some iterables and control loops can be multithreaded.  Worries that
> > it takes a syntax change.
>
> > for X in A:
> >     def f( x ):
> >         normal suite( x )
> >     start_new_thread( target= f, args= ( X, ) )
>
> > Perhaps a control-flow wrapper, or method on iterable.
>
> > @parallel
> > for X in A:
> >     normal suite( X )
>
> > for X in parallel( A ):
> >     normal suite( X )
>
> > Discussion presued about multi-core systems.  Allow user certain
> > control over what runs on multi-core.  Clearly, not generally
> > applicable.  -- But, from __future__ import does change syntax.
>
> Why not simply writing a function?
>
> def execute_parallel(f, A):
>     for args in A:
>         start_new_thread(target=f, args=args)
>
> def f(x):
>     normal_suit(x)
>
> parallel(f, A)
>
> Ciao,
>         Marc 'BlackJack' Rintsch- Hide quoted text -
>
> - Show quoted text -

Are code blocks first-class objects, without customizing syntax?

from functools import partial
from threading import Thread
from random import uniform

A= range(10)

@partial( partial, partial )
def execute_parallel( f, A ):
for args in A:
Thread( target= f, args= ( args, ) ).start()

from time import sleep

@execute_parallel( A= A )
def f(x):
sleep( uniform( 0, .2 ) )
print(x)

[Windows XP threading timing code, 10,000 calls]: [ allocate,
CreateEvent, CreateThread, WaitForMultipleObjects ]: 0.887416 secs