[lnkForumImage]
TotalShareware - Download Free Software
Confronta i prezzi di migliaia di prodotti.
Asp Forum
 Home | Login | Register | Search 


 

Forums >

comp.lang.python

start function in new process

wongjoekmeu@yahoo.com

3/5/2010 7:22:00 PM

Hello all,

I would like to run a python function completely in a new process. For
example I have a parent process. That parent process needs to start a
child process in which the function is called. After the child process
is finished however I want that the child process should stop and then
only the parent process remain. When I start the function I want to
pass arguments also which include a function reference. I tried
os.fork() as I need to write it in linux. However it seems like the
child process remain alive. I discovered this when I loop through the
function. I see that the child process continue with the loop and
starts again a new grand child untill the loop is finished. See code
snippet. f() is the function I want to run in a new process. It got as
one of the input a refence to a function c().
Function g() I called in a loop for 3 times. Now it seems that when
child process get started in the function g() it return to the loop
and it calls g() itself and starts itself a nwe child process. I do
not want thant. I want the child process to perform the function f()
and just finished. Does anyone know how I can achieve this ?


def c():
print "function c"

def f(a,b, d):
# function I want to run in a new child process
print a, b
d()

def g():
pidID = os.fork()
if pidID == 0:
# child do something here
f(2,3,c)
else:
# parent do something here
print "Parent

for i in range(3):
g()
9 Answers

Martin P. Hellwig

3/5/2010 7:40:00 PM

0

On 03/05/10 19:21, wongjoekmeu@yahoo.com wrote:
<cut os.fork problem>
Any specific reason why threading.Thread or multiprocessing is not
suitable to solve your problem?

--
mph

wongjoekmeu@yahoo.com

3/5/2010 7:46:00 PM

0

On 5 mrt, 20:40, "Martin P. Hellwig" <martin.hell...@dcuktec.org>
wrote:
> On 03/05/10 19:21, wongjoek...@yahoo.com wrote:
> <cut os.fork problem>
> Any specific reason why threading.Thread or multiprocessing is not
> suitable to solve your problem?
>
> --
> mph

Because I got a memory leak in my function f(). It uses scipy, numpy,
pylab, and I am not planning to solve the memory leak because its too
complicated. So I thought of just calling the function then when it is
finished the process is gone and all memory is released. With
threading I don't think I would solve this problem. I am not sure
though.

Martin P. Hellwig

3/5/2010 8:03:00 PM

0

On 03/05/10 19:45, wongjoekmeu@yahoo.com wrote:
> On 5 mrt, 20:40, "Martin P. Hellwig"<martin.hell...@dcuktec.org>
> wrote:
>> On 03/05/10 19:21, wongjoek...@yahoo.com wrote:
>> <cut os.fork problem>
>> Any specific reason why threading.Thread or multiprocessing is not
>> suitable to solve your problem?
>>
>> --
>> mph
>
> Because I got a memory leak in my function f(). It uses scipy, numpy,
> pylab, and I am not planning to solve the memory leak because its too
> complicated. So I thought of just calling the function then when it is
> finished the process is gone and all memory is released. With
> threading I don't think I would solve this problem. I am not sure
> though.

I would be surprised if you can't do the same with
subprocess/multiprocessing, since you seem to know how to identify the
memory leak it shouldn't be a problem scripting out a test to see if it
works this way. I would be interested though in your findings.

--
mph

wongjoekmeu@yahoo.com

3/5/2010 8:10:00 PM

0

On 5 mrt, 21:02, "Martin P. Hellwig" <martin.hell...@dcuktec.org>
wrote:
> On 03/05/10 19:45, wongjoek...@yahoo.com wrote:
>
>
>
> > On 5 mrt, 20:40, "Martin P. Hellwig"<martin.hell...@dcuktec.org>
> > wrote:
> >> On 03/05/10 19:21, wongjoek...@yahoo.com wrote:
> >> <cut os.fork problem>
> >> Any specific reason why threading.Thread or multiprocessing is not
> >> suitable to solve your problem?
>
> >> --
> >> mph
>
> > Because I got a memory leak in my function f(). It uses scipy, numpy,
> > pylab, and I am not planning to solve the memory leak because its too
> > complicated. So I thought of just calling the function then when it is
> > finished the process is gone and all memory is released. With
> > threading I don't think I would solve this problem. I am not sure
> > though.
>
> I would be surprised if you can't do the same with
> subprocess/multiprocessing, since you seem to know how to identify the
> memory leak it shouldn't be a problem scripting out a test to see if it
> works this way. I would be interested though in your findings.
>
> --
> mph

I can't use multiprocessing module since it comes only with python 2.6
and I am bound to python2.4. But subprocess does exist in python2.4,
but the question now is, how do I achieve that ? Any example ?

Jonathan Gardner

3/5/2010 8:40:00 PM

0

You're really close. See inline comment below.

On Fri, Mar 5, 2010 at 11:21 AM, wongjoekmeu@yahoo.com
<wongjoekmeu@yahoo.com> wrote:
>
> def c():
>    print "function c"
>
> def f(a,b, d):
>    # function I want to run in a new child process
>    print a, b
>    d()
>
> def g():
>    pidID = os.fork()
>    if pidID == 0:
>        # child do something here
>        f(2,3,c)

You'll need to exit here -- not return.
http://docs.python.org/library/os.htm...

>    else:
>        # parent do something here
>        print "Parent
>
> for i in range(3):
>    g()
>

--
Jonathan Gardner
jgardner@jonathangardner.net

Martin P. Hellwig

3/5/2010 8:41:00 PM

0

On 03/05/10 20:09, wongjoekmeu@yahoo.com wrote:
> On 5 mrt, 21:02, "Martin P. Hellwig"<martin.hell...@dcuktec.org>
> wrote:
>> On 03/05/10 19:45, wongjoek...@yahoo.com wrote:
>>
>>
>>
>>> On 5 mrt, 20:40, "Martin P. Hellwig"<martin.hell...@dcuktec.org>
>>> wrote:
>>>> On 03/05/10 19:21, wongjoek...@yahoo.com wrote:
>>>> <cut os.fork problem>
>>>> Any specific reason why threading.Thread or multiprocessing is not
>>>> suitable to solve your problem?
>>
>>>> --
>>>> mph
>>
>>> Because I got a memory leak in my function f(). It uses scipy, numpy,
>>> pylab, and I am not planning to solve the memory leak because its too
>>> complicated. So I thought of just calling the function then when it is
>>> finished the process is gone and all memory is released. With
>>> threading I don't think I would solve this problem. I am not sure
>>> though.
>>
>> I would be surprised if you can't do the same with
>> subprocess/multiprocessing, since you seem to know how to identify the
>> memory leak it shouldn't be a problem scripting out a test to see if it
>> works this way. I would be interested though in your findings.
>>
>> --
>> mph
>
> I can't use multiprocessing module since it comes only with python 2.6
> and I am bound to python2.4. But subprocess does exist in python2.4,
> but the question now is, how do I achieve that ? Any example ?

Sure, for example if I want to check the openssl version (didn't specify
I need to provide a useful example :-)

I would normally do on the command line:
[martin@aspire8930 /usr/home/martin/Desktop]$ /usr/bin/openssl version
OpenSSL 0.9.8k 25 Mar 2009

The python subprocess equivalent is:
[martin@aspire8930 /usr/home/martin/Desktop]$ python
Python 2.6.4 (r264:75706, Jan 31 2010, 20:52:16)
[GCC 4.2.1 20070719 [FreeBSD]] on freebsd8
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess as _sp
>>> ssl_version = _sp.Popen(['/usr/bin/openssl', 'version'],
stdout=_sp.PIPE)
>>> print(ssl_version.stdout.readlines())
['OpenSSL 0.9.8k 25 Mar 2009\n']
>>> quit()
[martin@aspire8930 /usr/home/martin/Desktop]$

If you get any error in the Popen part, you probably did not use the
full path or an illegal parameter.

If you get a long wait in the readlines part, this usually means that
there is either nothing written to stdout or something is still being
written (I am not sure about this part though, it has been a while).

hth
--
mph

Robert Kern

3/5/2010 8:50:00 PM

0

On 2010-03-05 14:09 PM, wongjoekmeu@yahoo.com wrote:

> I can't use multiprocessing module since it comes only with python 2.6
> and I am bound to python2.4.

It is available as a third party package for Python 2.4:

http://pypi.python.org/pypi/multi...

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

wongjoekmeu@yahoo.com

3/6/2010 9:46:00 AM

0

On 5 mrt, 21:40, "Martin P. Hellwig" <martin.hell...@dcuktec.org>
wrote:
> On 03/05/10 20:09, wongjoek...@yahoo.com wrote:
>
>
>
> > On 5 mrt, 21:02, "Martin P. Hellwig"<martin.hell...@dcuktec.org>
> > wrote:
> >> On 03/05/10 19:45, wongjoek...@yahoo.com wrote:
>
> >>> On 5 mrt, 20:40, "Martin P. Hellwig"<martin.hell...@dcuktec.org>
> >>> wrote:
> >>>> On 03/05/10 19:21, wongjoek...@yahoo.com wrote:
> >>>> <cut os.fork problem>
> >>>> Any specific reason why threading.Thread or multiprocessing is not
> >>>> suitable to solve your problem?
>
> >>>> --
> >>>> mph
>
> >>> Because I got a memory leak in my function f(). It uses scipy, numpy,
> >>> pylab, and I am not planning to solve the memory leak because its too
> >>> complicated. So I thought of just calling the function then when it is
> >>> finished the process is gone and all memory is released. With
> >>> threading I don't think I would solve this problem. I am not sure
> >>> though.
>
> >> I would be surprised if you can't do the same with
> >> subprocess/multiprocessing, since you seem to know how to identify the
> >> memory leak it shouldn't be a problem scripting out a test to see if it
> >> works this way. I would be interested though in your findings.
>
> >> --
> >> mph
>
> > I can't use multiprocessing module since it comes only with python 2.6
> > and I am bound to python2.4. But subprocess does exist in python2.4,
> > but the question now is, how do I achieve that ? Any example ?
>
> Sure, for example if I want to check the openssl version (didn't specify
> I need to provide a useful example :-)
>
> I would normally do on the command line:
> [martin@aspire8930 /usr/home/martin/Desktop]$ /usr/bin/openssl version
> OpenSSL 0.9.8k 25 Mar 2009
>
> The python subprocess equivalent is:
> [martin@aspire8930 /usr/home/martin/Desktop]$ python
> Python 2.6.4 (r264:75706, Jan 31 2010, 20:52:16)
> [GCC 4.2.1 20070719  [FreeBSD]] on freebsd8
> Type "help", "copyright", "credits" or "license" for more information.
>  >>> import subprocess as _sp
>  >>> ssl_version = _sp.Popen(['/usr/bin/openssl', 'version'],
> stdout=_sp.PIPE)
>  >>> print(ssl_version.stdout.readlines())
> ['OpenSSL 0.9.8k 25 Mar 2009\n']
>  >>> quit()
> [martin@aspire8930 /usr/home/martin/Desktop]$
>
> If you get any error in the Popen part, you probably did not use the
> full path or an illegal parameter.
>
> If you get a long wait in the readlines part, this usually means that
> there is either nothing written to stdout or something is still being
> written (I am not sure about this part though, it has been a while).
>
> hth
> --
> mph

Yes, I saw this example also before. HOwever what I want is to call an
internal function which gets a reference of another internal function
as input and not calling an external program. Do you have any example
on that with subprocess module ?
Thanks.

Martin P. Hellwig

3/6/2010 12:28:00 PM

0

On 03/06/10 09:45, wongjoekmeu@yahoo.com wrote:
<cut>
>
> Yes, I saw this example also before. HOwever what I want is to call an
> internal function which gets a reference of another internal function
> as input and not calling an external program. Do you have any example
> on that with subprocess module ?
> Thanks.

Well yes that is possible but you would more or less be just reinventing
parts of the multiprocessing module, so I would use that one (at the
location Robert Kern suggested).

--
mph