[lnkForumImage]
TotalShareware - Download Free Software
Confronta i prezzi di migliaia di prodotti.
Asp Forum
 Home | Login | Register | Search 


 

Forums >

comp.lang.python

popen question

Robert Latest

1/8/2008 9:20:00 AM

Hello,

look at this function:

--------------
def test():
child = os.popen('./slow')
for line in child:
print line
-------------

The program "slow" just writes the numbers 0 through 9 on stdout, one line a
second, and then quits.

I would have expected the python program to spit out a numbers one by one,
instead I see nothing for 10 seconds and then the whole output all at once.

How can I get and process the pipe's output at the pace it is generated?

Thanks,

robert
7 Answers

Marc 'BlackJack' Rintsch

1/8/2008 9:39:00 AM

0

On Tue, 08 Jan 2008 09:20:16 +0000, Robert Latest wrote:

> The program "slow" just writes the numbers 0 through 9 on stdout, one line a
> second, and then quits.
>
> I would have expected the python program to spit out a numbers one by one,
> instead I see nothing for 10 seconds and then the whole output all at once.
>
> How can I get and process the pipe's output at the pace it is generated?

Both processes have to make their communication ends unbuffered or line
buffered. See the documentation of `os.popen()` for the `bufsize`
argument. And do whatever is needed to output the numbers from ``slow``
unbuffered or line buffered.

Ciao,
Marc 'BlackJack' Rintsch

Robert Latest

1/8/2008 9:47:00 AM

0

Marc 'BlackJack' Rintsch wrote:

> Both processes have to make their communication ends unbuffered or line
> buffered.

Yeah, I figured something like that.

> And do whatever is needed to output the numbers from ``slow``
> unbuffered or line buffered.

Hm, "slow" of course is just a little test program I wrote for this purpose.
In reality I want to call another program whose behavior I can't influence
(well, technically I could because it's open-source, but let's assume it to
be a black box for now).

If 'slow' or some other program does buffered output, how come I can see
its output line-by-line in the shell?

robert

Hrvoje Niksic

1/8/2008 10:00:00 AM

0

Robert Latest <boblatest@yahoo.com> writes:

> If 'slow' or some other program does buffered output, how come I can
> see its output line-by-line in the shell?

stdio uses different buffering strategies depending on the output
type. When the output is a TTY, line buffering is used; when the
output goes to a pipe or file, it is fully buffered.

> In reality I want to call another program whose behavior I can't
> influence (well, technically I could because it's open-source, but
> let's assume it to be a black box for now).

To test whether your black box buffers output to pipe, simply start it
like this:

$ ./slow | cat

If you see lines one by one, you are in luck, and you can fix things
on the Python level simply by avoiding buffering in popen. If not,
you will need to resort to more advanced hackery (e.g. fixing stdio
using LD_PRELOAD).

Robert Latest

1/8/2008 10:10:00 AM

0

Hrvoje Niksic wrote:

> stdio uses different buffering strategies depending on the output
> type. When the output is a TTY, line buffering is used; when the
> output goes to a pipe or file, it is fully buffered.

Makes sense.

> If you see lines one by one, you are in luck, and you can fix things
> on the Python level simply by avoiding buffering in popen. If not,
> you will need to resort to more advanced hackery (e.g. fixing stdio
> using LD_PRELOAD).

Do I really? After all, the shell itself doesn't hack stdio, does it?
Anyway, I'm taking this over to comp.unix.programmer since it really isn't a
python problem.

Thanks,
robert

Robert Latest

1/8/2008 10:43:00 AM

0

pexpect is the solution. Seems to wrap quite a bit of dirty pseudo-tty
hacking.

robert

Hrvoje Niksic

1/8/2008 11:53:00 AM

0

Robert Latest <boblatest@yahoo.com> writes:

>> If you see lines one by one, you are in luck, and you can fix things
>> on the Python level simply by avoiding buffering in popen. If not,
>> you will need to resort to more advanced hackery (e.g. fixing stdio
>> using LD_PRELOAD).
>
> Do I really? After all, the shell itself doesn't hack stdio, does
> it?

Have you tried the "./slow | cat" test? It's not about the shell
doing anything special, it's about slow's stdio choosing buffering
"appropriate" for the output device. The hackery I referred to would
be necessary to force slow's stdio to use line buffering over
file/pipe output simply because there is no documented way to do that
(that I know of).

> Anyway, I'm taking this over to comp.unix.programmer since it really
> isn't a python problem.

It's still an often-encountered problem, so I believe a summary of the
solution(s) would be appreciated.

kar1107@gmail.com

1/9/2008 12:59:00 AM

0

On Jan 8, 1:20 am, Robert Latest <boblat...@yahoo.com> wrote:
> Hello,
>
> look at this function:
>
> --------------
> def test():
> child = os.popen('./slow')
> for line in child:
> print line
> -------------
>
> The program "slow" just writes the numbers 0 through 9 on stdout, one line a
> second, and then quits.
>
> I would have expected the python program to spit out a numbers one by one,
> instead I see nothing for 10 seconds and then the whole output all at once.
>
> How can I get and process the pipe's output at the pace it is generated?

I've seen this problem and it took me a while to figure out what is
happening.
As other posts, I too first suspected it's a problem related to line/
full buffering on the sender side (./slow here).

It turned out that the "for line in child:" line in the iterator is
the culprit. The iterator does something like a child.readlines()
underneath (in it's __iter__ call) instead of a more logical single
line read.

Change your reading to force line-by-line read
e.g.
While True:
line = child.readline()
if not line: break
print line

Karthik

>
> Thanks,
>
> robert