Asp Forum - Re: Reading stdout & stderr from a pipe with popen3

Matt Mencel

7/17/2008 4:54:00 PM

OK...using what ara.t.howard suggested, I've rewritten a block of my code and it looks something like this...

>>>>CODE
require "open3"
require "thread"

def runner(stdin, stdout, stderr, cmd)
queue = Queue.new
stdin.puts(cmd)
errthd = Thread.new do
Thread.current.abort_on_exception = true
while(( line = stderr.gets ))
queue.push(line)
end
queue.push :stderr_done
end

outthd = Thread.new do
Thread.current.abort_on_exception = true
while(( line = stdout.gets ))
queue.push(line)
end
queue.push :stdout_done
end

inthd = Thread.new do
Thread.current.abort_on_exception = true
while(( stuff = queue.pop ))
puts stuff
break if :stdout_done and :stderr_done
end
end

inthd.join
errthd.exit
outthd.exit
end

stdin, stdout, stderr = Open3.popen3("zmprov")
runner(stdin, stdout, stderr, "selectMailbox shares")
runner(stdin, stdout, stderr, "getAllFolders")
<<<<CODE

It almost works. I found that if I didn't add the errthd.exit and outthd.exit lines, the program would get stuck on inthd.join(waiting for the break conditions?) most of the time...very rarely would it complete. But the output is always incomplete. If I run the commands directly from the command line it look something like this...

>>>>GOOD OUTPUT
$ zmprov
prov> selectMailbox shares
mailbox: shares@domain.com, size: 174.36 KB, messages: 115, unread: 59
mbox shares@domain.com> getAllFolders
Id View Unread Msg Count Path
---------- ---- ---------- ---------- ----------
1 conv 0 0 /
16 docu 0 0 /Briefcase
10 appo 0 0 /Calendar
14 mess 0 0 /Chats
7 cont 0 0 /Contacts
6 mess 0 0 /Drafts
13 cont 0 2 /Emailed Contacts
257 appo 0 0 /evite
2 mess 0 0 /Inbox
4 mess 0 0 /Junk
12 wiki 0 0 /Notebook
5 mess 0 2 /Sent
15 task 0 0 /Tasks
3 conv 0 0 /Trash
<<<<GOOD OUTPUT

When I run my program though it looks more like this...

>>>>>BAD OUTPUT
prov> mailbox: shares@domain.com, size: 174.36 KB, messages: 115, unread: 59
mbox shares@domain.com> Id View Unread Msg Count Path
<<<<<BAD OUTPUT

The first line always seems to come out right... but the 'getAllFolders' command never comes out correctly. Maybe 1 in 10 times it will return a line or two more than just the 'mbox' line, but most of the time it's just that line of the output...the first line returned from the 'getAllFolders' command. So the thread for standard out doesn't seem to be grabbing all the data out of the pipe? I tried adding a short pause 'sleep 2' above the threads to see if maybe there was some slow response from them getting in the way, but that didn't really change anything.

Any help or ideas anyone can offer?

Thanks,
Matt

----- "ara.t.howard" <ara.t.howard@gmail.com> wrote:

>
> the issue is even worse than you describe, the program can easily
> become blocked if it's stdout or stderr pipes get full - to fix the
> situation you need to use threads, one processing both stdout and
> stderr asynchronously where each may have to trigger actions on
> stdin. the general pattern is
>
>
> q = Queue.new
>
> err = Thread.new do
> Thread.current.abort_on_exception = true
>
> while(( line = stderr.gets ))
> ...
> q.push :somthing if some_condition_on(line)
> end
> q.push :stderr_done
> end
>
> out = Thread.new do
> Thread.current.abort_on_exception = true
>
> while(( line = stdout.gets ))
> ...
> q.push :something if some_condition_on(line)
> end
> q.push :stdout_done
> end
>
> in = Thread.new do
> Thread.current.abort_on_exception
>
> while(( command = q.pop ))
> ...
> break if stdout_done and stderr_done
> end
> end
>
> in.join
>
> so basically have one thread sending commands down stdin. start a
> thread each for stdout and stderr, each doing their own processing, if
>
> they encounter something which means input needs to be send push it
> onto a queue to allow the stdin thread to do it on their behave. this
>
> of course ignores exceptional conditions and coordination between the
>
> stdout and stderr threads, but it's one approach.
>
>
> the big conditions any solution needs to handle are having no output
>
> on either stderr or stdout or being blocked on a write to either due a
>
> full pipe, which is why this cannot work safely:
>
> loop do
> handle stdout.gets
> handle stderr.gets
> end
>
> check out open4 and session for examples of using threads to process
>
> both stdout and stderr concurrently.
>
> http://codeforpeople.com...
> # gem install open4 session
>
> cheers.
>
>
> a @ http://codeforp...
> --
> we can deny everything, except that we have the possibility of being
>
> better. simply reflect on that.
> h.h. the 14th dalai lama

1 Answer

ara.t.howard

7/17/2008 5:02:00 PM

On Jul 17, 2008, at 10:53 AM, Matt Mencel wrote:

> def runner(stdin, stdout, stderr, cmd)
> queue = Queue.new
> stdin.puts(cmd)

stdin.flush

>
> errthd = Thread.new do
> Thread.current.abort_on_exception = true
> while(( line = stderr.gets ))
> queue.push(line)
> end
> queue.push :stderr_done
> end
>
> outthd = Thread.new do
> Thread.current.abort_on_exception = true
> while(( line = stdout.gets ))
> queue.push(line)
> end
> queue.push :stdout_done
> end
>
> inthd = Thread.new do
> Thread.current.abort_on_exception = true
> while(( stuff = queue.pop ))
> puts stuff
> break if :stdout_done and :stderr_done

this test reads "break if true and true" !?!

you are testing truth on two symbols. you need to keep vars and set
them when :stdout_done and :stderr_done are seen on the queue.

if stuff == :stdout_done
stdout_done == true
next
end

if stuff == :stderr_done
stderr_done = true
next
end

if stdout_done and stderr_done
end

etc...

btw - you are simply re-writing the open4 gem - why not use it?

>
> end
> end
>
> inthd.join
> errthd.exit
> outthd.exit
> end

a @ http://codeforp...
--
we can deny everything, except that we have the possibility of being
better. simply reflect on that.
h.h. the 14th dalai lama

comp.lang.ruby

Re: Reading stdout & stderr from a pipe with popen3

Matt Mencel

ara.t.howard

x Login to ForumsZone