Intercepting stdout of a subprocess while it is running

Question

If this is my subprocess:

import time, sys for i in range(200): sys.stdout.write( 'reading %i\n'%i ) time.sleep(.02)

And this is the script controlling and modifying the output of the subprocess:

import subprocess, time, sys print 'starting' proc = subprocess.Popen( 'c:/test_apps/testcr.py', shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE ) print 'process created' while True: #next_line = proc.communicate()[0] next_line = proc.stdout.readline() if next_line == '' and proc.poll() != None: break sys.stdout.write(next_line) sys.stdout.flush() print 'done'

Why is readline and communicate waiting until the process is done running? Is there a simple way to pass (and modify) the subprocess' stdout real-time?

I'm on Windows XP.

Related: How to flush output of Python print?

Piotr Dobrogost
– Piotr Dobrogost

2012-10-30 20:31:42 +00:00
Commented Oct 30, 2012 at 20:31 — Piotr Dobrogost
– Piotr Dobrogost, Commented Oct 30, 2012 at 20:31

Kamil Kisiel · Accepted Answer · 2009-02-09 06:12:49Z

16

As Charles already mentioned, the problem is buffering. I ran in to a similar problem when writing some modules for SNMPd, and solved it by replacing stdout with an auto-flushing version.

I used the following code, inspired by some posts on ActiveState:

class FlushFile(object): """Write-only flushing wrapper for file-type objects.""" def __init__(self, f): self.f = f def write(self, x): self.f.write(x) self.f.flush() # Replace stdout with an automatically flushing version sys.stdout = FlushFile(sys.__stdout__)

answered Feb 9, 2009 at 6:12

Kamil Kisiel

20.6k13 gold badges50 silver badges57 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Paul Over a year ago

I don't see how that is any different than calling sys.stdout.flush() after every sys.stdout.readline(), which is what I do. I also tried setting bufsize=0 for the subprocess.

bobince Over a year ago

The flush is needed in the subprocess, not the parent process.

Kamil Kisiel Over a year ago

Yes, in the example the subprocess is also a python script. So replace the stdout in the subprocess. Calling sys.stdout.flush() in the parent process doesn't do anything.

Paul Over a year ago

Ok. I see what i did there. Of course this child process is just a sample. My real process is a giant piece of compiled FORTRAN to which I do not have access to source. In which case I just need to hope that the child does not have buffered output? What then does subprocess.Popen's bufsize do?

Kamil Kisiel Over a year ago

As far as I know, it's the application code that determines the size of the output buffer. I don't think you can do anything externally, unless it's dynamically linked and you preload a library that replaces the system calls. But that's a huge hack and out of the scope of this question :)

|

Charles Duffy · Accepted Answer · 2009-02-09 08:20:50Z

Process output is buffered. On more UNIXy operating systems (or Cygwin), the pexpect module is available, which recites all the necessary incantations to avoid buffering-related issues. However, these incantations require a working pty module, which is not available on native (non-cygwin) win32 Python builds.

In the example case where you control the subprocess, you can just have it call sys.stdout.flush() where necessary -- but for arbitrary subprocesses, that option isn't available.

See also the question "Why not just use a pipe (popen())?" in the pexpect FAQ.

Collectives™ on Stack Overflow

Intercepting stdout of a subprocess while it is running

2 Answers 2

6 Comments

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

6 Comments

Comments

Linked

Related