real time subprocess.Popen via stdout and PIPE

Actually, the real solution is to directly redirect the stdout of the subprocess to the stdout of your process.

Indeed, with your solution, you can only print stdout, and not stderr, for instance, at the same time.

import sys from subprocess import Popen Popen("./slow_cmd_output.sh", stdout=sys.stdout, stderr=sys.stderr).communicate() 

The communicate() is so to make the call blocking until the end of the subprocess, else it would directly go to the next line and your program might terminate before the subprocess (although the redirection to your stdout will still work, even after your python script has closed, I tested it).

That way, for instance, you are redirecting both stdout and stderr, and in absolute real time.

For instance, in my case I tested with this script slow_cmd_output.sh:

#!/bin/bash for i in 1 2 3 4 5 6; do sleep 5 && echo "${i}th output" && echo "err output num ${i}" >&2; done 

To get output "in real time", subprocess is unsuitable because it can't defeat the other process's buffering strategies. That's the reason I always recommend, whenever such "real time" output grabbing is desired (quite a frequent question on stack overflow!), to use instead pexpect (everywhere but Windows -- on Windows, wexpect).

As this is a question I searched for an answer to for days, I wanted to leave this here for those who follow. While it is true that subprocess cannot combat the other process's buffering strategy, in the case where you are calling another Python script with subprocess.Popen, you CAN tell it to start an unbuffered python.

command = ["python", "-u", "python_file.py"] p = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) for line in iter(p.stdout.readline, ''): line = line.replace('\r', '').replace('\n', '') print line sys.stdout.flush() 

I have also seen cases where the popen arguments bufsize=1 and universal_newlines=True have helped with exposing the hidden stdout.

Drop the readlines() which is coalescing the output. Also you'll need to enforce line buffering since most commands will interally buffer output to a pipe. For details see: http://www.pixelbeat.org/programming/stdio_buffering/

cmd = subprocess.Popen(["ls", "-l"], stdout=subprocess.PIPE) for line in cmd.stdout: print line.rstrip("\n") 

The call to readlines is waiting for the process to exit. Replace this with a loop around cmd.stdout.readline() (note singular) and all should be well.

As stated already the issue is in the stdio library's buffering of printf like statements when no terminal is attached to the process. There is a way around this on the Windows platform anyway. There may be a similar solution on other platforms as well.

On Windows you can force create a new console at process creation. The good thing is this can remain hidden so you never see it (this is done by shell=True inside the subprocess module).

cmd = subprocess.Popen('ls -l', shell=True, stdout=PIPE, creationflags=_winapi.CREATE_NEW_CONSOLE, bufsize=1, universal_newlines=True) for line in cmd.stdout.readlines(): print line 

or

A slightly more complete solution is that you explicitly set the STARTUPINFO params which prevents launching a new and unnecessary cmd.exe shell process which shell=True did above.

class PopenBackground(subprocess.Popen): def __init__(self, *args, **kwargs): si = kwargs.get('startupinfo', subprocess.STARTUPINFO()) si.dwFlags |= _winapi.STARTF_USESHOWWINDOW si.wShowWindow = _winapi.SW_HIDE kwargs['startupinfo'] = si kwargs['creationflags'] = kwargs.get('creationflags', 0) | _winapi.CREATE_NEW_CONSOLE kwargs['bufsize'] = 1 kwargs['universal_newlines'] = True super(PopenBackground, self).__init__(*args, **kwargs) process = PopenBackground(['ls', '-l'], stdout=subprocess.PIPE) for line in cmd.stdout.readlines(): print line