subprocess.Popen.stdout - reading stdout in real-time (again)

Trying to write to and read from pipes to a sub-process is tricky because of the default buffering going on in both directions. It's extremely easy to get a deadlock where one or the other process (parent or child) is reading from an empty buffer, writing into a full buffer or doing a blocking read on a buffer that's awaiting data before the system libraries flush it.

For more modest amounts of data the Popen.communicate() method might be sufficient. However, for data that exceeds its buffering you'd probably get stalled processes (similar to what you're already seeing?)

You might want to look for details on using the fcntl module and making one or the other (or both) of your file descriptors non-blocking. In that case, of course, you'll have to wrap all reads and/or writes to those file descriptors in the appropriate exception handling to handle the "EWOULDBLOCK" events. (I don't remember the exact Python exception that's raised for these).

A completely different approach would be for your parent to use the select module and os.fork() ... and for the child process to execve() the target program after directly handling any file dup()ing. (Basically you'd be re-implement parts of Popen() but with different parent file descriptor (PIPE) handling.

Incidentally, .communicate, at least in Python's 2.5 and 2.6 standard libraries, will only handle about 64K of remote data (on Linux and FreeBSD). This number may vary based on various factors (possibly including the build options used to compile your Python interpreter, or the version of libc being linked to it). It is NOT simply limited by available memory (despite J.F. Sebastian's assertion to the contrary) but is limited to a much smaller value.

Push reading from the pipe into a separate thread that signals when a chunk of output is available:

How can I read all availably data from subprocess.Popen.stdout (non blocking)?

The bufsize=256 argument prevents 12345\n from being sent to the child process in a chunk smaller than 256 bytes, as it will be when omitting bufsize or inserting p.stdin.flush() after p.stdin.write(). Default behaviour is line-buffering.

In either case you should at least see one empty line before blocking as emitted by the first printf(\n...) in your example.

Your particular example doesn't require "real-time" interaction. The following works:

from subprocess import Popen, PIPE p = Popen(["./a.out"], stdin=PIPE, stdout=PIPE) output = p.communicate(b"12345")[0] # send input/read all output print output, 

where a.out is your example C program.

In general, for a dialog-based interaction with a subprocess you could use pexpect module (or its analogs on Windows):

import pexpect child = pexpect.spawn("./a.out") child.expect("input>>") child.sendline("12345.67890") # send a number child.expect(r"\d+\.\d+") # expect the number at the end print float(child.after) # assert that we can parse it child.close() 

I had the same problem, and "proc.communicate()" does not solve it because it waits for process terminating.

So here is what is working for me, on Windows with Python 3.5.1 :

import subprocess as sp myProcess = sp.Popen( cmd, creationflags=sp.CREATE_NEW_PROCESS_GROUP,stdout=sp.PIPE,stderr=sp.STDOUT) while i<40: i+=1 time.sleep(.5) out = myProcess.stdout.readline().decode("utf-8").rstrip() 

I guess creationflags and other arguments are not mandatory (but I don't have time to test), so this would be the minimal syntax :

myProcess = sp.Popen( cmd, stdout=sp.PIPE) for i in range(40) time.sleep(.5) out = myProcess.stdout.readline()