Python C program subprocess hangs at "for line in iter"

It is a block buffering issue.

What follows is an extended for your case version of my answer to Python: read streaming input from subprocess.communicate() question.

Fix stdout buffer in C program directly

stdio-based programs as a rule are line buffered if they are running interactively in a terminal and block buffered when their stdout is redirected to a pipe. In the latter case, you won't see new lines until the buffer overflows or flushed.

To avoid calling fflush() after each printf() call, you could force line buffered output by calling in a C program at the very beginning:

setvbuf(stdout, (char *) NULL, _IOLBF, 0); /* make line buffered stdout */ 

As soon as a newline is printed the buffer is flushed in this case.

Or fix it without modifying the source of C program

There is stdbuf utility that allows you to change buffering type without modifying the source code e.g.:

from subprocess import Popen, PIPE process = Popen(["stdbuf", "-oL", "./main"], stdout=PIPE, bufsize=1) for line in iter(process.stdout.readline, b''): print line, process.communicate() # close process' stream, wait for it to exit 

There are also other utilities available, see Turn off buffering in pipe.

Or use pseudo-TTY

To trick the subprocess into thinking that it is running interactively, you could use pexpect module or its analogs, for code examples that use pexpect and pty modules, see Python subprocess readlines() hangs. Here's a variation on the pty example provided there (it should work on Linux):

#!/usr/bin/env python import os import pty import sys from select import select from subprocess import Popen, STDOUT master_fd, slave_fd = pty.openpty() # provide tty to enable line buffering process = Popen("./main", stdin=slave_fd, stdout=slave_fd, stderr=STDOUT, bufsize=0, close_fds=True) timeout = .1 # ugly but otherwise `select` blocks on process' exit # code is similar to _copy() from pty.py with os.fdopen(master_fd, 'r+b', 0) as master: input_fds = [master, sys.stdin] while True: fds = select(input_fds, [], [], timeout)[0] if master in fds: # subprocess' output is ready data = os.read(master_fd, 512) # <-- doesn't block, may return less if not data: # EOF input_fds.remove(master) else: os.write(sys.stdout.fileno(), data) # copy to our stdout if sys.stdin in fds: # got user input data = os.read(sys.stdin.fileno(), 512) if not data: input_fds.remove(sys.stdin) else: master.write(data) # copy it to subprocess' stdin if not fds: # timeout in select() if process.poll() is not None: # subprocess ended # and no output is buffered <-- timeout + dead subprocess assert not select([master], [], [], 0)[0] # race is possible os.close(slave_fd) # subproces don't need it anymore break rc = process.wait() print("subprocess exited with status %d" % rc) 

Or use pty via pexpect

pexpect wraps pty handling into higher level interface:

#!/usr/bin/env python import pexpect child = pexpect.spawn("/.main") for line in child: print line, child.close() 

Q: Why not just use a pipe (popen())? explains why pseudo-TTY is useful.