python popen.stdout.readline() hangs

Does doing a nonblocking read help you out?

import fcntl import os def nonBlockReadline(output): fd = output.fileno() fl = fcntl.fcntl(fd, fcntl.F_GETFL) fcntl.fcntl(fd, fcntl.F_SETFL, fl | os.O_NONBLOCK) try: return output.readline() except: return '' working_file = subprocess.Popen(["/pyRoot/iAmACrashyProgram"], stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.PIPE) line = nonBlockReadline(working_file.stdout) working_file.stdout.flush() while working_file != "" : print(line) line = nonBlockReadline(working_file.stdout) working_file.stdout.flush() 

I'm not sure exactly what you're trying to do, but will this work better? It just reads all the data, instead of reading only one line at a time. It's a little more readable to me.

import fcntl import os def nonBlockRead(output): fd = output.fileno() fl = fcntl.fcntl(fd, fcntl.F_GETFL) fcntl.fcntl(fd, fcntl.F_SETFL, fl | os.O_NONBLOCK) try: return output.read() except: return '' working_file = subprocess.Popen(["/pyRoot/iAmACrashyProgram"], stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.PIPE) stdout = '' while working_file.poll() is None: stdout += nonBlockRead(working_file.stdout) # we can probably save some time and just print it instead... #print(stdout) stdout = stdout.splitlines() for line in stdout: print(line) 

Edit: A generalized script which should be more suited for your use case:

import fcntl import os def nonBlockRead(output): fd = output.fileno() fl = fcntl.fcntl(fd, fcntl.F_GETFL) fcntl.fcntl(fd, fcntl.F_SETFL, fl | os.O_NONBLOCK) try: return output.read() except: return '' working_file = subprocess.Popen(["/pyRoot/iAmACrashyProgram"], stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.PIPE) while working_file.poll() is None: stdout = nonBlockRead(working_file.stdout) # make sure it returned data if stdout: # process data working_file.stdin.write(something) 

In addition to the previous answers, there is another case which might be blocking your stdout.readline(). I see you have redirected both stdout and stderr to subprocess.pipe but you're only ever reading stdout. The stderr is never read.

If your program throws enough errors to fill up the buffer (which has a default size of 8192 bytes), and the program still needs to write errors, the child process will wait until the buffer is empty, which in this case is never. The stdout.readline() further completes the deadlock. Consider this scenario:

1. you read the last line in stdout using stdout.readline()
2. subprocess writes stderr and fills up the buffer
3. you attempt the next stdout read
4. readline() is blocked since it is waiting for data to be written into stdout, which in turn is waiting for stderr to be written, which is further waiting for the buffer to be emptied.

If you have nothing to do with stdout, simply flush it out before readline() or pass it to /dev/null instead of a pipe.

working_file = subprocess.Popen(["/pyRoot/iAmACrashyProgram"], stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.DEVNULL) 

otherwise if stderr is imp, you can handle it separately using:

if stderr.readline(): do something 

You could also spawn a thread to read stderr, or perform a a non blocking read as suggested above.