Python: How to read stdout non blocking from another process?

Question

During the runtime of a process I would like to read its stdout and write it to a file. Any attempt of mine however failed because no matter what I tried as soon as I tried reading from the stdout it blocked until the process finished.

Here is a snippet of what I am trying to do. (The first part is simply a python script that writes something to stdout.)

import subprocess p = subprocess.Popen('python -c \'\ from time import sleep\n\ for i in range(3):\n\ sleep(1)\n\ print "Hello", i\ \'', shell = True, stdout = subprocess.PIPE) while p.poll() == None: #read the stdout continuously pass print "Done"

I know that there are multiple questions out there that deal with the same subject. However, none of the ones I found was able to answer my question.

@Keith Well, this is kind of my question. How do you read the stream without blocking it. I hope to find that part of the code in an answer somewhere below. ^^ — Woltan
– Woltan, Commented Jun 20, 2011 at 15:50

Keith · Accepted Answer · 2011-06-20 16:07:33Z

What is happening is buffering on the writer side. Since you are writing such small chunks from the little code snippet the underlying FILE object is buffering the output until the end. The following works as you expect.

#!/usr/bin/python import sys import subprocess p = subprocess.Popen("""python -c ' from time import sleep ; import sys for i in range(3): sleep(1) print "Hello", i sys.stdout.flush() '""", shell = True, stdout = subprocess.PIPE) while True: inline = p.stdout.readline() if not inline: break sys.stdout.write(inline) sys.stdout.flush() print "Done"

However, you may not be expecting the right thing. The buffering is there to reduce the number of system calls in order to make the system more efficient. Does it really matter to you that the whole text is buffered until the end before you write it to a file? Don't you still get all the output in the file?

Hi keith, what if I wanted to send some commands to my process like p.stdin.write(command1), then read the chunk of output from that (as in the output resulting only from this input!) and then, p.stdin.write(command1) and read that chunk and so on?
@Jack Yes, you can do that. You should use shell=False in that case, and stdin=PIPE. You can also use the communicate method to make that easier. See the docs.

Corey Goldberg · Accepted Answer · 2011-06-21 13:34:17Z

the following code would print stdout line by line as the subprocess runs until the readline() method returns an empty string:

p = subprocess.Popen(cmd, stdout=subprocess.PIPE) for line in iter(p.stdout.readline, ''): print line p.stdout.close() print 'Done'

update relating to your question better:

import subprocess p = subprocess.Popen(['python'], stdout=subprocess.PIPE, stdin=subprocess.PIPE) p.stdin.write(""" from time import sleep ; import sys for i in range(3): sleep(1) print "Hello", i sys.stdout.flush() """) p.stdin.close() for line in iter(p.stdout.readline, ''): print line p.stdout.close() print 'Done'

Removing the parameter shell = True from the Popen constructor results in a OSError: [Errno 2] No such file or directory exception on my machine. And if I leave the shell = True parameter in there, your reading routine still blocks the reading process.
I pass it exactly like posted in my question. I simply removed the shell = True statement.

iceaway · Accepted Answer · 2011-06-20 15:41:57Z

You can use subprocess.communicate() to get the output from stdout. Something like:

while(p.poll() == None): #read the stdout continuously print(p.communicate()[0]) pass

More info available at: http://docs.python.org/library/subprocess.html

Unfortunately the communicate method also blocks and only returns the stdout after the process p has finished. No continuous readout is possible.

Collectives™ on Stack Overflow

Python: How to read stdout non blocking from another process?

3 Answers 3

2 Comments

2 Comments

1 Comment

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

2 Comments

1 Comment

Related