Real-time intercepting of stdout from another process in Python

Question

I'd like to run a system process, intercept the output, and modify it real-time, line by line, in a Python script.

My best attempt, which waits for the process to complete before printing, is:

#!/usr/bin/env python import subprocess cmd = "waitsome.py" proc = subprocess.Popen(cmd, shell=True, bufsize=256, stdout=subprocess.PIPE) for line in proc.stdout: print ">>> " + line.rstrip()

The script waitsome.py simply prints a line every half a second:

#!/usr/bin/env python import time from sys import stdout print "Starting" for i in range(0,20): time.sleep(0.5) print "Hello, iteration", i stdout.flush()

Is there an easy solution to get subprocess to allow iterating over the output in real time? Do I have to use threads?

Once upon a time, I scripted in Perl, and this was a piece of cake:

open(CMD, "waitsome.py |"); while (<CMD>) { print ">>> $_"; } close(CMD);

Duplicate: stackoverflow.com/search?q=%5Bpython%5D+subprocess+real-time, specifically stackoverflow.com/questions/527197/…, stackoverflow.com/questions/803265/…, — S.Lott
– S.Lott, Commented Jul 5, 2009 at 23:34
Sorry, I only saw the first one and understood it to be a problem with the subprocess' buffering, not that of the parent python script. — Seth Johnson
– Seth Johnson, Commented Jul 6, 2009 at 2:13

Alex Martelli · Accepted Answer · 2009-07-06 01:51:07Z

Looping over a file unavoidably buffers things in pretty large chunks -- a known issue with all Python 2.* implementations. It works as you intend in Python 3.1, with the final loop being slightly different:

for line in proc.stdout: print(">>> " + str(line.rstrip()))

If upgrading to Python 3.1 is impractical (and I know it will often be!), go the other way and write the loop in an old-fashioned manner -- the following version of the loop does work as you intend in Python 2.*:

while True: line = proc.stdout.readline() if not line: break print ">>> " + line.rstrip()

Yep, but now that Python 3.1 is out (with a much better implementation of the whole I/O stack) there's no reason to stick with 3.0 (it was definitely a transitional version;-).
for line in iter(proc.stdout.readline, ''): print ">>>", line, could be used instead of the while loop on Python 2.
People talking about impractical Python 3 upgrade in 2009: Python 3 is bad... . Everyone else in 2024: Why would you use Python 2? Are you using legacy computers or something.

Jeff Younker · Accepted Answer · 2020-03-10 00:03:09Z

0

This whole thing can be encapsulated in an iterator as:

def subprocess_readlines(out): while True: line = out.readline() if not line: return yield line

And called as:

for line in subprocess_readlines(proc.stdout): print ">>>", line.rstrip()

edited Mar 10, 2020 at 0:03

answered Mar 15, 2017 at 15:35

Jeff Younker

1552 silver badges5 bronze badges

3 Comments

Seth Johnson Over a year ago

Buffering in Python has changed a lot since I wrote the original question :) Yes, what took a lot of effort for python 2.5 can now be done in a few lines.

0-0 Over a year ago

I dont understand where subprocess_readlines needs to be defined for this approach. Can you elaborate?

Jeff Younker Over a year ago

Ooops. I'd not added the call to subprocess_readlines(). I just updated that.

Collectives™ on Stack Overflow

Real-time intercepting of stdout from another process in Python

2 Answers 2

3 Comments

3 Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

3 Comments

Linked

Related