1

I am trying to run a python file that prints something, waits 2 seconds, and then prints again. I want to catch these outputs live from my python script to then process them. I tried different things but nothing worked.

process = subprocess.Popen(cmd, stdout=subprocess.PIPE) while True: output = process.stdout.readline() if process.poll() is not None and output == '': break if output: print(output.strip())

I'm at this point but it doesn't work. It waits until the code finishes and then prints all the outputs.

I just need to run a python file and get live outputs from it, if you have other ideas for doing it, without using the print function let me know, just know that I have to run the file separately. I just thought of the easiest way possible but, from what I'm seeing it can't be done.

Improve this question
8
  • Does this answer your question? live output from subprocess command Commented May 3, 2022 at 14:47
  • i already tried every there, it doesn't work Commented May 3, 2022 at 14:51
  • 1
    There are three layers of buffering here, and you need to limit all three of them to get live data: 1) Use stdbuf or alter the program itself to change the buffering of the program to line-oriented mode (or add fflushs); without that, everything is stuck in the subprocess's user-mode buffers. 2) Add bufsize=1 to the Popen arguments (probably not needed since you don't send stdin, but harmless). 3) Add flush=True to the print arguments (if you're connected to a terminal, the line-buffering will flush it for you, so it's only if stdout is piped to a file that this will matter). Commented May 3, 2022 at 14:58
  • To be clear, stdbuf in #1 is the *NIX-world solution; you'd just change the Popen to run ['stdbuf', '-oL'] + cmd. Commented May 3, 2022 at 14:59
  • Thank you @ShadowRanger, the problem was the flush=True on the print. Commented May 3, 2022 at 15:06

3 Answers 3

Reset to default
3

There are three layers of buffering here, and you need to limit all three of them to guarantee you get live data:

  1. Use the stdbuf command (on Linux) to wrap the subprocess execution (e.g. run ['stdbuf', '-oL'] + cmd instead of just cmd), or (if you have the ability to do so) alter the program itself to either explicitly change the buffering on stdout (e.g. using setvbuf for C/C++ code to switch stdout globally to line-buffered mode, rather than the default block buffering it uses when outputting to a non-tty) or to insert flush statements after critical output (e.g. fflush(stdout); for C/C++, fileobj.flush() for Python, etc.) the buffering of the program to line-oriented mode (or add fflushs); without that, everything is stuck in user-mode buffers of the sub-process.

  2. Add bufsize=0 to the Popen arguments (probably not needed since you don't send anything to stdin, but harmless) so it unbuffers all piped handles. If the Popen is in text=True mode, switch to bufsize=1 (which is line-buffered, rather than unbuffered).

  3. Add flush=True to the print arguments (if you're connected to a terminal, the line-buffering will flush it for you, so it's only if stdout is piped to a file that this will matter), or explicitly call sys.stdout.flush().

Between the three of these, you should be able to guarantee no data is stuck waiting in user-mode buffers; if at least one line has been output by the sub-process, it will reach you immediately, and any output triggered by it will also appear immediately. Item #1 is the hardest in most cases (when you can't use stdbuf, or the process reconfigures its own buffering internally and undoes the effect of stdbuf, and you can't modify the process executable to fix it); you have complete control over #2 and #3, but #1 may be outside your control.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

2

This is the code I use for that same purpose:

def run_command(command, **kwargs): """Run a command while printing the live output""" process = subprocess.Popen( command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, **kwargs, ) while True: # Could be more pythonic with := in Python3.8+ line = process.stdout.readline() if not line and process.poll() is not None: break print(line.decode(), end='')

An example of usage would be:

run_command(['git', 'status'], cwd=Path(__file__).parent.absolute())
Improve this answer

Comments

0

If you want to treat stdout and stderr separately, you can spawn two threads that handle them concurrently (live as the output is produced).

Adapted from my more detailed answer:

import logging from collections import deque from concurrent.futures import ThreadPoolExecutor from functools import partial from subprocess import PIPE, CalledProcessError, CompletedProcess, Popen def stream_command( args, *, stdout_handler=logging.info, stderr_handler=logging.error, # for a version that redirects stderr to stdout (without multithreading), see https://stackoverflow.com/a/76626021/5511061 check=True, text=True, **kwargs, ): """Mimic subprocess.run, while processing the command output in real time ref https://stackoverflow.com/a/76634163/5511061.""" with ( Popen(args, text=text, stdout=PIPE, stderr=PIPE, **kwargs) as process, ThreadPoolExecutor(2) as pool, # two threads to handle the (live) streams separately ): exhaust = partial(deque, maxlen=0) # collections recipe: exhaust an iterable at C-speed exhaust_async = partial(pool.submit, exhaust) # exhaust non-blocking in a background thread # exhaust both streams (stdout and stderr iterables) asynchronously, each in its own thread exhaust_async(stdout_handler(line[:-1]) for line in process.stdout) exhaust_async(stderr_handler(line[:-1]) for line in process.stderr) retcode = process.poll() # block until both iterables are exhausted (process finished) if check and retcode: raise CalledProcessError(retcode, process.args) return CompletedProcess(process.args, retcode)

Call with simple print handlers:

stream_command(["echo", "test"], stdout_handler=print, stderr_handler=print) # test

Or with custom handlers:

outs, errs = [], [] def stdout_handler(line): outs.append(line) print(line) def stderr_handler(line): errs.append(line) print(line) stream_command( ["echo", "test"], stdout_handler=stdout_handler, stderr_handler=stderr_handler, ) # test print(outs) # ['test']
Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.