Python: Subprocess.popen stdout: not in real time buffer

Question

I was tring for so long to catch the stdout of a python scrypt in subprocess in real time.

SOLUTION

main.py

import subprocess import io import time import sys if __name__ == '__main__': command = ["python", "-u", "createValue.py"] # "-u" changed my life p = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) for line in iter(p.stdout.readline, b''): print(">>> " + str(line.rstrip()))

createValue.py

 import time import sys i = 0 while i < 5: print("I like pizza "+ str(i)) # sys.stdout.flush() # This is an another solution time.sleep(1.5) i += 1

Why every solution on Internet work without "-u" for them but not on my PC ? I use Python 3.6.5

AFAIK, -u means unbuffered which translates to: Python will not buffer the output, it will send back the data as soon as it is pointed to stdout. Isn't that what you want? I don't see any issues with your code. — Bruno Assis
– Bruno Assis, Commented Jun 18, 2021 at 15:05
Yes, does-it work in real time without "-u" for you ? Every solution on Internet doesn't use "-u" Exemple 1: stackoverflow.com/questions/1606795/… Exemple 2: stackoverflow.com/questions/18421757/… — Syl
– Syl, Commented Jun 18, 2021 at 15:17

Bruno Assis · Accepted Answer · 2021-06-18 16:05:25Z

Yes, it's because you need to add a environment variable called PYTHONUNBUFFERED and set it to = 1

"...Setting PYTHONUNBUFFERED to a non empty value ensures that the python output is sent straight to terminal..."

Check the solution below using your code:

import os # Added this line import subprocess import io import time import sys if __name__ == '__main__': os.environ["PYTHONUNBUFFERED"] = "1" # Added command = ["python", "createValue.py"] # Removed "-u" p = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) print("init") for line in iter(p.stdout.readline, b''): print(">>> " + str(line.rstrip()))

This pollutes the environment of the parent process. A cleaner solution is to copy the environment and pass the copy to Popen: env = dict(os.environ); env["PYTHONUNBUFFERED"] = "1" and Popen(command, env=env, ...)
I do agree that your suggestion would be the best in the majority of the professional (specially enterprise) scenarios but I disagree with you when you used the term 'pollutes', because this flexibility might be the intention of the programmer (Set an environment variable for ALL processes).

Collectives™ on Stack Overflow

Python: Subprocess.popen stdout: not in real time buffer

1 Answer 1

2 Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Linked

Related