11

I am using a python script to run a process using subprocess.Popen and simultaneously store the output in a text file as well as print it on the console. This is my code:

result = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE) for line in result.stdout.readlines(): #read and store result in log file openfile.write("%s\n" %line) print("%s" %line)

Above code works fine, but what it does is it first completes the process and stores the output in result variable. After that for loop stores the output as well as print it.

But i want the output at runtime (as my process can take hours to complete, i don't get any output for all these hours).

So is there any other function that gives me the output dynamically (at runtime), means as soon as the process gives first line, it should get printed.

Improve this question
5
  • 1
    I've never actually tried it, but I think you are supposed to send your own Python file object to the stdout (or stdin, or stderr) argument. Then you have to poll that file. Subprocess was invented to spare you that pain, but it looks like you have no choice. Good luck. Commented May 15, 2013 at 6:13
  • 1
    "it first completes the process and stores the output in result" - that's just not true. Commented May 15, 2013 at 6:50
  • 1
    "it first completes the process and stores the output in result" - that's just not true. Well, when i am running Popen command, it keeps on running until the process finishes and then it executes further coding. If its work some other way, i am eager to know that. Commented May 15, 2013 at 8:13
  • @AdrianRatnapala Sounds too complicated, and the "no choice" claim is wrong. Commented May 15, 2013 at 9:38
  • you may consider using fcntl to set result.stdout as non blocking, this will allow to read in real time and if there is no data yet, IOError will be raised Commented Mar 13, 2018 at 12:04

3 Answers 3

Reset to default
10

The problem here is that .readlines() gets the entire output before returning, as it constructs a full list. Just iterate directly:

for line in result.stdout: print(line)
Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

note: for line in result.stdout does not return lines in "real time" on Python 2, use for line in iter(result.stdout.readline, b'') instead (your print line statement suggests that you expect it to work on Python 2. On Python 3 it works fine
Yes, its working, i am getting output at real time, Thank you.
4

.readlines() returns a list of all the lines the process will return while open, i.e., it doesn't return anything until all output from the subprocess is received. To read line by line in "real time":

import sys from subprocess import Popen, PIPE proc = Popen(cmd, shell=True, bufsize=1, stdout=PIPE) for line in proc.stdout: openfile.write(line) sys.stdout.buffer.write(line) sys.stdout.buffer.flush() proc.stdout.close() proc.wait()

Note: if the subprocess uses block-buffering when it is run in non-interactive mode; you might need pexpect, pty modules or stdbuf, unbuffer, script commands.

Note: on Python 2, you might also need to use iter(), to get "real time" output:

for line in iter(proc.stdout.readline, ""): openfile.write(line) print line,
Improve this answer

Comments

1

You can iterate over the lines one by one by using readline on the pipe:

while True: line = result.stdout.readline() print line.strip() if not line: break

The lines contain a trailing \n which I stripped for printing. When the process terminates, readline returns an empty string, so you know when to stop.

Improve this answer

3 Comments

I am sorry but you didn't get my question. What you have suggested i am already accomplishing it using for loop, what i want is line by line output while process is still in running state.
did you try? You're using readlines, which returns all lines. That's only possible after termination. With readline, you get only the next
result.stdout works just fine for me, but your line.strip() functions is helpful as i was getting too many "b'\r\n'" before it, Thank you

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.