1

I have a program written in Python that at some point creates a subprocess and then has to get its std output in "real time" through a file (the process takes a while and some output is needed while it is still running). The problem is that, sometimes, a chunk of the output is lost (actually a chunk starting at the begining). The code looks like this: 

import subprocess import tempfile import time .. tf_out = tempfile.TemporaryFile() tf_err = tempfile.TemporaryFile() tf_in = tempfile.TemporaryFile() tf_in.write(some_stdin) tf_in.seek(0) # create the process process = subprocess.Popen(command, shell=True, stdin=tf_in, stdout=tf_out, stderr=tf_err) running = True while running: # last iteration if process ended if process.poll() is not None: running = False # read output tf_out.seek(0) # 1 stdout = tf_out.read() # 2 # do something with the partial output do_something(stdout) time.sleep(0.1) tf_out.close() tf_err.close() tf_in.close() ..

I wonder if it is possible that the problem may be between # 1 and # 2, in case that: the subprocess write something, then seek to zero at # 1, another write starting at zero because of the seek (wrong one), and finally the read at point # 2 doesn't see the first write. If that's the problem, any idea how to do it?

Thanks in advance!

Improve this question
2
  • 2
    Is there a reason you can't use subprocess.PIPE, which would allow you to read/write from process.stdout/process.stdin directly? Commented Jul 7, 2014 at 18:21
  • Dano, as far as I know those objects are written when the process ends so I cannot use them to get the output in "real time". Commented Jul 8, 2014 at 17:22

2 Answers 2

Reset to default
1

Getting realtime data from Popen is quite finicky. In addition, good old print also generally buffers output.

The following is adapted from another SO question. It's set up so you can just run it.

source

#!/usr/bin/env python # adapted from http://stackoverflow.com/questions/2804543/read-subprocess-stdout-line-by-line import subprocess, sys, time def test_proc(cmd): start = time.time() print 'COMMAND:',cmd proc = subprocess.Popen( cmd, shell=True, stdout=subprocess.PIPE, ) for line in iter(proc.stdout.readline, ''): sys.stdout.write('{:.2f} {}\n'.format( time.time() - start, line.rstrip() )) sys.stdout.flush() if __name__=='__main__': test_proc('ping -c3 8.8.8.8')

output

COMMAND: ping -c3 8.8.8.8 0.05 PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data. 0.05 64 bytes from 8.8.8.8: icmp_seq=1 ttl=43 time=49.0 ms 1.05 64 bytes from 8.8.8.8: icmp_seq=2 ttl=43 time=46.8 ms 2.05 64 bytes from 8.8.8.8: icmp_seq=3 ttl=43 time=49.1 ms 2.05 2.05 --- 8.8.8.8 ping statistics --- 2.05 3 packets transmitted, 3 received, 0% packet loss, time 2003ms 2.05 rtt min/avg/max/mdev = 46.826/48.334/49.143/1.097 ms

The numbers on the left side are seconds since the start of the command. Since the ping command sends out a packet once per second, this verifies that the data is processed in realtime. The two "0.05" lines are spit put when ping starts. Next line is after a second, when the first ping response is received. At time 2.05 the last response is received, and ping outputs a footer.

Very convenient!

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

Actually I have solved that in a really simple way. Thanks a lot for the answer anyway!
1

I just figured out how to fix that. An easy solution is just open the temporary file twice, one to be passed to the subprocess and the other one to actually do the reads. In addition, you can use the tempfile module to create a named temporary file like this:

import tempfile tf_in = tempfile.TemporaryFile() tf_in.write(some_stdin) tf_in.seek(0) tf_out = tempfile.NamedTemporaryFile() tf_out2 = open(tf_out.name, 'r') tf_err = tempfile.NamedTemporaryFile() tf_err2 = open(tf_err.name, 'r') # create the process process = subprocess.Popen(command, shell=True, stdin=tf_in, stdout=tf_out, stderr=tf_err) # same loop but reading from tf_out2 and tf_err2 # close files

Using two file objects for the writes and the reads solves the problem.

Thanks for the answers anyway!

Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.