Work with the stdout of some other process (created with Popen)

Question

I would like to count the number of lines written to stdout by a process (here unrar.exe) created with Popen.

import time from subprocess import Popen, PIPE, STDOUT p = Popen('unrar.exe x -y myfile.rar', stdout=PIPE) while (p is not finished): # pseudo code here and next lines... time.sleep(0.100) print 'Number of lines written to STDOUT by unrar' + len(PIPE.split('\n'))

How to do this properly ?

Remark : I already looked at p.communicate() (https://python.readthedocs.org/en/v2.7.2/library/subprocess.html#subprocess.Popen.communicate) but this has the effect of blocking the execution of the Python until p has terminated, which is not what I want : I want to be able to print the number of lines written by p when it's running.

possible duplicate of Non-blocking read on a subprocess.PIPE in python — Martijn Pieters
– Martijn Pieters, Commented Feb 24, 2014 at 13:41
@MartijnPieters I'm looking for an easier solution without using another thread, etc. I don't mind if the process is blocked every 100 ms — Basj
– Basj, Commented Feb 24, 2014 at 13:49
@Basj: you can't do it without blocking unless you use threads or select (poll, kqueue, named pipe on Windows) or fcntl or similar. — jfs
– jfs, Commented Feb 24, 2014 at 15:08

jfs · Accepted Answer · 2014-02-24 15:30:20Z

I'm looking for an easier solution without using another thread, etc. I don't mind if the process is blocked every 100 ms

If it is a hard requirement that the process must not block then you need threads (or other asynchronous techniques). To emulate non-blocking wc --lines <(cmd):

#!/usr/bin/env python import io import shlex from functools import partial from subprocess import Popen, PIPE from threading import Thread from Queue import Queue def count_lines(pipe, queue, chunksize=io.DEFAULT_BUFFER_SIZE): #NOTE: you could write intermediate results here (just drop `sum()`) queue.put(sum(chunk.count(b'\n') for chunk in iter(partial(pipe.read, chunksize), b''))) pipe.close() p = Popen(shlex.split('unrar.exe x -y myfile.rar'), stdout=PIPE, bufsize=-1) result = Queue() Thread(target=count_lines, args=[p.stdout, result]).start() p.wait() # you can omit it if you want to do something else and call it later number_of_lines = result.get() # this blocks (you could pass `timeout`)

On the other hand if all you need is "to print the number of lines written by p when it's running." then you could count lines in the main thread:

#!/usr/bin/env python import shlex from subprocess import Popen, PIPE p = Popen(shlex.split('unrar.exe x -y myfile.rar'), stdout=PIPE, bufsize=1) count = 0 for line in iter(p.stdout.readline, b''): count += 1 print count p.stdout.close() p.wait()

Basj · Accepted Answer · 2014-02-24 23:01:54Z

0

I don't know if this is the cleanest way to do it, but it works:

from subprocess import Popen, PIPE import io, time p = Popen('unrar.exe x -y myfile.rar', stdout = PIPE) b = ' ' i = 0 while b: b = p.stdout.readline() i += 1 print i print 'FINISHED'

edited Feb 24, 2014 at 23:01

answered Feb 24, 2014 at 14:23

Basj

47.6k113 gold badges467 silver badges819 bronze badges

2 Comments

Basj Over a year ago

@J.F.Sebastian : in which situation may it be blocked at readline() ? If there is no more lines, wouln't it give '' and then wouldn't this end the loop ?

jfs Over a year ago

p.stdout.readline() blocks until a line is available or EOF e.g., in a typical case of a block buffered output to a pipe you won't see anything until the pipe buffer overflows or it is flushed (print the current time to see that the parent sees the output in bursts). If child process doesn't produce much output; it may take a while before you see the first line.

Collectives™ on Stack Overflow

Work with the stdout of some other process (created with Popen)

2 Answers 2

Comments

2 Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

2 Comments

Linked

Related