How can I call a command using Python, capturing stderr and stdout, without waiting for stderr/stdout to be closed?

Question

Right now I have some code that uses Popen.communicate() from subprocess (setting stdin=PIPE and stderr=PIPE) to run a command and capture both stderr and stdout.

The problem is that communicate() not only waits for the command to exit, it waits for stdout and stderr to be closed. The command I'm running spawns a child process which keeps stderr open, so even though the command is finished running (and shows as "defunct" in ps) communicate() is still hung.

I want to only wait for the command to finish without waiting on stderr/stdout. But I still want to capture any stderr/stdout output given while the command was running. The documentation for wait() is accompanied by a red box with a disclaimer:

This will deadlock when using stdout=PIPE and/or stderr=PIPE and the child process generates enough output to a pipe such that it blocks waiting for the OS pipe buffer to accept more data. Use communicate() to avoid that.

Obviously, I also want to avoid deadlocks.

What is the right way to accomplish this task?

Maybe you'll get interesting information from stackoverflow.com/questions/803265/… or stackoverflow.com/questions/3140189/… or stackoverflow.com/questions/17411966/… — damienfrancois
– damienfrancois, Commented Jan 15, 2014 at 17:36
Could spawn separate threads to read the process's stdout and stderr so that you can call wait() without worrying about buffers filling up. Complicated part is if you want to stop reading stderr. See stackoverflow.com/questions/3076542/… — Dan Getz
– Dan Getz, Commented Jan 16, 2014 at 23:51
@DanGetz thanks, I think that will work. Will report back if I get a working solution. — Caleb Spare
– Caleb Spare, Commented Jan 18, 2014 at 1:05

jfs · Accepted Answer · 2014-03-04 15:44:09Z

"shows as "defunct" in ps" implies that you might be on a system where select or fcntl would work i.e., you can read stdout/stderr without blocking easily.

Example: A starts B (cmd, the child), B starts C (grandchild), A reads output until B exits or EOF:

#!/usr/bin/env python import os from select import select from subprocess import Popen, PIPE p = Popen(cmd, stdout=PIPE, stderr=PIPE, bufsize=0) read_set = [p.stdout, p.stderr] pipename = {p.stdout: "stdout", p.stderr: "stderr"} timeout = 0.5 # ugly but it works while read_set and p.poll() is None: # while subprocess is running or until EOF for pipe in select(read_set, [], [], timeout)[0]: data = os.read(pipe.fileno(), 1<<30) if data: print("got from %s: %r" % (pipename[pipe], data)) else: # EOF pipe.close() read_set.remove(pipe) print("exit code %s" % (p.wait(),)) # child exited, wait for grandchild to print for pipe in read_set: print("read the rest of %s: %r" % (pipename[pipe], pipe.read())) pipe.close()

where cmd:

import sys from textwrap import dedent cmd = [sys.executable, '-u', '-c', dedent(""" # inception style import os import sys from subprocess import Popen from textwrap import dedent Popen([sys.executable, '-u', '-c', dedent(''' import os import sys import time time.sleep(60) print("grandchild %d done" % os.getpid()) sys.stderr.write("grandchild stderr") sys.exit(20) ''')]) # stdout/stderr are not redirected print('child %d done' % os.getpid()) sys.stderr.write('child stderr') sys.exit(19) """)]

Nodari L · Accepted Answer · 2014-03-04 13:12:49Z

I've tried a simple example. And all I've got so far is capturing of the std.err I've created two files. The contents of the first is (example1.py):

import time import sys for i in xrange(1000): print >>sys.stderr, "hello", i time.sleep(1)

The contents of the second one(example2.py):

import subprocess cmd = "python example1.py" print subprocess.check_output(cmd, shell=True)

And that I've just called the example2.py script:

python example2.py And I've realtime output ( I assume it is true:) ):

hello 0 hello 1 hello 2 hello 3 hello 4 hello 5 hello 6

Still, I have no idea, how to deal with standard output, but if I manage it, I will post the answer here

Collectives™ on Stack Overflow

How can I call a command using Python, capturing stderr and stdout, without waiting for stderr/stdout to be closed?

2 Answers 2

Comments

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Linked

Related