0

I did basic due diligence and couldn't find a good answer to this anywhere.

I want to call subprocess.Popen in a way that they will still raise a Python exception when control flow is interrupted, but I want to redirect stderr at the same time.

The use case is for a command line client that shouldn't output warnings, but whose internal logic still wants to know about subprocess problems.

As an example, this silently redirects all errors:

subprocess.Popen(command, stderr=subprocess.PIPE)

Calling a Python module as a subprocess with contents:

raise(Exception("AVAST!"))

Doesn't raise anything.

It would be great if it redirected all error text, but still raised on anything that halted control flow prematurely. Do I need to use return code and hope that all subprocesses I call have implemented this correctly?

The best thing I've thought of so far is manually parsing the redirected errors, which is a pretty poor implementation in my mind.

Is there a clean canonical way to do this?

Improve this question

1 Answer 1

Reset to default
1

There's no way to pass exceptions across a text pipe like stderr, because all you can pass across a text pipe is text.

But there are a few options:

  1. Make sure all of your children exit with non-zero status on exception (which is the default, if you don't do anything) and don't do so in any other cases (unless there are cases you want to treat the same as an exception).
  2. Parse for exceptions in stderr.
  3. Create some other communication channel for the parent and children to share.
  4. Don't use subprocess. For example, if the only reason you're running Python scripts via subprocess is to get core parallelism (or memory space isolation), it may be a lot easier to use multiprocessing or concurrent.futures, which have already built the machinery to propagate exceptions for you.

From your comment:

My use case is calling a bunch of non-Python third party things. If return codes are how the standard library module propagates errors, I'm happy enough using them.

No, the Python standard library propagates errors using Exceptions. And so should you.

Return codes are how non-Python third party things propagate errors. (Actually, how they propagate both errors and unexpected signals, but… don't worry about that.) That's limited to 7 bits worth of data, and the meanings aren't really standardized, so it's not as good. But it's all you have in the POSIX child process model, so that's what generic programs use.

What you probably want to do is the same thing subprocess.check_call does—if the return code is not zero, raise an exception. (In fact, if you're not doing anything asynchronous, ask yourself whether you can use check_call in the first place, instead of using a Popen object explicitly.)

For example, if you were doing this:

output, errors = p.communicate()

Change it to this:

output, errors = p.communicate() if p.returncode: raise subprocess.CalledProcessError(p.returncode, 'description')

(The description is usually the subprocess path or name.)

Then, the rest of your code can just handle exceptions.

Improve this answer
Sign up to request clarification or add additional context in comments.

7 Comments

Thanks for the response @Abernert. My use case is calling a bunch of non-Python third party things. If return codes are how the standard library module propagates errors, I'm happy enough using them. I'll go skim.
Your reply is a bit confused. Let me edit my answer to explain better.
That makes sense. Possibly the following is the source of confusion? 2,3,4 were not extremely attractive options because of some limitations of the development process that I'll omit for brevity. With 1, my concern was that some of the things I am calling may return a 0 even when an error occurs, and that Python may use some other method to determine failure and raise an Exception. My comment was written in the context of intending to redirect err, then check the return code and raise an Exception in the case that it isn't 0. I could certainly have asked if Python uses only rc more clearly.
Looks good. Propagate was definitely the wrong word to use there. What I had intended to convey was curiosity over how Python decides that a given subprocess has errored in a way that warrants raising an Exception. I want my usage to be consistent with the norm rather than rolling my own logic, and it seems like checking the return code will be?
The incorrectness of the term coming from my desire to refer to a single step of an error moving from the subprocess to Python's error handling vs. a reference to the entire path of the error/Exception.
|

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.