Trouble with subprocess.check_output()

Question

I'm having some strange issues using subprocess.check_output(). At first I was just using subprocess.call() and everything was working fine. However when I simply switch out call() for check_output(), I receive a strange error.

Before code (works fine):

def execute(hosts): ''' Using psexec, execute the batch script on the list of hosts ''' successes = [] wd = r'c:\\' file = r'c:\\script.exe' for host in hosts: res = subprocess.call(shlex.split(r'psexec \\\\%s -e -s -d -w %s %s' % (host,wd,file))) if res.... # Want to check the output here successes.append(host) return successes

After code (doesn't work):

def execute(hosts): ''' Using psexec, execute the batch script on the list of hosts ''' successes = [] wd = r'c:\\' file = r'c:\\script.exe' for host in hosts: res = subprocess.check_output(shlex.split(r'psexec \\\\%s -e -s -d -w %s %s' % (host,wd,file))) if res.... # Want to check the output here successes.append(host) return successes

This gives the error: Error

I couldnt redirect this because the program hangs here and I can't ctrl-c out. Any ideas why this is happening? What's the difference between subprocess.call() and check_output() that could be causing this?

Here is the additional code including the multiprocessing portion:

PROCESSES = 2 host_sublists_execute = [.... list of hosts ... ] poolE = multiprocessing.Pool(processes=PROCESSES) success_executions = poolE.map(execute,host_sublists_execute) success_executions = [entry for sub in success_executions for entry in sub] poolE.close() poolE.join()

Thanks!

Both code samples are the same. Looks like you made a copy/paste error... — dano
– dano, Commented Aug 21, 2014 at 21:05
Yeah, sorry about that - fixed. Only difference is call > check_output — DJMcCarthy12
– DJMcCarthy12, Commented Aug 21, 2014 at 21:08
Is execute running inside of a multiprocessing.Pool? Are you trying to do a raise CalledProcessError later in execute? If so, please include that code. — dano
– dano, Commented Aug 21, 2014 at 21:13
Yes I am using multiprocessing, I will update the original code with everything else — DJMcCarthy12
– DJMcCarthy12, Commented Aug 21, 2014 at 21:28
Also included the rest of execute function, all I do is track successes and return them. Thanks for the help! — DJMcCarthy12
– DJMcCarthy12, Commented Aug 21, 2014 at 21:34

skrrgwasme · Accepted Answer · 2014-09-10 22:31:54Z

You are encountering Python Issue 9400.

There is a key distinction you have to understand about subprocess.call() vs subprocess.check_output(). subprocess.call() will execute the command you give it, then provide you with the return code. On the other hand, subprocess.check_output() returns the program's output to you in a string, but it tries to do you a favor and check the program's return code and will raise an exception (subprocess.CalledProcessError) if the program did not execute successfully (returned a non-zero return code).

When you call pool.map() with a multiprocessing pool, it will try to propagate exceptions in the subprocesses back to main and raise an exception there. Apparently there is an issue with how the subprocess.CalledProcessError exception class is defined, so the pickling fails when the multiprocessing library tries to propagate the exception back to main.

The program you're calling is returning a non-zero return code, which makes subprocess.check_output() raise an exception, and pool.map() can't handle it properly, so you get the TypeError that results from the failed attempt to retrieve the exception.

As a side note, the definition of subprocess.CalledProcessError must be really screwed up, because if I open my 2.7.6 terminal, import subprocess, and manuallly raise the error, I still get the TypeError, so I don't think it's merely a pickling problem.

Collectives™ on Stack Overflow

Trouble with subprocess.check_output()

1 Answer 1

Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Related