Python subprocess get output as process finishes

Question

Currently I am getting MAC addresses from devices via Bluetooth and I pass these mac addresses one at a time to a method that calls a subprocess and assigns the output to a variable. With that variable I run some filtering functions to get a value from the command called in the subprocess. I then return that value if it finds it from the output.

What I would like to do is pass all mac addresses to the method at once and run them all at one time. How do I capture the output of each process and run my filtering script as the processes complete, and at the same time notify me if the process fails or errors.

Here is the current method that handles one mac at a time. Lets assume now that I am passing a list of mac addresses.

 def getchan(mac): a = subprocess.Popen(["sdptool", "search","--bdaddr", mac, "OPUSH"], stdout = subprocess.PIPE).communicate()[0].split()[2:] if a==[]: print "ERROR" return "ERROR" else: count = 0 for item in a: if item == "Channel:": return a[count + 1] count += 1 return "Could not find the OPUSH channel"

It should look something like

def getchan(macs): processes = set() for mac in macs: processes.add(subprocess.Popen(["sdptool", "search","--bdaddr", mac, "OPUSH"], stdout = subprocess.PIPE).communicate()[0].split()[2:]) #this is where I need the help

Thank you for taking a look. Any help or clarification of Subprocesses would be much appreciated.

kriomant · Accepted Answer · 2013-08-16 03:39:30Z

import select import subprocess def in_parallel(processes): pipes = {p.stdout.fileno(): (i, p.stdout) for i, p in enumerate(processes)} poller = select.poll() for fd, pipe in pipes.iteritems(): poller.register(fd, select.POLLIN) outputs = [''] * len(processes) while pipes: active = poller.poll() for fd, event in active: idx, pipe = pipes[fd] o = pipe.read() if o: outputs[idx] += o else: poller.unregister(fd) pipe.close() del pipes[fd] for p in processes: p.wait() return outputs args = ['a', 'b', 'c'] processes = [subprocess.Popen(['sleep 5; echo ' + arg], stdout=subprocess.PIPE, shell=True) for arg in args] outputs = in_parallel(processes) print outputs

$ time python test.py ['a\n', 'b\n', 'c\n'] real 0m5.042s user 0m0.016s sys 0m0.016s

jfs · Accepted Answer · 2013-08-16 08:29:12Z

The simplest way to get output from subprocesses in parallel in a portable manner is to use threads:

from multiprocessing.dummy import Pool # use threads from subprocess import check_output def getchan_mp(mac): try: output = check_output(["sdptool", "search","--bdaddr", mac, "OPUSH"]) result = parse(output) return result, mac, None except Exception as err: return None, mac, err results = Pool(processes=len(macs)).map(getchan_mp, macs) failed_macs = [mac for result, mac, error in results if error is not None]

Collectives™ on Stack Overflow

Python subprocess get output as process finishes

2 Answers 2

Comments

Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Related