How can I access the stdout of child processes prior to sending them to main process? I am using multiprocessing.Pool module to generate child process pools.
3
- maybe this could help you: stackoverflow.com/questions/7714868/…Udy– Udy2013-10-17 07:41:17 +00:00Commented Oct 17, 2013 at 7:41
- @Udy the question you reference seems like a windows specific stdout redirection issue. Also, they're trying to redirect the stdout to a file. I on the other hand want to just modify the child process's stdout.KT100– KT1002013-10-17 08:00:08 +00:00Commented Oct 17, 2013 at 8:00
- true. but this will defantly help: stackoverflow.com/questions/4227808/…Udy– Udy2013-10-17 08:14:42 +00:00Commented Oct 17, 2013 at 8:14
Add a comment |
1 Answer
The main process and the children all share the same standard input and standard output file descriptors. They have no control over what the other writes to them. The only thing you can do is replace stdin and stdout in the children with something else that the main process can control. As an example you could subclass a dummy file object like StringIO and redirect the data that the children write to this object to the parent via a Queue:
import sys from multiprocessing import Queue, Pool, current_process from StringIO import StringIO class MyStringIO(StringIO): def __init__(self, queue, *args, **kwargs): StringIO.__init__(self, *args, **kwargs) self.queue = queue def flush(self): self.queue.put((current_process().name, self.getvalue())) self.truncate(0) def initializer(queue): sys.stderr = sys.stdout = MyStringIO(queue) def task(num): print num sys.stdout.flush() return num ** 2 q = Queue() pool = Pool(3, initializer, [q]) for _ in pool.map(task, range(5)): proc, out = q.get() print proc, "got", out This should print something like this:
PoolWorker-1 got 0 PoolWorker-1 got 3 PoolWorker-1 got 4 PoolWorker-2 got 1 PoolWorker-3 got 2 Don't forget to call sys.{stdout,stderr}.flush() at the end of task otherwise, nothing will be written to the queue.
1 Comment
kouk
the children's output is redirected to the
MyStringIO object which then sends the data to the main process via the queue. The main process then does all the processing it needs and outputs it to sys.stdout (which, as I explain in my updated answer, is shared by all processes attached to the terminal)