Blocking multiprocessing pool.map called in a process

Question

I want to call a multiprocessing.pool.map inside a process.

When initialized inside the run() function, it works. When initialized at instantiation, it does not.

I cannot figure the reason for this behavior ? What happens in the process ? I am on python 3.6

from multiprocessing import Pool, Process, Queue def DummyPrinter(key): print(key) class Consumer(Process): def __init__(self, task_queue): Process.__init__(self) self.task_queue = task_queue self.p = Pool(1) def run(self): p = Pool(8) while True: next_task = self.task_queue.get() if next_task is None: break p.map(DummyPrinter, next_task) # Works #self.p.map(DummyPrinter, next_task) # Does not Work return if __name__ == '__main__': task_queue = Queue() Consumer(task_queue).start() task_queue.put(range(5)) task_queue.put(None)

can you show the code that doesn't work, not the one that works? I think I solved an issue like that. Let me find it — Jean-François Fabre
– Jean-François Fabre ♦, Commented Sep 4, 2017 at 19:27
just uncomment #self.p.map(DummyPrinter, next_task) # Does not Work — mxdbld
– mxdbld, Commented Sep 4, 2017 at 19:27

noxdafox · Accepted Answer · 2017-09-04 19:38:30Z

multiprocessing.Pool cannot be shared by multiple processes because it relies on pipes and threads for its functioning.

The __init__ method gets executed in the parent process whereas the run logic belongs to the child process.

I usually recommend against sub-classing the Process object as it's quite counter intuitive.

A logic like the following would better show the actual division of responsibilities.

def function(task_queue): """This runs in the child process.""" p = Pool(8) while True: next_task = self.task_queue.get() if next_task is None: break p.map(DummyPrinter, next_task) # Works def main(): """This runs in the parent process.""" task_queue = Queue() process = Process(target=function, args=[task_queue]) process.start()

Collectives™ on Stack Overflow

Blocking multiprocessing pool.map called in a process

1 Answer 1

Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Related