Python: multiprocessing - terminate other processes after one process finished

Question

I have some programm in which multiple processes try to finish some function. My aim now is to stop all the other processes after one process has successfully finished the function.

The python program shown below unfortunately waits until all the processes successfully solved the question given in find function. How can I fix my problem?

import multiprocessing import random FIND = 50 MAX_COUNT = 100000 INTERVAL = range(10) def find(process, initial, return_dict): succ = False while succ == False: start=initial while(start <= MAX_COUNT): if(FIND == start): return_dict[process] = f"Found: {process}, start: {initial}" succ = True break; i = random.choice(INTERVAL) start = start + i print(start) processes = [] manager = multiprocessing.Manager() return_code = manager.dict() for i in range(5): process = multiprocessing.Process(target=find, args=(f'computer_{i}', i, return_code)) processes.append(process) process.start() for process in processes: process.join() print(return_code.values())

output can be for example:

['Found: computer_0, start: 0', 'Found: computer_4, start: 4', 'Found: computer_2, start: 2', 'Found: computer_1, start: 1', 'Found: computer_3, start: 3']

But this output shows me the program is waiting until all processes are finished ...

You could use a "multiprocessing event" that signals a solution has been found in another process to make the other processes know they should exit cleanly pymotw.com/3/multiprocessing/… — Mark Setchell
– Mark Setchell, Commented Feb 20, 2022 at 10:07

julaine · Accepted Answer · 2023-06-28 21:24:18Z

Use an Event to govern if the processes should keep running.

Basically, it replaces succ with something that works over all processes.

import multiprocessing import random FIND = 50 MAX_COUNT = 1000 def find(process, initial, return_dict, run): while run.is_set(): start = initial while start <= MAX_COUNT: if FIND == start: return_dict[process] = f"Found: {process}, start: {initial}" run.clear() # Stop running. break start += random.randrange(0, 10) print(start) if __name__ == "__main__": processes = [] manager = multiprocessing.Manager() return_code = manager.dict() run = manager.Event() run.set() # We should keep running. for i in range(5): process = multiprocessing.Process( target=find, args=(f"computer_{i}", i, return_code, run) ) processes.append(process) process.start() for process in processes: process.join() print(return_code.values())

Note that using __name__ is mandatory for multiprocessing to work properly when the process-creation method is set to 'spawn' which is the default on ms-windows and macOS but also available on linux. On those systems, the main module is imported into newly created Python processes. This needs to happen without side effects such as starting a process, and the __name__ mechanism ensures that.

Andrew Wei · Accepted Answer · 2022-02-20 10:12:57Z

You can do this using multiprocessing.Queue and multiprocessing.Queue.get. How this works is that get by default blocks until there's something in the queue. So it will return the first result that gets appended to the queue, i.e. one of the processes finishing the search. After that, we can iterate over the processes and terminate each one (note that terminating a process doesn't kill child processes spawned by the process unless daemon is set to True).

import multiprocessing import random import time FIND = 50 MAX_COUNT = 100000 INTERVAL = range(10) queue = multiprocessing.Queue(maxsize=1) def find(process, initial): succ = False while succ == False: start=initial while(start <= MAX_COUNT): if(FIND == start): queue.put(f"Found: {process}, start: {initial}") break; i = random.choice(INTERVAL) start = start + i print(process, start) processes = [] manager = multiprocessing.Manager() for i in range(5): process = multiprocessing.Process(target=find, args=(f'computer_{i}', i)) processes.append(process) process.start() ret = queue.get() for i in range(5): process = processes[i] process.terminate() print(f'terminated {i}') print(ret)

You might also want to look into setting the daemon, which kills the processes after the main process exits.

Fixed it. You still need an Event, kind of like in the example above, to properly exit the processes. Just queue doesn't seem to work.

Collectives™ on Stack Overflow

Python: multiprocessing - terminate other processes after one process finished

2 Answers 2

Comments

2 Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

2 Comments

Linked

Related