Wait until all threads are finished in Python

Put the threads in a list, .start() each thread, and then .join() each thread:

threads = [ Thread(...), Thread(...), Thread(...), ] # Start all threads. for t in threads: t.start() # Wait for all threads to finish. for t in threads: t.join() 

In Python3, since Python 3.2 there is a new approach to reach the same result, that I personally prefer to the traditional thread creation/start/join, package concurrent.futures: https://docs.python.org/3/library/concurrent.futures.html

Using a ThreadPoolExecutor the code would be:

from concurrent.futures.thread import ThreadPoolExecutor import time def call_script(ordinal, arg): print('Thread', ordinal, 'argument:', arg) time.sleep(2) print('Thread', ordinal, 'Finished') args = ['argumentsA', 'argumentsB', 'argumentsC'] with ThreadPoolExecutor(max_workers=2) as executor: ordinal = 1 for arg in args: executor.submit(call_script, ordinal, arg) ordinal += 1 print('All tasks has been finished') 

The output of the previous code is something like:

Thread 1 argument: argumentsA Thread 2 argument: argumentsB Thread 1 Finished Thread 2 Finished Thread 3 argument: argumentsC Thread 3 Finished All tasks has been finished 

One of the advantages is that you can control the throughput setting the max concurrent workers.

To use multiprocessing instead, you can use ProcessPoolExecutor.

I prefer using list comprehension based on an input list:

inputs = [scriptA + argumentsA, scriptA + argumentsB, ...] threads = [Thread(target=call_script, args=(i)) for i in inputs] [t.start() for t in threads] [t.join() for t in threads] 

You can have class something like below from which you can add 'n' number of functions or console_scripts you want to execute in parallel passion and start the execution and wait for all jobs to complete..

from multiprocessing import Process class ProcessParallel(object): """ To Process the functions parallely """ def __init__(self, *jobs): """ """ self.jobs = jobs self.processes = [] def fork_processes(self): """ Creates the process objects for given function deligates """ for job in self.jobs: proc = Process(target=job) self.processes.append(proc) def start_all(self): """ Starts the functions process all together. """ for proc in self.processes: proc.start() def join_all(self): """ Waits untill all the functions executed. """ for proc in self.processes: proc.join() def two_sum(a=2, b=2): return a + b def multiply(a=2, b=2): return a * b #How to run: if __name__ == '__main__': #note: two_sum, multiply can be replace with any python console scripts which #you wanted to run parallel.. procs = ProcessParallel(two_sum, multiply) #Add all the process in list procs.fork_processes() #starts process execution procs.start_all() #wait until all the process got executed procs.join_all() 

From the threading module documentation

There is a “main thread” object; this corresponds to the initial thread of control in the Python program. It is not a daemon thread.

There is the possibility that “dummy thread objects” are created. These are thread objects corresponding to “alien threads”, which are threads of control started outside the threading module, such as directly from C code. Dummy thread objects have limited functionality; they are always considered alive and daemonic, and cannot be join()ed. They are never deleted, since it is impossible to detect the termination of alien threads.

So, to catch those two cases when you are not interested in keeping a list of the threads you create:

import threading as thrd def alter_data(data, index): data[index] *= 2 data = [0, 2, 6, 20] for i, value in enumerate(data): thrd.Thread(target=alter_data, args=[data, i]).start() for thread in thrd.enumerate(): if thread.daemon: continue try: thread.join() except RuntimeError as err: if 'cannot join current thread' in err.args[0]: # catchs main thread continue else: raise 

Whereupon:

>>> print(data) [0, 4, 12, 40] 

I just came across the same problem where I needed to wait for all the threads which were created using the for loop.I just tried out the following piece of code.It may not be the perfect solution but I thought it would be a simple solution to test:

for t in threading.enumerate(): try: t.join() except RuntimeError as err: if 'cannot join current thread' in err: continue else: raise 

Maybe, something like

for t in threading.enumerate(): if t.daemon: t.join() 

Create a ThreadPoolExecutor (or ProcessPoolExecutor).
Then, call .map on your desired function func, and a list of arguments xs:

from concurrent.futures import ThreadPoolExecutor with ThreadPoolExecutor(max_workers=len(xs)) as executor: results = list(executor.map(func, xs)) 

.map returns an iterator containing the return value of each function, which we collect into a list.

In your case:

from concurrent.futures import ThreadPoolExecutor argss = [ ["python", "scriptA.py", "a"], ["python", "scriptA.py", "b"], ["python", "scriptA.py", "c"], ] with ThreadPoolExecutor(max_workers=len(argss)) as executor: results = list(executor.map(call_script, argss)) 

using only join can result in false-possitive interaction with thread. Like said in docs :

When the timeout argument is present and not None, it should be a floating point number specifying a timeout for the operation in seconds (or fractions thereof). As join() always returns None, you must call isAlive() after join() to decide whether a timeout happened – if the thread is still alive, the join() call timed out.

and illustrative piece of code:

threads = [] for name in some_data: new = threading.Thread( target=self.some_func, args=(name,) ) threads.append(new) new.start() over_threads = iter(threads) curr_th = next(over_threads) while True: curr_th.join() if curr_th.is_alive(): continue try: curr_th = next(over_threads) except StopIteration: break