I have a thread list in python code which is quiet big
threadlist = [thread1, thread2, thread3..........threadn] where n is greater than 200 and I need to process 10 threads at a time using python queue, how do I do that. Suggestions deeply appreciated.
This seems like a great place to use the threading.Semaphore class. It's designed explicitly to allow only a limited number of threads to access a resource at one time. You create a Semaphore(10) in the main thread, then have each child thread call acquire() at the start of its execution, and release at the end. Then only ten will run at a time.
Here's an example which abstracts out the semaphore handling to a thread subclass; but you could just as easily do it yourself inside the target function, if you don't mind a little less encapsulation.
from threading import Thread, Semaphore import time class PatientThread(Thread): MAXIMUM_SIMULTANEOUS_THREADS = 10 _semaphore = Semaphore(MAXIMUM_SIMULTANEOUS_THREADS) def run(self): PatientThread._semaphore.acquire() super().run() PatientThread._semaphore.release() def exampleTargetFunc(x): print(f"Thread #{x} is starting.") time.sleep(1) print(f"Thread #{x} is completing.") threads = [] for i in range(200): threads.append(PatientThread(target=exampleTargetFunc, args=(i,))) for t in threads: t.start() for t in threads: t.join() Result:
Thread #0 is starting. Thread #1 is starting. Thread #2 is starting. Thread #3 is starting. Thread #4 is starting. Thread #5 is starting. Thread #6 is starting. Thread #7 is starting. Thread #8 is starting. Thread #9 is starting. <a one second pause occurs here...> Thread #2 is completing. Thread #0 is completing. Thread #1 is completing. Thread #10 is starting. Thread #11 is starting. Thread #12 is starting. Thread #4 is completing. Thread #5 is completing. Thread #3 is completing. Thread #7 is completing. Thread #13 is starting. Thread #6 is completing. Thread #16 is starting. Thread #14 is starting. Thread #15 is starting. Thread #17 is starting. Thread #9 is completing. Thread #8 is completing. Thread #18 is starting. Thread #19 is starting. <... And so on for the next 20 seconds> This demonstrates that threads 10, 11, and 12 did not start until 0, 1, and 2 finished. And likewise for threads 3-9 and threads 13-19.
