0

Python ThreadPoolExecutor not running parallelly, it is calling cube method in sequence and waiting for complete,

where I need to run 20 parallell threads

from concurrent.futures import ThreadPoolExecutor from time import sleep def cube(x): sleep(2) print(f'Cube of {x}:{x*x*x}') count = 0 while True: with ThreadPoolExecutor(max_workers=20) as exe: exe.submit(cube,2) count += 1 if count > 50: break 
1
  • This is because cube function is CPU bound function. I think you have to use multiprocessing instead of multithreading. Commented Mar 27, 2023 at 7:52

3 Answers 3

1

"with" statement uses __enter__ method for ThreadPoolExecutor, it initialize thread pool at this moment. So, for function execution in thread pool we should use it inside "with" statement.

with ThreadPoolExecutor(max_workers=20) as exe: # for default it will be number of os.cpu_count() futures = [] for i in range(50): futures.append(executor.submit(cube, i)) for future in concurrent.futures.as_completed(futures): print(future.result()) 
Sign up to request clarification or add additional context in comments.

Comments

0

I guess want something like this:

from concurrent import futures from time import sleep THREADS = 20 def cube(x): sleep(2) res = x*x*x print(f'Cube of {x}:{res}') return res with futures.ThreadPoolExecutor(max_workers=THREADS) as exe: jobs = {exe.submit(cube, i): i for i in range(THREADS)} for job in futures.as_completed(jobs): try: data = job.result() except Exception as exc: print('%r generated an exception: %s' % (jobs[job], exc)) else: print(f"Result of thread {jobs[job]}: {data})") 

Out:

Cube of 2:8 Cube of 1:1 Cube of 3:27 Cube of 0:0 Cube of 4:64 Result of thread 2: 8) Result of thread 1: 1) Result of thread 0: 0) Result of thread 4: 64) Cube of 5:125 Cube of 6:216 Cube of 8:512 Cube of 14:2744 Cube of 12:1728 Cube of 17:4913 Result of thread 3: 27) Cube of 13:2197 Cube of 7:343 Cube of 16:4096 Cube of 15:3375 Cube of 18:5832 Cube of 11:1331 Cube of 9:729 Cube of 19:6859 Cube of 10:1000 Result of thread 5: 125) Result of thread 6: 216) Result of thread 8: 512) Result of thread 14: 2744) Result of thread 12: 1728) Result of thread 17: 4913) Result of thread 13: 2197) Result of thread 7: 343) Result of thread 16: 4096) Result of thread 15: 3375) Result of thread 18: 5832) Result of thread 11: 1331) Result of thread 10: 1000) Result of thread 19: 6859) Result of thread 9: 729) 

1 Comment

I am submitting jobs as I received from yield output (one output) from other method, if I am checking jobs.completed, its not going to run, any solution for that ?
0

This might be something you want to achieve:

from concurrent.futures import ThreadPoolExecutor from time import sleep def cube(x): print(f'Starting {x}') sleep(2) print(f'Cube of {x}:{x*x*x}') count = [] for i in range (0,50): count.append(i) with ThreadPoolExecutor(max_workers=20) as exe: exe.map(cube,count) 

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.