Parallel a process in Python

Question

I am learning how to run simple function on multiple threads in Python.

Assume this simple code:

 from itertools import product all_combinations = [] for cas in range(3): target_sum = 10 combinations = product(range(target_sum + 1), repeat=4) valid_combinations = [combo for combo in combinations if sum(combo) == target_sum] all_combinations.append(valid_combinations) compute_combo = [] for a in all_combinations[2]: for b in all_combinations[1]: for c in all_combinations[0]: compute_combo.append([a, b, c]) def foo(bar): max_combo = 0 for j in bar: for a in j: suma = a[0] + a[1] + 3*a[2] + a[3] if suma > max_combo: max_combo = suma return max_combo import time start_time_simple = time.time() result_simple = foo(compute_combo) print(f"1 CPU result: {result_simple}") end_time_simple = time.time() execution_time_simple = end_time_simple - start_time_simple print(f"1 CPU run time: {execution_time_simple} s")

This is how I try to run the foo() function on multiple threads:

 from multiprocessing.dummy import Pool as ThreadPool start_time_par = time.time() pool = ThreadPool(4) result_par = pool.map(foo, [compute_combo]) pool.close() pool.join() end_time_par = time.time() execution_time_par = end_time_par - start_time_par print(f"4 CPUs result {result_par}") print(f"4 CPUs run time: {execution_time_par} s")

But the run time is the same, what is the problem here, please? Thanks a lot

This is expected due to the Global Interpreter Lock (GIL). Multithreading is only useful in CPython for computation releasing the GIL (very few) or ones doing IOs. Classical computations cannot be faster. You should use multiprocessing (which as also significant overheads and it is the only standard alternative). Note there are many posts opened on this topic. — Jérôme Richard
– Jérôme Richard, Commented Apr 23, 2024 at 9:00
Does this answer your question? Multi-threading and Single-threading performance issues in CPU-bound task — Serge Hauri
– Serge Hauri, Commented Apr 23, 2024 at 9:06

Martin · Accepted Answer · 2024-04-23 08:49:00Z

Use threading instead:

import time import threading def func1(): while True: time.sleep(1) var += 1 def main(): time.sleep(100) print("hello world")

Then:

timer_thread = threading.Thread(target=func1) timer_thread.start() main()

func1 and main function will run simultaneously. A time.sleep() function without threading would otherwise prevent any code in your script from running, however you can replace this with a time intensive script.

How this is supposed to solve the OP issue? Threading is still limited by the GIL... time.sleep is very bad for multithreading benchmark as it is not realistic at all. For example, with that, you can show that multi threading scale well with 100 threads on a 4-core machine and observe a low startup latency... while in almost all real-world application it will never be true (or even close to).

Collectives™ on Stack Overflow

Parallel a process in Python

1 Answer 1

1 Comment

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Linked

Related