ThreadPool and Pool for parallel processing

Question

Is there a way to use both ThreadPool and Pool in python to parallelise a loop by specifying the number of CPUs and cores you wish to use?

For example I would have a loop execute as:

from multiprocessing.dummy import Pool as ThreadPool from tqdm import tqdm import numpy as np def my_function(x): return x + 1 pool = ThreadPool(4) my_array = np.arange(0,1e6,1) results = list(tqdm(pool.imap(my_function, my_array),total=len(my_array)))

For 4 cores but it I wanted to spread these out on multiple CPUs as well, is there a simple way to adapt the code?

I know that, I am trying to carry out a process on multiple cores across multiple CPUs — user8188120
– user8188120, Commented Jul 31, 2018 at 12:04

Deepak Saini · Accepted Answer · 2018-07-31 11:54:15Z

You are confusing between a core and a CPU. Generally, for all purposes both are considered to be the same(let's call them processor from now on).

When creating a thread pool in python, the threads are user level threads and are run on the same processor, due to Global Interpreter Lock(GIL) in python. As only one thread can control the python interpreter at a time. So, using (python)threads we don't get any real concurrency in data-intensive tasks.

How to solve this? Easy. Spawn multiple python processes running on different processors(each with its own interpreter). This is where the multi processing(mp) module is used, to spawn multiple processes from the parent python process in which it is called.

You can verify this by running htop(on linux, mac) and analysing the number of python processes. In case of mp module, they all will have the same name as the parent script where the pool.map function is called.

Timings for your code on a 8 core mac: 39.7s
Timing for this code on the same machine : 2.9s(note I can use 8 cores at max, but for comparison purposes using only 4)

Below is the modified code:

from multiprocessing.dummy import Pool as ThreadPool from tqdm import tqdm import numpy as np import time import multiprocessing as mp def my_function(x): return x + 1 pool = ThreadPool(4) my_array = np.arange(0,1e6,1) t1 = time.time() # results = list(tqdm(pool.imap(my_function, my_array),total=len(my_array))) pool = mp.Pool(processes=4) # Generally, set to 2*num_cores you have res = pool.map(my_function, my_array) print("Time taken = ", time.time() - t1)

This makes sense, I suspect that because I am trying to run the code on a cluster of CPUs there is some confusion in how it handles multiprocessing as it spawns operations on multiple CPUs but it will work on my stand alone laptop

Khanh Le · Accepted Answer · 2018-07-31 12:42:28Z

multiprocessing.dummy.Pool is exactly simple ThreadPool, which don't use multicores and multicpus (because of GIL). you must use multiprocessing.Pool to run Process, which is process in your OS (if you define Pool(N) - N is number of this processes, if no - number of your cores in OS is default). Arguments this processes get from internal queue of Pool. 'case of that U will use all cpu and all core in your OS

If I use multiprocessing.Pool i get operations spread out on multiple CPUs but when I use ThreadPool it only uses one CPU. Perhaps this is because I am running the code on a cluster of CPUs

Collectives™ on Stack Overflow

ThreadPool and Pool for parallel processing

2 Answers 2

1 Comment

1 Comment

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

1 Comment

Linked

Related