Python multiprocessing library Pool.map()

Question

def myfun(a): return a*2 p=Pool(5) k0=time.time() p.map(myfun,[1,2,3,4,5,6,7,8,9,10,1,2,3,4,5,6,7,8,9,10]) k1=time.time() print(k1-k0) k0=time.time() for i in [1,2,3,4,5,6,7,8,9,10,1,2,3,4,5,6,7,8,9,10]: myfun(i) k1=time.time() print(k1-k0)

I am using the multiprocessing package in python. So as you can see i have executed two different snippets of code separately.The first one that uses Pool.map takes more time than the second one which is executed serially. Can anyone explain to me why so? I thought the p.map() would be much faster. Is it not executed parallely?

I never worked with Pool.map() but I can tell you for sure from Java or C, that on small data, no matter how many threads you are using, it will always run slower than with one thread, because it takes time to 'start' those threads and etc. Try running the code with a serious for, like thousands of steps and you will see some good results. If I'm wrong, I'm sorry, I'm just telling you what I've experienced in Java or C. — Mr. Wizard
– Mr. Wizard, Commented Aug 24, 2018 at 7:45

Mike McKerns · Accepted Answer · 2018-08-27 16:33:40Z

Indeed as noted in the comments, it takes longer to run in parallel for some tasks with multiprocessing. This is expected for very small tasks. The reason is that you have to spin up a python instance on each process for each worker used, and you also have to serialize and ship both the function and the data you are sending with map. This takes some time, so there's an overhead associated with using a multiprocessing.Pool. For very quick tasks, I suggest multiprocessing.dummy.Pool, which uses threads -- and thus minimizes setup overhead.

Try putting a time.sleep(x) in your function call, and varying x. You'll see that as x increases, the function becomes more suitable to run in a thread pool, and then in a process pool for even more expensive x.

Collectives™ on Stack Overflow

Python multiprocessing library Pool.map()

1 Answer 1

Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Related