Python - Multiprocessing pool.map(): Processes don't do anything

Question

I am trying to use the python multiprocessing library in order to parallize a task I am working on:

import multiprocessing as MP def myFunction((x,y,z)): ...create a sqlite3 database specific to x,y,z ...write to the database (one DB per process) y = 'somestring' z = <large read-only global dictionary to be shared> jobs = [] for x in X: jobs.append((x,y,z,)) pool = MP.Pool(processes=16) pool.map(myFunction,jobs) pool.close() pool.join()

Sixteen processes are started as seen in htop, however no errors are returned, no files written, no CPU is used.

Could it happen that there is an error in myFunction that is not reported to STDOUT and blocks execution?

Perhaps it is relevant that the python script is called from a bash script running in background.

did you place the extra "," in "jobs.append((x,y,z,))" in purpose? — Yaron
– Yaron, Commented Sep 5, 2016 at 12:35
In the function definition: def myFunction((x,y,z)): are the extra parenthesis on purpose? — Reut Sharabani
– Reut Sharabani, Commented Sep 5, 2016 at 12:35
Have you tried import multiprocessing.dummy as MP instead of import multiprocessing as MP? dummy uses threads instead of actual processes which helps with debugging a. — janbrohl
– janbrohl, Commented Sep 5, 2016 at 12:43

Radio Controlled · Accepted Answer · 2016-09-05 14:19:13Z

1

The lesson learned here was to follow the strategy suggested in one of the comments and use multiprocessing.dummy until everything works.

At least in my case, errors were not visible otherwise and the processes were still running as if nothing had happened.

answered Sep 5, 2016 at 14:19

Radio Controlled

98012 silver badges26 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Radio Controlled Over a year ago

Now I have the case that for two datasets, it totally works. For another (very large one), it works with threads but with processes same problem as described above occurs. Are there some kind of size limitations? Memory was not overflowing...

Collectives™ on Stack Overflow

Python - Multiprocessing pool.map(): Processes don't do anything

1 Answer 1

1 Comment

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Related