multithreading/processing in Python

Question

Hello I am trying define values of a dictionary in parallel using multiprocessing. When the function f() is called outside "pool" the dictionary value is set correctly. In the pool call however it fails.

What am I doing wrong? Thanks.

from multiprocessing import Pool hits={} def f(x): hits[x] = x #this will involve a complex operation f('000') print hits['000'] if __name__ == '__main__': pool = Pool(processes=2) inputs = ['a','b','c'] result = pool.map(f, inputs) print hits['a']

Sharing state between processes is hard. Perhaps perform the complex operation in child processes and maintain the global state of hits in the parent. — Tim McNamara
– Tim McNamara, Commented Oct 25, 2011 at 21:57

Sven Marnach · Accepted Answer · 2011-10-25 21:56:40Z

You spawn a bunch of subprocesses which inherit the current value of hits. Each subprocess get its own copy of hits and modifies it, then exits, dropping the process-local copy of hits.

The intended use of multiprocessing.Pool.map() is to use its return value, which you are ignoring. Each process could return its copy of hits, and you finally join these copies.

I see, then the answer is: from multiprocessing import Pool hits={} def f(x): return x if name == 'main': pool = Pool(processes=2) inputs = ['a','b','c'] result = pool.map(f, inputs) result=dict(zip(result,result)) print result.items()

Collectives™ on Stack Overflow

multithreading/processing in Python

1 Answer 1

1 Comment

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Related