Python: can't access memory changed with multiprocessing

Question

Here's a python module in a python package:

import multiprocessing as mp class Test(object): def __init__(self): self.dict = dict() def fill_dict(self): self.dict = {'testing': 123} print self.dict if __name__ == "__main__": tests = [Test() for i in xrange(3)] jobs = [mp.Process(target=test.fill_dict, args=()) for test in tests] for job in jobs: job.start() for job in jobs: job.join() print "RESULTS:" for test in tests: print test.dict

I run the module and get the results as follows:

C:\path\to\package>python -m package.path.to.module {'testing': 123} {'testing': 123} {'testing': 123} RESULTS: {} {} {}

From the printout, it seems that each test.dict was filled in parallel by multiprocessing. However, when I try to recover the results, the test.dicts seems to be empty. Can someone explain why this is happening and what I can do to recover the nonempty test.dicts?

EDIT: @Valentin Lorentz, @Moinuddin Quadri, @zstewart

Even if I change the latter part of the module to

if __name__ == "__main__": test0 = Test() test1 = Test() test2 = Test() jobs = list() jobs.append(mp.Process(target=test0.fill_dict, args=())) jobs.append(mp.Process(target=test1.fill_dict, args=())) jobs.append(mp.Process(target=test2.fill_dict, args=())) for job in jobs: job.start() for job in jobs: job.join() print "RESULTS:" print test0.dict print test1.dict print test2.dict

I get the same results. Each process is completely independent of the others (they don't share memory and don't need to). So when answers speak of sharing memory between processes, does this mean between the main module and the multiprocessing processes?

Multiprocessing launches separate python processes. You have to use a multiprocessing.queue to send data between the processes. — zstewart
– zstewart, Commented Dec 18, 2016 at 17:31

Community · Accepted Answer · 2017-05-23 12:08:46Z

You can not transmit data between threads using built-in types. You need to use multiprocessing.Queue or mutiprocess.Pipes to perform that. Check: Exchanging objects between processes.

You may refer: Running multiple asynchronous function and get the returned value of each function. It is having the example on how to use Queue() instead of list.

Valentin Lorentz · Accepted Answer · 2016-12-18 17:32:56Z

Processes do not share memory by default. That's on purpose, because sharing memory between execution threads is a very complicated topic (synchronization between threads, communications, ...)

The most direct solution to your problem is to explicitely share this object: https://docs.python.org/3/library/multiprocessing.html#sharing-state-between-processes

However, a cleaner way is to make your function return a result, and use multiprocessing.Pool.map to call the function in multiple processes.

Collectives™ on Stack Overflow

Python: can't access memory changed with multiprocessing

2 Answers 2

Comments

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Linked

Related