5

Python docs of the multiprocessing module state:

Changed in version 3.6: Shared objects are capable of being nested. For example, a shared container object such as a shared list can contain other shared objects which will all be managed and synchronized by the SyncManager.

This does work with list and dict. However, if I try to create a shared Queue inside a shared dict, I get an error:

>>> from multiprocessing import Manager >>> m = Manager() >>> d = m.dict() >>> d['a'] = m.list() >>> d['b'] = m.dict() >>> d['c'] = m.Queue() Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<string>", line 2, in __setitem__ File "/usr/lib/python3.6/multiprocessing/managers.py", line 772, in _callmethod raise convert_to_error(kind, result) multiprocessing.managers.RemoteError: --------------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python3.6/multiprocessing/managers.py", line 228, in serve_client request = recv() File "/usr/lib/python3.6/multiprocessing/connection.py", line 251, in recv return _ForkingPickler.loads(buf.getbuffer()) File "/usr/lib/python3.6/multiprocessing/managers.py", line 881, in RebuildProxy return func(token, serializer, incref=incref, **kwds) TypeError: AutoProxy() got an unexpected keyword argument 'manager_owned' ---------------------------------------------------------------------------

Seems like https://hg.python.org/cpython/rev/39e7307f9aee is the changeset which introduced nested shared objects.

Improve this question
0

2 Answers 2

Reset to default
5

The error is caused by AutoProxy currently not handling all BaseProxy arguments. There's a pull request which has not been merged yet. You either need to monkey-patch AutoProxy, or you look into multiprocessing.managers.py and apply the changes in the patch here directly to your source code.

It's really important to fix both lines in the patch to prevent a memory leak in the server process. The manager_owned-flag is used to let the BaseProxy code know, when to skip a reference increment for a proxy the manager owns himself (through nesting).

Improve this answer
Sign up to request clarification or add additional context in comments.

5 Comments

Is this solution sufficient, or does it risk a memory leak? stackoverflow.com/questions/46779860/…
@DavidParks I had a play with the source code before I wrote this answer and the answer (both) you linked is not sufficient as it currently is. Created objects will always have a reference in the manager-process and hence never be destroyed.
Thanks, when you say created objects, you mean objects created by the manager? If so, then if we create a fixed number of objects (say a few manager.Queue objects) then it sounds like it's not an issue, right? Or do you mean every object that is added to a manager.Queue, or other manager data structures, will leave a residue?
@DavidParks Sorry, with "objects" I meant only the nested queues itself, other objects you put inside it should not be affected. So unless you create masses of nested queues, it's not going to be a big issue. But the additional change to make it a clean solution is small enough, so I would just implement the patch as a whole and tick it off forever.
For what it’s worth, I created a monkey patch that handles this all properly (handling the manager_owned flag properly and re-registering the existing SyncManager registrations that use AutoProxy). See my answer on the duplicate post.
0

If you don't want to patch the underlying python library, you can apply this patch in your own code using the following.

I've copied the change from the pull request reference by @Darkonaut and made package name modifications so it works outside the original package. This is placed at the module level of any module that uses multiprocessing.managers.

Note that the solution I referenced in comments in @Darkonaut's answer produced seg faults in my own testing, but this solution did not.

import multiprocessing.managers def AutoProxy(token, serializer, manager=None, authkey=None, exposed=None, incref=True, manager_owned=False): ''' Return an auto-proxy for `token` ''' _Client = multiprocessing.managers.listener_client[serializer][1] if exposed is None: conn = _Client(token.address, authkey=authkey) try: exposed = dispatch(conn, None, 'get_methods', (token,)) finally: conn.close() if authkey is None and manager is not None: authkey = manager._authkey if authkey is None: authkey = multiprocessing.process.current_process().authkey ProxyType = multiprocessing.managers.MakeProxyType('AutoProxy[%s]' % token.typeid, exposed) proxy = ProxyType(token, serializer, manager=manager, authkey=authkey, incref=incref, manager_owned=manager_owned) proxy._isauto = True return proxy multiprocessing.managers.AutoProxy = AutoProxy
Improve this answer

2 Comments

In W10_64, with Py3.7.7 adding this in the code raise an error when you try to run a SyncManager: _pickle.PicklingError: Can't pickle <function AutoProxy at 0x000001DAF1615EE8>: it's not the same object as multiprocessing.managers.AutoProxy. It seems to work in WSL.
Patching the function is not enough. My answer to the older post this one is a duplicate of also re-registers all SyncManager types that used the unpatched AutoProxy, because otherwise you get pickling errors.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.