6

The function store_in_shm writes a numpy array to the shared memory while the second function read_from_shm creates a numpy array using data in the same shared memory space and returns the numpy array.

However, running the code in Python 3.8 gives the following segmentation error:

zsh: segmentation fault python foo.py

Why is there no problem accessing the numpy array from inside the function read_from_shm, but a segmentation error appears when accessing the numpy array again outside of the function?

Output:

From read_from_shm(): [0 1 2 3 4 5 6 7 8 9] zsh: segmentation fault python foo.py % /Users/athena/opt/anaconda3/envs/test/lib/python3.8/multiprocessing/resource_tracker.py:203: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d '

foo.py

import numpy as np from multiprocessing import shared_memory def store_in_shm(data): shm = shared_memory.SharedMemory(name='foo', create=True, size=data.nbytes) shmData = np.ndarray(data.shape, dtype=data.dtype, buffer=shm.buf) shmData[:] = data[:] shm.close() return shm def read_from_shm(shape, dtype): shm = shared_memory.SharedMemory(name='foo', create=False) shmData = np.ndarray(shape, dtype, buffer=shm.buf) print('From read_from_shm():', shmData) return shmData if __name__ == '__main__': data = np.arange(10) shm = store_in_shm(data) shmData = read_from_shm(data.shape, data.dtype) print('From __main__:', shmData) # no seg fault if we comment this line shm.unlink()
Improve this question
5
  • I've typically run into problems with python's multiprocessing managers, and just use multiprocessing.sharedctypes.RawArray and a multiprocessing.Lock for manual synchronization. With your example I actually get different answers with Mac vs Windows where windows can't seem to re-open the file after being close()d within the same process. The error you get suggests that it wants you to also close shm from within read_from_shm because it's saying you're not close()ing something. Commented Sep 3, 2020 at 3:39
  • Apparently with Windows unlink actually does nothing, and the file is destroyed once all currently open copies are closed, so the memory is destroyed in-between store_in_shm and read_from_shm resulting in the file not found error. Commented Sep 3, 2020 at 4:15
  • 1
    @Aaron if we do shm.close() inside read_from_shm, I think the numpy array shmData have problem accessing its buffer in shm as shm is now closed. Commented Sep 3, 2020 at 4:18
  • 1
    yah, I'm still looking to figure out how it's supposed to work. I'll post an actual answer if I figure it out for real... Commented Sep 3, 2020 at 4:19
  • I may have to eventually retract my statement on preferring sharedctypes. After diving the code, it seems like shared_memory may be a bit simpler and more lightweight (although they do work in a similar manner under the hood). There are just apparently a few bugs to still work out... Commented Sep 3, 2020 at 6:18

1 Answer 1

Reset to default
7

Basically the problem seems to be that the underlying mmap'ed file (owned by shm within read_from_shm) is being closed when shm is garbage collected when the function returns. Then shmData refers back to it, which is where you get the segfault (for referring to a closed mmap) This seems to be a known bug, but it can be solved by keeping a reference to shm.

Additionally all SharedMemory instances want to be close()'d with exactly one of them being unlink()'ed when it is no longer necessary. If you don't call shm.close() yourself, it will be called at GC as mentioned, and on Windows if it is the only one currently "open" the shared memory file will be deleted. When you call shm.close() inside store_in_shm, you introduce an OS dependency as on windows the data will be deleted, and MacOS and Linux, it will retain until unlink is called.

Finally though this doesn't appear in your code, another problem currently exists where accessing data from independent processes (rather than child processes) can similarly delete the underlying mmap too soon. SharedMemory is a very new library, and hopefully all the kinks will work out soon.

You can re-write the given example to retain a reference to the "second" shm and just use either one to unlink:

import numpy as np from multiprocessing import shared_memory def store_in_shm(data): shm = shared_memory.SharedMemory(name='foo', create=True, size=data.nbytes) shmData = np.ndarray(data.shape, dtype=data.dtype, buffer=shm.buf) shmData[:] = data[:] #there must always be at least one `SharedMemory` object open for it to not # be destroyed on Windows, so we won't `shm.close()` inside the function, # but rather after we're done with everything. return shm def read_from_shm(shape, dtype): shm = shared_memory.SharedMemory(name='foo', create=False) shmData = np.ndarray(shape, dtype, buffer=shm.buf) print('From read_from_shm():', shmData) return shm, shmData #we need to keep a reference of shm both so we don't # segfault on shmData and so we can `close()` it. if __name__ == '__main__': data = np.arange(10) shm1 = store_in_shm(data) #This is where the *Windows* previously reclaimed the memory resulting in a # FileNotFoundError because the tempory mmap'ed file had been released. shm2, shmData = read_from_shm(data.shape, data.dtype) print('From __main__:', shmData) shm1.close() shm2.close() #on windows "unlink" happens automatically here whether you call `unlink()` or not. shm2.unlink() #either shm1 or shm2
Improve this answer
Sign up to request clarification or add additional context in comments.

13 Comments

Logically, you must still have a reference to the SharedMemory object to close it, and you wouldn't probably assume you can still access the data it's backing (the numpy array) after closing. It seems like this may be the underlying reason the dereference is missed, and segfault is thrown.
"keeping a reference to shm" Is there a way to store a reference somewhere to workaround this bug automatically? I want to create a large number of shared_memory backed numpy arrays. I guess I could keep a huge list of pointers and append a reference to it whenever I make a new array? Is there a more Pythonic way of doing that?
@ike maybe create a subclass of ndarray which handles creating the array from the buffer then hangs on to the reference for you? Probably should then define __del__ too to call unlink so the file gets deleted when it's not needed anymore on *nix systems
The problem is, I'm creating different ndarrays backed by the same buffer in different threads. I need to store these references somewhere, but I run into the same sharing issue all over again - if I only keep them in the thread that allocated them I seem to be getting segfaults.
@ike obligatory: "be very careful mixing threads and processes: fork does not play nice at all with multithreading" This is basically the same issue as OP had... you must always have a reference somewhere or the temporary file backing it will get deleted by the os (delete file when all handles are closed). Honestly in this case I'd just make a list at the module level in the parent process, and stuff all shm objects into it, then do something like get the length of the list (anything really that refers to it) at the end of the process so it doesn't ever get garbage collected.
|

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.