3

I am writing an interface to allow communication between a main program written in C and extension scripts written in python and run in a separate python interpreter process. The interface uses a UNIX socket for small amounts of data and POSIX shared memory for large arrays. The C program handles all creation, resource tracking and final unlinking of shared memory.

This works perfectly on Linux. I can transfer data between the two processes as expected using the shm.

However when exactly the same code runs on MacOS, although it runs without error the shared memory is always full of zeroes when read from the other process to the one that populated the memory. e.g. if I write image data into the shm from C, and read it from python, it's all zero. If I write image data into the shm from python and read it from C, again it's all zero. I create the shm in C as follows: (some error handling lines removed for clarity)

 void *shm_ptr = NULL; snprintf(shm_name_ptr, 30, "/%08x%08x%08x%04x", my_random_int(), my_random_int(), my_random_int(), my_random_int()); debug_print("shm name: %s\n", shm_name_ptr); *fd = shm_open(shm_name_ptr, O_CREAT | O_RDWR | O_EXCL, S_IRUSR | S_IWUSR); ftruncate(*fd, aligned_size) == -1); shm_ptr = mmap(NULL, (size_t) aligned_size, PROT_READ | PROT_WRITE, MAP_SHARED, *fd, 0); *shm_ptr_ptr = shm_ptr;

And then the details are passed to python through the socket. I haven't reproduced the details of that because the code is fairly long with #ifdefs for Windows etc., but the socket mechanism provably works and I can print the shm name as it is created in C and as it is received in python and show that they are the same:

shm name: /b6c31655f708d0e20760b60bc483 SHM allocation: Original size: 25941632, Aligned size: 25944064, Page size: 4096 Truncating shm file to 25941632 bytes log: b'/b6c31655f708d0e20760b60bc483'

(The first lines of text are printed from C, the last is printed from python.)

The __init__ function from the wrapper class I'm using in python to open the shm is shown below:

class SharedMemoryWrapper: """ Wrapper class to handle shared memory creation and cleanup across platforms. """ def __init__(self, name: str, size: int): self.name = name self.size = size # Store intended size separately self._shm = None try: # First try to attach to existing shared memory self._shm = shared_memory.SharedMemory(name=self.name) unregister(self._shm._name, "shared_memory") except FileNotFoundError: # If it doesn't exist, create new shared memory print("Existing SHM not found, creating a new one...") self._shm = shared_memory.SharedMemory(name=self.name, create=True, size=self.size)

(The unregister() call is there because the SHM is always allocated, tracked and cleaned up by the C program and passed to the python program, so this suppresses warning messages about shm leaks when the python script exits.)

Once the name and size are received over the socket the shared memory object is initialized using SharedMemoryWrapper(name=name_from_socket, size=size_from_socket)

On Linux the try block works and the shm is opened correctly, however on MacOS I see the message "Existing SHM not found, creating a new one..."

Please can someone explain what is different about MacOS here, and how can I fix it so that the existing shm is opened correctly?

Improve this question
5
  • Howe do you transfer the name? How do you create the SharedMemoryWrapper object? Please try to create a minimal reproducible example to show us. Commented Jan 7 at 15:13
  • 2
    I have a suspicion this may be due to MacOS's sandboxing mechanism. You may need to give the program full disk access. Commented Jan 7 at 16:08
  • @Someprogrammerdude the name gets transferred through a UNIX socket. I've tried to keep the details in the question minimal but sufficient, because you don't want to read pages of the actual code that sets up the socket, monitors it for connections, and the matching code on the python side. That bit works: the entire mechanism works in Linux and on MacOS I can printf the name sent from the C program and I can print the name as received by python, and they match. Example confirmation to follow in next comment... Commented Jan 7 at 16:31
  • Are you checking for errors from shm_open()? Commented Jan 7 at 20:18
  • Yes, I'm checking for errors from shm_open and ftruncate and mmap, and there are none reported. Commented Jan 7 at 21:50

1 Answer 1

Reset to default
3

I'm a colleague of OP and have looked into the issue we were having.

When creating a shared memory object on the C side, use a leading slash in its name:

 int fd = shm_open("/mem", O_RDWR|O_CREAT, S_IRUSR|S_IWUSR);

When accessing it on the Python side, do not use a leading slash as it is added automatically on POSIX systems:

shm = shared_memory.SharedMemory(name="mem", create=False)

And now Python is able to find the object.

supplemental edit by OP: this is indeed the answer, and having updated the start of the __init__ function above to:

 def __init__(self, name: str, size: int): if os.name != "nt": name = name.lstrip('/') # Remove leading '/' on POSIX systems # because SharedMemory.__init__ will add it back self.name = name self.size = size # Store intended size separately self._shm = None ...

everything now works.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.