Shared memory across processes on Linux/x86_64

Question

I have a few questions on using shared memory with processes. I looked at several previous posts and couldn't glean the answers precisely enough. Thanks in advance for your help.

I'm using shm_open + mmap like below. This code works as intended with parent and child alternating to increment g_shared->count (the synchronization is not portable; it works only for certain memory models, but good enough for my case for now). However, when I change MAP_SHARED to MAP_ANONYMOUS | MAP_SHARED, the memory isn't shared and the program hangs since the 'flag' doesn't get flipped. Removing the flag confirms what's happening with each process counting from 0 to 10 (implying that each has its own copy of the structure and hence the 'count' field). Is this the expected behavior? I don't want the memory to be backed by a file; I really want to emulate what might happen if these were threads instead of processes (they need to be processes for other reasons).
Do I really need shm_open? Since the processes belong to the same hierarchy, can I just use mmap alone instead? I understand this would be fairly straightforward if there wasn't an 'exec,' but how do I get it to work when there is an 'exec' following the 'fork?'
I'm using kernel version 3.2.0-23 on x86_64 (Intel i7-2600). For this implementation, does mmap give the same behavior (correctness as well as performance) as shared memory with pthreads sharing the same global object? For example, does the MMU map the segment with 'cacheable' MTRR/TLB attributes?
Is the cleanup_shared() code correct? Is it leaking any memory? How could I check? For example, is there an equivalent of System V's 'ipcs?'

thanks, /Doobs

shmem.h:

#ifndef __SHMEM_H__ #define __SHMEM_H__ //includes #define LEN 1000 #define ITERS 10 #define SHM_FNAME "/myshm" typedef struct shmem_obj { int count; char buff[LEN]; volatile int flag; } shmem_t; extern shmem_t* g_shared; extern char proc_name[100]; extern int fd; void cleanup_shared() { munmap(g_shared, sizeof(shmem_t)); close(fd); shm_unlink(SHM_FNAME); } static inline void init_shared() { int oflag; if (!strcmp(proc_name, "parent")) { oflag = O_CREAT | O_RDWR; } else { oflag = O_RDWR; } fd = shm_open(SHM_FNAME, oflag, (S_IREAD | S_IWRITE)); if (fd == -1) { perror("shm_open"); exit(EXIT_FAILURE); } if (ftruncate(fd, sizeof(shmem_t)) == -1) { perror("ftruncate"); shm_unlink(SHM_FNAME); exit(EXIT_FAILURE); } g_shared = mmap(NULL, sizeof(shmem_t), (PROT_WRITE | PROT_READ), MAP_SHARED, fd, 0); if (g_shared == MAP_FAILED) { perror("mmap"); cleanup_shared(); exit(EXIT_FAILURE); } } static inline void proc_write(const char* s) { fprintf(stderr, "[%s] %s\n", proc_name, s); } #endif // __SHMEM_H__

shmem1.c (parent process):

#include "shmem.h" int fd; shmem_t* g_shared; char proc_name[100]; void work() { int i; for (i = 0; i &lt ITERS; ++i) { while (g_shared->flag); ++g_shared->count; sprintf(g_shared->buff, "%s: %d", proc_name, g_shared->count); proc_write(g_shared->buff); g_shared->flag = !g_shared->flag; } } int main(int argc, char* argv[], char* envp[]) { int status, child; strcpy(proc_name, "parent"); init_shared(argv); fprintf(stderr, "Map address is: %p\n", g_shared); if (child = fork()) { work(); waitpid(child, &status, 0); cleanup_shared(); fprintf(stderr, "Parent finished!\n"); } else { /* child executes shmem2 */ execvpe("./shmem2", argv + 2, envp); } }

shmem2.c (child process):

#include "shmem.h" int fd; shmem_t* g_shared; char proc_name[100]; void work() { int i; for (i = 0; i &lt ITERS; ++i) { while (!g_shared->flag); ++g_shared->count; sprintf(g_shared->buff, "%s: %d", proc_name, g_shared->count); proc_write(g_shared->buff); g_shared->flag = !g_shared->flag; } } int main(int argc, char* argv[], char* envp[]) { int status; strcpy(proc_name, "child"); init_shared(argv); fprintf(stderr, "Map address is: %p\n", g_shared); work(); cleanup_shared(); return 0; }

The tmpfs that Linux uses to implement shared memory is a pseudo filesystem. The content of the files there resides entirely in memory and mmap-ing them directly maps the same physical pages that tmpfs uses to store the content of the shared memory "files". What shm_open does is creating/opening a file in /dev/shm (or wherever tmpfs is mounted) and giving you back the file descriptor. You can create the file on your own with open(..., O_CREAT) but it won't be portable. The open file descriptor will survive the execve but the mappings won't. — Hristo Iliev
– Hristo Iliev, Commented Aug 17, 2012 at 17:02

Andy Ross · Accepted Answer · 2012-08-17 05:37:51Z

Passing MAP_ANONYMOUS causes the kernel to ignore your file descriptor argument and give you a private mapping instead. That's not what you want.
Yes, you can create an anonymous shared mapping in a parent process, fork, and have the child process inherit the mapping, sharing the memory with the parent and any other children. That obvoiusly doesn't survive an exec() though.
I don't understand this question; pthreads doesn't allocate memory. The cacheability will depend on the file descriptor you mapped. If it's a disk file or anonymous mapping, then it's cacheable memory. If it's a video framebuffer device, it's probably not.
That's the right way to call munmap(), but I didn't verify the logic beyond that. All processes need to unmap, only one should call unlink.

Thanks Andy. To clarify further: 2. <snip> That obviously doesn't survive an exec() though This is my problem. Can I hack around it? Can I derive the address that MAP_ANONYMOUS ended up mapping to and pass it to the child process explicitly? 3. I don't understand this question <snip> I'm specifically asking for the shared memory shown in the example code. Will '*g_shared' be cached. In contrast, when pthreads, the object would be allocated in the global segment shared by the threads and hence cached. I think your answer suggests that it will even if I use MAP_SHARED like above?
@user1605883, mmap(NULL, length, PROT_READ|PROT_WRITE, MAP_ANONYMOUS|MAP_SHARED, -1, 0) creates a non-copy-on-write shared anonymous mapping that will survive fork but not execve. execve (the syscall that backs all exec*() library calls) installs a fresh page table and thus destroys all existing mappings.

Matthew Moskewicz · Accepted Answer · 2014-09-12 22:17:20Z

2b) as a middle-ground of a sort, it is possible to call:

int const shm_fd = shm_open(fn,...); shm_unlink(fn);

in a parent process, and then pass fd to a child process created by fork()/execve() via argp or envp. since open file descriptors of this type will survive the fork()/execve(), you can mmap the fd in both the parent process and any dervied processes. here's a more complete code example copied and simplified/sanitized from code i ran successfully under Ubuntu 12.04 / linux kernel 3.13 / glibc 2.15:

int create_shm_fd( void ) { int oflags = O_RDWR | O_CREAT | O_TRUNC; string const fn = "/some_shm_fn_maybe_with_pid"; int fd; neg_one_fail( fd = shm_open( fn.c_str(), oflags, S_IRUSR | S_IWUSR ), "shm_open" ); if( fd == -1 ) { rt_err( strprintf( "shm_open() failed with errno=%s", str(errno).c_str() ) ); } // for now, we'll just pass the open fd to our child process, so // we don't need the file/name/link anymore, and by unlinking it // here we can try to minimize the chance / amount of OS-level shm // leakage. neg_one_fail( shm_unlink( fn.c_str() ), "shm_unlink" ); // by default, the fd returned from shm_open() has FD_CLOEXEC // set. it seems okay to remove it so that it will stay open // across execve. int fd_flags = 0; neg_one_fail( fd_flags = fcntl( fd, F_GETFD ), "fcntl" ); fd_flags &= ~FD_CLOEXEC; neg_one_fail( fcntl( fd, F_SETFD, fd_flags ), "fcntl" ); // resize the shm segment for later mapping via mmap() neg_one_fail( ftruncate( fd, 1024*1024*4 ), "ftruncate" ); return fd; }

it's not 100% clear to me if it's okay spec-wise to remove the FD_CLOEXEC and/or assume that after doing so the fd really will survive the exec. the man page for exec is unclear; it says: "POSIX shared memory regions are unmapped", but to me that's redundant with the general comments earlier that mapping are not preserved, and doesn't say that shm_open()'d fd will be closed. any of course there's the fact that, as i mentioned, the code does seem to work in at least one case.

the reason i might use this approach is that it would seem to reduce the chance of leaking the shared memory segment / filename, and it makes it clear that i don't need persistence of the memory segment.

Collectives™ on Stack Overflow

Shared memory across processes on Linux/x86_64

2 Answers 2

2 Comments

Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Related