How to know shared memory between two processes?

You can look at /proc/<pid>/maps, /proc/<pid>/smaps (or pmap -x <pid> if your OS supports) of interested process ID's and compare outputs to determine shared memory regions. That includes shared memory segments via shmget calls, as well as any shared libraries, files.

Edit: As mr.spuratic pointed out his answer here has more details on kernel side

You can look at a process RSS using ps, however it doesn't take into consideration all the shared pages. To see RSS for specific process, see below

cv@thunder:~$ ps -o rss,pid,comm -p $$,7023 RSS PID COMMAND 22060 7023 xfwm4 6876 18094 bash 

smem tool provides more detailed information, taking into consideration of shared pages. See below output for the same above process

cv@thunder:~$ smem -t |egrep "RSS|$$|7023" PID User Command Swap USS PSS RSS 9852 cv grep -E RSS|18094|7023 0 340 367 2220 18094 cv bash 0 3472 4043 6876 7023 cv xfwm4 --display :0.0 --sm-c 0 5176 7027 22192 

From man smem:

 smem reports physical memory usage, taking shared memory pages into account. Unshared memory is reported as the USS (Unique Set Size). Shared memory is divided evenly among the processes sharing that memory. The unshared memory (USS) plus a process's proportion of shared memory is reported as the PSS (Proportional Set Size). The USS and PSS only include physical memory usage. They do not include memory that has been swapped out to disk.