import rdma: zero-copy networking with RDMA and Python

import rdma: zero-copy networking with RDMA and Python

• 1.
  import rdma: zero-copynetworking with RDMA and Python Andy Grover @groveronline http://groveronline.com http://blogs.oracle.com/linuxnstuff
• 2.
  Plan Sockets, RDMA,and RDMA Sockets
• 3.
  Python and performance
• 4.
  Issues I raninto
• 5.
  (Questions anytime.)
• 6.
  Socket Example client:sock.sendto(server, “get data.tgz”)
• 7.
  server: recvfrom() ->(“get data.tgz”)
• 8.
  server: data =open(“data.tgz”).read()
• 9.
  server: sock.sendto(client, “OK“ + data)
• 10.
  client: recvfrom() ->data
• 11.
  Sockets Sending datafrom server (S) to client (C), how many buffer copies are performed on S? On C?
• 12.
  2 on S,2 on C
• 13.
  S: read fromuser buffer, write to kernel buffer by OS
• 14.
  S: read fromkernel buffer by HW
• 15.
  C: write tokernel buffer by HW
• 16.
  C: read fromkernel buffer, write to user buffer (OS)
• 17.
  So what? Socketinterface is easy to use
• 18.
  Extra copy oneach side consumes CPU
• 19.
  Also consumes 3xRAM bandwidth!
• 20.
  What do wedo???
• 21.
  Direct Data Placement(RDMA)
• 22.
  RDMA? Target locksdown user memory region and gives sender a key to reference it
• 23.
  Sender tells HWdata buffer and key, Target HW uses key to place received data directly in user buffer
• 24.
  Done!
• 25.
  Way complicated
• 26.
  Aside: InfiniBand Cheapand high speed
• 27.
  Supports RDMA
• 28.
  RHEL 5.4+ supportsnatively
• 29.
  RDMA Sockets (RDS)Reliable Datagram Sockets
• 30.
  Full disclosure: myday job
• 31.
  Guaranteed delivery ofdatagrams
• 32.
  Allows RDMA opsvia sendmsg() and CMSGs
• 33.
  Hides complexity ofIB Verbs
• 34.
  Still pretty complex!
• 35.
  What is thesimplest possible interface to use RDMA? Let's try Python
• 36.
  Learning opportunity
• 37.
  Can it bedone?
• 38.
  Pythonically?
• 39.
  Can Python doefficient networking? Hell yes. Well, pretty sure
• 40.
  Interpreted, but fitsin per-CPU cache
• 41.
  Many CPU coresthese days
• 42.
  Shared RAM
• 43.
  Cache misses onthe data
• 44.
  Implementation Issues.
• 45.
  Python strings Immutable
• 46.
  Buffers shared behindthe scenes
• 47.
  Solution: mmap modulemap a file or anonymous memory
• 48.
  sliceable etc.
• 49.
  Python has nopointers We need addresses of things To pin it
• 50.
  To map itto the hardware Solution: C extension module using “new buffer protocol” added in 2.6
• 51.
  Python stdlib doesn'tsupport sendmsg/recvmsg WHAT??
• 52.
  Solution: external library,python-eunuchs
• 53.
  Native support RSN
• 54.
  Python AF_RDS supportCan't extend socket.socket
• 55.
  Solution: forget inheritance,just implement socket object methods
• 56.
  Implementing RdmaSocket Pythonas much as possible
• 57.
  ctypes used heavily
• 58.
  C Module solelyto return an address
• 59.
  RDMA Socket Exampleclient: m = mmap(-1, 8192)
• 60.
  client: cookie =sock.get_mr(m)
• 61.
  client: sock.sendto(server. “getdata.tgz, my cookie is <cookieval>”)
• 62.
  server: recvmsg() ->(“get data.tgz, my cookie is <cookieval>”
• 63.
  server: m =mmap(“data.tgz”)
• 64.
  server: sock.rdma_sendmsg(client, m,cookieval, length, token, “OK”)
• 65.
  client: recvmsg() ->“OK”
• 66.
  OK... Extra overheadnot worth it for small sizes
• 67.
  Copied “OK” insteadof “OK”+8K, CPU and cache win
• 68.
  It worked!
• 69.
  Future investigations Actualperformance data
• 70.
  Dogfood it --simplify RDS utility apps
• 71.
  RDMA loves async:go Twisted
• 72.
  Summary Sysadmins IBis fast, cheap networking, even without RDMA DB cluster & Storage cluster developers A new tool in the toolbox coming soon, even to Ethernet Python coders Shared-something could be good if your I/O is good enough C coders Writing a Python or xyz wrapper is straightforward, and enables a much wider pool of users
• 73.
  Thanks! http://github.com/agrover/python-rds