How to be more efficient using redis-py in multithreading program?

Redis is now thread-safe: https://github.com/andymccurdy/redis-py

Redis client instances can safely be shared between threads. Internally, connection instances are only retrieved from the connection pool during command execution, and returned to the pool directly after. Command execution never modifies state on the client instance.

Connection pool is automatically created by each instance of Redis()

Well, there are a couple questions here, so I will attempt to answer each separately.

1. If you are going to use threads, then you certainly want individual connections. Redis connections will not be thread safe if shared as you could send a request on one thread and read on another potentially (unless you use a mutex / lock). The connection pool implementation appears to me to be thread safe, so you should opt for that, or create your own connection pool using something like a queue.Queue where the individual connections are placed and threads get and put onto the queue.

2. Client side sharding is the "poor man's" way to shard data on multiple redis instances. A typical strategy involves applying some sort of hash, like a crc32 to the target key and the taking the modulus of that value and the number of shards. An example would look something like this:

--

>>> binascii.crc32("foo") % 3 1 >>> binascii.crc32("bar") % 3 2 >>> binascii.crc32("baz") % 3 0 

This is assuming we have 3 shards (or individual redis server instances). The key foo lives in the 2nd shard (index 1). The key bar lives on the 3rd shard (index 2) and baz is on the 1st shard.

Redis Cluster (in beta of Redis 3.0.0) aims to do the sharding server side vs client side.