Table of Contents (中文说明)
- 1. Redis-replicator
- 2. Installation
- 3. Simple Usage
- 4. Advanced Topics
- 5. Other Topics
- 6. Contributors
- 7. Consulting
- 8. References
- 9. Supported By
Redis Replicator is an implementation of the Redis Replication protocol written in Java. It can parse, filter, and broadcast RDB and AOF events in real-time. It can also synchronize Redis data to a local cache or a database. In this document, Command refers to writable commands (e.g., set, hmset) and excludes readable commands (e.g., get, hmget). Supports Redis 8.4.x and older versions.
- Compile: JDK 9+
- Runtime: JDK 8+
- Maven: 3.3.1+
- Redis: 2.6 - 8.4
<dependency> <groupId>com.moilioncircle</groupId> <artifactId>redis-replicator</artifactId> <version>3.11.0</version> </dependency># Step 1: Install JDK 11+ for compilation # Step 2: Clone the repository git clone https://github.com/leonchen83/redis-replicator.git # Step 3: Navigate to the project directory cd redis-replicator # Step 4: Build the project mvn clean install package -DskipTests| Redis Version | redis-replicator Version |
|---|---|
| [2.6, 8.4.x] | [3.11.0, ] |
| [2.6, 8.2.x] | [3.10.0,3.10.0] |
| [2.6, 8.0.x] | [3.9.0, 3.9.0] |
| [2.6, 7.2.x] | [3.8.0, 3.8.1] |
| [2.6, 7.0.x] | [3.6.4, 3.7.0] |
| [2.6, 7.0.x-RC2] | [3.6.2, 3.6.3] |
| [2.6, 7.0.0-RC1] | [3.6.0, 3.6.1] |
| [2.6, 6.2.x] | [3.5.2, 3.5.5] |
| [2.6, 6.2.0-RC1] | [3.5.0, 3.5.1] |
| [2.6, 6.0.x] | [3.4.0, 3.4.4] |
| [2.6, 5.0.x] | [2.6.1, 3.3.3] |
| [2.6, 4.0.x] | [2.3.0, 2.5.0] |
| [2.6, 4.0-RC3] | [2.1.0, 2.2.0] |
| [2.6, 3.2.x] | [1.0.18] (not supported) |
Replicator replicator = new RedisReplicator("redis://127.0.0.1:6379"); replicator.addEventListener(new EventListener() { @Override public void onEvent(Replicator replicator, Event event) { if (event instanceof KeyStringValueString) { KeyStringValueString kv = (KeyStringValueString) event; System.out.println(new String(kv.getKey())); System.out.println(new String(kv.getValue())); } else { // ... } } }); replicator.open();You can use DumpRdbVisitor to convert an RDB file to the Redis DUMP format.
Replicator r = new RedisReplicator("redis:///path/to/dump.rdb"); r.setRdbVisitor(new DumpRdbVisitor(r)); r.addEventListener(new EventListener() { @Override public void onEvent(Replicator replicator, Event event) { if (!(event instanceof DumpKeyValuePair)) return; DumpKeyValuePair dkv = (DumpKeyValuePair) event; byte[] serialized = dkv.getValue(); // We can use the Redis RESTORE command to migrate this serialized value to another Redis instance. } }); r.open();You can use SkipRdbVisitor to check the correctness of an RDB file.
Replicator r = new RedisReplicator("redis:///path/to/dump.rdb"); r.setRdbVisitor(new SkipRdbVisitor(r)); r.open();By default, redis-replicator uses the PSYNC command, pretending to be a replica, to receive commands. An example is as follows:
Replicator r = new RedisReplicator("redis://127.0.0.1:6379"); r.addEventListener(new EventListener() { @Override public void onEvent(Replicator replicator, Event event) { System.out.println(event); } }); r.open();However, on some cloud services, the PSYNC command is prohibited. In such cases, you can use the SCAN command instead:
Replicator r = new RedisReplicator("redis://127.0.0.1:6379?enableScan=yes&scanStep=256"); r.addEventListener(new EventListener() { @Override public void onEvent(Replicator replicator, Event event) { System.out.println(event); } }); r.open();See examples
@CommandSpec(command = "APPEND") public static class YourAppendCommand extends AbstractCommand { private final String key; private final String value; public YourAppendCommand(String key, String value) { this.key = key; this.value = value; } public String getKey() { return key; } public String getValue() { return value; } }public class YourAppendParser implements CommandParser<YourAppendCommand> { @Override public YourAppendCommand parse(Object[] command) { return new YourAppendCommand(new String((byte[]) command[1], UTF_8), new String((byte[]) command[2], UTF_8)); } }Replicator replicator = new RedisReplicator("redis://127.0.0.1:6379"); replicator.addCommandParser(CommandName.name("APPEND"), new YourAppendParser());replicator.addEventListener(new EventListener() { @Override public void onEvent(Replicator replicator, Event event) { if(event instanceof YourAppendCommand){ YourAppendCommand appendCommand = (YourAppendCommand)event; // Your code goes here } } });See CommandExtensionExample.java
cd /path/to/redis-4.0-rc2/src/modules makeloadmodule /path/to/redis-4.0-rc2/src/modules/hellotype.so public class HelloTypeModuleParser implements ModuleParser<HelloTypeModule> { @Override public HelloTypeModule parse(RedisInputStream in, int version) throws IOException { DefaultRdbModuleParser parser = new DefaultRdbModuleParser(in); int elements = parser.loadUnsigned(version).intValue(); long[] ary = new long[elements]; int i = 0; while (elements-- > 0) { ary[i++] = parser.loadSigned(version); } return new HelloTypeModule(ary); } } public class HelloTypeModule implements Module { private final long[] value; public HelloTypeModule(long[] value) { this.value = value; } public long[] getValue() { return value; } }public class HelloTypeParser implements CommandParser<HelloTypeCommand> { @Override public HelloTypeCommand parse(Object[] command) { String key = new String((byte[]) command[1], Constants.UTF_8); long value = Long.parseLong(new String((byte[]) command[2], Constants.UTF_8)); return new HelloTypeCommand(key, value); } } @CommandSpec(command = "hellotype.insert") public class HelloTypeCommand extends AbstractCommand { private final String key; private final long value; public long getValue() { return value; } public String getKey() { return key; } public HelloTypeCommand(String key, long value) { this.key = key; this.value = value; } }public static void main(String[] args) throws IOException { Replicator replicator = new RedisReplicator("redis://127.0.0.1:6379"); replicator.addCommandParser(CommandName.name("hellotype.insert"), new HelloTypeParser()); replicator.addModuleParser("hellotype", 0, new HelloTypeModuleParser()); replicator.addEventListener(new EventListener() { @Override public void onEvent(Replicator replicator, Event event) { if (event instanceof KeyStringValueModule) { System.out.println(event); } if (event instanceof HelloTypeCommand) { System.out.println(event); } } }); replicator.open(); }See ModuleExtensionExample.java
Since Redis 5.0, a new data structure called STREAM has been added. Redis-replicator parses STREAM data as follows:
Replicator r = new RedisReplicator("redis://127.0.0.1:6379"); r.addEventListener(new EventListener() { @Override public void onEvent(Replicator replicator, Event event) { if (event instanceof KeyStringValueStream) { KeyStringValueStream kv = (KeyStringValueStream)event; // Key String key = kv.getKey(); // Stream Stream stream = kv.getValueAsStream(); // Last stream ID stream.getLastId(); // Entries NavigableMap<Stream.ID, Stream.Entry> entries = stream.getEntries(); // Optional: Groups for (Stream.Group group : stream.getGroups()) { // Group PEL (Pending Entries List) NavigableMap<Stream.ID, Stream.Nack> gpel = group.getPendingEntries(); // Consumers for (Stream.Consumer consumer : group.getConsumers()) { // Consumer PEL (Pending Entries List) NavigableMap<Stream.ID, Stream.Nack> cpel = consumer.getPendingEntries(); } } } } }); r.open();- Write a
YourRdbVisitorthat extendsRdbVisitor. - Register your
RdbVisitorwith theReplicatorusing thesetRdbVisitormethod.
Before version 2.4.0, RedisReplicator was constructed as follows:
Replicator replicator = new RedisReplicator("127.0.0.1", 6379, Configuration.defaultSetting()); Replicator replicator = new RedisReplicator(new File("/path/to/dump.rdb"), FileType.RDB, Configuration.defaultSetting()); Replicator replicator = new RedisReplicator(new File("/path/to/appendonly.aof"), FileType.AOF, Configuration.defaultSetting()); Replicator replicator = new RedisReplicator(new File("/path/to/appendonly.aof"), FileType.MIXED, Configuration.defaultSetting());Since version 2.4.0, we have introduced the Redis URI concept to simplify the RedisReplicator construction process:
Replicator replicator = new RedisReplicator("redis://127.0.0.1:6379"); Replicator replicator = new RedisReplicator("redis:///path/to/dump.rdb"); Replicator replicator = new RedisReplicator("redis:///path/to/appendonly.aof"); // Configuration setting example Replicator replicator = new RedisReplicator("redis://127.0.0.1:6379?authPassword=foobared&readTimeout=10000&ssl=yes"); Replicator replicator = new RedisReplicator("redis:///path/to/dump.rdb?rateLimit=1000000"); Replicator replicator = new RedisReplicator("rediss://user:pass@127.0.0.1:6379?rateLimit=1000000");| Command | Command | Command | Command | Command | Command |
|---|---|---|---|---|---|
| PING | APPEND | SET | SETEX | MSET | DEL |
| SADD | HMSET | HSET | LSET | EXPIRE | EXPIREAT |
| GETSET | HSETNX | MSETNX | PSETEX | SETNX | SETRANGE |
| HDEL | UNLINK | SREM | LPOP | LPUSH | LPUSHX |
| LREM | RPOP | RPUSH | RPUSHX | ZREM | ZINTERSTORE |
| INCR | DECR | INCRBY | PERSIST | SELECT | FLUSHALL |
| FLUSHDB | HINCRBY | ZINCRBY | MOVE | SMOVE | BRPOPLPUSH |
| PFCOUNT | PFMERGE | SDIFFSTORE | RENAMENX | PEXPIREAT | SINTERSTORE |
| ZADD | BITFIELD | SUNIONSTORE | RESTORE | LINSERT | ZREMRANGEBYLEX |
| GEOADD | PEXPIRE | ZUNIONSTORE | EVAL | SCRIPT | ZREMRANGEBYRANK |
| PUBLISH | BITOP | SETBIT | SWAPDB | PFADD | ZREMRANGEBYSCORE |
| RENAME | MULTI | EXEC | LTRIM | RPOPLPUSH | SORT |
| EVALSHA | ZPOPMAX | ZPOPMIN | XACK | XADD | XCLAIM |
| XDEL | XGROUP | XTRIM | XSETID | COPY | LMOVE |
| BLMOVE | ZDIFFSTORE | GEOSEARCHSTORE | FUNCTION | SPUBLISH | HPERSIST |
| HSETEX | HPEXPIREAT | XACKDEL | XDELEX | MSETEX |
When event consumption is too slow and the backlog of events exceeds the Redis backlog limit, Redis will actively disconnect from the slave. When Redis-replicator reconnects, it will perform a full synchronization. To avoid this situation, you need to set the parameter client-output-buffer-limit slave 0 0 0.
For more details, please refer to redis.conf.
client-output-buffer-limit slave 0 0 0 WARNING: This setting may cause the Redis server to run out of memory in some cases.
- Set the log level to debug.
- If you are using Log4j2, add a logger as shown below:
<Logger name="com.moilioncircle" level="debug"> <AppenderRef ref="YourAppender"/> </Logger>Configuration.defaultSetting().setVerbose(true); // As a Redis URI parameter "redis://127.0.0.1:6379?verbose=yes"System.setProperty("javax.net.ssl.keyStore", "/path/to/keystore"); System.setProperty("javax.net.ssl.keyStorePassword", "password"); System.setProperty("javax.net.ssl.keyStoreType", "your_type"); System.setProperty("javax.net.ssl.trustStore", "/path/to/truststore"); System.setProperty("javax.net.ssl.trustStorePassword", "password"); System.setProperty("javax.net.ssl.trustStoreType", "your_type"); Configuration.defaultSetting().setSsl(true); // Optional settings Configuration.defaultSetting().setSslSocketFactory(sslSocketFactory); Configuration.defaultSetting().setSslParameters(sslParameters); Configuration.defaultSetting().setHostnameVerifier(hostnameVerifier); // As a Redis URI parameter "redis://127.0.0.1:6379?ssl=yes" "rediss://127.0.0.1:6379"If you prefer not to use System.setProperty, you can configure it programmatically as follows:
RedisSslContextFactory factory = new RedisSslContextFactory(); factory.setKeyStorePath("/path/to/redis/tests/tls/redis.p12"); factory.setKeyStoreType("pkcs12"); factory.setKeyStorePassword("password"); factory.setTrustStorePath("/path/to/redis/tests/tls/redis.p12"); factory.setTrustStoreType("pkcs12"); factory.setTrustStorePassword("password"); SslConfiguration ssl = SslConfiguration.defaultSetting().setSslContextFactory(factory); Replicator replicator = new RedisReplicator("rediss://127.0.0.1:6379", ssl);Configuration.defaultSetting().setAuthUser("default"); Configuration.defaultSetting().setAuthPassword("foobared"); // As a Redis URI parameter "redis://127.0.0.1:6379?authPassword=foobared&authUser=default" "redis://default:foobared@127.0.0.1:6379"Adjust the Redis server settings as follows:
repl-backlog-size repl-backlog-ttl repl-ping-slave-period The repl-ping-slave-period MUST be less than Configuration.getReadTimeout(). The default Configuration.getReadTimeout() is 60 seconds.
Replicator replicator = new RedisReplicator("redis://127.0.0.1:6379"); final long start = System.currentTimeMillis(); final AtomicInteger acc = new AtomicInteger(0); replicator.addEventListener(new EventListener() { @Override public void onEvent(Replicator replicator, Event event) { if(event instanceof PreRdbSyncEvent) { System.out.println("pre rdb sync"); } else if(event instanceof PostRdbSyncEvent) { long end = System.currentTimeMillis(); System.out.println("time elapsed:" + (end - start)); System.out.println("rdb event count:" + acc.get()); } else { acc.incrementAndGet(); } } }); replicator.open();As mentioned in 4.4. Write Your Own RDB Parser, this tool has a built-in Iterable Rdb Parser to handle huge key-value pairs. For more details, please refer to: [1] HugeKVFileExample.java [2] HugeKVSocketExample.java
cd /path/to/redis ./utils/gen-test-certs.sh cd tests/tls openssl pkcs12 -export -CAfile ca.crt -in redis.crt -inkey redis.key -out redis.p12 cd /path/to/redis ./src/redis-server --tls-port 6379 --port 0 --tls-cert-file ./tests/tls/redis.crt \ --tls-key-file ./tests/tls/redis.key --tls-ca-cert-file ./tests/tls/ca.crt \ --tls-replication yes --bind 0.0.0.0 --protected-mode noSystem.setProperty("javax.net.ssl.keyStore", "/path/to/redis/tests/tls/redis.p12"); System.setProperty("javax.net.ssl.keyStorePassword", "password"); System.setProperty("javax.net.ssl.keyStoreType", "pkcs12"); System.setProperty("javax.net.ssl.trustStore", "/path/to/redis/tests/tls/redis.p12"); System.setProperty("javax.net.ssl.trustStorePassword", "password"); System.setProperty("javax.net.ssl.trustStoreType", "pkcs12"); Replicator replicator = new RedisReplicator("rediss://127.0.0.1:6379");Replicator replicator = new RedisReplicator("redis://user:pass@127.0.0.1:6379");Since Redis 7.0, FUNCTION is supported, and its structure is stored in the RDB file. You can use the following method to parse a FUNCTION.
Replicator replicator = new RedisReplicator("redis://127.0.0.1:6379"); replicator.addEventListener(new EventListener() { @Override public void onEvent(Replicator replicator, Event event) { if (event instanceof Function) { Function function = (Function) event; function.getCode(); // Your code goes here } } }); replicator.open();You can also parse a FUNCTION into serialized data and use FUNCTION RESTORE to restore it to a target Redis instance.
Replicator replicator = new RedisReplicator("redis://127.0.0.1:6379"); replicator.setRdbVisitor(new DumpRdbVisitor(replicator)); replicator.addEventListener(new EventListener() { @Override public void onEvent(Replicator replicator, Event event) { if (event instanceof DumpFunction) { DumpFunction function = (DumpFunction) event; byte[] serialized = function.getSerialized(); // Your code goes here // You can use FUNCTION RESTORE to restore the serialized data to a target Redis instance } } }); replicator.open();Since Redis 7.4, TTL HASH is supported, and its structure is stored in the RDB file. You can use the following method to parse a TTL HASH.
Replicator replicator = new RedisReplicator("redis://127.0.0.1:6379"); replicator.addEventListener(new EventListener() { @Override public void onEvent(Replicator replicator, Event event) { if (event instanceof KeyStringValueTTLHash) { KeyStringValueTTLHash skv = (KeyStringValueTTLHash) event; // Key byte[] key = skv.getKey(); // TTL Hash Map<byte[], TTLValue> ttlHash = skv.getValue(); for (Map.Entry<byte[], TTLValue> entry : ttlHash.entrySet()) { System.out.println("field: " + Strings.toString(entry.getKey())); System.out.println("value: " + Strings.toString(entry.getValue().getValue())); System.out.println("field ttl: " + entry.getValue().getExpires()); } } } }); replicator.open();- Leon Chen
- Adrian Yao
- Trydofor
- Argun
- Sean Pan
- René Kerner
- Maplestoria
- Special thanks to Kevin Zheng
Commercial support for redis-replicator is available. The following services are currently offered:
- Onsite consulting: $10,000 per day
- Onsite training: $10,000 per day
You may also contact Baoyi Chen directly at chen.bao.yi@gmail.com.
- rdb.c
- Redis RDB File Format
- Redis Protocol specification
- Redis Replication
- Redis-replicator Design and Implementation
January 27, 2023, was a sad day as I lost my mother, 宁文君. She was always encouraging and supportive of my work on this tool. Every time a company started using it, she would get as excited as a child and motivate me to continue. Without her, I could not have maintained this tool for so many years. Even though I haven't achieved much, she was always proud of me. R.I.P, and may God bless her.
YourKit is kindly supporting this open source project with its full-featured Java Profiler. YourKit, LLC is the creator of innovative and intelligent tools for profiling Java and .NET applications. Take a look at YourKit's leading software products: YourKit Java Profiler and YourKit .NET Profiler.
IntelliJ IDEA is a Java Integrated Development Environment (IDE) for developing computer software. It is developed by JetBrains (formerly known as IntelliJ), and is available as an Apache 2 Licensed community edition, and in a proprietary commercial edition. Both can be used for commercial development.
Redisson, a Redis-based In-Memory Data Grid for Java, offers distributed objects and services (BitSet, Set, Multimap, SortedSet, Map, List, Queue, BlockingQueue, Deque, BlockingDeque, Semaphore, Lock, AtomicLong, CountDownLatch, Publish / Subscribe, Bloom filter, Remote service, Spring cache, Executor service, Live Object service, Scheduler service) backed by a Redis server. Redisson provides a more convenient and easier way to work with Redis. Redisson objects provide a separation of concerns, allowing you to focus on data modeling and application logic.
