Memcached client does not perform DNS lookup even after cluster is replaced #52
Description
We use AWS route53 to create a CNAME record pointing memcached cluster. When the cluster is updated in cloudformation, sometimes the cluster is replaced and we update the new configuration endpoint in route53 record
However, memcached client keeps trying to connect to the old IP address belonging to the cluster that is gone.
Seems like ConfigurationPoller has a retry loop for MAX_RETRY_ATTEMPT (=3). On one of the retry attempts, it seems to be resolving hostname to IP address.
} else { //Reresolve on retry attempt socketAddressToGetConfig = endpointToGetConfig.getInetSocketAddress(true); } But this retry loop only catches OperationTimeoutException and in this case, we noticed an IllegalArgumentException was thrown and hence there are no retries. ConfigurationPoller keeps trying to use the same IP address that was resolved when it first connected to the original memcached cluster
Stacktrace captured is below and it is thrown from line ConfigurationPoller.java:120 and being caught at the outer catck where this line is logged
2025-10-09 02:19:46.593 ERROR net.spy.memcached.ConfigurationPoller: Error encountered in the poller. Current cluster configuration: 1 java.lang.IllegalArgumentException: The specified address does not belong to the cluster at net.spy.memcached.MemcachedConnection.addOperation(MemcachedConnection.java:1550) at net.spy.memcached.MemcachedConnection.enqueueOperation(MemcachedConnection.java:1478) at net.spy.memcached.MemcachedClient.asyncGetConfig(MemcachedClient.java:1999) at net.spy.memcached.MemcachedClient.getConfig(MemcachedClient.java:1949) at net.spy.memcached.ConfigurationPoller.run(ConfigurationPoller.java:120) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:545) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:369) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:310) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) at java.base/java.lang.Thread.run(Thread.java:1474) This requires all application containers to be restarted whenever memcached cluster is replaced.