Strange Prod issue: java lettuce Redis client is trying to connect with strange host IP intermittently.

[anandmehta-fareye]

We connect with AWS Redis elastic cache using lettuce java client but it's throwing below error intermittently, surprisingly we are not passing this IP anywhere in the code

Below is the code to create the connect bean

@Configuration
public class RedisConfig {

    @Value("${spring.redis.host}")
    private String redisService;

    @Value("${spring.redis.port}")
    private Integer redisPort;

    /**
     * Redis connection factory lettuce connection factory.
     *
     * @return the lettuce connection factory
     */
    @Bean
    public RedisConnectionFactory redisConnectionFactory() {
        LettuceClientConfiguration clientConfig = LettuceClientConfiguration.builder()
                .readFrom(ReadFrom.REPLICA_PREFERRED)
                .build();
        RedisStandaloneConfiguration serverConfig = new RedisStandaloneConfiguration(redisService, redisPort);
        return new LettuceConnectionFactory(serverConfig, clientConfig);
    }
}

if i am sending single request then its working fine, i am getting below error whenever i am trying to send multiple concurrent request.

[ioEventLoop-6-1] io.lettuce.core.AbstractRedisClient      : Connecting to Redis at 10.X.X.193/<unresolved>:6379: 10.X.X.193/<unresolved>:6379

io.netty.channel.ConnectTimeoutException: **connection timed out: /10.X.X.193:6379**
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:261) ~[netty-transport-4.1.100.Final.jar!/:4.1.100.Final]
	at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) ~[netty-common-4.1.100.Final.jar!/:4.1.100.Final]
	at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:153) ~[netty-common-4.1.100.Final.jar!/:4.1.100.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[netty-common-4.1.100.Final.jar!/:4.1.100.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[netty-common-4.1.100.Final.jar!/:4.1.100.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[netty-common-4.1.100.Final.jar!/:4.1.100.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569) ~[netty-transport-4.1.100.Final.jar!/:4.1.100.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.100.Final.jar!/:4.1.100.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.100.Final.jar!/:4.1.100.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.100.Final.jar!/:4.1.100.Final]
	at java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]
2025-02-03 17:40:44.182	
2025-02-03T12:10:44.182Z DEBUG 6 --- [ioEventLoop-6-2] io.lettuce.core.AbstractRedisClient      : Connecting to Redis at **REDIS HOST**/<unresolved>:6379: **Success**
2025-02-03 17:40:44.178	
2025-02-03T12:10:44.178Z DEBUG 6 --- [ioEventLoop-6-1] io.lettuce.core.AbstractRedisClient      : Connecting to Redis at 10.X.X.193/<unresolved>:6379

It's a prod issue, any help would be highly appreciated

[tishun]

Hey @anandmehta-fareye ,

Have you checked the AWS ElasiCache Best Practises guide and specifically the section Java DNS cache TTL?
Did you configure the JVM accordingly?

[anandmehta-fareye]

@tishun I have checked the guide and also reached out to AWS regarding this. I learned that sometimes, when there is a parallel connection, the Lettuce Redis client tries to connect using the IP of the ENI(Elastic Network Interfaces), which causes connection time out. However, most of the time, it connects to the specified Elastic host, allowing us to establish a successful connection.

It's escalated to aws team , but still we haven't find the solution.Do you have any insights on this?

[tishun]

@tishun I have checked the guide and also reached out to AWS regarding this. I learned that sometimes, when there is a parallel connection

Do you know what "parallel" means in this case?

, the Lettuce Redis client tries to connect using the IP of the ENI(Elastic Network Interfaces), which causes connection time out. However, most of the time, it connects to the specified Elastic host, allowing us to establish a successful connection.

It's escalated to aws team , but still we haven't find the solution.Do you have any insights on this?

I am afraid not. Seems like a very specific issue, related to AWS infrastructure, I think the AWS team would better help here.

If anything could be done from the side of the driver please let me know.