Growing number of NioEventLoopGroup threads

[oklahomer]

Currently I am working on Redis Cluster interaction, so I tried out a couple of client modules: Jedis and lettuce. So far I prefer lettuce over Jedis in terms of handiness and robustness.

I, however, bumped into a strange behavior when I closed connection(s) via RedisAdvancedClusterConnection#close. I understand it creates new thread(s) for new connection handling, but it does not seem to be removed when the connection is closed. Below is the code fragment my colleague gave to me.

@Slf4j
public class Main {
    public static void main(String[] args) throws Exception {
        List<RedisURI> redisUris = IntStream.rangeClosed(7000, 7005)
                .mapToObj(i -> "redis://localhost:" + i)
                .map(RedisURI::create)
                .collect(Collectors.toList());
        RedisClusterClient redisClusterClient = new RedisClusterClient(redisUris);

        //noinspection InfiniteLoopStatement
        while (true) {
            RedisAdvancedClusterConnection<String, String> connection = redisClusterClient.connectCluster();
            String key = Long.toString(System.currentTimeMillis(), 10);
            connection.set(key, "foo");
            assert connection.get(key).equals("foo");
            connection.close();
            Thread.sleep(Duration.ofSeconds(5).toMillis());
        }
    }
}

Try trace the count of threads and see it grows.
3.3.1.Final and 3.3.2.Final both reproduce this issue.
I tried to see if they were reused after reaching some kind of threshold, but no luck.

$ jps | grep -F AppMain | cut -f1 -d' ' | xargs jstack | grep '^"nio' | wc -l
8
$ jps | grep -F AppMain | cut -f1 -d' ' | xargs jstack | grep '^"nio' | wc -l
10
$ jps | grep -F AppMain | cut -f1 -d' ' | xargs jstack | grep '^"nio' | wc -l
12

I also found a similar issue on the old repo, wg/lettuce#34 .
Is it going to be fixed? Or is there any solution or workaround?

[mp911de]

Hi @oklahomer
Threads in lettuce are bound to a particular RedisClient/RedisClusterClient instance (in 3.4 and 4.1 to ClientResources, but that's a new thing) and not to connections. Threads are not removed when you close a connection but when a client instance is shutdown. Please test your case with calling RedisClusterClient.shutdown() at the end.

The EventLoop threads are an infrastructure resource that is expensive to create and to remove so they are kept alive until the client resource is closed. A thread is not bound to a particular connection, but a connection binds to one EventLoop thread which means that one thread can be used for more than one connection. To answer your question: This behavior is by design to facilitate scalability.

wg/lettuce#34 is different because it's about daemon threads vs. regular threads. 3.4 and 4.1 are using daemon threads.

[oklahomer]

Thanks @mp911de for quick reply.
That should help.

[oklahomer]

@mp911de Just to make sure, let me ask a bit more.

With the code fragment above, we only have one client instance at all time. In the while loop, it calls connectCluster and try establish connection with the same setting every time. Based on my understanding about how client tries to reuse EventLoop threads, it should reuse the created EventLoop threads to minimize the resource cost. At least that's what I thought when I took a glance at AbstractRedisClient#getEventLoopGroup; reuse event loop group for the same connection point. However the number grows with both RedisClient and RedisClusterClient whether close is called or not.
Am I right?

I think it's neat that client instance manages those threads' life-cycle, but got a bit confused to see how it's actually working.

[mp911de]

The answer is yes and no.

The underlying facility that provides Threads to the client is the NioEventLoopGroup (for TCP connections). That's a Thread group with a max-capacity that is currently set to Runtime.getRuntime().availableProcessors() * 4. It's not a pool; Threads can be requested by different "users" such as Channels or custom events but are not exclusively used by that "user". The number of Threads cannot grow beyond Runtime.getRuntime().availableProcessors() * 4 for one RedisClient or one RedisClusterClient instance. Multiple instances of RedisClient create their own NioEventLoopGroups and can cause unlimited threads.

The thing of multithreaded EventLoopGroups is that the client has no (or only very little) control over the Thread on which a particular thing runs on. There is an EventExecutorChooser but it does not work like a pool when a Thread is "free" that it is reused (see https://github.com/netty/netty/blob/43ebbc3fa065155fa67732b0cbd7c12843b0f3f7/common/src/main/java/io/netty/util/concurrent/MultithreadEventExecutorGroup.java#L234), but the Threads are selected in a particular order. This means that you probably end up with up to Runtime.getRuntime().availableProcessors() * 4 Threads after using the client a while.

There are two things you can do to limit Threads:

1. Wait for lettuce 3.4/4.1 with an implementation you can control the max number of Threads programmatically
2. You can set the system property io.netty.eventLoopThreads to specify the number of Threads, but you could also run into issues if you use netty for other things in your JVM instance

Come back to me if you need further support.

[oklahomer]

Thanks again for detailed comment. I checked with the code fragment above and the number actually stopped at 32, where Runtime.getRuntime().availableProcessors() * 4 = 32. I'd steer away from modifying system property and wait for next version.

Thanks.