Skip to content

fix memory leak issue on thread local in sharedLock - PR for main branch#3640

Open
kandogu wants to merge 1 commit intoredis:mainfrom
kandogu:fix/thread-local-memory-leak
Open

fix memory leak issue on thread local in sharedLock - PR for main branch#3640
kandogu wants to merge 1 commit intoredis:mainfrom
kandogu:fix/thread-local-memory-leak

Conversation

@kandogu
Copy link

@kandogu kandogu commented Feb 3, 2026

fix: SharedLock ThreadLocal memory leak in connection pooling scenarios #3489

Summary

This PR fixes the ThreadLocal memory leak in SharedLock that causes high CPU usage from ThreadLocalMap.expungeStaleEntries() when using connection pooling (shareNativeConnection=false).

Fixes #3489

Problem

The original implementation created a per-instance ThreadLocal<Integer> for each SharedLock:

private final ThreadLocal<Integer> threadWriters = ThreadLocal.withInitial(() -> 0);

With connection pooling:

  • Each connection has its own DefaultEndpointSharedLock
  • Each thread accessing any SharedLock creates an entry in its ThreadLocalMap
  • When connections are recycled, SharedLock instances are garbage collected
  • But stale entries remain in each thread's ThreadLocalMap
  • Over time, thousands of stale entries accumulate
  • expungeStaleEntries() becomes expensive, causing CPU spikes

Solution

Replace per-instance ThreadLocal with a static ThreadLocal<WeakHashMap<SharedLock, Integer>>:

private static final ThreadLocal<WeakHashMap<SharedLock, Integer>> THREAD_WRITERS = 
    ThreadLocal.withInitial(WeakHashMap::new);

Key benefits:

  1. Only ONE ThreadLocal entry per thread (not per SharedLock)
  2. WeakHashMap keys allow SharedLock to be garbage collected, automatically removing entries
  3. Explicit cleanup when writer count reaches zero for immediate memory reclamation
  4. Preserves reentrant lock semantics from fix:deadlock when reentrant exclusive lock #2905 #2879 #2961

Memory Comparison

Scenario Original Fixed
200 threads × 100 connections 20,000 ThreadLocalMap entries 200 entries
After connection churn Stale entries accumulate Automatic cleanup

Changes

  • SharedLock.java: Refactored to use static ThreadLocal<WeakHashMap<SharedLock, Integer>>
  • SharedLockUnitTests.java: Added tests for new behavior

Checklist

  • You have read the contribution guidelines.
  • You have created a feature request first to discuss your contribution intent. Please reference the feature request ticket number in the pull request. → SharedLock ThreadLocal threadWriters memory leak may cause CPU usage to reach 100% #3489
  • You applied code formatting rules using the mvn formatter:format target. Don't submit any formatting related changes.
  • You submit test cases (unit or integration tests) that back your changes.

@kandogu
Copy link
Author

kandogu commented Feb 3, 2026

@a-TODO-rov Hello,
I opened a PR for main branch after your comments on previous PR

@kandogu kandogu changed the title fix memory leak issue on thread local in sharedLock #3633 - PR for main branch fix memory leak issue on thread local in sharedLock - PR for main branch Feb 3, 2026
@a-TODO-rov
Copy link
Contributor

I think this might work.
I ran my local reproducer:
Both have ~120 active connections at the time of the snapshot

Metric Without Fix (7.2.x) With Fix
ThreadLocalMap$Entry count 18,615 635

@tishun what are your thoughts on this ?

@tishun
Copy link
Collaborator

tishun commented Feb 6, 2026

Awesome job @kandogu; and thanks for testing this @a-TODO-rov.

IMHO we can merge this.

@a-TODO-rov a-TODO-rov force-pushed the fix/thread-local-memory-leak branch from c495257 to 2f3ce1e Compare February 6, 2026 12:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SharedLock ThreadLocal threadWriters memory leak may cause CPU usage to reach 100%

3 participants