[JENKINS-76095] Kubernetes client leaks Thread Pool Executors

[jenkins-infra-bot]

Kubernetes plugin may leak thread pool executor overtime like the following:

"-109914906-pool-56-thread-10" id=6547 state=TIMED_WAITING
    - waiting on <0x7a4dbe2c> (a java.util.concurrent.SynchronousQueue$Transferer)
    - locked <0x7a4dbe2c> (a java.util.concurrent.SynchronousQueue$Transferer)
    at java.base@​21.0.8/jdk.internal.misc.Unsafe.park(Native Method)
    at java.base@​21.0.8/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410)
    at java.base@​21.0.8/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452)
    at java.base@​21.0.8/java.util.concurrent.SynchronousQueue$Transferer.xferLifo(SynchronousQueue.java:194)
    at java.base@​21.0.8/java.util.concurrent.SynchronousQueue.xfer(SynchronousQueue.java:235)
    at java.base@​21.0.8/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:338)
    at java.base@​21.0.8/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069)
    at java.base@​21.0.8/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
    at java.base@​21.0.8/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
    at java.base@​21.0.8/java.lang.Thread.runWith(Thread.java:1596)
    at java.base@​21.0.8/java.lang.Thread.run(Thread.java:1583)

The rate at which this happens is unclear. The impact depends on the activity and the number of Kubernetes Cloud configured. And seems related to the expiration of Cached clients that does not cleanup resources. More details below.

How to Reproduce

• Spin Up Jenkins
• Configure a Kubernetes Cloud with a fake credentials ( credentials that doe snot work and will make the kubernetes client fail)
• Create one template in that Kubernetes Cloud
• Create a Pipeline that request that Pod template
• Build the pipeline

--> The pipeline will be stuck in queue and Kubernetes plugin will keep on try to spin up and agent and fail. About every 10 minutes, your will notice a leaked thread.

Workaround

If the environment allow it, the cache expiry time can be increased to one hour org.csanchez.jenkins.plugins.kubernetes.clients.cacheExpiration=3600 or more. Whether this is suitable depends on the environment.

Notes

With a heapdump, it was evident that the ThreadFactory that generated the thread is the one

• https://github.com/fabric8io/kubernetes-client/blob/v7.4.0/kubernetes-client-api/src/main/java/io/fabric8/kubernetes/client/utils/Utils.java#L451-L463

The first term in the prefixed name is empty because it originates from an anonymous class in BaseClient:

• https://github.com/fabric8io/kubernetes-client/blob/v7.4.0/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/impl/BaseClient.java#L64-L79

Noticing that with a simple 1 Kubernetes Cloud scenario, one thread is leaked every 10 minutes, I narrowed this down to be linked to the management of cached clients in the KubernetesClientProvider that expires Kubernetes clients after 10 minutes without explicitly closing them

• https://github.com/jenkinsci/kubernetes-plugin/blob/4371.vb_33b_086d54a_1/src/main/java/org/csanchez/jenkins/plugins/kubernetes/KubernetesClientProvider.java#L30-L48

---

Originally reported by allan_burdajewicz, imported from: Kubernetes client leaks Thread Pool Executors

• assignee: allan_burdajewicz
• status: Open
• priority: Major
• component(s): kubernetes-plugin
• resolution: Unresolved
• votes: 0
• watchers: 4
• imported: 2025-12-02

Raw content of original issue

Kubernetes plugin may leak thread pool executor overtime like the following:


"-109914906-pool-56-thread-10" id=6547 state=TIMED_WAITING
    - waiting on <0x7a4dbe2c> (a java.util.concurrent.SynchronousQueue$Transferer)
    - locked <0x7a4dbe2c> (a java.util.concurrent.SynchronousQueue$Transferer)
    at java.base@21.0.8/jdk.internal.misc.Unsafe.park(Native Method)
    at java.base@21.0.8/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410)
    at java.base@21.0.8/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452)
    at java.base@21.0.8/java.util.concurrent.SynchronousQueue$Transferer.xferLifo(SynchronousQueue.java:194)
    at java.base@21.0.8/java.util.concurrent.SynchronousQueue.xfer(SynchronousQueue.java:235)
    at java.base@21.0.8/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:338)
    at java.base@21.0.8/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069)
    at java.base@21.0.8/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
    at java.base@21.0.8/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
    at java.base@21.0.8/java.lang.Thread.runWith(Thread.java:1596)
    at java.base@21.0.8/java.lang.Thread.run(Thread.java:1583)


The rate at which this happens is unclear. The impact depends on the activity and the number of Kubernetes Cloud configured. And seems related to the expiration of Cached clients that does not cleanup resources. More details below.
How to Reproduce

	
Spin Up Jenkins
	
Configure a Kubernetes Cloud with a fake credentials ( credentials that doe snot work and will make the kubernetes client fail)
	
Create one template in that Kubernetes Cloud
	
Create a Pipeline that request that Pod template
	
Build the pipeline

--> The pipeline will be stuck in queue and Kubernetes plugin will keep on try to spin up and agent and fail. About every 10 minutes, your will notice a leaked thread.
Workaround
If the environment allow it, the cache expiry time can be increased to one hour org.csanchez.jenkins.plugins.kubernetes.clients.cacheExpiration=3600 or more. Whether this is suitable depends on the environment.
Notes
With a heapdump, it was evident that the ThreadFactory that generated the thread is the one 

	
https://github.com/fabric8io/kubernetes-client/blob/v7.4.0/kubernetes-client-api/src/main/java/io/fabric8/kubernetes/client/utils/Utils.java#L451-L463

The first term in the prefixed name is empty because it originates from an anonymous class in BaseClient:

	
https://github.com/fabric8io/kubernetes-client/blob/v7.4.0/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/impl/BaseClient.java#L64-L79

Noticing that with a simple 1 Kubernetes Cloud scenario, one thread is leaked every 10 minutes, I narrowed this down to be linked to the management of cached clients in the KubernetesClientProvider that expires Kubernetes clients after 10 minutes without explicitly closing them

	
https://github.com/jenkinsci/kubernetes-plugin/blob/4371.vb_33b_086d54a_1/src/main/java/org/csanchez/jenkins/plugins/kubernetes/KubernetesClientProvider.java#L30-L48

• environment: kubernetes:4371.vb_33b_086d54a_1

2 attachments

• Screenshot 2025-09-11 at 13.18.53.png

  

• image-2025-11-03-13-16-57-214.png

  

[jenkins-infra-bot]

• Remote links associated with this issue:

  • CloudBees Internal Issue
  • kubernetes-plugin #​1747

[jenkins-infra-bot]

allan_burdajewicz:

• Original comment link
• Raw content of original comment:

  It is unclear if the leak can happen for functional Kubernetes Client. I thought that maybe it happens when an expired client is still use while already removed from the cache. And the reason we can reproduce it easily with a client that fails is because the Kubernetes client is gonna be used again and again to try to provision a node and that improve the likelihood of having an expired client still in use.
  
  FWIW we cannot just close the client on expiry because it would interrupt active connections. We could either schedule its closure. Or maybe use a java.lang.ref.Cleaner to take care of it when garbage collected.
  

It is unclear if the leak can happen for functional Kubernetes Client. I thought that maybe it happens when an expired client is still use while already removed from the cache. And the reason we can reproduce it easily with a client that fails is because the Kubernetes client is gonna be used again and again to try to provision a node and that improve the likelihood of having an expired client still in use.

FWIW we cannot just close the client on expiry because it would interrupt active connections. We could either schedule its closure. Or maybe use a java.lang.ref.Cleaner to take care of it when garbage collected.

[jenkins-infra-bot]

allan_burdajewicz:

• Original comment link
• Raw content of original comment:

  I also created Fix BaseClient thread executor name fabric8io/kubernetes-client#7284 (comment) by the way.
  

I also created fabric8io/kubernetes-client#7284 (comment) by the way.

[jenkins-infra-bot]

jhmed:

• Original comment link
• Raw content of original comment:

  After I updated to the newest plugin version (4383.vb_76fc9d3c4dc), I started experiencing a memory leak in the kubernetes plugin. The heapdump analysis seams to indicate `jdk.internal.ref.CleanerImpl` objects that pile up. See heap analysis screenshot below.
  
  Could this have something to do with this ticket? Is the change from this ticket already rolled out? Since the comment above talked about using `java.lang.ref.Cleaner`, maybe that implementation caused a new leak now?
  
  
  

After I updated to the newest plugin version (4383.vb_76fc9d3c4dc), I started experiencing a memory leak in the kubernetes plugin. The heapdump analysis seams to indicate `jdk.internal.ref.CleanerImpl` objects that pile up. See heap analysis screenshot below.

Could this have something to do with this ticket? Is the change from this ticket already rolled out? Since the comment above talked about using `java.lang.ref.Cleaner`, maybe that implementation caused a new leak now?

[jenkins-infra-bot]

jglick:

• Original comment link
• Raw content of original comment:

  Revert "[JENKINS-76095] Schedule cleanup of expired clients" #1766 filed to revert the change pending investigation.
  

#1766 filed to revert the change pending investigation.

[jenkins-infra-bot]

jgarciacloudbees:

• Original comment link
• Raw content of original comment:

  Changed back to Open, as the current changes were reverted, and we need to further investigate this issue.
  

Changed back to Open, as the current changes were reverted, and we need to further investigate this issue.

[jenkins-infra-bot]

allan_burdajewicz:

• Original comment link
• Raw content of original comment:

  It could be due to the Cleaner action that refer to the reigstered object. Per the javadoc:
  
  > Note that the cleaning action must not refer to the object being registered. If so, the object will not become phantom reachable and the cleaning action will not be invoked automatically.
  
  In the PR, we register the wrapper Client and we also refer to it in the cleaning action (we close it). The Kubernetes Client is closed (associated OKHttp threads are gone) but the ref is retained in memory. We should register the Client but then refer to the KubernetesClient in the cleaning method...
  

It could be due to the Cleaner action that refer to the reigstered object. Per the javadoc:

> Note that the cleaning action must not refer to the object being registered. If so, the object will not become phantom reachable and the cleaning action will not be invoked automatically.

In the PR, we register the wrapper Client and we also refer to it in the cleaning action (we close it). The Kubernetes Client is closed (associated OKHttp threads are gone) but the ref is retained in memory. We should register the Client but then refer to the KubernetesClient in the cleaning method...