Spring DMLC - SQSSessionCallbackScheduler too many zombie threads

[yeshodhan]

We've been facing high CPU utilization after putting in SQS JMS based implementation on production.

The root cause turned out to SQS connectivity issues and the use of CacheLevel: CACHE_CONSUMER and a Fixed BackOff. Turns out CACHE_CONSUMER is too aggressive and even with an ExponentialBackOff, it will try to retry quite aggressively. After changing the CacheLevel to CACHE_CONNECTION, the CPU utilization is stable in our test environments.

We are facing another issue though. To simulate SQS connectivity issues, we kill the network connection and monitor using VisualVM. The no. of threads created for SQSSessionCallbackScheduler are quite high and they never exit.

See screenshot below:

After resuming network connectivity, the threads don't seem to go down and new threads are created after this point.

We are using Spring DMLC with the below configuration:

ConcurrentConsumers: 10
MaxConcurrentConsumers: 25
CacheLevel: CACHE_CONNECTION (1)
AcknowledgeMode: CLIENT
BackOff: Exponential ( InitialInterval: 5000, Multiplier: 1.5)
IdleConsumerLimit: 1
IdleTaskExecutionLimit: 5
ReceiveTimeout: 1000

Any help?

[yeshodhan]

I've created #48 to solve this issue.