SQSSessionCallbackScheduler race condition at close leave threads in waiting state for ever

[wigtor]

I have found a race condition when you create a SQSSession and inmediatly close it (For example when I only send a message I don't need any thread for listen messages).
In a production system after 2 days sending around 10000 messages, I have more than 100 threads in waiting state in line 102 of SQSSessionCallbackScheduler class (I see it using jstack command)
At construct a SQSSession, it construct an object SQSSessionCallbackScheduler and execute in a ExecutorService object tha make a thread. So, I have 2 threads: main thread and SQSSessionCallbackScheduler thread, the execution order is:

• SQSSessionCallbackScheduler thread check "closed" variable and it is false (line 95 to line 97), context switch occurs
  before the synchronized section in line 98.
• main thread call close() method, change "closed" variable to true
• main thread enter to synchronized section
• main thread call notify() and it is never received by the QSSessionCallbackScheduler thread
• main thread exit of synchronized section, context switch occurs
• SQSSessionCallbackScheduler thread enter to synchronized section (line 98)
• SQSSessionCallbackScheduler thread check callbackEntry == null (line 100)
• SQSSessionCallbackScheduler thread call wait() (line 102)

I propose check "closed" variable too into the synchronized section (after line 98)
I attach an image of code explaining it.

[vghero]

That looks similar to what I saw during an AWS SQS outage a couple of days ago, where the DNS lookup of the SQS endpoint were failing with UnknownHostException. The number of threads was rising very quickly until OOM errors occured after 15 minutes - reaching 30k of those threads. Since the DNS lookup fails quickly, I guess it has the same issue like explained here. I will take a deeper look.

This also looks similar: #47

[tomhunte]

Official update to fix this issue has been released today. Please let us know if this continues.

[ziyanli-amazon]

Please see the (latest release https://github.com/awslabs/amazon-sqs-java-messaging-lib/releases/tag/1.0.9)