Jetty Connection Closed When Application Attempts to Write - intermittent issue under load #3684
Comments
|
I don't think there is nothing strange in these stack traces. The server is trying to write to the client, but the network connection has been interrupted. You have DEBUG enabled, so it's typical to see that much of stack traces - that does not mean there is something wrong. |
|
EofException is very common in a production environment. This part of the stacktrace tell us the most ... That stacktrace says the server was attempting to commit/close/write/flush a response, but the remote side disconnected before it could complete. |
|
At first we were also suspecting client mis-behavior. We are continuing to investigate what causes the jetty connection to close. Any thoughts on how we can track jetty connection management (other than enabling jetty debug which we have tried). Thanks. |
|
I've seen in the past such major Jetty upgrades and the new Jetty version was so much faster that the client was overwhelmed trying to cope with the server replying faster than before. I've also seen old clients that fail to parse more up-to-date HTTP responses that newer versions of Jetty now produce. If the broken pipes are generated by the client I would investigate the client first, no matter the server version. |
|
@abennet news? |
|
Is there a way we can change these ERRORS into WARN? |
|
@devilethon consider opening a new issue and describe exactly your problem. Jetty does not log at ERROR level, so you must detail what is your issue in much more details. |
@joakime how can I find this change? Any specific class? |
|
@rhamedy the code between Jetty 8.0 and Jetty 9.4 (a difference of 4 major versions) with regards to IO has changed massively. If you want to verify that theory, just enable ALL (if using Jetty Logging) or TRACE (if using logback, log4j, slf4j, etc) or FINEST (if using java.util.logging) logging for |
Done. Jetty never logs at ERROR level anyway. |
|
This issue has been automatically marked as stale because it has been a full year without activity. It will be closed if no further activity occurs. Thank you for your contributions. |
|
This issue has been closed due to it having no activity. |
Since upgrading jetty (7.x to 9.14) our server has been experiencing intermittent jetty EofExceptions under load with https traffic.
The stack traces are only seen as part of GET operations, and our analysis shows that our server is attempting to write the response to the jetty response, and the exception is triggered at this point because the jetty ResponseWriter already has an exception when the application attempts the first write (see ResponseWriter:isOpen, _ioException is non-null).
We have two concerning stack traces we are trying to understand/analyze.
Here is the first stack trace we see from failing to write the response of get requests:
Additionally and possibly more concerning is that we were able to catch a stack trace that originates in the reserved thread management (see attached). It seems to be a known issue in jetty (#3652)
Details about our jetty configuration:
The jetty configuration is setup with 10 minimum threads and 255 maximum threads, with idleTimeout of 30000. reservedThreadsLifeSpan is set to default. We have separate http and https connectors (with the same thread sizing).
Any direction is much appreciated.
The text was updated successfully, but these errors were encountered: