Outages reported, starting from 2024-12-21

[kltm]

On the GO Helpdesk mailing list, we have been receiving frequent reports of outages, starting on 12/21.

These seem to mostly be 500 errors, with screenshots like:

And:

Type Exception Report

Message An exception occurred processing [webservices/go/overrep.jsp] at line [16]

Description The server encountered an unexpected condition that prevented it from fulfilling the request.

Exception

org.apache.jasper.JasperException: An exception occurred processing [webservices/go/overrep.jsp] at line [16]

13:   // Set standard HTTP/1.0 no-cache header.
14:   response.setHeader("Pragma", "no-cache");
15: %>
16: <jsp:forward page="/Overrepresentation" />


Stacktrace:
	org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServletWrapper.java:626)
	org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:500)
	org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:385)
	org.apache.jasper.servlet.JspServlet.service(JspServlet.java:329)
	javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
	org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
	com.ab.scienceSite.filter.IPAddrFilter.doFilter(IPAddrFilter.java:340)

Root Cause

javax.servlet.ServletException: Servlet execution threw an exception
	org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
	org.apache.jasper.runtime.PageContextImpl.forward(PageContextImpl.java:530)
	org.apache.jsp.webservices.go.overrep_jsp._jspService(overrep_jsp.java:134)
	org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
	javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
	org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:477)
	org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:385)
	org.apache.jasper.servlet.JspServlet.service(JspServlet.java:329)
	javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
	org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
	com.ab.scienceSite.filter.IPAddrFilter.doFilter(IPAddrFilter.java:340)

Root Cause

java.lang.NoClassDefFoundError: Could not initialize class com.ab.scienceSite.list.ResultItemizer
	com.ab.scienceSite.listAnalysis.FileUploadClassifications.getClassificationItems(FileUploadClassifications.java:582)
	com.ab.scienceSite.listAnalysis.FileUploadClassifications.mapToClassifications(FileUploadClassifications.java:436)
	com.ab.scienceSite.listAnalysis.FileUploadClassifications.mapUploadToClassifications(FileUploadClassifications.java:179)
	com.ab.scienceSite.listAnalysis.FileUploadClassifications.<init>(FileUploadClassifications.java:109)
	com.ab.scienceSite.listAnalysis.ListMappingLogic.<init>(ListMappingLogic.java:159)
	com.ab.scienceSite.servlet.Overrepresentation.doPost(Overrepresentation.java:399)
	javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
	javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
	org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
	org.apache.jasper.runtime.PageContextImpl.forward(PageContextImpl.java:530)
	org.apache.jsp.webservices.go.overrep_jsp._jspService(overrep_jsp.java:134)
	org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
	javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
	org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:477)
	org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:385)
	org.apache.jasper.servlet.JspServlet.service(JspServlet.java:329)
	javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
	org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
	com.ab.scienceSite.filter.IPAddrFilter.doFilter(IPAddrFilter.java:340)

Note The full stack trace of the root cause is available in the server logs.

Tagging @huaiyumi @mugitty @dustine32

[dustine32]

Thanks for reporting @kltm! I'm also currently scratching my head at this one while searching the server logs for any clues as to what's going on. Probably will need @mugitty to solve this.

[kltm]

@dustine32 Cheers! Just a heads-up: we've gotten six messages on the GO Helpdesk so far. I've pointed them at this issue.

[dustine32]

@kltm This problem seems to have cleared up with a long stop-wait-start of the web server. It seems to have been a "too many DB connections" issue but then it's odd that this hasn't happened more often.

[kltm]

@dustine32 Thank you all for taking a look at this. Maybe just an end-of-the-year rush kind of thing?

[kltm]

Single report 2024-12-24

[dustine32]

Thanks for the alert @kltm! I was also seeing the issue return so I just did a full restart of both web server and DB. It's working now.

[mugitty]

Thanks for reporting @kltm, I am also keeping an eye on it.