java应用挂死故障排查

提示：扫一扫查出行
【扫一扫了解最新限行尾号】
复制提示

现象：

java开发的web应用无法访问

排查：

1．从resin/log/watchdog-manager.log的日志里可以看出来，jvm的内存满，无法创建新进程

java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:574)
at com.caucho.util.ThreadPool$ThreadLauncher.startConnection(ThreadPool.java:812)
at com.caucho.util.ThreadPool$ThreadLauncher.run(ThreadPool.java:851)
at java.lang.Thread.run(Thread.java:595)

2．用jstat –gcutil pid可以看出jvm的OLD堆内存区域已经100%，与上述判断符合

S0 S1 E O P YGC YGCT FGC FGCT GCT 0.00 0.00 99.19 100.00 20.72 577 62.517 322 2953.102 3015.619

3．用jstack pid查看jvm的内部的执行线程情况，可以看到存在大量的下述线程，阻塞

Thread 15810: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
- java.util.concurrent.locks.LockSupport.park() @bci=5, line=118 (Interpreted frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=41, line=1767 (Interpreted frame)
- com.ctsi.nmp.flux.ecfchart.ECFluxChartService.writeChart(com.ctsi.nmp.flux.ecfchart.ECFluxChartType, com.ctsi.nmp.flux.ecfchart.ECFluxType, java.lang.String, java.util.Date, java.util.Date, java.util.List, boolean, java.io.OutputStream) @bci=295, line=189 (Compiled frame) - com.ctsi.nmp.flux.ecfchart.ECFluxChartClient.executeChart() @bci=1051, line=203 (Compiled frame)
- sun.reflect.GeneratedMethodAccessor49.invoke(java.lang.Object, java.lang.Object[]) @bci=36 (Compiled frame)
- sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=25 (Compiled frame)
- com.ctsi.nmp.flux.ecfchart.ECFluxChartClient.execute() @bci=94, line=230 (Interpreted frame)
- _jsp._nos._flux._flux._chart._ECFluxChart__jsp._jspService(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletR
esponse) @bci=130, line=39 (Compiled frame)
- com.caucho.jsp.JavaPage.service(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=9, line=61 (Interpreted frame)
- com.caucho.jsp.Page.pageservice(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=252, line=578 (Compiled frame)
- com.caucho.server.dispatch.PageFilterChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=262, line=195 (Compiled frame)
- com.zhongying.util.EncodingFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain
) @bci=39, line=80 (Compiled frame)
- com.caucho.server.webapp.WebAppFilterChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=140, line=187 (Compiled frame)
- com.caucho.server.dispatch.ServletInvocation.service(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=97, line=266 (Compiled frame)
- com.caucho.server.hmux.HmuxRequest.handleRequest() @bci=478, line=435 (Compiled frame)
- com.caucho.server.port.TcpConnection.run() @bci=375, line=678 (Compiled frame)
- com.caucho.util.ThreadPool$Item.runTasks() @bci=246, line=721 (Compiled frame)
- com.caucho.util.ThreadPool$Item.run() @bci=106, line=643 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=595 (Interpreted frame)