websphere application server goes down automatically /frequently

HI All,

Please help me
websphere application server goes down automatically /frequently
Server name & Version : IBM-WebSphere-Application-Server-6.1.0.25
OS Name :Linux 2008 x86_64 x86_64 x86_64 GNU/Linux
we have verified in SystemOut.log file it is showing : [11/22/14 21:46:01:558 IST]
00000053 AdminHelper A ADMN1020I: An attempt is made to stop the server1 server. (User ID = defaultWIMFileBasedRealm/estore)

It is encounter evey 4 hours or 5 hours max day it is down around 4 to 5 times day it is production servers issues ,



Please find the below Logs message (SystemOut.log file)
[11/22/14 21:39:45:176 IST] 0001aedd WebApp E [Servlet Error]-[InjectionServlet]: java.lang.IllegalStateException: Response already committed.
at com.ibm.ws.webcontainer.webapp.WebAppDispatcherContext.sendError(WebAppDispatcherContext.java:518)
at com.ibm.ws.webcontainer.srt.SRTServletResponse.sendError(SRTServletResponse.java:1002)
at com.iwinner.estore.webapp.servlet.InjectionServlet.handleException(InjectionServlet.java:300)
at com.iwinner.estore.webapp.servlet.InjectionServlet.serveAds(InjectionServlet.java:144)
at com.iwinner.estore.webapp.servlet.InjectionServlet.doPost(InjectionServlet.java:61)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:763)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1146)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1087)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:145)
at com.iwinner.estore.webapp.gui.CharsetFilter.doFilter(CharsetFilter.java:47)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:190)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:130)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:87)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:837)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:680)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:588)
at com.ibm.ws.wswebcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:481)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:90)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:748)
at com.ibm.ws.wswebcontainer.WebContainer.handleRequest(WebContainer.java:1466)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:122)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:458)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewInformation(HttpInboundLink.java:387)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:267)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:136)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:196)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:751)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:881)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1497)


[11/22/14 21:39:45:176 IST] 0001aedd SRTServletRes W WARNING: Cannot set status. Response already committed.

[11/22/14 21:39:45:177 IST] 0001aedd SRTServletRes W WARNING: Cannot set header. Response already committed.

[11/22/14 21:45:51:122 IST] 00000053 AdminHelper A ADMN1020I: An attempt is made to stop the server1 server. (User ID = defaultWIMFileBasedRealm/iwinner)

[11/22/14 21:45:51:196 IST] 0000000f TCPChannel I TCPC0002I: TCP Channel TCP_4 has stopped listening on host * (IPv6) port 9443.

[11/22/14 21:45:51:196 IST] 0000000f TCPChannel I TCPC0002I: TCP Channel TCP_1 has stopped listening on host * (IPv6) port 9060.

[11/22/14 21:45:51:197 IST] 0000000f TCPChannel I TCPC0002I: TCP Channel TCP_2 has stopped listening on host * (IPv6) port 8080.

[11/22/14 21:45:51:198 IST] 0000000f TCPChannel I TCPC0002I: TCP Channel TCP_3 has stopped listening on host * (IPv6) port 9043.

[11/22/14 21:46:01:558 IST] 00000053 AdminHelper A ADMN1020I: An attempt is made to stop t
************ Start Display Current Environment ************
WebSphere Platform 6.1 [ND 6.1.0.25 cf250922.06] running with process name estoreNode01Cell\estoreNode01\server1 and process id 405
Detailed IFix information: No IFixes applied to this build
Host Operating System is Linux, version 2.6.18-128.el5
Java version = 1.5.0, Java Compiler = j9jit23, Java VM name = IBM J9 VM
was.install.root = /iwinner/estore/WebSphere/AppServer
user.install.root = /iwinner/estore/WebSphere/AppServer/profiles/AppSrv01

Thanks
Anji

It is difficult to guess what may happen to tour websphere installation. Versione 6 isn't supporter anymore if i am noto wrong. You are using a Network deployment version, right? How manu nodes are installed ?

Hi Claude Moore

Thanks for your help we are using only one Node in this server.It is continuosly down the server every 2 days once.



Thanks
Anji

Using ND with a single nide doesn't male much dense... You should use at least two nodes with it. Anyway, try and have a look at deployment manager. Do you find any hints?

HI Claude,

we observed some errors in SystemOut.log file but these errors coming last 6 months ,but last 20day on words started application going down frequently

Please explain what is cause ,application servers going frequently it is production issue?



[11/22/14 21:53:23:464 IST] 00000022 ServletWrappe I SRVE0242I: [iwinner_war] [/iwinner] [Location Based Ads Servlet]: Initialization successful.

[11/22/14 21:54:17:244 IST] 00000052 JDBCException W org.hibernate.util.JDBCExceptionReporter logExceptions SQL Error: 1, SQLState: 23000

[11/22/14 21:54:17:246 IST] 00000052 JDBCException E org.hibernate.util.JDBCExceptionReporter logExceptions ORA-00001: unique constraint (winnerUSER.SYS_C0022260) violated


[11/22/14 21:54:17:249 IST] 00000052 JDBCException W org.hibernate.util.JDBCExceptionReporter logExceptions SQL Error: 1, SQLState: 23000

[11/22/14 21:54:17:251 IST] 00000052 JDBCException E org.hibernate.util.JDBCExceptionReporter logExceptions ORA-00001: unique constraint (winnerUSER.SYS_C0022260) violated


[11/22/14 21:54:17:252 IST] 00000052 AbstractFlush E org.hibernate.event.def.AbstractFlushingEventListener performExecutions Could not synchronize database state with session
org.hibernate.exception.ConstraintViolationException: Could not execute JDBC batch update
at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:71)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:43)
at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:253)
at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:237)
at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:141)
at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:298)
at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:27)
at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1000)
at org.hibernate.impl.SessionImpl.managedFlush(SessionImpl.java:338)
at org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:106)
at com.iwinner.winner.common.hibernate.HibernateUtilities.closeSession(HibernateUtilities.java:61)
at com.iwinner.winner.common.hibernate.HibernateUtilities.simplePersist(HibernateUtilities.java:119)
at com.iwinner.iwinner.hibernate.dal.HibernateSubscribersDal.persistSubscriber(HibernateSubscribersDal.java:37)
at sun.reflect.GeneratedMethodAccessor67.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:618)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:77)
at com.iwinner.iwinner.hibernate.HibernateAspectWrapper.runHibernateMethod(HibernateAspectWrapper.java:38)
at com.iwinner.iwinner.hibernate.HibernateAspectWrapper.wrapHibernateAction(HibernateAspectWrapper.java:21)
at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:618)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:627)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:616)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:64)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy38.persistSubscriber(Unknown Source)
at com.iwinner.iwinner.webapp.logic.NotifyTransactionManager.persistSubscriber(NotifyTransactionManager.java:149)
at com.iwinner.iwinner.webapp.servlet.InjectionServlet.autoCommitView(InjectionServlet.java:213)
at com.iwinner.iwinner.webapp.servlet.InjectionServlet.serveAds(InjectionServlet.java:121)
at com.iwinner.iwinner.webapp.servlet.InjectionServlet.doPost(InjectionServlet.java:61)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:763)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1146)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1087)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:145)
at com.iwinner.iwinner.webapp.gui.CharsetFilter.doFilter(CharsetFilter.java:47)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:190)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:130)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:87)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:837)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:680)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:588)
at com.ibm.ws.wswebcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:481)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:90)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:748)
at com.ibm.ws.wswebcontainer.WebContainer.handleRequest(WebContainer.java:1466)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:122)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:458)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewInformation(HttpInboundLink.java:387)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:267)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture$1.run(AsyncChannelFuture.java:205)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1497)
Caused by: java.sql.BatchUpdateException: ORA-00001: unique constraint (winnerUSER.SYS_C0022260) violated

at oracle.jdbc.driver.DatabaseError.throwBatchUpdateException(DatabaseError.java:674)
at oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePreparedStatement.java:9394)
at oracle.jdbc.driver.OracleStatementWrapper.executeBatch(OracleStatementWrapper.java:211)
at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeBatch(NewProxyPreparedStatement.java:1723)
at org.hibernate.jdbc.BatchingBatcher.doExecuteBatch(BatchingBatcher.java:48)
at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:246)
... 60 more
---- Begin backtrace for Nested Throwables
java.sql.SQLException: ORA-00001: unique constraint (winnerUSER.SYS_C0022260) violated

at oracle.jdbc.driver.SQLStateMapping.newSQLException(SQLStateMapping.java:74)
at oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:131)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:204)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:455)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:413)
at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:1034)

[11/22/14 21:54:17:288 IST] 00000052 ServletWrappe E SRVE0068E: Uncaught exception thrown in one of the service methods of the servlet: InjectionServlet. Exception thrown : java.lang.IllegalStateException: Response already committed.
at com.ibm.ws.webcontainer.webapp.WebAppDispatcherContext.sendError(WebAppDispatcherContext.java:518)
at com.ibm.ws.webcontainer.srt.SRTServletResponse.sendError(SRTServletResponse.java:1002)
at com.iwinner.estore.webapp.servlet.InjectionServlet.handleException(InjectionServlet.java:300)
at com.iwinner.estore.webapp.servlet.InjectionServlet.serveAds(InjectionServlet.java:144)
at com.iwinner.estore.webapp.servlet.InjectionServlet.doPost(InjectionServlet.java:61)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:763)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1146)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1087)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:145)
at com.iwinner.estore.webapp.gui.CharsetFilter.doFilter(CharsetFilter.java:47)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:190)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:130)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:87)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:837)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:680)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:588)
at com.ibm.ws.wswebcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:481)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:90)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:748)
at com.ibm.ws.wswebcontainer.WebContainer.handleRequest(WebContainer.java:1466)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:122)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:458)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewInformation(HttpInboundLink.java:387)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:267)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture$1.run(AsyncChannelFuture.java:205)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1497)


[11/22/14 21:54:17:296 IST] 00000052 ServiceLogger I com.ibm.ws.ffdc.IncidentStreamImpl initialize FFDC0009I: FFDC opened incident stream file /iwinner/iwinner/WebSphere/AppServer/profiles/AppSrv01/logs/ffdc/server1_00000052_14.11.22_21.54.17_0.txt

[11/22/14 21:54:17:316 IST] 00000052 ServiceLogger I com.ibm.ws.ffdc.IncidentStreamImpl resetIncidentStream FFDC0010I: FFDC closed incident stream file /iwinner/iwinner/WebSphere/AppServer/profiles/AppSrv01/logs/ffdc/server1_00000052_14.11.22_21.54.17_0.txt

[11/22/14 21:54:17:319 IST] 00000052 ServiceLogger I com.ibm.ws.ffdc.IncidentStreamImpl open FFDC0009I: FFDC opened incident stream file /iwinner/iwinner/WebSphere/AppServer/profiles/AppSrv01/logs/ffdc/server1_00000052_14.11.22_21.54.17_1.txt



Thanks
Anji

Of course I cannot know what may be the cause for your problem. If 20 days before, application server was running without issues, the only thing I can say is that something must have changed recentely. Can't you figure out what ? I can only see that something goes wrong with Hibernate (looking at your stacktrace), but it seems a normal violation of any constraint on an Oracle DBMS server.
I cannot say what happens under the hood just because I have no idea of which apps your appserver is hosting.
From this stacktrace:

[11/22/14 21:45:51:122 IST] 00000053 AdminHelper A ADMN1020I: An attempt is made to stop the server1 server. (User ID = defaultWIMFileBasedRealm/iwinner)

[11/22/14 21:45:51:196 IST] 0000000f TCPChannel I TCPC0002I: TCP Channel TCP_4 has stopped listening on host * (IPv6) port 9443.

[11/22/14 21:45:51:196 IST] 0000000f TCPChannel I TCPC0002I: TCP Channel TCP_1 has stopped listening on host * (IPv6) port 9060.

[11/22/14 21:45:51:197 IST] 0000000f TCPChannel I TCPC0002I: TCP Channel TCP_2 has stopped listening on host * (IPv6) port 8080.

[11/22/14 21:45:51:198 IST] 0000000f TCPChannel I TCPC0002I: TCP Channel TCP_3 has stopped listening on host * (IPv6) port 9043.

[11/22/14 21:46:01:558 IST] 00000053 AdminHelper A ADMN1020I: An attempt is made to stop t

i presumed that your appserver did not do harakiri (i.e, it was not brutely killed or coredumped), it looks like if it were shutdown clearly. My advice is to clean up every logs, keep on monitoring it and verify if a shutdown is issued. By
the way: a shutdown on a single node may be performed by Deployment manager, not only by stopping WAS service by hand.

HI Claude Moore,

Thanks for your help ,When application stopped ,we killed all process ,

As you said previous one any configuration changes ,Please explain briefly , is this configuration related to WAS Servers or Application related or DB related ,Please explain briefly, this issue occurs around last 20 days .

If this changes related to WAS Servers please explain where we need changes in WAS Servers levels .


Thanks
Anji

Anji,
Sorry but I don't have answer to your question. You are the one that can check what changed in your environment, not me. I can only help you guessing what may be wrong. You stated that your was installation has been going down since twenty days ago, now you wrote that is the application is stopped and you need to restart appserver. Maybe it's only your application that goes down; if it's so, what happens if you try to restart it ? What is the appserver status before restarting it ? It's frozen? How is the jvm at memory level ? Try and enable verbose garbage collection. Try and verify which is the coon pool status.
Try, in simple words, to detect the root of the problem. I'm afraid not being able to help you further.