4

I am getting continuously outofmemory error for 1 of my tomcat where my application is deployed after given the error tomcat is exit(shutdow).

I took the log file and found this

SEVERE: Error allocating socket processor java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:597) at org.apache.tomcat.util.net.JIoEndpoint$Worker.start(JIoEndpoint.java:513) at org.apache.tomcat.util.net.JIoEndpoint.newWorkerThread(JIoEndpoint.java:744) at org.apache.tomcat.util.net.JIoEndpoint.createWorkerThread(JIoEndpoint.java:723) at org.apache.tomcat.util.net.JIoEndpoint.getWorkerThread(JIoEndpoint.java:757) at org.apache.tomcat.util.net.JIoEndpoint.processSocket(JIoEndpoint.java:789) at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:355) at java.lang.Thread.run(Thread.java:619) 18 Feb, 2015 5:43:30 PM org.apache.tomcat.util.net.JIoEndpoint createWorkerThread INFO: Maximum number of threads (750) created for connector with address null and port 80

I am using this connector settings in the server.xml

<Connector port="80" protocol="HTTP/1.1" connectionTimeout="10000" maxThreads="750" minSpareThreads="50" redirectPort="8443" />

Can anybody suggest me what can i do?

Thread Dump

"http-80-123" daemon prio=6 tid=0x5f5e7400 nid=0xcfc runnable [0x619be000..0x619bf9e8] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at com.microsoft.sqlserver.jdbc.DBComms.receive(Unknown Source) at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(Unknown Source) at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PreparedStatementExecutionRequest.executeStatement(Unknown Source) at com.microsoft.sqlserver.jdbc.CancelableRequest.execute(Unknown Source) at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeRequest(Unknown Source) - locked <0x15c69398> (a com.microsoft.sqlserver.jdbc.TDSWriter)

I have taken 4 thread dump over 20 seconds duration and found this tid alive all the time

Can we find something with this dump?

I got this from thread dump is this indicating the deadlock condition 'BLOCKED' since i am getting this several times 80- 90 with same state 

"http-80-342" daemon prio=6 tid=0x5c0d7c00 nid=0x1d0c waiting for monitor entry [0x6ee0e000..0x6ee0fce8] java.lang.Thread.State: BLOCKED (on object monitor) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:247) at java.sql.DriverManager.getCallerClass(DriverManager.java:477) at java.sql.DriverManager.getConnection(DriverManager.java:576) at java.sql.DriverManager.getConnection(DriverManager.java:185) at t4u.common.DBConnection.getConnectionToDB(DBConnection.java:32) at t4u.functions.CommonFunctions.getProcessID(CommonFunctions.java:1465)
Improve this question

1 Answer 1

Reset to default
3

java.lang.OutOfMemoryError: unable to create new native thread

This means you've hit some limit on the number of threads the JVM can create. This could be an OS imposed limit or it could be due to lack of resources (like memory).

If it's an OS imposed limit, you may be able to raise it, assuming you have that level of access to the system.

If it's memory related, you need to either increase the available memory (add more physical or increase memory on your VM) or lower the memory requirement for each thread. To do the later, you can change the thread stack size with the -Xss argument to the JVM. To find the ideal value, start with it as very low, like 128K or 256K, and see if you get any StackOverflow exceptions. If you do, increase until they go away.

One other possibility here is to use a different Connector implementation. You haven't said which one you're currently using, but the BIO (blocking IO) Connector consumes one thread per request. Depending on what your application is doing, you may see better thread utilization (i.e. less threads required to handle the same amount of requests) if you use the NIO or APR connectors. There's not enough information here for me to say one way or another, but you might want to test it out with your apps and see if the number of threads required drops.

INFO: Maximum number of threads (750) created for connector with address null and port 80

This looks like you might be hitting a limit on your thread pool in Tomcat. Check your configuration in conf/server.xml. If you're using an Executor, look for maxThreads on that element. If not, look for maxThreads on the Connector element.

Improve this answer
Sign up to request clarification or add additional context in comments.

6 Comments

Thanks for you response i have edited the question and i have also mentioned the connector configuration details done in server.xml
That's helpful. You're definitely hitting maxThreads, which you set to 750 on your Connector. It won't help with the OOME, in fact it'll probably make that more likely to occur, but increasing maxThreads would allow you to go past 750 threads in the pool. Again, that won't help unless you have the memory to do so. See my suggestions above for handling that.
Oh, you're also using the BIO connector. NIO or APR might help reduce the number of threads required. How much, depends on your workload. Lots of IO, you'll see more benefit; lots of app processing or CPU intensive tasks, then probably not. Either way, if you have a test environment I'd suggest giving it a shot to see what happens.
so you are suggesting me to change the thread stack size with the -Xss argument to the JVM . But, in my test environment i am not getting this error so i am stuck how to proceed further. Will thread dump will he helpful in this case.
I can't tell you what will work here, I don't know enough about your apps and your environment. My suggestions above are things that you can try and that might help to address the problem. Ultimately you need to investigate further and test to find the solution. You have a dev environment, which is good. Try throwing some load against that environment and see if you can replicate the problem. If that fails, try to make the problem worse by increasing the stack size or lowering the memory in your dev environment, that might make it easier to replicate. Then you can try the solutions above
|

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.