5

I am running a Java process with Xmx2000m, the host OS is linux centos, jdk 1.6 update 22. Lately I have been experiencing a weird behavior in the process, it becomes totally unresponsive with no apparent reason, no logs, no errors, nothing.. I am using jconsole to monitor the processor, heap and Perm memory are not full, threads and loaded classes are not leaking.. Explanation anyone?

Improve this question
3
  • 1
    Impossible to answer. You need to identify where in your code the hang is occurring (Lots of logging may help here). Then maybe someone can help. Commented Dec 7, 2010 at 12:29
  • I thought so. Just asking if anybody is familiar with debugging similar situations. Commented Dec 7, 2010 at 12:55
  • Just recently, I had the same problem on several of my server applications. They had been running for years without problems. Then, one by one, stopped working. The first occurrence I can reconstruct by logs happened Nov 22 in 2020, the last around early December 2020. Didn't realize that all instances had that problem, found out one by one on different occasions. Could not even SIGTERM them, had to SIGKILL them. Scripts restarted them, any app that was manually restarted once has been running fine ever since... Commented Dec 15, 2020 at 11:09

7 Answers 7

Reset to default
7

I doubt anyone can give you an explanation since there are lots of possible reasons and not nearly enough information. However, I suggest that you jstack the process once it's hung to figure out what the threads are doing, and take it from there. It sounds like a deadlock or thrashing of some sort.

Improve this answer
Sign up to request clarification or add additional context in comments.

4 Comments

+1 because I'm an old chimp and yet I didn't know about the word "trashing". Being an old chimp I've witnessed (and sometimes I've been guilty of ;) that behavior, but I didn't know it had a name :)
And that name is thrashing ... not trashing.
If you are experience virtual memory thrashing, it will show up using top or (my favorite) vmstat 5.
Thrashing is directly related to the OS, the JVM has nothing to do with it.
4

Do a thread dump. If you have access to the foreground process on Linux, use ctrl-\. Or use jstack to dump stack remotely. Or you can actually poke it through JMX via jconsole at MBeans/java.lang/Threading/Operations/dumpAllThreads.

Without knowing more about your app, it's hard to speculate about the cause. Presumably your threads are either a) blocked or b) exited. If they are blocked, they could be waiting for I/O on a database or other operation OR they could be waiting on a lock or monitor (deadlocked). If a deadlock exists, the thread dump will tell you which threads are deadlocked, which lock, and (in Java 6) annotate the stack with where locks have been taken. You can also search for deadlocks with the JMX method, available through jconsole at MBeans/java.lang/Threading/Operations/find[Monitor]DeadlockedThreads().

Or your threads may have received unhandled exceptions and exited. Check out Thread's uncaughtExceptionHandlers or (better) use Executors in java.util.concurrent.

And finally, the other classic source of pauses in Java is GC. Run with -verbose:gc and other GC flags to see if it's doing a full GC collection. You can also turn this on dynamically in jconsole by flipping the flag at MBeans/java.lang/Memory/Attributes/Verbose.

Improve this answer

Comments

2

Agree with aix, but would like to add a couple of recommendataions. 1. check your system. Run top to see whether the system itself is healthy, CPU is not 100% and memory is available. If not, fix this. 2. application may freeze as a result of dead lock. Check this.

Improve this answer

Comments

1

Ok here are some updates I wanted to share:

There is an incompatability between NTPL (Linux’s new thread library) and the Java 1.6+ JVM. A random bug causes the JVM to hang and eat up 100% CPU.

To work around it set LD_ASSUME_KERNEL=2.4.1 before running the JVM, export LD_ASSUME_KERMEL=2.4.1 . This disables NTPL: problem solved!

But for compatibility reasons, I'm still looking for a solution that uses NTPL.

Improve this answer

Comments

0

Threads can be traced using jvisualvm and jconsole, and deadlocks can be avoided too. Note that there are several network services each with separate thread pools, and they all become unreachable.

Improve this answer

Comments

0

Check the jvisualvm of the process right before the crash. http://www.jadyounan.com/wp-content/uploads/2010/12/process.png

Improve this answer

2 Comments

adding comments/edits as answers is not the correct way of using SO. Better use the edit button and add this to your original question.
if you want to keep a track of that info, put it as a comment saying something like "This ain't working for me because my process ain't crashing" or something like that, but do not make it an answer.
0

Could you elaborate more on what you are doing ? 2000 for memory is rather a lot.

Improve this answer

4 Comments

It is a memory-intensive statistics service. data structures have to be scanned on daily basis.
Is it possible to increase RAM memory ? Maybe that may help a bit. Look through logs and find out parts of codes that are not smoothly running or causing the problems. Break down into small modules to test if possible.
jvm is 64-bit it could be increased. Although i dont expect it has something to do. check this jadyounan.com/wp-content/uploads/2010/12/process.png
Just to clarify it: the process that is freezing is a remote process. This is my desktop here. Nothing fancy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.