Kubernetes Pod memory usage does not fall when jvm runs garbage collection

Question

I'm struggling to understand why my Java application is slowly consuming all memory available to the pod causing Kubernetes to mark the pod as out of memory. The JVM (OpenJDK 8) is started with the following arguments:

-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=2

I'm monitoring the memory used by the pod and also the JVM memory and was expecting to see some correlation e.g. after major garbage collection the pod memory used would also fall. However I don't see this. I've attached some graphs below:

Pod memory: Total JVM memory Detailed Breakdown of JVM (sorry for all the colours looking the same...thanks Kibana)

What I'm struggling with is why when there is a significant reduction in heap memory just before 16:00 does the pods memory not also fall?

In general, JVM does not return memory reserved for Java Heap back to OS. However, it can do this depending on GC algorithm. G1 releases heap memory, but only after Full GC. See this post for more information. — apangin
– apangin, Commented Jan 20, 2019 at 21:00

Eytan Avisror · Accepted Answer · 2019-01-20 19:45:11Z

It looks like you are creating a pod with a resource limit of 1GB Memory. You are setting -XX:MaxRAMFraction=2 which means you are allocating 50% of available memory to the JVM which seem to match what you are graphing as Memory Limit.

JVM then reserves around 80% of that which is what you are graphing in Memory Consumed.

When you look at Memory Consumed you will not see internal garbage collection (as in your second graph), because that GC memory is released back to JVM but is still reserved by it.

Is it possible that there is a memory leak in your java application? it is possibly causing more memory to get reserved over time, until the JVM limit (512MB) is met and your pod gets OOM killed.

Realised I never came back to this! Your hunch was correct, there was a memory leak at the root cause! A very delayed thanks and accepted answer!

Collectives™ on Stack Overflow

Kubernetes Pod memory usage does not fall when jvm runs garbage collection

1 Answer 1

1 Comment

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Related