meminfo

Yes, your meminfo reports 9.1 GB AnonPages of 15.5 GB total.

Small amounts of Shmem and file pages. This is consistent with an application doing big private allocations, and indeed you are running some java. Note java does not tend to directly allocate shared memory for JVM purposes. Contrast to databases, which often have shared memory segments, and of course do lots of file I/O.

No, you do not get all of the remainder to use. MemAvailable is the most useful estimate of this, and your is relatively low at 0.7 GB, or 4% of MemTotal. Upstream's notes on what this is:

MemAvailable

An estimate of how much memory is available for starting new applications, without swapping. Calculated from MemFree, SReclaimable, the size of the file LRU lists, and the low watermarks in each zone. The estimate takes into account that the system needs some page cache to function well, and that not all reclaimable slab will be reclaimable, due to items being in use. The impact of those factors will vary from system to system.

So, actually free memory, and a guess at easy to reclaim pages from file cache and kernel objects. Easy stuff. New memory allocations beyond this size may do terribly slow direct reclaim, or anger the oom killer.

Some very difficult to reclaim items from meminfo are SUnreclaim, KernelStack, and PageTables, totaling maybe 0.8 GB. Not unreasonable, I tend to assume a non-trivial sized host needs at least a GB or two for the kernel.

Still leaves 5 GB or so not easily bucketed. I assume it was in use at some point, things changed like processes terminating, and the kernel has yet to finish the work of reclaiming it as free.

Capacity planning estimates

Do some rough, back-of-the-napkin estimates of how much memory this application can use. Based on your experience with it, and approximately how big its working size is.

Java's memory use and the knobs to limit it are well documented. Here a Red Hat developer blog takes a stab at it. Heap is most of it, but there is a bit more: JVM memory = Heap memory+ Metaspace + CodeCache + (ThreadStackSize * Number of Threads) + DirectByteBuffers + Jvm-native

Also note the suggestion to round up. "production heap size should be at least 25% to 30% higher than the tested maximum to allow room for overhead."

Containers

I find it odd that dockerd and other processes are ones to hit oom killer, and apparently not the big java processes. Admittedly, the oom killer is guessing based on arbitrary heuristics.

Containers, via cgroups, can set memory use limits and tuning on different groups. Also enables accurate reporting of memory use per cgroup, a good tool on systemd systems is systemd-cgtop -m

Review all memory related configuration for containers, for docker see resource constraints documentation. (Although at least one thing on that docker page is incorrect: swappiness is not a percentage.)

Reliably not running low of memory host wide may mean setting per container memory limits. If they contain say some java threads, set the maximum to more than the estimate from capacity planning.

Guessing here, but maybe --oom-kill-disable is on a java container. Under memory pressure but the big consumer is protected, oom killer might remove other system processes on the host. I would in some ways prefer it take out the big java thread, as a hint to the administrator JVM memory might need to be compared to available memory.

Overall, this host could use a bit more memory. Could make do with what it has, but would need more capacity planning, a closer look at what's running and what limits their resource use.