1

I think, there should be some type of memory corruption. The problems happen mainly on processes allocating large amount of ram (especially browsers, and cc1plus). JVMs with intensive ram usage die nearly on the start, Eclipse can't even create its first window. The problem is nearly always segfault caused by nullpointer-dereference.

What I experienced:

  • Windows and OSX running the same processes in ESXi, doesn't show the phenomenon.
  • But: Linux with different kernels (between 3.6 and 4.1) shows. Linux with different distributions (I've tried debian, ubuntu and opensuse) also shows.
  • 32-bit guests aren't problematic even if they are Linux.
  • Although PCI passthrough is activated, without it (and even with turned off IOMMU in the BIOS) the problem is coming.
  • Upgrading esxi 5.5 to 6 didn't help anything.

Other infos:

  • I've runned memtest a night long on the host machine, and it didn't find any problem.
  • Using 64-bit OS on the host (thus, without virtualization), the problem isn't coming.
  • The crashes are happening mainly after the process allocated a large amount of memory.
  • The problem happens nearly always in user space (but its reason could be that kernelspace allocates large amount of ram only seldom).
  • Turning off acceleration, or switching the guest os to "other linux" or "other 64-bit OS" didn't help.
  • Turning off the SMP on the guest machine (i.e. giving them only a single CPU core) didn't help. Playing with other CPU settings (various multi socket / multicore settings, limiting the guest to various CPU cores) didn't had any effect.
  • Limiting the Guest OS memory also didn't help, although Guests running only a few processes with low memory consumption work seamslessly.
  • Just after poweron the problem doesn't happen. It is coming only after there is some large memory allocation-free cycle happened (for example, only after some minutes of clicking in a firefox).
  • Changing the virtual SCSI controller type from LSI to vmware paravirtual, or turning off swap didn't help.

I think, something corrupts (zeroes) the guest OSes memory on large brk() calls.

Did anybody meet this problem? What other could be done to make the hunting more efficient?

Improve this question
6
  • This sounds like a bug in vmware. Have you tried contacting their support people? Commented Aug 8, 2015 at 10:39
  • @WouterVerhelst Thanks. I didn't contact them yet, I am using a free test version now. I think if there is a such major problem in esxi, it should be known already. Commented Aug 8, 2015 at 10:43
  • Rather interesting post. How did it go? Did you ever find out? Were you overprovisioning RAM? Commented Jun 6, 2016 at 12:10
  • 1
    @RuiFRibeiro Yes. It was a damaged memory module, which memtest couldn't detect. :-) And the problem happened even with Windows and OSX, but later. Commented Jun 6, 2016 at 12:20
  • answer to yourself and close the post. Commented Jun 6, 2016 at 12:21

1 Answer 1

Reset to default
1

Problem solved.

It was a damaged memory module, which memtest couldn't detect. :-)

And the problem happened even with Windows and OSX, but later.

Never forget: memtest can't find always every ram problem! The reason: memtest cycles through the memory and makes complex operations on it. But leaving memory areas a long time unused, that it doesn't do.

DIMM memory is essentially a large array of condensators. These condensators slowly lose their charge. Thus, DIMM requires periodic refreshes in the memory, i.e. its content need to be periodically read out and written back.

If it doesn't happen, the memory loses its content. This memory refresh happens by hardware, mostly in the memory controller of the motherboard/cpu.

They have also another feature: reading out the content of the memory clears the condensator (they are very small, because it is the only way to integrate as many as possible in a memory chip). Thus, after every memory read its content must be written back.

Problems related to this refresh mechanism are undetectable by the memtest.

Although it would be detectable if the memtest would work on a method, that it checkes a half of a the memory and lets the other half in peace, and switches between them only after a long delay. But it wasn't implemented in the memtest until now.

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.