3

I have a VERY simple Ubuntu 16 x64 VM setup on my ESXi host which acts as a fileserver. It has NFS/SMB and MDADM installed. It is fully updated.

Twice in the last week it has hung with an error about “rcu_sched detected stalls on CPUs and something about not enough jiffies.

I took a screen shot this time, but its so bad that ESXi can't kill the VM and after reboot is causing a rebuild of my MDADM Array. I worry this will cause unnecessary damage to my HDD's and wonder what the problem could be? The VM is given plenty of extra resources w/ 1vCPU and 4vThreads w/ 6GB Ram.

Any ideas? The VM is back running now, so I can debug any information requested. I'm thinking about just moving to a RHEL based Distro, but I'd like to figure out the problem vs rebuilding on a different Linux OS.

enter image description here

PS: I am the main user and as far as I remember there were no intensive R/W operations going on at the time.

Improve this question
3
  • 1
    If you're not committed to ESXI (and it sounds like you're not, and this is a relatively new build), I'd try converting to a real linux distro and using LVM or preferably ZFS to provide the storage for your VMs using KVM. If you want something with a nice management GUI over the top, proxmox seems to have a lot of happy users (haven't tried it myself, I do my VMs on ZFS ZVOLs, wit KVM & libvirt - virsh and virt-manager are good enough for me, and i don't mind the rough edges from doing it all "manually") Commented Mar 14, 2018 at 0:42
  • This question has been receiving a lot of views, Im interested tohear others follow-up on there situation and root cause and solution. Commented Jul 18, 2018 at 20:20
  • @cas I have moved to KVM due to it's mainstream device temp monitoring, but I hate LVM/ZFS etc. I use simple RAID either via Raid Card of Software Raid. I tried ProxMox but was unable to get my BluRay drive to correctly passthrough so I am on Fedora. Been testing GUI's, lots of half good ones out there, but nothing like the HTML5 Web Interface for ESXi! I did get Virt-Manager working on Windows with Cygwin so that works, but not much on the Web front ATM. Commented Nov 20, 2018 at 20:45

1 Answer 1

Reset to default
2

TLDR; About a Week Later I lost a CPU Core likely due to overheating/badly placed heatsink/fan.

  • If you are using ESXi I would fire up another OS and check on Temp's and/or consider re-seating your CPU heat sink.

  • This post has got a lot of views, and when I had the issue, google had very little information for me. Please share your experiences in comments or answers!

TimeLine:

  • I get error about Jiffies
  • Have to use power button to reboot
  • MDADM Array Rebuild --> Successful.
  • I get another error next day
  • Reboot/Rebuild Successful.
  • Another error!
  • Rebuild VM w/ new OS
  • Stable for about a week
  • Single Core in CPU dies!

Further research into ESXi showed me that ESXi does NOT gather device Temps without some sort of advanced hardware addition that I didn't have (Possibly because I wasn't use a computer from the "Hardware Compatibility List". (https://communities.vmware.com/thread/547244). If it had, ESXi could have likely throttled my CPU. I now use KVM which checks all my device Temps via normal methods and reacts accordingly. Not just that but my RW speed has greatly increased as my Hypervisor is now also my FileServer vs before I had to passthrough the disks to a FileServer VM since ESXi doesn't support SMB/NFS/MDADM etc. (I'm talking about a 2 or 3 fold increase in RW speeds now that my clients talk directly to the Hypervisor/FileServer).

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.