OS = CentOS 7.3
If I try to run the 'df' command on this server with an option (e.g. -h -l) it hangs and leaves a zombie process. I cannot Ctrl+z out back to a prompt. If I run 'df' against specific mount points that are found in FSTAB the command works successfully (e.g. df /home).
How do I go about troubleshooting this?
This is the symptom of a system call locked on one of your local filesystem:
a local disk who is dying and cause any read or write to loop on retries.
You might be able to sort which of these is your culprit by looking at the system log within /var/log if the root or /var filesystem isn't the origin of your df locked.
A reboot resolved the problem. Whether the issue returns or not is yet to be seen. A complex configuration script ran that configured docker containers and made various IPTables changes was executed and no reboot had ever taken place. I will dig a bit deeper and perform and RCA if the issue returns.
I did notice an entry in the dmesg log - Thanks @Bob for the suggestion.
[252869.529789] INFO: task df:14958 blocked for more than 120 seconds. [252869.529838] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
straceorltraceto see where the process is hanging up. Also, is it really a zombie, i.e., it's unkillable from another window?dmesgandcat /etc/mtab?