5

My linux server reports high disk space usage for device /dev/sda4 mounted on / as shown below: 

[root@stormtrooper03 /]# df -h Filesystem Size Used Avail Use% Mounted on /dev/sda4 126G 114G 5.5G 96% / tmpfs 32G 0 32G 0% /dev/shm /dev/sda2 239M 118M 109M 53% /boot /dev/sda1 150M 264K 150M 1% /boot/efi /dev/sda5 63G 52M 60G 1% /home /dev/sda6 63G 54M 60G 1% /tmp /dev/sda7 63G 2.6G 58G 5% /usr /dev/sda3 539G 11G 501G 3% /var /dev/sdb1 917G 857G 51G 95% /data/1 /dev/sdc1 917G 861G 47G 95% /data/2 /dev/sdd1 917G 282G 627G 31% /data/3 /dev/sde1 917G 861G 47G 95% /data/4 /dev/sdf1 917G 858G 50G 95% /data/5

However, I know there is much less than this amount used by running du -sh /* as shown below. I need to clean files setting on this device but I can't find them!!

[root@stormtrooper03 /]# du -sh /* 6.7M /bin 116M /boot 3.7T /data 356K /dev 30M /etc 172K /home 638M /lib 20M /lib64 16K /lost+found 4.0K /media 4.0K /mnt 3.8G /opt 0 /proc 42M /root 9.2M /sbin 4.0K /selinux 4.0K /srv 0 /sys 2.1M /tmp 2.5G /usr 11G /var

Update I have found reason for this: a new disk was mounted on /data/3 which had 110GB of data and became hidden from the filesystem after mounting a disk on it.

The answer to the question why does mount happen over an existing directory explains how to uncover original data using mount --bind but it doesn't explain how to find if your system has such "shadowed" directories if you don't know ahead of time.

Is there a quick way to a list of disks mounted on non-empty directories without checking each one individually by mount --bind?

Improve this question
1
  • /var is on a different device (/dev/sda3) and even if it was on the same device it's only 11G while df reports 114G usage Commented Feb 18, 2017 at 2:21

3 Answers 3

Reset to default
1

I figured out the problem! By tracing actions on this server that are different from other servers I found that at one point the disk mounted on /data/3 stopped working so it was commented out in /etc/fstab. However processes in the system didn't know about this and continued to write to /data/3. which at this time is a directory in /. Then later the bad disk was replaced and the fstab entry was updated so now /data/3 is mounted on the new disk. Somehow the old data was hidden as it seems there is no pointer to it or something.

Once I knew this, I unmounted the disk /data/3 and now I was able to see 110GB of data setting there which I deleted and then remounted the disk and things are back to normal.

I don't know if there is any tool to find such orphan directory so to speak.

Improve this answer
2
  • I have heard of this as "shadowing" a directory. It is often done on purpose when mounting volumes within containers. I am not aware of a program that would have detected this, unless there is a special flag for it in du. Maybe re-word the question in order to attract people that may know a way of discovering the issue sooner, or who know of a method of detecting shadowed directories more readily. Commented Feb 20, 2017 at 19:47
  • Thanks for the term "shadowing". I have updated the question for future readers. Commented Feb 20, 2017 at 20:36
0

[EDIT]: Leaving this answer as a general note, it does not answer the refined version of the question.

KDirStat is a good graphical utility for seeing where large files and directories are on a system. It gives a heatmap and a sorted list.

Improve this answer
4
  • This could not uncover those hiding files either. Commented Feb 18, 2017 at 3:06
  • @MajidAlfifi, did you run it as root and tell it start from /? If I remember correct, by default it only analyzes the user's home directory. Commented Feb 20, 2017 at 19:37
  • 1
    KDirStat will show the mount point as such, and only using the size of 1 inode when the shadowing device is mounted. It will show the real used space if the device is unmounted. Same results, only graphical, as du. Commented Feb 20, 2017 at 21:42
  • How about: du -ak | sort -n Commented Feb 22, 2017 at 19:20
0

If you mount the file system that might have the hidden data elsewhere, then you can go within the location to the relative spot where mounts might hide the data and see if there any any data there.

If sudo ls -A $MOUNTPOINT produces any results, then there is something there at $MOUNTPOINT.

To see what's mounted run the mount command by itself.

Here's some code that you can use to look for hidden data.

#!/bin/bash TMP="/media/root" # temp location to mount the directory that might have data hidden by a mount point sudo mkdir -p "$TMP" sudo mount --bind --read-only -- "/" "$TMP" # if "/" holds the mount point with the hidden by mountpoint data, replace "/" if needed. # replace `$(mount | awk '{print $3}')` below with the directories you wish to test for hidden data for LOCATION in $(mount | awk '{print $3}') ; do if [ -n "$(sudo ls -A "$TMP$LOCATION" 2>/dev/null)" ]; then echo "data on unmounted \"$LOCATION\""; fi; done; sudo umount "$TMP"

sample output:

data on unmounted "/run" data on unmounted "/" data on unmounted "/var"
Improve this answer
3
  • I was hoping there maybe a way to check the whole system without unmounting/mounting every disk one by one. Commented Feb 21, 2017 at 23:24
  • Just saw this after my post, maybe it'll help using --bind. unix.stackexchange.com/questions/267629/… Commented Feb 21, 2017 at 23:27
  • @MajidAlfifi I've changed the code so the directories don't have to be unmounted. Commented Feb 22, 2017 at 4:52

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.