7

Suddenly all the available disk space on / has disappeared.

If I make room in the disk (by deleting ~50GB of stuff, for example), after a few minutes I am back to 0 available disk space (according to df).

Clearly, some process is eating up disk space at a rapid rate, but I can't figure out what it is.

One thing is certain, though: whatever it is, it must be creating many small files, because there are no files bigger than 10GB on the disk, and all the ones bigger than 1GB are much older than today.

How can I find what's eating up disk space?

FWIW, only df sees the problem, not du.

For example, below I show several "snapshots" from du and df taken 60s. apart. (I did this after I had made some room in the disk.) Notice how du's output remains steady (at 495G), but df shows a steadily shrinking amount of available space. (I've followed the recommendation given here. IOW, /mnt/root is pointing to /.)

# while true; do du -sh /mnt/root && df -h /mnt/root; sleep 60; done 495G /mnt/root Filesystem Size Used Avail Use% Mounted on /dev/sdb1 880G 824G 12G 99% /mnt/root 495G /mnt/root Filesystem Size Used Avail Use% Mounted on /dev/sdb1 880G 825G 11G 99% /mnt/root 495G /mnt/root Filesystem Size Used Avail Use% Mounted on /dev/sdb1 880G 827G 8.9G 99% /mnt/root 495G /mnt/root Filesystem Size Used Avail Use% Mounted on /dev/sdb1 880G 827G 8.1G 100% /mnt/root 495G /mnt/root Filesystem Size Used Avail Use% Mounted on /dev/sdb1 880G 828G 7.5G 100% /mnt/root
Improve this question
10
  • What happens if you restart syslog? Commented May 22, 2017 at 19:15
  • 1
    @RuiFRibeiro: thanks for the idea... I've never restarted syslog, need to look into that... Commented May 22, 2017 at 19:16
  • @RuiFRibeiro: FWIW, /var is tiny (3.1G), and doesn't seem to be growing particularly fast. Commented May 22, 2017 at 19:19
  • You dealing with a deleted file, that is why du does not register it. The /var size, if syslog is the culprit, wont register it too. Commented May 22, 2017 at 19:20
  • 1
    @RuiFRibeiro: a deleted file that is growing? Commented May 22, 2017 at 19:21

2 Answers 2

Reset to default
10

You are dealing with deleted files, that is why du does not register used space, but dfdoes.

Deleted files only disappear after the owner process is stopped; they remain in use while that does not happen.

So to find the culprit process, I recommend you doing:

sudo lsof -nP | grep '(deleted)'

Then for killing the process.

sudo kill -9 $(lsof | grep deleted | cut -d " " -f4)
Improve this answer
2

You could use iotop to see which processes are performing the most disk write operations.

Example:

Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND 1 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % init 2 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kthreadd] 3 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/0] 6 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [migration/0] 7 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [watchdog/0] 8 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [migration/1]
Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.