I inherited two servers, a production server (wglsbi02) and a mirror/backup server (wglsbi01), that were originally set up by someone who has since left the company. I’m not an experienced system administrator, but I’m trying to understand why the /dev/mapper/ol_wglsbi02-root is almost full. There is around 40G I can not explain.
Filesystem Size Used Avail Use% Mounted on devtmpfs 47G 0 47G 0% /dev tmpfs 47G 0 47G 0% /dev/shm tmpfs 47G 26M 47G 1% /run tmpfs 47G 0 47G 0% /sys/fs/cgroup /dev/mapper/ol_wglsbi02-root 70G 60G 11G 85% / /dev/sda 3.0T 1.8T 1.2T 61% /mnt/scratch1 /dev/sdb 3.0T 2.2T 757G 75% /mnt/data1 /dev/sdc2 1014M 670M 345M 67% /boot /dev/mapper/ol_wglsbi02-home 372G 89G 284G 24% /home /dev/sdc1 599M 6.0M 593M 1% /boot/efi //rhmfile1/Share01/DivC/Path/Genetics 101T 71T 30T 71% /mnt/Genetics overlay 70G 60G 11G 85% /var/lib/docker/overlay2/4b8e7ff542c2285b7f2e01dba49034c7beb3304cb6a3107c07ac1e5578b1192b/merged overlay 70G 60G 11G 85% /var/lib/docker/overlay2/3eada26813165853a6cc770b370fbbf14c2e56db8ca0a030735af98ebf2550bf/merged overlay 70G 60G 11G 85% /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged //genetix-nas-pth/sft-uhs-transfer/htsf-working 5.5T 4.1T 1.5T 75% /home/auto/htsf-working //rhmgensftp/sftproot 500G 285G 216G 57% /mnt/rhmgensftp //rhmqarep/ArchiveData/Genetics 51T 37T 14T 73% /mnt/ArchiveData //genetix-nas-pth/sft-uhs-transfer/htsf-working 5.5T 4.1T 1.5T 75% /mnt/htsf-working //rhmfile1/Share01/DivC/Path/GeneticsData 101T 71T 30T 71% /mnt/GeneticsData tmpfs 9.4G 0 9.4G 0% /run/user/1001 tmpfs 9.4G 0 9.4G 0% /run/user/0 sudo du -xh --max-depth=1 / | sort -h 0 /media 0 /mnt 0 /opt 0 /srv 4.9M /tmp 29M /etc 140M /root 4.1G /var 5.5G /usr 9.7G / I have also use ncdu but again, this didn't help. Also
docker system df TYPE TOTAL ACTIVE SIZE RECLAIMABLE Images 3 3 2.841GB 0B (0%) Containers 3 3 377.7kB 0B (0%) Local Volumes 0 0 0B 0B Build Cache 0 0 0B 0B journalctl --disk-usage Archived and active journals take up 24.0M in the file system. apptainer cache list There are 1 container file(s) using 19.41 MiB and 5 oci blob file(s) using 19.73 MiB of space Total space used: 39.14 MiB Unable to figure out what is taking up that approximate. 40 GB, I tried using the backup server to see if I could find the culprit by comparison. On wglsbi01, the usage of /dev/mapper/ol_wglsbi01-root makes sense.
df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 47G 0 47G 0% /dev tmpfs 47G 4.0K 47G 1% /dev/shm tmpfs 47G 84M 47G 1% /run tmpfs 47G 0 47G 0% /sys/fs/cgroup /dev/mapper/ol_wglsbi01-root 70G 30G 41G 43% / /dev/sda 3.0T 948G 2.0T 32% /mnt/scratch1 /dev/sdb 3.0T 1.2T 1.8T 41% /mnt/data1 /dev/sdc2 1014M 343M 672M 34% /boot /dev/mapper/ol_wglsbi01-home 372G 222G 150G 60% /home /dev/sdc1 599M 6.0M 593M 1% /boot/efi //genetix-nas-pth/sft-uhs-transfer/htsf-working 5.5T 4.1T 1.5T 75% /mnt/htsf-working //rhmfile1/Share01/DivC/Path/GeneticsData 101T 71T 30T 71% /mnt/GeneticsData //genetix-nas-pth/sft-uhs-transfer/htsf-working 5.5T 4.1T 1.5T 75% /home/auto/htsf-working //rhmfile1/Share01/DivC/Path/Genetics 101T 71T 30T 71% /mnt/Genetics overlay 70G 30G 41G 43% /var/lib/docker/overlay2/f841c18cb9fc1689833e01fb71a9e6be2ce74a679c78604e6299b55e68cbcaf5/merged overlay 70G 30G 41G 43% /var/lib/docker/overlay2/00775a1fc4f8b26cd12853c1c59cad7c615bf1dbc0531df96023b0be84d0b6cd/merged overlay 70G 30G 41G 43% /var/lib/docker/overlay2/32f5d04ee399d2130982c2f0fe209fba056757fb061b082f8a72902a83d02d9e/merged overlay 70G 30G 41G 43% /var/lib/docker/overlay2/5668f6a8a629c7f94d15c1667c5fdcd4ac1aa6c62d671f4cd4d05789428e6f1e/merged //rhmgensftp/sftproot 500G 285G 216G 57% /mnt/rhmgensftp //rhmqarep/ArchiveData/Genetics 51T 37T 14T 73% /mnt/ArchiveData tmpfs 9.4G 0 9.4G 0% /run/user/1004 overlay 70G 30G 41G 43% /var/lib/docker/overlay2/8a7c217db3531825c2c4d0ddb49940d686d03e656f606f2ad3fc3b1395632c90/merged tmpfs 9.4G 0 9.4G 0% /run/user/0 du -xh --max-depth=1 / | sort -h 0 /media 0 /mnt 0 /opt 0 /srv 1.6M /tmp 28M /etc 6.4G /usr 11G /root 13G /var 30G / I need at least 10 GB available in /dev/mapper/ol_wglsbi02-root to be able to run Slurm, so I’m concerned that this issue might cause it to fail.
If I’m looking at the disk space incorrectly, or if there’s any additional information I should provide to help identify the issue, please let me know — I’m happy to update the question.
EDIT: lsof commands doesn't report anything
the command find / -type f ! -size 0 -exec perl -le 'for(@ARGV){open(A,"<",$_)or next;seek A,0,4;$p=tell A;seek A,0,2;print if$p!=tell A;close A}' {} + -printf '%s %p\n' | sort -nk1,1 | tail -n 20 takes a lot so I will do this targeting specific directories in / and I will update here.
find /var/ -path /mnt -prune -o -type f ! -size 0 \ > -exec perl -le 'for(@ARGV){open(A,"<",$_)or next;seek A,0,4;$p=tell A;seek A,0,2;print if$p!=tell A;close A}' {} + \ > -printf '%s %p\n' | sort -nk1,1 | tail -n 200 | \ > awk '{printf "%.2f GB\t%s\n", $1/1024/1024/1024, $2}' 0.00 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/lib/x86_64-linux-gnu/perl/5.36.0/auto/Encode/KR/KR.so 0.00 GB /var/lib/docker/overlay2/112af818162d47f1f60409980f5176d2916a9693b43995c2ff342b15ea0de7ce/diff/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.33 0.00 GB /var/lib/docker/overlay2/4b8e7ff542c2285b7f2e01dba49034c7beb3304cb6a3107c07ac1e5578b1192b/merged/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.33 0.00 GB /var/lib/docker/overlay2/3928cd955a5797b25d4394fecbb209bf029f60ca15f9e124557190bf086e16de/diff/usr/lib/x86_64-linux-gnu/libgprofng.so.0.0.0 0.00 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/lib/x86_64-linux-gnu/libgprofng.so.0.0.0 0.00 GB /var/lib/docker/overlay2/3e2daefe94ba3663907aa148a11b660882b5eb9cca26335824316bc67c1a17b0/diff/usr/lib/x86_64-linux-gnu/libtk8.6.a 0.00 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/lib/x86_64-linux-gnu/libtk8.6.a 0.00 GB /var/lib/docker/overlay2/3928cd955a5797b25d4394fecbb209bf029f60ca15f9e124557190bf086e16de/diff/usr/lib/x86_64-linux-gnu/libMagickWand-6.Q16.a 0.00 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/lib/x86_64-linux-gnu/libMagickWand-6.Q16.a 0.00 GB /var/lib/docker/overlay2/112af818162d47f1f60409980f5176d2916a9693b43995c2ff342b15ea0de7ce/diff/usr/lib/x86_64-linux-gnu/libapt-pkg.so.7.0.0 0.00 GB /var/lib/docker/overlay2/4b8e7ff542c2285b7f2e01dba49034c7beb3304cb6a3107c07ac1e5578b1192b/merged/usr/lib/x86_64-linux-gnu/libapt-pkg.so.7.0.0 0.00 GB /var/lib/docker/overlay2/4b8e7ff542c2285b7f2e01dba49034c7beb3304cb6a3107c07ac1e5578b1192b/merged/usr/local/lib/python3.11/site-packages/pandas/_libs/groupby.cpython-311-x86_64-linux-gnu.so 0.00 GB /var/lib/docker/overlay2/ofkm96by1ucifl84c1bcqlnrt/diff/usr/local/lib/python3.11/site-packages/pandas/_libs/groupby.cpython-311-x86_64-linux-gnu.so 0.00 GB /var/lib/docker/overlay2/2gef88e6y5d8ppgp28viu1rzr/diff/usr/local/lib/python3.11/site-packages/numpy.libs/libgfortran-040039e1.so.5.0.0 0.00 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/local/lib/python3.11/site-packages/numpy.libs/libgfortran-040039e1.so.5.0.0 0.00 GB /var/lib/docker/overlay2/3928cd955a5797b25d4394fecbb209bf029f60ca15f9e124557190bf086e16de/diff/usr/lib/x86_64-linux-gnu/libubsan.so.1.0.0 0.00 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/lib/x86_64-linux-gnu/libubsan.so.1.0.0 0.00 GB /var/log/slurm/slurmd.log-20250928 0.00 GB /var/lib/docker/overlay2/67554dcbb8d2196fd3f5ff94a44a98df811f39f9e01ceff8928cf70ed58f762d/diff/usr/lib/x86_64-linux-gnu/perl/5.36.0/auto/Encode/JP/JP.so 0.00 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/lib/x86_64-linux-gnu/perl/5.36.0/auto/Encode/JP/JP.so 0.00 GB /var/lib/docker/overlay2/3928cd955a5797b25d4394fecbb209bf029f60ca15f9e124557190bf086e16de/diff/usr/lib/x86_64-linux-gnu/libxml2.a 0.00 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/lib/x86_64-linux-gnu/libxml2.a 0.00 GB /var/lib/docker/overlay2/3928cd955a5797b25d4394fecbb209bf029f60ca15f9e124557190bf086e16de/diff/usr/lib/gcc/x86_64-linux-gnu/12/libasan.a 0.00 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/lib/gcc/x86_64-linux-gnu/12/libasan.a 0.00 GB /var/lib/docker/overlay2/3928cd955a5797b25d4394fecbb209bf029f60ca15f9e124557190bf086e16de/diff/usr/lib/x86_64-linux-gnu/libMagickCore-6.Q16.so.6.0.0 0.00 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/lib/x86_64-linux-gnu/libMagickCore-6.Q16.so.6.0.0 0.00 GB /var/lib/docker/overlay2/4b8e7ff542c2285b7f2e01dba49034c7beb3304cb6a3107c07ac1e5578b1192b/merged/usr/local/lib/python3.11/site-packages/numpy.libs/libgfortran-040039e1-0352e75f.so.5.0.0 0.00 GB /var/lib/docker/overlay2/ofkm96by1ucifl84c1bcqlnrt/diff/usr/local/lib/python3.11/site-packages/numpy.libs/libgfortran-040039e1-0352e75f.so.5.0.0 0.00 GB /var/lib/docker/overlay2/4b8e7ff542c2285b7f2e01dba49034c7beb3304cb6a3107c07ac1e5578b1192b/merged/usr/local/lib/python3.11/site-packages/numpy/_core/_simd.cpython-311-x86_64-linux-gnu.so 0.00 GB /var/lib/docker/overlay2/ofkm96by1ucifl84c1bcqlnrt/diff/usr/local/lib/python3.11/site-packages/numpy/_core/_simd.cpython-311-x86_64-linux-gnu.so 0.00 GB /var/cache/dnf/ol8_baseos_latest-e4c6155830ad002c/repodata/57a55ddf91813a5f60113ec64e22f75cbe6126f21bec81e8c23406b641a9c7dd-updateinfo.xml.gz 0.00 GB /var/lib/docker/overlay2/3928cd955a5797b25d4394fecbb209bf029f60ca15f9e124557190bf086e16de/diff/usr/lib/x86_64-linux-gnu/liblsan.so.0.0.0 0.00 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/lib/x86_64-linux-gnu/liblsan.so.0.0.0 0.00 GB /var/lib/docker/overlay2/3928cd955a5797b25d4394fecbb209bf029f60ca15f9e124557190bf086e16de/diff/usr/lib/x86_64-linux-gnu/libdb-5.3.a 0.00 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/lib/x86_64-linux-gnu/libdb-5.3.a 0.00 GB /var/lib/docker/overlay2/3928cd955a5797b25d4394fecbb209bf029f60ca15f9e124557190bf086e16de/diff/usr/lib/gcc/x86_64-linux-gnu/12/libgcc.a 0.00 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/lib/gcc/x86_64-linux-gnu/12/libgcc.a 0.00 GB /var/lib/docker/overlay2/3928cd955a5797b25d4394fecbb209bf029f60ca15f9e124557190bf086e16de/diff/usr/bin/x86_64-linux-gnu-ld.gold 0.00 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/bin/x86_64-linux-gnu-ld.gold 0.00 GB /var/lib/docker/overlay2/3928cd955a5797b25d4394fecbb209bf029f60ca15f9e124557190bf086e16de/diff/usr/lib/x86_64-linux-gnu/libOpenEXR-3_1.so.30.5.1 0.00 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/lib/x86_64-linux-gnu/libOpenEXR-3_1.so.30.5.1 0.00 GB /var/lib/docker/overlay2/3928cd955a5797b25d4394fecbb209bf029f60ca15f9e124557190bf086e16de/diff/usr/lib/x86_64-linux-gnu/libicui18n.so.72.1 0.00 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/lib/x86_64-linux-gnu/libicui18n.so.72.1 0.00 GB /var/cache/dnf/ol8_codeready_builder-updateinfo.solvx 0.00 GB /var/lib/docker/overlay2/3e2daefe94ba3663907aa148a11b660882b5eb9cca26335824316bc67c1a17b0/diff/usr/lib/x86_64-linux-gnu/libtcl8.6.a 0.00 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/lib/x86_64-linux-gnu/libtcl8.6.a 0.00 GB /var/lib/docker/overlay2/2gef88e6y5d8ppgp28viu1rzr/diff/usr/local/lib/python3.11/site-packages/numpy/core/_simd.cpython-311-x86_64-linux-gnu.so 0.00 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/local/lib/python3.11/site-packages/numpy/core/_simd.cpython-311-x86_64-linux-gnu.so 0.00 GB /var/log/anaconda/journal.log 0.00 GB /var/cache/dnf/ol8_codeready_builder.solv 0.00 GB /var/lib/docker/overlay2/67554dcbb8d2196fd3f5ff94a44a98df811f39f9e01ceff8928cf70ed58f762d/diff/usr/bin/git 0.00 GB /var/lib/docker/overlay2/67554dcbb8d2196fd3f5ff94a44a98df811f39f9e01ceff8928cf70ed58f762d/diff/usr/lib/git-core/git 0.00 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/bin/git 0.00 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/lib/git-core/git 0.00 GB /var/lib/samba/lock/leases.tdb 0.00 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/bin/perl 0.00 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/bin/perl5.36.0 0.00 GB /var/lib/docker/overlay2/f45fa45f06c098c189da997b02795fd6b6c3cf60032d79413d56c66ad749bcc5/diff/usr/bin/perl 0.00 GB /var/lib/docker/overlay2/f45fa45f06c098c189da997b02795fd6b6c3cf60032d79413d56c66ad749bcc5/diff/usr/bin/perl5.36.0 0.00 GB /var/lib/docker/overlay2/67554dcbb8d2196fd3f5ff94a44a98df811f39f9e01ceff8928cf70ed58f762d/diff/usr/lib/x86_64-linux-gnu/libperl.so.5.36.0 0.00 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/lib/x86_64-linux-gnu/libperl.so.5.36.0 0.00 GB /var/lib/docker/overlay2/112af818162d47f1f60409980f5176d2916a9693b43995c2ff342b15ea0de7ce/diff/usr/bin/perl 0.00 GB /var/lib/docker/overlay2/112af818162d47f1f60409980f5176d2916a9693b43995c2ff342b15ea0de7ce/diff/usr/bin/perl5.40.1 0.00 GB /var/lib/docker/overlay2/4b8e7ff542c2285b7f2e01dba49034c7beb3304cb6a3107c07ac1e5578b1192b/merged/usr/bin/perl 0.00 GB /var/lib/docker/overlay2/4b8e7ff542c2285b7f2e01dba49034c7beb3304cb6a3107c07ac1e5578b1192b/merged/usr/bin/perl5.40.1 0.00 GB /var/lib/docker/overlay2/3928cd955a5797b25d4394fecbb209bf029f60ca15f9e124557190bf086e16de/diff/usr/lib/x86_64-linux-gnu/libicuuc.a 0.00 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/lib/x86_64-linux-gnu/libicuuc.a 0.00 GB /var/lib/dnf/history.sqlite-wal 0.00 GB /var/lib/docker/buildkit/containerdmeta.db 0.00 GB /var/lib/docker/overlay2/3928cd955a5797b25d4394fecbb209bf029f60ca15f9e124557190bf086e16de/diff/usr/lib/x86_64-linux-gnu/libdjvulibre.a 0.00 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/lib/x86_64-linux-gnu/libdjvulibre.a 0.00 GB /var/lib/docker/overlay2/67554dcbb8d2196fd3f5ff94a44a98df811f39f9e01ceff8928cf70ed58f762d/diff/usr/lib/x86_64-linux-gnu/perl/5.36.0/CORE/charclass_invlists.h 0.00 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/lib/x86_64-linux-gnu/perl/5.36.0/CORE/charclass_invlists.h 0.00 GB /var/lib/docker/overlay2/3928cd955a5797b25d4394fecbb209bf029f60ca15f9e124557190bf086e16de/diff/usr/lib/x86_64-linux-gnu/libgio-2.0.a 0.00 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/lib/x86_64-linux-gnu/libgio-2.0.a 0.00 GB /var/lib/docker/overlay2/5094e2385b9b4e31d230f75a957039803f4d005360f8680b38e460769295b5ca/diff/usr/lib/x86_64-linux-gnu/libcrypto.so.3 0.00 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/lib/x86_64-linux-gnu/libcrypto.so.3 0.00 GB /var/log/bcl2fq.log 0.00 GB /var/lib/docker/overlay2/2gef88e6y5d8ppgp28viu1rzr/diff/usr/local/lib/python3.11/site-packages/pydantic_core/_pydantic_core.cpython-311-x86_64-linux-gnu.so 0.00 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/local/lib/python3.11/site-packages/pydantic_core/_pydantic_core.cpython-311-x86_64-linux-gnu.so 0.00 GB /var/lib/rpm/Basenames 0.00 GB /var/lib/docker/overlay2/4b8e7ff542c2285b7f2e01dba49034c7beb3304cb6a3107c07ac1e5578b1192b/merged/usr/local/lib/python3.11/site-packages/pydantic_core/_pydantic_core.cpython-311-x86_64-linux-gnu.so 0.00 GB /var/lib/docker/overlay2/ofkm96by1ucifl84c1bcqlnrt/diff/usr/local/lib/python3.11/site-packages/pydantic_core/_pydantic_core.cpython-311-x86_64-linux-gnu.so 0.00 GB /var/cache/dnf/ol8_baseos_latest-updateinfo.solvx 0.00 GB /var/lib/docker/overlay2/3928cd955a5797b25d4394fecbb209bf029f60ca15f9e124557190bf086e16de/diff/usr/lib/x86_64-linux-gnu/libMagickCore-6.Q16.a 0.00 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/lib/x86_64-linux-gnu/libMagickCore-6.Q16.a 0.00 GB /var/lib/docker/overlay2/4b8e7ff542c2285b7f2e01dba49034c7beb3304cb6a3107c07ac1e5578b1192b/merged/usr/local/lib/libpython3.11.so.1.0 0.00 GB /var/lib/docker/overlay2/fdffcf58d56bdf7b553f9140aa5186718ea74139158cfa5f2f9dd39b191a055d/diff/usr/local/lib/libpython3.11.so.1.0 0.01 GB /var/lib/docker/overlay2/3928cd955a5797b25d4394fecbb209bf029f60ca15f9e124557190bf086e16de/diff/usr/lib/x86_64-linux-gnu/libc.a 0.01 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/lib/x86_64-linux-gnu/libc.a 0.01 GB /var/lib/docker/overlay2/3928cd955a5797b25d4394fecbb209bf029f60ca15f9e124557190bf086e16de/diff/usr/lib/x86_64-linux-gnu/libaom.so.3.6.0 0.01 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/lib/x86_64-linux-gnu/libaom.so.3.6.0 0.01 GB /var/cache/dnf/ol8_appstream-d4d5cd0a0a095846/repodata/e7e4a7bcbcedec298e83d94357d23199a5fcc041b9a14795550448d92cbcdb48-updateinfo.xml.gz 0.01 GB /var/lib/docker/overlay2/3928cd955a5797b25d4394fecbb209bf029f60ca15f9e124557190bf086e16de/diff/usr/lib/gcc/x86_64-linux-gnu/12/libstdc++.a 0.01 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/lib/gcc/x86_64-linux-gnu/12/libstdc++.a 0.01 GB /var/cache/dnf/ol8_UEKR7.solv 0.01 GB /var/log/audit/audit.log 0.01 GB /var/lib/rpm/Providename 0.01 GB /var/lib/docker/overlay2/112af818162d47f1f60409980f5176d2916a9693b43995c2ff342b15ea0de7ce/diff/usr/lib/x86_64-linux-gnu/libcrypto.so.3 0.01 GB /var/lib/docker/overlay2/4b8e7ff542c2285b7f2e01dba49034c7beb3304cb6a3107c07ac1e5578b1192b/merged/usr/lib/x86_64-linux-gnu/libcrypto.so.3 0.01 GB /var/lib/docker/overlay2/67554dcbb8d2196fd3f5ff94a44a98df811f39f9e01ceff8928cf70ed58f762d/diff/usr/bin/python3.11 0.01 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/bin/python3.11 0.01 GB /var/lib/docker/overlay2/2gef88e6y5d8ppgp28viu1rzr/diff/usr/local/lib/python3.11/site-packages/numpy/core/_multiarray_umath.cpython-311-x86_64-linux-gnu.so 0.01 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/local/lib/python3.11/site-packages/numpy/core/_multiarray_umath.cpython-311-x86_64-linux-gnu.so 0.01 GB /var/lib/docker/overlay2/3928cd955a5797b25d4394fecbb209bf029f60ca15f9e124557190bf086e16de/diff/usr/lib/x86_64-linux-gnu/libtsan.so.2.0.0 0.01 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/lib/x86_64-linux-gnu/libtsan.so.2.0.0 0.01 GB /var/cache/dnf/ol8_developer_EPEL-6205dfaa445bcb4b/repodata/2f409cd05f0fb731476c94e31c264f544ad5724479b32d56a276839a03b135ac-primary.xml.gz 0.01 GB /var/lib/docker/overlay2/3928cd955a5797b25d4394fecbb209bf029f60ca15f9e124557190bf086e16de/diff/usr/lib/x86_64-linux-gnu/libicui18n.a 0.01 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/lib/x86_64-linux-gnu/libicui18n.a 0.01 GB /var/lib/docker/overlay2/3928cd955a5797b25d4394fecbb209bf029f60ca15f9e124557190bf086e16de/diff/usr/lib/x86_64-linux-gnu/libasan.so.8.0.0 0.01 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/lib/x86_64-linux-gnu/libasan.so.8.0.0 0.01 GB /var/lib/docker/overlay2/3928cd955a5797b25d4394fecbb209bf029f60ca15f9e124557190bf086e16de/diff/usr/lib/file/magic.mgc 0.01 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/lib/file/magic.mgc 0.01 GB /var/log/audit/audit.log.3 0.01 GB /var/log/audit/audit.log.1 0.01 GB /var/log/audit/audit.log.2 0.01 GB /var/log/audit/audit.log.4 0.01 GB /var/lib/docker/overlay2/3928cd955a5797b25d4394fecbb209bf029f60ca15f9e124557190bf086e16de/diff/usr/lib/x86_64-linux-gnu/libcrypto.a 0.01 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/lib/x86_64-linux-gnu/libcrypto.a 0.01 GB /var/lib/selinux/targeted/active/policy.kern 0.01 GB /var/lib/selinux/targeted/active/policy.linked 0.01 GB /var/lib/samba/lock/locking.tdb 0.01 GB /var/cache/dnf/ol8_appstream-d4d5cd0a0a095846/repodata/9fa06ae901f29c46a7d06cf86a51afebd29a3255d9eaea28d8a56416d489a13f-primary.xml.gz 0.01 GB /var/lib/docker/overlay2/5094e2385b9b4e31d230f75a957039803f4d005360f8680b38e460769295b5ca/diff/usr/bin/sq 0.01 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/bin/sq 0.01 GB /var/cache/dnf/ol8_codeready_builder-filenames.solvx 0.01 GB /var/cache/dnf/ol8_codeready_builder-0274a0056455208d/repodata/745f3ce2b9520d8c09aa8fd99270e4f7c483b864868401b845a67bdb16144c05-filelists.xml.gz 0.01 GB /var/lib/docker/overlay2/4b8e7ff542c2285b7f2e01dba49034c7beb3304cb6a3107c07ac1e5578b1192b/merged/usr/local/lib/python3.11/site-packages/numpy/_core/_multiarray_umath.cpython-311-x86_64-linux-gnu.so 0.01 GB /var/lib/docker/overlay2/ofkm96by1ucifl84c1bcqlnrt/diff/usr/local/lib/python3.11/site-packages/numpy/_core/_multiarray_umath.cpython-311-x86_64-linux-gnu.so 0.01 GB /var/lib/samba/lock/smbXsrv_open_global.tdb 0.01 GB /var/lib/docker/overlay2/3928cd955a5797b25d4394fecbb209bf029f60ca15f9e124557190bf086e16de/diff/usr/lib/x86_64-linux-gnu/librsvg-2.so.2.48.0 0.01 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/lib/x86_64-linux-gnu/librsvg-2.so.2.48.0 0.01 GB /var/cache/dnf/packages.db 0.01 GB /var/cache/dnf/ol8_appstream-updateinfo.solvx 0.01 GB /var/lib/docker/overlay2/2gef88e6y5d8ppgp28viu1rzr/diff/root/.cache/pip/http-v2/d/5/e/2/b/d5e2b8da29a1f6235b99a9fb4695ba6e487ee7eb5386b489ccb994a7.body 0.01 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/root/.cache/pip/http-v2/d/5/e/2/b/d5e2b8da29a1f6235b99a9fb4695ba6e487ee7eb5386b489ccb994a7.body 0.01 GB /var/cache/dnf/ol8_developer_EPEL.solv 0.01 GB /var/cache/dnf/ol8_developer_EPEL-6205dfaa445bcb4b/repodata/d15d2880410a489871b6d13018abc58b25f933c663df0e180582ed453e3ffd5b-updateinfo.xml.gz 0.01 GB /var/cache/dnf/ol8_baseos_latest.solv 0.02 GB /var/cache/dnf/ol8_appstream.solv 0.02 GB /var/lib/docker/overlay2/3928cd955a5797b25d4394fecbb209bf029f60ca15f9e124557190bf086e16de/diff/usr/lib/x86_64-linux-gnu/libx265.so.199 0.02 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/lib/x86_64-linux-gnu/libx265.so.199 0.02 GB /var/lib/docker/overlay2/2gef88e6y5d8ppgp28viu1rzr/diff/root/.cache/pip/http-v2/4/5/3/4/3/453430c3a43f9419e4f6ab6e2ff6ad11b6420baf9c7e972b02afb4ad.body 0.02 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/root/.cache/pip/http-v2/4/5/3/4/3/453430c3a43f9419e4f6ab6e2ff6ad11b6420baf9c7e972b02afb4ad.body 0.02 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/local/lib/libpython3.11.so.1.0 0.02 GB /var/lib/docker/overlay2/e69249722d56d7f5ef85bbb528f75282896b9274d28378108ae84f9184612db4/diff/usr/local/lib/libpython3.11.so.1.0 0.02 GB /var/lib/docker/containers/a02590184a8fa90ca049f72d860a93bcdfd87003e4c551829f0a95812947c57b/a02590184a8fa90ca049f72d860a93bcdfd87003e4c551829f0a95812947c57b-json.log 0.02 GB /var/cache/dnf/ol8_developer_EPEL-updateinfo.solvx 0.02 GB /var/lib/docker/overlay2/4b8e7ff542c2285b7f2e01dba49034c7beb3304cb6a3107c07ac1e5578b1192b/merged/usr/local/lib/python3.11/site-packages/numpy.libs/libscipy_openblas64_-8fb3d286.so 0.02 GB /var/lib/docker/overlay2/ofkm96by1ucifl84c1bcqlnrt/diff/usr/local/lib/python3.11/site-packages/numpy.libs/libscipy_openblas64_-8fb3d286.so 0.03 GB /var/cache/dnf/ol8_UEKR7-cb3953d2e04e80fd/repodata/523d7e96e051db47b9820b8f5ea0723e801010e2e22cbd444810a3fe16e46fe6-filelists.xml.gz 0.03 GB /var/cache/dnf/ol8_UEKR7-filenames.solvx 0.03 GB /var/lib/docker/overlay2/3928cd955a5797b25d4394fecbb209bf029f60ca15f9e124557190bf086e16de/diff/usr/lib/x86_64-linux-gnu/libicudata.a 0.03 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/lib/x86_64-linux-gnu/libicudata.a 0.03 GB /var/lib/docker/overlay2/3928cd955a5797b25d4394fecbb209bf029f60ca15f9e124557190bf086e16de/diff/usr/lib/x86_64-linux-gnu/libicudata.so.72.1 0.03 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/lib/x86_64-linux-gnu/libicudata.so.72.1 0.03 GB /var/lib/docker/overlay2/3928cd955a5797b25d4394fecbb209bf029f60ca15f9e124557190bf086e16de/diff/usr/bin/x86_64-linux-gnu-lto-dump-12 0.03 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/bin/x86_64-linux-gnu-lto-dump-12 0.03 GB /var/lib/docker/overlay2/3928cd955a5797b25d4394fecbb209bf029f60ca15f9e124557190bf086e16de/diff/usr/lib/gcc/x86_64-linux-gnu/12/lto1 0.03 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/lib/gcc/x86_64-linux-gnu/12/lto1 0.03 GB /var/lib/docker/overlay2/3928cd955a5797b25d4394fecbb209bf029f60ca15f9e124557190bf086e16de/diff/usr/lib/gcc/x86_64-linux-gnu/12/cc1 0.03 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/lib/gcc/x86_64-linux-gnu/12/cc1 0.03 GB /var/lib/docker/overlay2/2gef88e6y5d8ppgp28viu1rzr/diff/usr/local/lib/python3.11/site-packages/numpy.libs/libopenblas64_p-r0-0cf96a72.3.23.dev.so 0.03 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/local/lib/python3.11/site-packages/numpy.libs/libopenblas64_p-r0-0cf96a72.3.23.dev.so 0.03 GB /var/lib/docker/overlay2/3928cd955a5797b25d4394fecbb209bf029f60ca15f9e124557190bf086e16de/diff/usr/lib/gcc/x86_64-linux-gnu/12/cc1plus 0.03 GB /var/lib/docker/overlay2/aa11ead53f35ff83e1f94480cd8a30597718274797804d93a520c0e1eb189c9b/merged/usr/lib/gcc/x86_64-linux-gnu/12/cc1plus 0.05 GB /var/cache/dnf/ol8_UEKR7-cb3953d2e04e80fd/repodata/7fefb4e721b861dcb903a22766a0a27d7bb802b4f0c6d30567952369bbc80900-primary.xml.gz 0.05 GB /var/cache/dnf/ol8_baseos_latest-e4c6155830ad002c/repodata/af71333f4c2e8d131581dc49e07347e212d76f95d0baeca3b51f1dd7dd4b56b3-filelists.xml.gz 0.05 GB /var/cache/dnf/ol8_baseos_latest-filenames.solvx 0.06 GB /var/cache/dnf/ol8_appstream-filenames.solvx 0.06 GB /var/cache/dnf/ol8_appstream-d4d5cd0a0a095846/repodata/6ec05cf2ff2f2db8a30aea74a520610b877f3f02300edc6b19af2d2179c0f08f-filelists.xml.gz 0.06 GB /var/cache/dnf/ol8_baseos_latest-e4c6155830ad002c/repodata/05c172de373b3c9abd19299942a4fd6adc963e6b63fe5564085da2f3fd098a38-primary.xml.gz 0.07 GB /var/cache/dnf/ol8_developer_EPEL-6205dfaa445bcb4b/repodata/fd642c5c3863a959196d56a1100deff70f1ef7a45179411096df9bf052865117-filelists.xml.gz 0.07 GB /var/cache/dnf/ol8_developer_EPEL-filenames.solvx 0.32 GB /var/lib/rpm/Packages
find / -type f ! -size 0 -exec perl -le 'for(@ARGV){open(A,"<",$_)or next;seek A,0,4;$p=tell A;seek A,0,2;print if$p!=tell A;close A}' {} + -printf '%s %p\n' | sort -nk1,1 | tail -n 20and add the output to your question. This is a slightly modified version of the command given in this answer, and will find sparse files on your system, order them by size and print the 20 largest ones.lsof | grep -E 'COMMAND|\(deleted\)'to identify thosefuser -m /to see all processes with open files on the root fs. orlsof | grep deletedto see processes with deleted files. Unix doesn't actually delete a file and return the space it's using until all processes which have opened it either close it or exit. This is not uncommon when, for example, a log file is deleted but the process that was writing to it hasn't closed it been killed, restarted...which is why it's standard practice to reload or restart processes when rotating log files.lsof +L1works too. orfind /proc/*/fd -ls | grep deleted. If there is such a process, either restart the process or truncate the file (using the symlink in/proc/PID/fd/, where PID is the process id of the process(es) that have the file open). See Find and remove large files that are open but have been deleted for more details.lsof +L1new to me, thank you