4

I have a btrfs disk (without raid) that gives me a very big difference between du and df:

$ df -h /pgdata Filesystem Size Used Avail Use% Mounted on /dev/mapper/ubuntu--vg-pgdata 85G 77G 7.1G 92% /pgdata $ du -sh /pgdata 56G /pgdata

And btrfs commands give me the same result:

$ btrfs fi df -h /pgdata/ Data, single: total=82.94GiB, used=75.63GiB System, DUP: total=32.00MiB, used=16.00KiB Metadata, DUP: total=1.00GiB, used=365.73MiB GlobalReserve, single: total=228.33MiB, used=0.00B $ btrfs fi du -s /pgdata/ Total Exclusive Set shared Filename 55.84GiB 55.36GiB 491.34MiB /pgdata/

There is no subvolume (I think), btrfs subvolume list /pgdata is empty. I ran btrfs balance start /pgdata/ -dusage=66 but it didn't change much.

$ btrfs fi usage /pgdata/ Overall: Device size: 85.00GiB Device allocated: 85.00GiB Device unallocated: 1.00MiB Device missing: 0.00B Used: 76.50GiB Free (estimated): 7.15GiB (min: 7.15GiB) Data ratio: 1.00 Metadata ratio: 2.00 Global reserve: 228.52MiB (used: 0.00B) Data,single: Size:82.94GiB, Used:75.79GiB (91.38%) /dev/mapper/ubuntu--vg-pgdata 82.94GiB Metadata,DUP: Size:1.00GiB, Used:365.78MiB (35.72%) /dev/mapper/ubuntu--vg-pgdata 2.00GiB System,DUP: Size:32.00MiB, Used:16.00KiB (0.05%) /dev/mapper/ubuntu--vg-pgdata 64.00MiB Unallocated: /dev/mapper/ubuntu--vg-pgdata 1.00MiB

It's not deleted inodes that are the issue (the machine has been rebooted 3 hours ago) and:

$ lsof /pgdata |grep deleted postgres 4107 postgres 4u REG 0,58 16777216 238658 /pgdata/postgresql/13/main/pg_wal/00000001000001E40000008A (deleted) postgres 5589 postgres 45u REG 0,58 16777216 238753 /pgdata/postgresql/13/main/pg_wal/00000001000001E50000000E (deleted) postgres 5590 postgres 45u REG 0,58 16777216 222523 /pgdata/postgresql/13/main/pg_wal/00000001000001E40000006A (deleted) postgres 5591 postgres 26u REG 0,58 16777216 238992 /pgdata/postgresql/13/main/pg_wal/00000001000001E4000000F3 (deleted) postgres 5592 postgres 52u REG 0,58 16777216 238986 /pgdata/postgresql/13/main/pg_wal/00000001000001E4000000ED (deleted) postgres 5595 postgres 28u REG 0,58 16777216 238995 /pgdata/postgresql/13/main/pg_wal/00000001000001E4000000F6 (deleted) postgres 5596 postgres 19u REG 0,58 16777216 222523 /pgdata/postgresql/13/main/pg_wal/00000001000001E40000006A (deleted) postgres 5597 postgres 44u REG 0,58 16777216 222523 /pgdata/postgresql/13/main/pg_wal/00000001000001E40000006A (deleted) postgres 5598 postgres 12u REG 0,58 16777216 238986 /pgdata/postgresql/13/main/pg_wal/00000001000001E4000000ED (deleted) postgres 5604 postgres 85u REG 0,58 16777216 238988 /pgdata/postgresql/13/main/pg_wal/00000001000001E4000000EF (deleted) postgres 5605 postgres 61u REG 0,58 16777216 238986 /pgdata/postgresql/13/main/pg_wal/00000001000001E4000000ED (deleted) postgres 12936 postgres 58u REG 0,58 16777216 238995 /pgdata/postgresql/13/main/pg_wal/00000001000001E4000000F6 (deleted)

I found other posts about this problem, but not having a so big difference (around 40% more used than it should) and it was always metadata or snapshot. Here, it seems it's not the case.

Does anyone has an idea why 20GiB are missing from the free space ?

Improve this question
3
  • Did you check if unix.stackexchange.com/q/337422/377345 can explain it? Commented Nov 15, 2021 at 15:35
  • Yes, but there is no RAID on my side. Commented Nov 15, 2021 at 15:44
  • Any difference after btrfs scrub ...? Commented Nov 15, 2021 at 19:23

2 Answers 2

Reset to default
3

One possible cause of more btrfs space being allocated for data than there is visible data is unreachable parts of extents (data in extents containing older versions of file content which has since been overwritten).

To analyze problems like this, I created btdu, a disk usage profiler for btrfs:

https://github.com/CyberShadow/btdu

The tool will identify the true cause for the discrepancy.

Improve this answer
1

If your server is well configured, most of the storage IO should be write IO and all read IO except the (big) first one should be done from ram: your server should nearly only makes write IO to store to disk blocks which where modify in ram

As postgreSQL constantly makes some small write IO (depending of its page size), it would not be surprising that all postgreSQL datafiles would have an high fragmentation level

I did notice the same kind of "issue" (making lost more than 28% on a terabytes BTRFS filesystem) in a configuration where I have VM stored on a big BTRFS filesystem and each VM are using COW VMDK files as disk and some VM are running databases (specially MariaDB / postgreSQL)

The way I recover most of this space was to run :

$ sudo btrfs balance start -musage=100 -dusage=100 /mnt/vm $ sudo btrfs filesystem defrag -r -f -v /mnt/vm

and to run a balance again :

$ sudo btrfs balance start -musage=100 -dusage=100 /mnt/vm

This way I recover the most part of the "lost" space

Please also note that before all you should read :

btrfs filesystem

Here are the results in my case:

Real data (original and last state):

$ sudo btrfs fi du -s /mnt/vm Total Exclusive Set shared Filename 669.62GiB 669.22GiB 401.46MiB /mnt/vm

BTRFS filesystem usage (original state)

$ sudo btrfs fi usage /mnt/vm Overall: Device size: 1000.00GiB Device allocated: 955.07GiB Device unallocated: 44.93GiB Device missing: 0.00B Device slack: 0.00B Used: 950.71GiB Free (estimated): 46.74GiB (min: 24.28GiB) Free (statfs, df): 46.74GiB Data ratio: 1.00 Metadata ratio: 2.00 Global reserve: 512.00MiB (used: 0.00B) Multiple profiles: no Data,single: Size:949.01GiB, Used:947.19GiB (99.81%) /dev/mapper/vgu2nvme-lvvm 949.01GiB Metadata,DUP: Size:3.00GiB, Used:1.76GiB (58.56%) /dev/mapper/vgu2nvme-lvvm 6.00GiB System,DUP: Size:32.00MiB, Used:144.00KiB (0.44%) /dev/mapper/vgu2nvme-lvvm 64.00MiB Unallocated: /dev/mapper/vgu2nvme-lvvm 44.93GiB

Last state after all operation (the first balance only recover 10 GiB):

$ sudo btrfs fi usage /mnt/vm Overall: Device size: 1000.00GiB Device allocated: 711.07GiB Device unallocated: 288.93GiB Device missing: 0.00B Device slack: 0.00B Used: 708.02GiB Free (estimated): 291.72GiB (min: 147.26GiB) Free (statfs, df): 291.72GiB Data ratio: 1.00 Metadata ratio: 2.00 Global reserve: 512.00MiB (used: 0.00B) Multiple profiles: no Data,single: Size:709.01GiB, Used:706.21GiB (99.61%) /dev/mapper/vgu2nvme-lvvm 709.01GiB Metadata,DUP: Size:1.00GiB, Used:926.33MiB (90.46%) /dev/mapper/vgu2nvme-lvvm 2.00GiB System,DUP: Size:32.00MiB, Used:144.00KiB (0.44%) /dev/mapper/vgu2nvme-lvvm 64.00MiB Unallocated: /dev/mapper/vgu2nvme-lvvm 288.93GiB

So not perfect (still about 4% of lost space is not recoverd) but the best I achieve to do !

NB: All those operations were done online with filesystem mounted and with about ~ 20 running VM on it Maybe the only way to recover the last 4% would be to do a cold copy of data to another newly BTRFS formated filesystem (= stopping ~20 production VM and doing cp -a )...

So if someone know how to recover the last 4% lost space without copying the data to another filesystem, it would help a lot

2025-02-12 EDIT Using the 'new cache' option of btrfs (space_cache=v2) did recover the 4% of lost space: mounting the filesystem with option : ssd,discard=async,space_cache=v2

mount -o ssd,discard=async,space_cache=v2 /dev/vgu2nvme/lvvm /mnt/vm

Now the difference is less than 0.5% :

$ df /mnt/vm/ Sys. de fichiers Type Taille Utilisé Dispo Uti% Monté sur /dev/mapper/vgu2nvme-lvvm btrfs 1,0T 776G 248G 76% /mnt/vm $ du -sh /mnt/vm/ 772G /mnt/vm/ $ sudo btrfs fi du -s /mnt/vm Total Exclusive Set shared Filename 771.85GiB 771.85GiB 0.00B /mnt/vm

And output of btrfs fi usage confirm it:

$ sudo btrfs fi usage /mnt/vm/ Overall: Device size: 1.00TiB Device allocated: 780.07GiB Device unallocated: 243.93GiB Device missing: 0.00B Device slack: 0.00B Used: 774.75GiB Free (estimated): 247.33GiB (min: 125.37GiB) Free (statfs, df): 247.33GiB Data ratio: 1.00 Metadata ratio: 2.00 Global reserve: 512.00MiB (used: 0.00B) Multiple profiles: no Data,single: Size:776.01GiB, Used:772.60GiB (99.56%) /dev/mapper/vgu2nvme-lvvm 776.01GiB Metadata,DUP: Size:2.00GiB, Used:1.07GiB (53.57%) /dev/mapper/vgu2nvme-lvvm 4.00GiB System,DUP: Size:32.00MiB, Used:128.00KiB (0.39%) /dev/mapper/vgu2nvme-lvvm 64.00MiB Unallocated: /dev/mapper/vgu2nvme-lvvm 243.93GiB
Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.