Is there a way to monitor progress of a btrfs rebalance?

sudo btrfs fi show 

this will output something like so:

Label: none uuid: 2c97e7cd-06d4-4df0-b1bc-651397edf74c Total devices 16 FS bytes used 5.36TiB devid 1 size 931.51GiB used 770.48GiB path /dev/sdc devid 2 size 931.51GiB used 770.48GiB path /dev/sdg devid 3 size 931.51GiB used 770.48GiB path /dev/sdj devid 4 size 0.00 used 10.02GiB path devid 5 size 931.51GiB used 770.48GiB path /dev/sdh devid 6 size 931.51GiB used 770.48GiB path /dev/sdi devid 7 size 931.51GiB used 770.48GiB path /dev/sdd devid 8 size 931.51GiB used 770.48GiB path /dev/sdo devid 9 size 465.76GiB used 384.31GiB path /dev/sdn devid 10 size 931.51GiB used 770.48GiB path /dev/sdp devid 11 size 931.51GiB used 770.48GiB path /dev/sdr devid 12 size 931.51GiB used 770.48GiB path /dev/sdm devid 13 size 931.51GiB used 769.48GiB path /dev/sdq devid 14 size 931.51GiB used 770.48GiB path /dev/sdl devid 15 size 931.51GiB used 770.48GiB path /dev/sde devid 16 size 3.64TiB used 587.16GiB path /dev/sdf Btrfs v3.12 

And if you notice that device id #4 looks a little bit different than the rest. when you do "btrfs device delete missing /mntpoint" then it will start to regenerate the raid meta/data necessary to free up that "missing" drive.

if you do something like

"watch -n 10 sudo btrfs fi show" 

then you can see the space on the offending "missing" device gradually getting smaller and smaller until the operation completes and it will be removed from the fi.

BTRFS may take some time reading or rearranging data prior to writing data to the drive you expect it to write to.

You can see how much CPU time is being devoted to BTRFS operations including rebalance, add, delete, convert, etc:

ps -ef | grep btrfs 

To see how busy each drive is, install sysstat, and run:

iostat 

Add some options to make iostat show stats in megabytes and update every 30 seconds:

iostat -m -d 30 

Sample output from scrub so no writes during this interval:

Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn sda 700.30 170.10 0.00 6804 0 sdb 0.78 0.00 0.01 0 0 sdc 520.20 127.98 0.00 5119 0 sdd 405.72 92.02 0.00 3680 0 sde 630.05 153.66 0.00 6146 0 sdf 627.43 153.60 0.00 6144 0 

Install and run munin to see historical graphs of drive activity and lots of other info. https://www.digitalocean.com/community/tutorials/how-to-install-the-munin-monitoring-tool-on-ubuntu-14-04

I was also wondering when a long lasting delete would finish so I came up with this little piece of shell code:

get_bytes() { btrfs device usage --raw /mnt/data | egrep -- '-[0-9]+' | sed -E 's/[^0-9]+([0-9]+)/\1/' } prev=$(get_bytes) while [ 1 ]; do current=$(get_bytes) diff=$((current-prev)) if [ "$diff" -gt 0 ]; then dd if=/dev/zero iflag=count_bytes count="$diff" 2>/dev/null fi prev="$current" sleep 1 done | pv -petraW -s $(get_bytes) >/dev/null 

This will give you a nice progress bar like this:

0:13:54 [0,00 B/s] [16,0MiB/s] [> ] 1% ETA 19:23:19 

The general idea is to use pv to display progress. Since that command only allows to monitor bytes flowing through a pipe we use dd to generate an appropriate amount of zeros and pipe them into pv.

The advantage of this method is that you get a nice progress bar. However, since it seems btrfs always deletes data one GB at a time it takes some time until a new difference in byte sizes can be observed.

To address this issue the flag -a is added to the default flags of pv to make it display an average transmission rate (since the normal current transmission rate will be 0 most of the time).

I realize this is not the best solution but the best I could come up with. If someone has ideas for improvements please let me know! :)

For simple "monitoring of command's output" you can use watch (as it works almost everywhere), for example running as root:

watch -n1 btrfs balance status /

(The switch -n1 means "wait 1 second and then run the command again".)