I've copied a large file to a USB disk mounted on a Linux system with async. This returns to a command prompt relatively quickly, but when I type sync, of course, it all has to go to disk, and that takes a long time.
I understand that it's going to be slow, but is there somewhere where I can watch a counter go down to zero? Watching buffers in top doesn't help.
Looking at /proc/meminfo will show the Dirty number shrinking over time as all the data spools out; some of it may spill into Writeback as well. That will be a summary against all devices, but in the cases where one device on the system is much slower than the rest you'll usually end up where everything in that queue is related to it. You'll probably find the Dirty number large when you start and the sync finishes about the same time it approaches 0. Try this to get an interactive display:
watch -d grep -e Dirty: -e Writeback: /proc/meminfo With regular disks I can normally ignore Writeback, but I'm not sure if it's involved more often in the USB transfer path. If it just bounces up and down without a clear trend to it, you can probably just look at the Dirty number.
You can look at the /sys/block/<device>/stat file for the appropriate device while you're syncing. The 9th column will indicate the number of in-flight requests on the device, which should go down to zero when the sync is done.
Don't know of a way to translate that to a number of bytes, but it should give you a rough idea of how much "stuff" is still pending.
See the stat.txt file in the kernel documentation for a bit more information. (There's also an inflight file in that directory on my system which looks like it could contain read and write in-flight requests, but I can't find docs for that.)
watch -t -n1 'awk "{ print \$9 }" /sys/block/sdd/stat' sync afterwards. It does go to 0, but only at the very end. That makes the other answer more useful for impatiently watching progress. du to calculate the real data that is synced on drive. By using Greg's answer, you can simply have sync run in background while displaying the state of the Dirty block in memory.
To achieve this, simply run this command:
sync & watch -n 1 grep -e Dirty: /proc/meminfo This will call sync in the background while executing watch in the front. When the sync command will have finished (around when the size of the Dirty block has reached 0), you will have an output that looks like this :
1] + 27260 done sync This means that the command has finished and you can kill the watch command with Ctrl+C.
Based on the other answers I made this over-bloated one-liner that tracks summary progress as well as device-specific progress:
# Create the command watchSync() { watch -n1 'grep -E "(Dirty|Write)" /proc/meminfo; echo; ls /sys/block/ | while read device; do awk "{ print \"$device: \" \$9 }" "/sys/block/$device/stat"; done' } # Run the command watchSync The output looks like this:
Dirty: 848956 kB Writeback: 125948 kB WritebackTmp: 0 kB dm-0: 0 dm-1: 0 loop0: 0 loop1: 0 loop2: 0 loop3: 0 loop4: 0 loop5: 0 loop6: 0 loop7: 0 nvme0n1: 0 sda: 0 sdb: 0 sdc: 124 The above tells me that 1) I have a few 100,000 kBytes that need writing, and 2) the device being written to is sdc. Sync is complete when, in this case, sdc and Writeback hit zero (this will happen at the same time).
/etc/bash.bashrc or ~/.profile. That way, you don't need to copy-paste this into your terminal every time you need it. You save the function once, and then whenever you run watchSync it'll just work™. In Bash et al. a function actually just means "custom command that isn't an application." ls/while read/awk part can be replaced with a single grep invocation: grep -Po '^(?=( +\d+){9})( +\d+)' /sys/block/*/stat. You can watch current block device I/O traffic with nmon, as in:
NMON=ld nmon -s1 (this preselects display of load graph and disk graph with 1 second refresh time.. this can also be set by starting nmon without parameters, than pressing l, d, - to toggle graphs and reduce refresh time from default 2s .. + of course would increase refresh delay.)
I wrote a script for this based on the answers here:
watchSync
#!/bin/sh # Output the `Dirty` section of `/proc/meminfo` report() { local dirty="$(grep -w Dirty: /proc/meminfo | cut -d ':' -f 2 | tr -d ' ')" echo -n -e "\e[2K\rSyncing ${dirty}... " } report # Start syncing sync & SYNC_PID=$! # Give a short sleep in case there's not much sleep 0.125 # While sync is running, report while ps -p $SYNC_PID > /dev/null 2>&1; do report sleep 1 done echo "Done!" 
repr () { if [[ $# -gt 0 ]]; then local result="$(printf -- '%q ' "$@")" printf -- '%s' "${result::-1}" fi return $? } sync () { local args if [[ $- = *i* ]]; then args="$(repr "$@")" bash -sc "$(shopt -p); $(declare -f); ${FUNCNAME[0]} $args" return $? fi local initial_value local percentage local backspace='' local i command sync "$@" & initial_value="$(grep -e 'Dirty:' /proc/meminfo | awk '{print $2}')" if [[ $initial_value = 0 ]]; then initial_value=1; fi sleep 0.1 while ps -p $! > /dev/null; do percentage="$(((($initial_value-$(grep -e 'Dirty:' /proc/meminfo | awk '{print $2}'))*100)/$initial_value))" percentage="$(( $percentage < 0 ? 0 : $percentage ))%" printf -- '%b' "$backspace$percentage" backspace='' for (( i=1; i<=${#percentage}; i++ )); do backspace+='\b \b'; done sleep 1 done printf -- '%b' "$backspace" return $(wait $!) } If anyone is looking for any tools that automate rsync and sync monitoring afterwards. I've made one, based on others great answers, check it out:
https://github.com/satk0/usbdrivetools
Using the tool you can simply run:
usbcp /path/to/src /path/to/dest To have a source file copied to USB Drive mounted destination.
