0

Progress and estimated time to write without and with sync

I have found no tool (or straightforward method) that will include flushing the buffers when showing the progress and estimating the estimated time for the whole write process, ETA (Estimated Time of Arrival).

  • pv can show the time for the progress as seen by the operating system, but if the target drive is slow and there is a lot of RAM, it shows only the time until the data are written to a buffer. This time can be a small fraction of the the real time until the buffers are flushed.

  • dd writes a final report about amount of data used time and transfer rate. It can also be made to write 'progress' reports. It used to give a much better estimate than pv, but nowadays the USB drives and memory cards are still very slow, while the other processes are fast and the available memory for buffers big. So dd will also finish long before the buffers are flushed.

  • I can 'time' the write process including sync with the time command

    time ( write command; sync )

    and it will give me the real time used which is useful, but only after it has finished. It does not show the progress and does not estimate the total remaining time.

  • I can run iotop to show read and write processes and how fast things are read and written, but it does not estimate the remaining time.

How to show progress and estimated time for the whole write process?

How can I show progress and estimated time for the whole write process, ETA (Estimated Time of Arrival), including flushing the buffers with sync?

Link to related question

Improve this question
4
  • 1
    You could install and use dcfldd instead of using dd. It works exactly like dd on its paremeters but with the addition that it has a progress bar while the writing command is executing... Commented Nov 26, 2019 at 18:23
  • 1
    I believe you could read (periodically) the dirty / writeback (don't know which one at this very moment), here's what I'm watching the flushing process: alias sync-watch='watch grep -e Dirty: -e Writeback: /proc/meminfo'. I think that could help you as it's the raw information you need. Commented Nov 26, 2019 at 18:29
  • @RafaelMuynarsk, Thanks for the tip :-) I will look into it. Commented Nov 26, 2019 at 19:36
  • @LinuxSecurityFreak, Please write an answer. I would like to upvote and accept an answer by you :-) Commented Nov 27, 2019 at 13:53

1 Answer 1

Reset to default
3

Shellscript

Kudos to @LinuxSecurityFreak for the advice to use the amount of 'Dirty' data reported in /proc/meminfo.

I made the following shellscript flusher. It shows progress and estimated time to flush the buffers. It can be used for example after cloning from an iso file to a USB drive or memory card in order to create a live drive with a linux operating system.

#!/bin/bash timeorig=$(date '+%s') deltat=5 # checking time interval dirtorig=$(grep -e 'Dirty:' /proc/meminfo | tr -s ' ' '\t' | cut -f2) dirt0=$dirtorig echo -n "dirty = $dirt0 kB - before sync" sync & spid=$! while ps -A|grep "$spid" > /dev/null do sleep "$deltat" dirty=$(grep -e 'Dirty:' /proc/meminfo | tr -s ' ' '\t' | cut -f2) deltad=$((dirt0-dirty)) if [ $deltad -gt 0 ] then eta="$((dirty*deltat/deltad)) s" rate="$(((deltad+500)/deltat/1000)) MB/s" else eta="n.a." rate="n.a." fi echo -en "\0033[2K\0033[1G" echo -n "dirty = $dirty kB -- syncing -- rate = $rate -- eta = $eta" dirt0="$dirty" done echo -e "\0033[2K\0033[1GDone syncing :-)" timefinal=$(date '+%s') timeused=$((timefinal-timeorig)) if [ $timeused -gt 0 ] then rate="$(((10*dirtorig+5)/timeused/10))" if [ $rate -ge 10000 ] then rate="$(((dirtorig+500)/timeused/1000)) MB/s" else rate="$rate kB/S" fi else rate="n.a." fi echo "syncing time = $timeused s -- rate = $rate"

Demo examples

Cloning to a slow drive (USB 2)

$ sudo dd if=xubuntu-18.04.1-desktop-amd64.iso of=/dev/sde bs=1M ; ./flusher [sudo] password for sudodus: 1367+1 posts in 1367+1 posts ut 1434386432 bytes (1.4 GB, 1.3 GiB) copied, 0.408724 s, 3.5 GB/s

Output from flusher:

dirty = 840600 kB -- syncing -- rate = 5 MB/s -- eta = 156 s ... Done syncing :-) syncing time = 302 s -- rate = 4639 kB/S

Cloning to a fast drive (eSATA)

$ sudo dd if=xubuntu-18.04.1-desktop-amd64.iso of=/dev/sda bs=1M ; ./flusher 1367+1 posts in 1367+1 posts ut 1434386432 bytes (1.4 GB, 1.3 GiB) copied, 0.404442 s, 3.5 GB/s

Output from flusher:

dirty = 727508 kB -- syncing -- rate = 59 MB/s -- eta = 12 s ... Done syncing :-) syncing time = 25 s -- rate = 56 MB/s

Edit:

There is an updated version of flusher, called watch-flush, in the current version in mkusb. It can be used separately from an own window via the following alias,

alias wf='xterm -title "watch-flush" -fa default -fs 13 -fg yellow -bg "#504030" -geometry 70x7 -e bash -c "watch-flush;read -n1" 2> /dev/null'
Improve this answer
1
  • 1
    Sorry for late reply. I managed to log in just now, was away the whole day, perfect answer. Commented Nov 27, 2019 at 14:17

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.