Timeline for Is there a way to determine the optimal value for the bs parameter to dd?
Current License: CC BY-SA 4.0
24 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Nov 2, 2021 at 8:57 | comment | added | Philip Couling | @Gilles'SO-stopbeingevil' the different syscall makes more sense to me than discussion of tuning read/write sizes especially with sizes >= 1mb. I can imagine that boxing up the data to pass to userland and visa-versa comes with overheads (extra copying?). In any case, thanks for the follow up. | |
| Nov 2, 2021 at 8:34 | comment | added | Gilles 'SO- stop being evil' | @PhilipCouling Also, cp and cat in modern Busybox (≥1.23, 2014 release) uses the sendfile system call, and the latest coreutils (≥9.0, Sep 2021 release) uses copy_file_range on Linux. Both give the kernel considerable opportunity to pick optimal buffer sizes and to interleave device reads and writes in an optimal way. | |
| Oct 29, 2021 at 14:25 | comment | added | Gilles 'SO- stop being evil' | @PhilipCouling I think it's due to the size and sequencing of read and write calls. dd forces a read of bs bytes followed by a write of bs bytes. Other tools can pick more optimal sequences if they want. But I don't know what cat and cp actually do. | |
| Oct 29, 2021 at 14:13 | comment | added | Philip Couling | The more times I return to this answer the more I question why cat or cp would be faster thandd; what's going on in userland or kernalland to make this happen. Is dd using a different syscall to write the data? If not then is there any possibility the benchmark is being unfair? Could dd be executing an fsync() that's not performed by cat? | |
| Jan 19, 2021 at 11:09 | comment | added | Gilles 'SO- stop being evil' | @Cadoiz The commands shown, plus the filename arguments or redirections. In cases where both a file name and redirection could be used (e.g. cat input), I don't remember which one was used, but it doesn't make any difference anyway (they don't use different open flags or anything that could affect the speed of the copy). | |
| Jan 19, 2021 at 7:50 | comment | added | Cadoiz | Which commands were used exactly for dd, cp and cat? | |
| May 9, 2020 at 18:03 | comment | added | Silv | @Gilles'SO-stopbeingevil' Thank you! I didn't expect you to remember what you were about 3 years ago. :) That's a useful reading (along with a couple of other SO questions I encountered reading it). | |
| May 9, 2020 at 11:27 | comment | added | Gilles 'SO- stop being evil' | @Silv unix.stackexchange.com/questions/32988/… | |
| May 8, 2020 at 23:24 | comment | added | Silv | PS. I've looked in the internet for dd being risky in this context, but with no luck. | |
| May 8, 2020 at 23:19 | comment | added | Silv | @Gilles'SO-stopbeingevil' If you remember, can you please explain what do (did, could) you mean by "risk of skipping data in the middle" using dd? | |
| Jan 29, 2020 at 12:33 | history | edited | Jonas Stein | CC BY-SA 4.0 | improve readability |
| May 6, 2017 at 22:12 | comment | added | sudo | @Gilles Ah, so if you don't know what bs to use, I now see that cat picks automatically based on the system buffer and filesystem block sizes. I still see dd perform much better in certain cases if you're careful, but I can see why you'd use cat. | |
| May 6, 2017 at 21:22 | comment | added | Gilles 'SO- stop being evil' | @sudo cat is faster if you don't get the bs parameter right, and unlike using dd with the bs parameter, there's no risk that it'll skip data in the middle, | |
| May 6, 2017 at 20:57 | comment | added | sudo | @Gilles In my experience, pv significantly bottlenecks the copy, so I only use it for slower things like network transfers. And how is cat better than dd at this point if it's not any faster, and you have to go manually calculate the throughput? | |
| May 6, 2017 at 18:36 | comment | added | Gilles 'SO- stop being evil' | @sudo You can get the amount of data copied with lsof. Instant speed isn't very relevant with a disk copy because it's uniform so you can divide bytes transferred by elapsed time; if you want something better, you can use pv. | |
| May 6, 2017 at 1:47 | comment | added | sudo | Well, dd reports the amount of data copied and the speed both during and after the copy, while cat does not, so it's a lot nicer when you're trying to clone a disk. | |
| May 11, 2016 at 12:52 | comment | added | Gilles 'SO- stop being evil' | @Masi cat just copies its input to its output. If you want to copy from unreliable media, and skip over unreadable parts or retry multiple times, that's a different problem, for which ddrescue works pretty nicely. | |
| May 11, 2016 at 12:26 | comment | added | Léo Léopold Hertz 준영 | @Gilles I am just a newbie who have HDDs which have some broken sectors so worried about copying with them. So what you propose is cat /media/HDD1 > /media/HDD2. Is there any limitation about Size? HDD1 is always smaller/equal than HDD2, which I think must be maintained. | |
| May 11, 2016 at 11:05 | comment | added | Gilles 'SO- stop being evil' | @Masi Yes, if I want to clone a whole disk, I'll use cat. Why are you looking for a better way? What's wrong with cat? | |
| May 11, 2016 at 8:51 | comment | added | Léo Léopold Hertz 준영 | @Gilles Would you cat whole HDD of 500GB too in cloning? Any better way? | |
| Dec 23, 2011 at 21:29 | comment | added | Gilles 'SO- stop being evil' | @Nikhil >| is the same as > except that under set -o noclobber, the shell will complain that the file exists if you use >. | |
| Dec 23, 2011 at 14:09 | comment | added | Nikhil Mulley | not in the context of the topic, @Gilles, what is the difference in echo 1 >|/proc/sys/vm/drop_caches and echo 1 > /proc/sys/vm/drop_caches | |
| Mar 17, 2011 at 23:00 | comment | added | ninjalj | I'd recommend the OP to do his own benchmarking, but anyway, nice answer! | |
| Mar 17, 2011 at 22:42 | history | answered | Gilles 'SO- stop being evil' | CC BY-SA 2.5 |