0

I just copied 1TB of files, zillions of them, from one external USB disk to another and it took 2 days over 5GBPS USB.

I have used rsync.

rsync seems to me to be serial, I mean, one file at a time.

I have confirmed that by looking at the traffic on the disk tab of activity monitor and the activity there was relatively low.

Is there some terminal app or some trick I can use to copy multiple files at a time and speed the process?

Improve this question
6
  • 3
    There's nothing stopping you from starting several rsync processes, as long as you ensure they don't try to copy overlapping sets of files. However, they would still use the same singular data bus, so speedup would definitely not be linear. Commented Jun 20, 2022 at 13:03
  • Do you happen to be using rsync's compress option? If so, try running without it. In my experience it slows down the transfer process tremendously. Commented Jun 20, 2022 at 14:46
  • You need to find the actual bottleneck of the copy. You made sure, it's not disk performance, but is it USB bandwidth (overhead for extremely large number of small files?) or CPU performance (unlikely, but maybe you have some virus scanner checking each file before proceeding) or what else? Maybe it's worth to test with only some small part of the data and compare rsync with cp -R. Commented Jun 20, 2022 at 16:50
  • External USB drives can be very slow. Typically they have a cache which makes them appear fast for the first 100MB or so, but their sustained write speed will disappoint. The 5Gb/s is the bus speed. All that this tells you is that you are not limited by the bus bandwidth. With over 1TB in 48 hours you are getting about 10MB/s which is not unreasonable. Top of the range drives can get 10 times faster random write performance, bottom of the range can be 10 times slower. Does the external drive call itself and "SSD"? Commented Jun 20, 2022 at 16:59
  • not SSD. The external disks are, unfortunately, HDD. Commented Jun 21, 2022 at 8:11

1 Answer 1

Reset to default
1

If you do

ps -ef | grep 'rsync' | grep -v 'grep'

you should see something like this:

root 7520 7514 12 20:50 pts/0 00:05:46 rsync --checksum --one-file-system --recursive --outbuf=Line --links --perms --times --group --owner --devices --specials --verbose --out-format=%t|%i|%M|%b|%f| --delete-delay --whole-file --human-readable --protect-args --ignore-errors --msgs2stderr ./ /DB001_F7/ root 7514 7512 0 20:50 pts/0 00:00:25 rsync --checksum --one-file-system --recursive --outbuf=Line --links --perms --times --group --owner --devices --specials --verbose --out-format=%t|%i|%M|%b|%f| --delete-delay --whole-file --human-readable --protect-args --ignore-errors --msgs2stderr ./ /DB001_F7/ root 7512 1 17 20:50 pts/0 00:08:27 rsync --checksum --one-file-system --recursive --outbuf=Line --links --perms --times --group --owner --devices --specials --verbose --out-format=%t|%i|%M|%b|%f| --delete-delay --whole-file --human-readable --protect-args --ignore-errors --msgs2stderr ./ /DB001_F7/"

In other words, had 3 processes running in parallel for file copying, maybe more if you have more than 4 CPUs (what I have in my core).

As Kefka said, compression option maybe forced the serialization of backup data into a single data stream, and that too would slow things down, on top of the CPU utilization for the compression.

For time consumption, there are a few modes that change the behaviour. Specifically, if you are using --checksum for every backup, backup times take up to 2.5 hours for a 250 GB partition, vs 15-20 minutes on what I call updates (cumulative delta from last full backup). Doing a full image copy (empty destination directory; copy everything) without --checksum would take about 1-1.5 hours, just because of the sheer volume of data.

My own backup device is via as well, a WD 4TB MyBook. I had to tweak the because of my physical configuration. Your own hardware configuration may need tweeking as well if the USB is slower than expected. This is what I have:

# quirk specification is to suppress UAS for external USB3 drive operating on USB2 channel. GRUB_CMDLINE_LINUX_DEFAULT="quiet splash scsi_mod.use_blk_mq=1 usb-storage.quirks=1058:25ee:u ipv6.disable=1"

If you are unsure of the performance hit using your USB drive, you can try, firstly a tar command of an ISO image from internal hard disk to another location on your internal hard disk, then tar that same ISO to a location on your USB drive to see if there is much difference in speed. Use the date command before and after to get a sense of the time usage for both scenarios. Then try doing rsync for that same ISO image, using your normal options, and see if there is much difference in time for completing that transfer to your USB.

I would try that rsync with the single ISO for a few different option for your rsync until you find what works best for you, balancing integrity of data at destination, with job duration for backups with rsync. Only once that best combination is found should you move on to directory, disk or system backups, at which point you have assurance that your transfer were completed as expected.

Improve this answer
2
  • thanks!!!!!!!!! Commented Sep 25, 2022 at 14:10
  • @Duck, just a quick comment. if this answer provides what you needed to know, I would encourage you to click on the checkmark next to the question to indicate that it is the answer that addresses the needs in your question. That way, other users will know to give it a closer review for their own needs. Commented Sep 25, 2022 at 19:47

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.