9

I need to copy one very large file (3TB) on the same machine from one external drive to another. This might take (because of low bandwidth) many days.

So I want to be prepared when I have to interrupt the copying and resume it after, say, a restart. From what I've read I can use

rsync --append

for this (with rsync version>3). Two questions about the --append flag here:

  1. Do I use rsync --append for all invocations? (For the first invocation when no interrupted copy on the destination drive yet exists and for the subsequent invocations when there is an interrupted copy at the destination.)

  2. Does rsync --append resume for the subsequent invocations the copying process without reading all the already copied data? (In other words: Does rsync mimic a dd-style seek-and-read operation?)

Improve this question

2 Answers 2

Reset to default
9

Do I use rsync --append for all invocations?

Yes, you would use it each time (the first time there is nothing to append, so it's a no-op; the second and subsequent times it's actioned). But do not use --append at all unless you can guarantee that the source is unchanged from the previous run (if any), because it turns off the checking of what has previously been copied.

Does rsync --append resume for the subsequent invocations… without reading all the already copied data?

Yes, but without rsync --partial would probably have first deleted the target file.

The correct invocation would be something like this:

rsync -a -vi --append --inplace --partial --progress /path/to/source/ /path/to/target

You could remove --progress if you didn't want to see a progress indicator, and -vi if you are less bothered about a more informational result (you'll still get told if it succeeds or fails). You may see -P used in other situations: this is the same as --partial --progress and can be used for that here too.

  • --append to continue after a restart without checking previously transferred data
  • --partial to keep partially transferred files
  • --inplace to force the update to be in-place

If you are in any doubt at all that the source might have changed since the first attempt at rsync, use the (much) slower --append-verify instead of --append. Or better still, remove the --append flag entirely and let rsync delete the target and start copying it again.

Improve this answer
3
  • "because it turns off the checking of what has previously been copied" -- Rather than turning off, isn't it because it checks by way of filesize? I would have expected that changes to either destination or source wouldn't cause corruption with --append if you use it with -c/--checksum. EDIT: Indeed it turns it off instead of checksumming to the minimum between the 2 sizes. That's a shame. Commented Sep 7, 2022 at 22:20
  • Apparently for --append to behave like --append-verify when used with -c. I hadn't noticed there was that separate option. EDIT: Yes. Commented Sep 7, 2022 at 22:27
  • Cool answer! Clears up all my questions in great detail. Thanks. Commented Sep 8, 2022 at 0:22
0

For this you'll want to use the --partial option. From the manpage:

--partial keep partially transferred files
Improve this answer
3
  • 1
    also --inplace option might be needed. --inplace This option changes how rsync transfers a file when its data needs to be updated: instead of the default method of creating a new copy of the file and moving it into place when it is com‐ plete, rsync instead writes the updated data directly to the destination file. Commented Sep 7, 2022 at 14:20
  • 1
    Tahnks for chiming in! - So I would need --partial --inplace for both, first and subsequent invocations? Commented Sep 7, 2022 at 14:35
  • I'd also care about the advice given 10 years ago. Which version of rsync in these times ? They possibly meant what more recent rsync would name append-verify. Anyway, I always use append-verify Commented Sep 7, 2022 at 14:39

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.