5

I have some files on remote host (in a directory) and I want to perform rsync in an atomic manner at directory level to pull files on local host (In a distributed setup). One way I could think of is a very trivial case when I can take files backup on local host and then replace the old files with the new files, but the approach is not efficient as far as disk space is concerned. e.g. files size is 10GB and diff is just 100 MB.

Is there a way to store just the rsync diff on local host in temporary location and then update the files on local host?

Improve this question

1 Answer 1

Reset to default
6

You could do it like this:

  1. Run rsync between local host and a temp folder in remote host. To make sure you only get the diff, use the --link-dest option and link to the real folder in remote host.

    You'd basically have a command like this:

    rsync --link-dest="/var/www" --archive "/localhost/path/www/" "[email protected]:/var/www_update_20131129"

    (With /var/www being the files to update and /var/www_update_20131129/ being the "temp" folder)

  2. Once the rsync operation is done, you can swap the www_update_20131129/ and real www/ folders in remote host (possibly by soft-linking www/ to www_update_20131129/).

Improve this answer
Sign up to request clarification or add additional context in comments.

4 Comments

I think you didn't get my question. My main concern is the temporary disk storage along with atomicity. I want something like I should be able to save just the diffs of the files in temp folder and after all the processing, somehow apply that diff to old files to update them. e.g. my host has just 5 GB of free space and files to be updated (in atomic way at directory level) have size 10GB with a diff of just 100MB.
@I_am_spiderman, do you mean some individual files take about 10GB, or is it the total size that is 10GB?
Let say, total size of all files is 10 GB
@I_am_spiderman, in that case, the solution I described would work. If only 100 MB have changed, then only 100 MB will be transferred, and the available space will only decrease by 100MB on the remote host. That's because the new folder "www_update_20131129" will be hard-linked (thanks to the link-dest option) to the previous one "/var/www". Once everything's transferred, you can delete the old files since you'll have the hard-linked copies in the new folder. You can test this quite easily actually. If you run rsync with --verbose, it will show what's actually been transferred.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.