3

I have an ever growing remote directory of files that I'd like to sort and filter before I rsync them, with the goal of always keeping only the latest N files in my destination directory (i.e. a rotation scheme, more or less). Since rsync doesn't seem to have options for this, I've been using the technique of "inserting an arbitrary remote command", described here:

https://stackoverflow.com/q/950062/787842

with which I came up with this command, that I execute as a cron on the destination host:

rsync -vrzO --delete \ -e ssh <remote_host>:'$(cd <remote_dir> && ls -t $PWD/* | head -n 25)' \ <destination_dir>

This works well the first time (i.e. when the dest dir is empty): only the 25 freshest files get copied. But then when the window "slides forward" (i.e. when the arrival of a newer file should push an older one out, locally), the problem is that the --delete option doesn't work as I'd expect. My guess is that since the filtered out remote files still exist (i.e. they're just being temporarily hidden by the inserted '$(..)' command), then rsync simply cannot filter them out locally. Is my understanding correct, and is there a better way?

Improve this question
0

2 Answers 2

Reset to default
3

You can simulate your command as,

rsync -vrzO --delete -e ssh <remote_host>:'<remote_dir>/file1 <remote_dir>/file2' <destination_dir>/ i.e. substituting output of cd <remote_dir> && ls -t $PWD/* | head -n 2.

rsync's --delete works on directories and your command substitution is providing list of files. So --delete is not working.

Excerpt from man rsync:

--delete

This tells rsync to delete extraneous files from the receiving side (ones that aren’t on the sending side), but only for the directories that are being synchronized. You must have asked rsync to send the whole directory (e.g. "dir" or "dir/") without using a wildcard for the directory’s contents (e.g. "dir/*") since the wildcard is expanded by the shell and rsync thus gets a request to transfer individual files, not the files’ parent directory.

Improve this answer
2
  • I'm not sure I understand the first part of our answer: what do you mean by "simulating my command"? So the conclusion is that what I want to do (with a single rsync call) is simply impossible, is that it? Commented Nov 25, 2011 at 23:36
  • That means you can try your command by explicitly putting few file names in place of the command to be substituted which is giving you the list of latest files. Commented Nov 26, 2011 at 5:39
0

Mount the remote directory over sshfs. Then you can treat it as a local directory.

Now, to sort files by modification time, the easiest way is zsh's glob qualifiers. The 25 most recent files in a directory are

*(NDom[1,25])

So, if you want to copy first, then delete the extraneous files at the destination:

for x in source/*(NDom[1,25]); do y=destination/${x:t} if [[ ! -e $y || $x -nt $y ]]; then cp -p $x $y; fi done destination_files=(destination/*(NDom[1,25])) if [[ ${#destination_files} -gt 25 ]]; then shift $((${#destination_files} - 25)) destination_files rm $destination_files fi

Here's another method that deletes files before copying. This one deletes the oldest file when it's about to copy a new file and there is already a full complement at the destination.

destination_files=(destination/*(ND)); destination_count=$#destination_files for x in source/*(NDom[1,25]); do y=destination/${x:t} if [[ $destination_count -gt 25 && ! -e $y ]]; then rm destination/*(NDom[-1]); fi if [[ ! -e $y || $x -nt $y ]]; then cp -p $x $y; fi done

(Warning: the code above was typed directly into the browser, I did not test it.)

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.