2

I want to rsync the latest two files of a certain pattern between two servers in different environments:

drwxrwxr-x 2 transfer transfer 4096 28. Nov 13:05 . drwxr-xr-x 7 transfer transfer 4096 28. Nov 13:05 .. -rw-rw-r-- 1 transfer transfer 11151643678 28. Nov 12:46 test_28-11-2023.dmp.gz -rw-rw-r-- 1 transfer transfer 29789 28. Nov 13:04 test_28-11-2023.log -rw-r--r-- 1 transfer transfer 10770251760 28. Nov 13:00 test_anonym_19-11-2023.dmp.gz -rw-r--r-- 1 transfer transfer 48408 28. Nov 13:02 test_anonym_19-11-2023.log -rw-r--r-- 1 transfer transfer 9329096219 28. Nov 13:03 test_anonym_21-11-2023.dmp.gz -rw-r--r-- 1 transfer transfer 48407 28. Nov 13:03 test_anonym_21-11-2023.log

Since only want to transfer the latest *.gz and *.log of the pattern \.\/test_anonym_.*\.* I can do something like this to identify them:

user@host:~/folder$ find . -type f -regex '\.\/test_anonym_.*\.*' | sort -r | head -n 2 ./test_anonym_21-11-2023.log ./test_anonym_21-11-2023.dmp.gz

Or directly identify them on the remote with ssh like this (which works fine):

user@host:~/folder$ ssh -qx user@host "ls -t /home/user/folder | sort -r | head -n 2" test_anonym_21-11-2023.log test_anonym_21-11-2023.dmp.gz

The following rsync command transfers too much because I am only using a pattern of one file type and not identifying the latest files of both types:

user@host:~/folder$ rsync -av --stats --include='test_anonym_*.gz' --exclude='test_*.*' user@host:/home/user/folder/ /home/user/folder/ test_anonym_19-11-2023.dmp.gz test_anonym_21-11-2023.dmp.gz

I tried using ssh and rsync together after sifting through many man pages and posts like this, but it fails in the ssh step which is never completed:

ssh user@host "ls -t /home/user/folder/ | sort -r | head -n 2" | rsync -0 --stats --files-from=- -av user@host:/home/user/folder/ /home/user/folder/

I tried using rsync with --files-from= option but have also failed because I am unsure of the syntax here in my two examples:

rsync -a --stats --files-from=<(ssh user@host 'find . -type f -regex '\./test_anonym_.*\.*' | sort -r | head -n 2') --exclude='test_*.*' user@host:/home/user/folder/ /home/folder/folder/ rsync -a --stats --files-from=<(find . -type f -regex '\.\/test_anonym_.*\.*' | sort -r | head -n 2) --exclude='test_*.*' user@host:/home/user/folder/ /home/folder/folder/

I'm not sure how to put it all together (or if it's even possible) to find and sync the desired files; I'm really stuck now. How can I make this work? My workstation uses zsh, my remotes all use bash.

To reiterate my expectations here: I want to rsync (with/without ssh) two files of a different type from a remote, i.E. test_anonym_21-11-2023.log and test_anonym_21-11-2023.dmp.gz witch match my pattern and are the latest two files of their type.

Improve this question
0

3 Answers 3

Reset to default
0

If you have GNU tools you can use find to identify the two most recently modified files and transfer just those. The process is complicated a little by the need to pull the files from a remote host, but that just means ssh gets more involved.

If we can assume that the two most recently modified files are the pair of files to be transferred, and that we have tools with GNU extensions (or similar):

ssh -qn remoteHost "find folder -name 'test_anonym_*' -printf '%T@ %p\0'" | sort -z -k1,1rn | head -z -n2 | cut -z -d' ' -f2- | rsync --dry-run -av --files-from - --from0 remoteHost: "$HOME/folder"

If you do not have GNU tools you have to be careful of unexpected file names but this pipeline will work provided you have control of the file names placed in folder:

ssh -qn remoteHost ls -t folder | head -n2 | rsync --dry-run -av --files-from - --from0 remoteHost:folder/ "$HOME/folder"

In both cases remove --dry-run when you're happy the code is doing what you expect.

If you can only guarantee (say) the dmp.gz file and you want to pull the associated log file, that gets more complex. But that doesn't seem to have been required by your question.

Improve this answer
1
  • Thanks for the answer, I'll give it a test under Debian and update the question need be. Commented Dec 18, 2023 at 8:24
0

If the login shell of the target user on remoteHost is zsh, that should just be matter of:

rsync --old-args 'remoteHost:folder/test_anonym*.(log|gz)(.om[1,2])' ~/folder/

With --old-args, needed in newer versions of rsync where that behaviour is no longer the default, the folder/test_anonym*.(log|gz)(.om[1,2]) is passed as-is in the remote shell command line that invokes rsync --server, and the shell will expand that glob which here matches on the two most recent files in folder whose name starts with test_anonym and ends in either .gz or .log.

You can find the difference in behaviour when using --old-args and not and when using --protect-args by running rsync under strace -e execve:

$ strace -fe execve rsync --old-args 'remoteHost:folder/test_anonym*.(log|gz)(.om[1,2])' ~/folder/ [...] [pid 8483] execve("/bin/ssh", ["ssh", "remoteHost", "rsync", "--server", "--sender", "-e.LsfxCIvu", ".", "folder/test_anonym*.(log|gz)(.om"...], 0x7ffceb4e2cf0 /* 52 vars */) = 0 [...]

Arg passed unmodified.

$ strace -fe execve rsync 'remoteHost:folder/test_anonym*.(log|gz)(.om[1,2])' ~/folder/ [...] [pid 8489] execve("/bin/ssh", ["ssh", "remoteHost", "rsync", "--server", "--sender", "-e.LsfxCIvu", ".", "folder/test_anonym*.\\(log\\|gz\\)\\"...], 0x7ffc5bced518 /* 52 vars */) = 0 [...]

Some characters were escaped with \, whether that's a valid thing to do for the remote shell or not.

$ strace -fe execve rsync --protect-args 'remoteHost:folder/test_anonym*.(log|gz)(.om[1,2])' ~/folder/ [...] [pid 8507] execve("/bin/ssh", ["ssh", "remoteHost", "rsync", "--server", "--sender", "-se.LsfxCIvu"], 0x7ffc404a9900 /* 52 vars */) = 0 [...]

The arg is not passed in the shell command line but inband in the newer rsync protocol. In general, that's the safest, but that wouldn't allow us to use advanced globbing operators of your shell.

If zsh is installed on the remote host, but the login shell of the user is bash instead, you can convince bash there to get zsh to run the rsync server command using a trick like:

rsync --rsync-path='zsh -c "${BASH_EXECUTION_STRING#*\#\ }" # rsync' \ --old-args 'remoteHost:folder/test_anonym*.(log|gz)(.om[1,2])' ~/folder/

Where bash on the remote host will be asked to evaluate:

zsh -c "${BASH_EXECUTION_STRING#*\#\ }" # rsync --server --sender -e.LsfxCIvu . folder/test_anonym*.(log|gz)(.om[1,2])

Where the whole rsync --server... command is just treated as a comment.

$BASH_EXECUTION_STRING is automatically set by bash to the inline script being executed, and we strip everthing from it up to the first occurrence of # , so zsh will get to interpret:

rsync --server --sender -e.LsfxCIvu . folder/test_anonym*.(log|gz)(.om[1,2])

If you don't need the syncing capability of rsync but just need to transfer those files, it may be easier just to do:

ssh remoteHost zsh << 'EOF' | (cd ~/folder && tar zxpvf -) cd folder && tar zcf - test_anonym*.(log|gz)(.om[1,2]) EOF
Improve this answer
1
  • Thanks for the in depth answer, I'll give it a test under Debian and comment accordingly. Commented Dec 18, 2023 at 8:25
0

My original question:

"I want to rsync the latest two files of a certain pattern between two servers in different environments."

My (transfer) host server with the latest files:

drwxrwxr-x 2 transfer 4,0K 26. Mär 13:30 . drwxr-xr-x 6 transfer 4,0K 20. Mär 11:01 .. -rw-rw-r-- 1 transfer 0 26. Mär 13:30 test_26-03-2024.dmp.gz -rw-rw-r-- 1 transfer 0 26. Mär 13:30 test_26-03-2024.log -rw-rw-r-- 1 transfer 0 28. Nov 12:46 test_28-11-2023.dmp.gz -rw-rw-r-- 1 transfer 21 28. Nov 13:04 test_28-11-2023.log -rw-rw-r-- 1 transfer 0 26. Mär 13:30 test_anonym_17-03-2024.dmp.gz -rw-rw-r-- 1 transfer 0 26. Mär 13:30 test_anonym_17-03-2024.log -rw-r--r-- 1 transfer 0 28. Nov 13:00 test_anonym_19-11-2023.dmp.gz -rw-r--r-- 1 transfer 5 29. Jan 13:35 test_anonym_19-11-2023.log -rw-r--r-- 1 transfer 0 28. Nov 13:03 test_anonym_21-11-2023.dmp.gz -rw-r--r-- 1 transfer 497 29. Jan 13:36 test_anonym_21-11-2023.log -rw-rw-r-- 1 transfer 0 29. Jan 13:38 test_anonym_29-01-2024.dmp.gz -rw-rw-r-- 1 transfer 201 29. Jan 13:37 test_anonym_29-01-2024.log user@transfer:~/folder$

The solution to my issue:

I identified the two latest files of the naming pattern test_anonym_*.* and the file types .log and .dmp.gz from my (devops) other server and save the output as a text file:

user@devops:~$ ssh -qx user@transfer 'ls -tr /home/user/folder | sort -k 8,8n -k 6,6M | grep test_anonym_* | head -n 2' > output.txt ... user@devops:~$ nl output.txt 1 test_anonym_17-03-2024.dmp.gz 2 test_anonym_17-03-2024.log

I can now rsync the files contained in my text file to my other server:

user@devops:~$ rsync -arv --stats --include-from=output.txt --exclude=* user@transfer:folder/ ./folder/ user@transfer's password: receiving incremental file list ./ test_anonym_17-03-2024.dmp.gz test_anonym_17-03-2024.log ... sent 139 bytes received 211 bytes 100,00 bytes/sec total size is 0 speedup is 0,00

The correct files were transferred and my main issue was resolved:

user@devops:~$ ls -la folder/ drwxrwxr-x 2 user user 4096 26. Mär 13:30 . drwxr-xr-x 4 user user 4096 26. Mär 14:55 .. -rw-rw-r-- 1 user user 0 26. Mär 13:30 test_anonym_17-03-2024.dmp.gz -rw-rw-r-- 1 user user 0 26. Mär 13:30 test_anonym_17-03-2024.log

Nice to have: The solution would now be to combine the ssh and rsync command to one single command via piping or saving the ls, sort, grep and head to tmp from the --include-from= but no one seems to be able to offer support for this.

Improve this answer
1
  • "The solution would now be to combine the ssh and rsync command to one single command via piping" - what, like in my answer or Stéphane's? Commented Mar 27, 2024 at 9:37

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.