I need to transfer some files via rsync and can´t tar them before. So I thought maybe there is a mode which checks the folders content and if it is thousand of small files inside compress that folder for the transfer. However I couldn´t find something like that in the man (http://www.manpagez.com/man/1/rsync/). Is it possible and I didn´t find it or would I have to do this before I use rsync?
1 Answer
You'll typically use one of the following methods when you need to push or pull files over SSH using tar.
Copy SOURCEDIR from localhost to remotehost
Conceptually I'll often refer to this as pushing files, since we're taking the output of a tar command and we're pushing the files to another system over SSH. The files are targeted for copying on the remote system into DESTDIR.
$ tar zcvf - SOURCEDIR | ssh user1@remotehost 'cd DESTDIR; tar zxvf - ' $ tar zcvf - SOURCEDIR | ssh user1@remotehost "(cd DESTDIR; tar zxvf -)" $ tar zcvf - SOURCEDIR | ssh -l user1 remotehost 'cd DESTDIR ; tar zxvf -' Copy SOURCEDIR from remotehost to localhost
Conceptually I'll often refer to this as pulling files, since we're making an SSH connection to a remote host, where we're kicking off a tar command. The output of this tar command is then targeted for copying on the local system into DESTDIR.
$ ssh remotehost 'tar zcvf - SOURCEDIR' | tar zxvf - $ ssh -n remotehost 'tar zcvf - SOURCEDIR' | tar zxvf - $ ssh remotehost "( cd SOURCEDIR ; tar zcvf - SOURCEFILES ) " | tar zxvf - References
- does that work in readonly mode as pull ?user2693017– user26930172014-03-19 02:33:51 +00:00Commented Mar 19, 2014 at 2:33
- Do you mean as in a dry run mode?2014-03-19 02:41:08 +00:00Commented Mar 19, 2014 at 2:41
- well the acc I use to pull the data does not have write permission. What means dry run mode in this case?user2693017– user26930172014-03-19 02:42:59 +00:00Commented Mar 19, 2014 at 2:42
- 2@user2693017 - there is no
tarfile actually created, as the contents oftaris created, it's redirected via the pipe,|to another receivingtarwhich will uncompress it.2014-03-19 03:05:38 +00:00Commented Mar 19, 2014 at 3:05 - 1@user2693017 - from my blog post that I referenced:
tar zcvf - SOURCEDIR | ssh user1@remotehost "cat > /DESTDIR/DESTFILE.tar.gz".2014-03-26 22:34:34 +00:00Commented Mar 26, 2014 at 22:34
-zoption. If you're thinking about performance, whatever tool you'd be using to “compress them together” (I guess you want to archive them together and compress the archive) would have to read all the files anyway, and that's what rsync is doing, I don't see what additional steps could make the transfer faster.rsync, it just won't perform well for large numbers of small files. Why can't you use atarpipe overssh?