5

I need to sync a directory to a directory on a remote server. at the moment i use rsync -ca to do this but the directory is quite large and the whole process fails with a timeout error.

Now I'm looking for a file by file sync instead of a 'all at once' approach. like find and for each found rsync to the server. What would be the most easy way to do this ?

I get the following error : 

receiving incremental file list Write failed: Broken pipe rsync: connection unexpectedly closed (1175057 bytes received so far) [receiver] rsync error: error in rsync protocol data stream (code 12) at io.c(601)[receiver=3.0.7] rsync: connection unexpectedly closed (287 bytes received so far) [generator] rsync error: unexplained error (code 255) at io.c(601) [generator=3.0.7]
Improve this question

4 Answers 4

Reset to default
5

It looks like the server times out on the SSH connection level.

So I added these settings:

KeepAlive yes ServerAliveInterval 20 ServerAliveCountMax 6

to the /etc/ssh/ssh_config (or ~/.ssh/config) and now it looks promising.

Improve this answer
3
  • 1
    It was definitely the problem of my ssh client timing out. after adding this to /etc/ssh/ssh_config the rsync stayed stable. Commented Mar 25, 2013 at 14:00
  • is ServerAliveInterval 20 mandatory here ? Commented Nov 26, 2015 at 0:45
  • Adding this to my ~/.ssh/config file solved the broken pipe error I was receiving with rsync. Thanks. Commented Sep 13, 2017 at 12:46
2

Update: Since the timeout appears to happen when rsync is creating the checksums, here are a few different approaches that may work.

First, to keep only one rsync open, you can (in addition to the options mentioned by Michael Kjörling) play around with the option --sockopts. That gives you the ability to set the same kind of socket options that you can set when creating a socket in code, e.g. SO_RCVTIMEOUT=0 to disable timeouts for receiving data.

Second, the --progress-file option might be enough to not trigger the timeout, by continuously sending back progress information

Third, you could do a file-by-file sync, e.g. like this:

for i in ls /path/to/dir; do rsync -ca $i remoteserver:/path/to/dir; done

Fourth, you could use ssh as your transfer mechanism, and use its TCPKeepAlive and ServerAliveInterval/ServerAliveCountMax options in some suitable way to keep the connection alive.

Original answer: rsync already does a file-by-file approach - it will check each file in the list of files to be synced, and will only sync those that don't exist on the target system, or which are different from those files on the target system.

What you could do to reduce transfer time is to use the option -z which will compress during transport, and the flag --partial so that if the transfer is interrupted, you'll keep any partially synced file so that rsync can continue where it was. Also the --timeout and --contimeout options that Michael Kjörling mentioned would probably be useful.

Improve this answer
4
  • it looks like it first calculates ALL the checksum before starting the transfer. is there a way it calculates the checksum of the first and if necessary transfers and then calculates the second ? Commented Mar 22, 2013 at 12:34
  • If you don't use the -c option it won't use checksums at all, so that'd be one way. Commented Mar 22, 2013 at 12:59
  • you’re right but i intended to check the integrity of all the files so the -c is a must for me Commented Mar 22, 2013 at 17:00
  • OK, I've added some possible ways to handle it. Commented Mar 25, 2013 at 10:10
1

Maybe something like this?

find . -type f -exec 'rsync' '{}' ';'

This will execute rsync once for each regular file under the current directory, passing the file name (represented by the {} token) as a command line parameter.

If you have symlinks, device files, etc. under the directory in question, you can try inverting the logic:

find . -not -type f -exec 'rsync' '{}' ';'

This should "work", in the sense of doing what you are asking (launching rsync once per file). But I get the feeling that you are going about it the wrong way.

The rsync man page lists --timeout (I/O timeout) and --contimeout (connection timeout), and that's just from grepping for timeout. Have you considered using those?

Improve this answer
2
  • Thanks for the help and you were right! i managed to resolve it with the timeout of ssh Commented Mar 22, 2013 at 16:26
  • @k.mooijman Glad if I was able to help. If you feel the question has been answered, please accept the answer that you found most useful by clicking the checkmark outline next to it. Commented Mar 23, 2013 at 0:50
0

For me this kept happening on resuming a large rsync transfer on a somewhat slow connection. I was even using rsyncd to avoid all the problems with ssh.

I found that enabling compression (-z or --compress) fixed the issue. Transfer resumed almost immediately.

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.