5

Summary:

I can't understand a peculiar discrepancy in network transfer.

  • Why is there such a discrepancy in syncing from one machine to the other and vice versa?

Also:

  • Given the maximum network transfer speed is about 110 M/sec, and the local disk speed for a similar operation is about 200 M/sec (so, no bottleneck there), why is there a much lower speed rsyncing between the two machines, than the theoretical 100M/sec?

Details:

First of all, server is

# uname -a FreeBSD das 10.1-RELEASE-p8 FreeBSD 10.1-RELEASE-p8 #25 d0fb866(releng/10.1): Thu Nov 13 07:57:26 EST 2014 root@bellicose:/usr/obj/root/pcbsd-build-10-STABLE/git/freebsd/ sys/GENERIC amd64

Client is:

# uname -a Darwin compute.internal 13.4.0 Darwin Kernel Version 13.4.0: Sun Aug 17 19:50:11 PDT 2014; root:xnu-2422.115.4~1/RELEASE_X86_64 x86_64

Both machines have 16GB of ram.

By doing a local rsync on the server, one kind of knows what to expect as disk speed, at least in that circumstance.

Using a binary file, test.bin, 732M, local rsync on FreeBSD server shows about 200 M/sec:

# rsync --stats -h test.bin copy.bin [....] sent 732.54M bytes received 35 bytes 209.30M bytes/sec total size is 732.36M speedup is 1.00

That is about 200 M/sec.

On the mac mini client I have almost 70M/sec:

# rsync --progress --stats -h test.bin copy.bin test.bin 732.36M 100% 70.06MB/s 0:00:09 (xfr#1, to-chk=0/1) [....] sent 732.54M bytes received 35 bytes 69.77M bytes/sec total size is 732.36M speedup is 1.00

Now, doing a network speed test with iperf:

On the server (the FreeBSD server):

# iperf -f M -s ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 0.06 MByte (default) ------------------------------------------------------------ [ 4] local 192.168.1.5 port 5001 connected with 192.168.1.226 port 50757 [ ID] Interval Transfer Bandwidth [ 4] 0.0-30.0 sec 3356 MBytes 112 MBytes/sec

On the client (OS X mac mini):

# iperf -f M -M 9000 -c 192.168.1.5 -t 30 -w 80K WARNING: attempt to set TCP maxmimum segment size to 9000 failed. Setting the MSS may not be implemented on this OS. ------------------------------------------------------------ Client connecting to 192.168.1.5, TCP port 5001 TCP window size: 0.08 MByte (WARNING: requested 0.08 MByte) ------------------------------------------------------------ [ 4] local 192.168.1.226 port 50757 connected with 192.168.1.5 port 5001 [ ID] Interval Transfer Bandwidth [ 4] 0.0-30.0 sec 3356 MBytes 112 MBytes/sec

So, I could assume that network connection (a straight cat 7 cable between the two nics) is about 110 M/sec.

Now, this is the puzzling situation:

If I rsync from the FreeBSD server to the mac mini, I get a transfer speed of about 50 M/sec:

# rsync --progress --stats -h test.bin -e "ssh -l gsl" '192.168.1.226:/tmp/' Password: test.bin 732.36M 100% 57.10MB/s 0:00:12 (xfr#1, to-chk=0/1) [....] sent 732.45M bytes received 46 bytes 50.51M bytes/sec total size is 732.36M speedup is 1.00

But rsync in the opposite direction gives a much lower transfer rate, 20M/sec:

# rsync --progress --stats -h test.bin -e "ssh -l gsl" '192.168.1.6:/tmp/' test.bin 732.36M 100% 19.55MB/s 0:00:35 (xfr#1, to-chk=0/1) [....] sent 732.54M bytes received 35 bytes 20.07M bytes/sec total size is 732.36M speedup is 1.00

My two questions:

  • Why is there such a discrepancy in syncing from one machine to the other and vice versa?

Also:

  • Given the maximum network transfer speed is about 110 M/sec, and the local disk speed for a similar operation is about 200 M/sec (so, no bottleneck there), why is there a much lower speed rsyncing between the two machines, than the theoretical 100M/sec?

Could somebody please help understand this, perhaps offering some advice on how to improve the transfer speed?

Update: Based on @dhag's answer, I tried to copy a file with netcat, ie using no compression:

On the "server" (pushing) side:

time cat test.bin | nc -l 2020 nc -l 2020 0.25s user 6.29s system 77% cpu 8.462 total

On the receiving side (FreeBSD):

time nc 192.168.1.226 2020 > test.bin nc 192.168.1.226 2020 > test.bin 0.09s user 4.00s system 62% cpu 6.560 total

If I am not mistaken, that should mean 732M/6.29s = 117M/sec, which kind of exceed theiperf stats. Perhaps a caching issue?

Update 2: Using rsync with no encryption at all (only possible if using a daemon, and the rsync:// protocol):

# rsync --progress --stats -h test.bin rsync://[email protected]/share ⏎ test.bin 732.36M 100% 112.23MB/s 0:00:06 (xfer#1, to-check=0/1) [....] sent 732.45M bytes received 38 bytes 112.69M bytes/sec total size is 732.36M speedup is 1.00

This also confirms @dhag's ideas.

Improve this question
2
  • Can you tell where you got the instructions about the update 2 ( rsync with no encryption, using a daemon and the rsync:// protocol )? Thanks! Commented Oct 5, 2017 at 17:49
  • I got the instructions here: banaspati.net/ubuntu/… Commented Oct 5, 2017 at 20:17

1 Answer 1

Reset to default
3

I can only provide a guess, which is that the discrepancy is explained by varying computational, memory, caching or disk characteristics of the two hosts:

  • If we assume that CPU is a bottleneck, then it would make some sense if the slower machine were slower at sending (this assumes that encrypting is more computationally heavy than decrypting). This can be tested by switching to a cipher that is lighter to compute (try adding -c arcfour to your SSH command line; in this case, passing --rsh="ssh -c arcfour" to rsync).

  • If we assume that files are being read / written straight from / to the disk, then the disk could conceivably be a bottleneck; read speeds of 100 MBps are well within reach of more modern computers, but not of older ones, or computers running on laptop-class drives (such as, I believe, the Mac Mini).

  • If we further assume that the operating system uses filesystem caches, the situation could be complicated further:

    • If the source file is contained in the filesystem cache, in RAM, then it can be read much faster than 100 MBps;

    • if the target system applies write-caching and is able to fit a significant part of the file in RAM (in your case it should be, since RAM is much bigger than your test file), then it can claim the write is complete before it has actually reached the disk (this could mean that your measured 200MBps is).

The disk-versus-cache unknown can be tested by flushing the filesystem cache prior to reading (how to do so is OS-dependent): then sending the file will be at least as slow as the disk dictates. Conversely, by reading the file completely before sending (perhaps with cat file.bin >/dev/null), one can influence the OS into caching it.

To further investigate whether CPU is an issue, it would make sense to run top while the transfer is ongoing; if the rsync or ssh process is taking 100% of a core, then that would be a bottleneck.

Improve this answer
2
  • You are right. I have tried sending a file with netcat, and it reach about 100M/sec. I have updated the question, with the data. Could you see if it makes sense? Thank you! Commented Mar 20, 2015 at 7:41
  • I have also added second update, using rsync with no encryption at all (only possible if using a daemon, and the rsync:// protocol). Thank you once again. Commented Mar 20, 2015 at 9:16

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.