Got a new NAS server I'm testing. It's basically CentOS 7.5.1804 with Areca hardware RAID cards and lots of HDDs, and XFS volumes.
Sporadic copying tests often succeed. But whenever I'm running large data copies from our SAN, they invariably eventually fail in the following manner:
rsync -ah --exclude="._*" --exclude=".DS_Store" --quiet --progress /Volumes/BEST* /Volumes/SAN_Backups/BEST/ rsync: writefd_unbuffered failed to write 32768 bytes [sender]: Broken pipe (32) rsync: write failed on "/Volumes/NEXIS_Backups/BEST/BEST_Media/FROM CLIENT/091224_BONUS CLEAN/02_YG_MAR_v2.mov": Is a directory (21) rsync: connection unexpectedly closed (4116 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at /AppleInternal/Library/BuildRoots/46298ee5-4a1c-11ef-a181-1aec23608739/Library/Caches/com.apple.xbs/Sources/rsync/rsync/io.c(453) [sender=2.6.9] writefd_unbuffered fails to write, broken pipe (32).
Write fails "Is a directory (21)" on a non-directory file
Connection unexpectedly closes, rsync error code 12.
- Running the rsync on an Apple Silicon Mac, as you noticed. The Mac has our SAN mounted, as well as this CentOS server over SMB.
- Needless to say, "02_YG_MAR_v2.mov" is a regular file, and not a directory.
- If I re-run the command, it will process the errored file correctly, and fail on another file in the same manner. Rinse and repeat.
- Running the same rsync command, on the same source data, on the same system, to a different CentOS (ZFS) NAS server, does not cause these errors. That server is quite different, it just supports my assertion that the problem is in this new NAS and not with the Mac client, rsync code, source data, nor the SAN.
- So far we tried blowing up and recreating the XFS filesystem (including the volume group and logical volume), and checked the RAID controller for errors. No change.
Any insights? I haven't found any info online to go off of, about files operations failing with the last file being mistaken for a directory…
Update: We gathered more verbose file I/O logs from the server end, and don't see any errors when the copying breaks:
editor closed file BEST/BEST_Media/FROM CLIENT/091224_BONUS CLEAN/02_YG_MAR_v1.mov (numopen=4) NT_STATUS_OK unix_mode(BEST/BEST_Media/FROM CLIENT/091224_BONUS CLEAN/.02_YG_MAR_v2.mov.kNey0S) inherit mode 40777 editor opened file BEST/BEST_Media/FROM CLIENT/091224_BONUS CLEAN/.02_YG_MAR_v2.mov.kNey0S read=Yes write=Yes (numopen=4) editor closed file BEST/BEST_Media/FROM CLIENT/091224_BONUS CLEAN/.02_YG_MAR_v2.mov.kNey0S (numopen=3) NT_STATUS_OK unix_mode(BEST/BEST_Media/FROM CLIENT/091224_BONUS CLEAN/.02_YG_MAR_v2.mov.kNey0S) inherit mode 40777 editor opened file BEST/BEST_Media/FROM CLIENT/091224_BONUS CLEAN/.02_YG_MAR_v2.mov.kNey0S read=No write=No (numopen=4) editor closed file BEST/BEST_Media/FROM CLIENT/091224_BONUS CLEAN/.02_YG_MAR_v2.mov.kNey0S (numopen=3) NT_STATUS_OK So i appears like the client gives up before even trying to rename the temp file to the final. Which makes it look like a macOS problem, not the server, except that I've never seen this happen before, e.g. another CentOS NAS server we have doesn't exhibit such an issue.
Next, I realized the rsync bundled with macOS to this day is ancient (2.6.9 from 18 years ago!), and tried v3.3.0. But that one destroyed the computer every time… in the exact same manner that's been documented around fairly well. For example: https://apple.stackexchange.com/questions/421007/rsync-3-2-3-keeps-crashing-on-mac-mini-m1
The conclusion that using SMB on Mac OS's from the past 2 years is broken and unusable would be senseless either. So I'm just more stumped…
