How can I transfer data by excluding files over 100MB but including files that are over 100MB if they match a pattern of known file extensions?
I've read through rsync options and I don't think I can achieve this with rsync, because --max-size= is not flexible like this, even in combination with --include or --exclude.
In two steps (for simplicity, even though these steps can definitely be combined).
First transfer "small" files:
find /source/path -type f -size -100M -print0 | rsync -av -0 --files-from=- / user@server:/destination/ Then transfer "big" files whose filenames match pattern:
find /source/path -type f -size +99M -name 'pattern' -print0 | rsync -av -0 --files-from=- / user@server:/destination/ This is, however, untested.
-print0 in GNU find (and others) will print the found names with a nul delimiter, and -0 with rsync will make --files-from- interpret this standard input stream in that particular way.
The file paths read with --files-from should be relative to the specified source, that's why I use / as the source in rsync (I'm assuming /source/path in find is an absolute path).
Combined variation (also not tested):
find /source/path -type f \ \( -size -100M -o -name 'pattern' \) -print0 | rsync -av -0 --files-from=- / user@server:/destination/ With more than one allowable pattern string for "big" files:
find /source/path -type f \ \( -size -100M -o -name 'pattern1' -o -name 'pattern2' -o -name 'pattern3' \) -print0 | rsync -av -0 --files-from=- / user@server:/destination/ Each pattern may be something like *.mp4 or whatever file extensions you use. Note that these needs to be quoted, as in -name '*.mp4'.
\( and \) inbetween the -size and final pattern, but it works with and without? Anyway, this example is still picking up files that do not match the pattern. \( ... \) groups the conditional so that -print0 happens for any file that matches any of the conditions. -100M so I assumed this was a mistake and made it +99M. However, the "or" in your response is the key bit of information that enables me to understand that it's copying all files less than 100M or files that match the pattern. Thank you!