Which one is more efficient over a very large set of files and should be used?
find . -exec cmd {} + or
find . | xargs cmd (Assume that there are no funny characters in the filenames)
- Related: stackoverflow.com/questions/9612090/…Mateusz Piotrowski– Mateusz Piotrowski2017-03-19 19:13:21 +00:00Commented Mar 19, 2017 at 19:13
Add a comment |
3 Answers
Speed difference will be insignificant.
But you have to make sure that:
Your script will not assume that no file will have space, tab, etc in file name; the first version is safe, the second is not.
Your script will not treat a file starting with "
-" as an option.
So your code should look like this:
find . -exec cmd -option1 -option2 -- {} + or
find . -print0 | xargs -0 cmd -option1 -option2 -- The first version is shorter and easier to write as you can ignore 1, but the second version is more portable and safe, as "-exec cmd {} +" is a relatively new option in GNU findutils (since 2005, lots of running systems will not have it yet) and it was buggy recently. Also lots of people do not know this "-exec cmd {} +", as you can see from other answers.
16 Comments
dannysauer
-print0 is also a GNU find (and GNU xargs) option which is missing from a lot of non-Linux systems, so the portability argument isn't as valid. Using just -print and leaving the -0 off of xargs, however, is very portable.
Tometzky
The point is that without -print0 it does not work if there is a file with a space or tab etc. This can be a security vulnerability as if there is a filename like "foo -o index.html" then -o will be treated as an option. Try in empty directory: "touch -- foo\ -o\ index.html; find . | xargs cat". You'll get: "cat: invalid option -- 'o'"
bobpaul
His example is a filename that contains a -. Without -print0, find will spit out ./foo -o index.html. So maybe starting with a - isn't a big deal, but the result is little changed, and on a multiuser system, could provide an attack vector if your script is world readable.
FarmerGedden
A note on something which tripped me up here - using
exec will output results as they are found, wheras xargs will, it seems, wait until the entire directory is searched before writing to stdout. If you're trying this on a large directory, and it seems that xargs isn't working, patience is advisable.Tometzky
@Motivated Without
-print0 find returns filenames separated with newline, but newline can also be part of a filename, making it ambiguous. Byte 0 can't, so it is a safe separator. Yes - adding -- to a command that supports it is a good practice when you can't control its arguments, even if not always strictly required or unsafe. |
find . | xargs cmd is more efficient (it runs cmd as few times as possible, unlike exec, which runs cmd once for each match). However, you will run into trouble if filenames contain spaces or funky characters.
The following is suggested to be used:
find . -print0 | xargs -0 cmd this will work even if filenames contain funky characters (-print0 makes find print NUL-terminated matches, -0 makes xargs expect this format.)
2 Comments
Tometzky
This is not "find . -exec cmd {} \;" but "find . -exec cmd {} +". The latter will not run one file at a time.
SamB
Note that the
xargs approach is actually significantly slower if there are no (or only a few) matching files and cmd doesn't have much to do for each file. For example, when run in an empty directory, the xargs version will take at least twice the time, since two processes must be started instead of just one. (Yes, the difference is usually imperceptible on *nix, but in a loop it could be important; or, try it on Windows some time ...)Modern xargs's versions often support parallel pipeline execution.
Obviously it might be a pivot point when it comes to choice between find … -exec and … | xargs
