0

Using Mac OS X command line I want to perform a simple find and replace in a large number of files, within the current directory and its many sub-directories.

I need to perform many replacements so I'd like the script to be as performant as possible.

Whatever I try seems to result in some random error so I'm finally asking for help.

So given I have two variables:

FIND=oldText REPLACE=newText

Here's what I've tried so far:

sed -i '' "s/${FIND}/${REPLACE}/g" * > sed: Build: in-place edditing only works for regular expressions

Apparently this is trying to sed on directory path's itself so I subsequently tried (to exclude directories from being sed'ed)

find * -type f -print | xargs sed -i '' "s/${FIND}/${REPLACE}/g" > xargs: sed: Argument list too long

So because I have such a large list of files to action xargs can't handle it. Apparently -exec is better at large lists..

find * -type f -print -exec sed -i '' "s/${FIND}/${REPLACE}/g" {} \;

Now this does actually work HOWEVER sed decides it must correct the missing eof/linefeed in all files in which they are missing, despite there being no replacements in the file. Unfortuntely there are thousands of files of this nature and its not an option for me to be making changes of this magnitude for this current piece of work. (Please don't preach about how I should correct the files, thats not the question I am asking).

So in an attempt to overcome this issue I have tried to first extract the list of files that do indeed contain my ${FIND} term and then only perform the sed on those files...

grep -r -l -e "${FIND}" "." | sed -i '' "s/${FIND}/${REPLACE}/g" > sed: -i may not be used with stdin

-

grep -r -l -e "${FIND}" "." | -exec sed -i '' "s/${FIND}/${REPLACE}/g" {} \; > ./file1.txt: line 10: -exec: command not found

-

$( grep -r -l -e "${FIND}" "." ) -exec sed -i '' "s/${FIND}/${REPLACE}/g" {} \; > ./file1.txt: line 10: -exec: command not found

-

FILEPATHS_CONTAINING_FIND=$( grep -r -l -e "${FIND}" "." ) sed -i '' "s/${FIND}/${REPLACE}/g" "${FILEPATHS_CONTAINING_FIND}" > sed: ./File1.txt ./File2.txt ./File3.txt: No such file or directory

I think here the its treating the variable ${FILEPATHS_CONTAINING_FIND} as a single long file path. If I remove the double quotes "" it doesn't handle paths with spaces so that's not an option either. Went back to trying xargs now that the list of files is shorter having filtered...

$( grep -r -l -e "${FIND}" "." ) | xargs sed -i '' "s/${FIND}/${REPLACE}/g" > ./Script.sh: line 10: ./File1.txt: Permission denied

Trying sudo in various places makes no difference.

Anyway I've resorted to using this for loop but I'd really rather something more succinct and performant. 

IFS=$'\n' # Ensure spaces don't mess up the for loop for FILEPATH_CONTAINING_FIND in $(grep -r -l -e "${FIND}" "."); do sed -i '' "s/${FIND}/${REPLACE}/g" "${FILEPATH_CONTAINING_FIND}" done

Can anyone help me with the problems I've experienced above?

Improve this question
9
  • 1
    Why on earth are you using -i '' to overwrite files while you still have not got your script working? Overwriting files without backup should only be done when you're pretty confident the whole thing will work. Well, never mind; they're your files — you can do as you please. But sanity dictates that you don't (normally) go around overwriting files until you're sure you're going to do it right. Commented Aug 5, 2016 at 15:10
  • If you're going to use -exec, use {} + instead of {} \; because it makes find behave like xargs and run with multiple file names as arguments. The error message xargs: sed: Argument list too long is pretty weird. Exactly how long are your ${FIND} and ${REPLACE} strings? Commented Aug 5, 2016 at 15:13
  • 1
    The sequence of 4 commands after 'So in an attempt to overcome this issue' are bizarre. They should fail. Do you have file names with spaces in them to deal with? If not, then grep -r -l … | xargs sed … should deal with the files containing the match. If you have spaces or other characters outside the portable file name character set in the names, say so. It makes your job harder. If you have file names containing newlines, it makes life even harder; that would be important information to have. Commented Aug 5, 2016 at 15:17
  • @JonathanLeffler, I use git so all the files are backed up and I can simply just reset my working copy in between attempts. Thanks for your concern. Commented Aug 5, 2016 at 15:22
  • The ${FIND} and ${REPLACE} are exceptionally short in this case, general single digits no of chars. Commented Aug 5, 2016 at 15:22

1 Answer 1

Reset to default
1

You can use find + grep + sed like this:

# cd to parent dir while IFS= read -d '' -r file; do grep -q "$FIND" "$file" && sed -i '' "s/${FIND}/${REPLACE}/g" "$file" done < <(find . -type f -print0)
  • Using print0 we generate null byte terminated filenames from find command
  • Using read -d '' we make read delimit on null byte
  • Using grep -q we make sure pattern $FIND is found in files before we run sed
  • If $FIND is just a plain string then you may consider using grep -F

EDIT: You can improve your for loop by using while loop and grep -r --null:

while IFS= read -d '' -r file; do sed -i '' "s/${FIND}/${REPLACE}/g" "$file" done < <(grep -lR --null "$FIND" .)
Improve this answer
Sign up to request clarification or add additional context in comments.

6 Comments

Is this better than the for loop I presented at the end? Seems far more complex than that.
Yes it is better as for loop is error prone due to word splitting when filenames have space or some other glob character. May be instead of find we can use grep -r with --null option (let me know if you want me to provide that option as well.
There's one problem with grep --null — it isn't supported by the BSD grep on Mac OS X; it is a GNU grep extension.
Ah; interesting. A discrepancy between the program (which lists --null when you ask grep --help) and the man page (which only mentions null in the context "The caret `^' matches the null string"). Given a choice, one assumes the program's self-documentation is more accurate than the 'printed' documentation. I live, I learn.
@EdMorton: :) Yes, I am familiar with views on grep -r. My original answer wasn't using grep -r at all, I just added because OP had attempted for loop using grep -r and I tried improving that in edited section. I would recommend using find always.
|

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.