2

I have several scripts that edit template text files, by removing a "tag" and replacing it with e.g. a number. To do this I use the

sed -i

command. However, I have an issue with write/read times on the server where I execute the scripts, making the scripts take a long time to run, since the sed -i command writes a temporary file to disk for every execution.

Is there an alternative approach that I could use, where a temporary file is not written to disk for every single replacement? Can the text file be edited in the memory and only written once all the replacements have been executed, or could I stack several replacements into the same sed command?

To clarify, the script is of the following form:

input=shiftLeft.txt while IFS= read -r line do sed -i "s/install, element = $line, at=/install, element = $line, at= -0.001 +/g" processedFiles/layoutDB.seq done < "$input"

That is, I read values from one text file, and then I do some changes in another text file depending on these values. This is done repeatedly for a large number of values.

Improve this question
12
  • 3
    Please add at least one script you are using. Commented Aug 30, 2021 at 10:29
  • Is there a reason you couldn't simply use ed? Commented Aug 30, 2021 at 10:31
  • 2
    "could I stack several replacements into the same sed command?" suggests that you're repeatedly running sed -i on the same file. Yes, you should add all the -e commands you want into a single sed invocation (or even into a sed script file - see sed -f). Commented Aug 30, 2021 at 10:32
  • @TobySpeight ed would potentially also use temporary files. Commented Aug 30, 2021 at 11:03
  • 1
    sed -i does not take a long time to run due to creating temporary files. It takes a long time to run because you're executing it in a tight loop. Please show what you are actually doing. Commented Aug 30, 2021 at 11:36

2 Answers 2

Reset to default
4

Your problem isn't that sed -i creates many temporary files, it's that you're running it many times with the same input file, and each of those creates a temporary file for the output, as strace shows:

execve("/bin/sed", ["sed", "-i", "-e", "", "/tmp/foo"], 0x7fff10da5288 /* 36 vars */) = 0 openat(AT_FDCWD, "/tmp/foo", O_RDONLY) = 3 openat(AT_FDCWD, "/tmp/sedVdjaBk", O_RDWR|O_CREAT|O_EXCL, 0600) = 4 rename("/tmp/sedVdjaBk", "/tmp/foo") = 0 +++ exited with 0 +++

The solution is to run sed -i just once.

To do that, start by writing a sed command that transforms your input file into a sed program. That would look something like:

sed -e 's!.*!s/install, element = &, at=/install, element = &, at= -0.001 +/g!"

(It's possible that we could improve this, if the input file includes regex-significant characters, e.g. s/install, element = &, at=/\& -0.001 +/g, but that's outside the scope of this question).

Test this to ensure you're happy with the resultant script.

Then we need to get another sed to use that transformed text as its program file. We can do that by telling it to read its program from standard input (although there are alternatives¹):

sed -e 's!.*!s/install, element = &, at=/install, element = &, at= -0.001 +/g!' \ shiftLeft.txt | sed -f - -i processedFiles/layoutDB.seq

Again, test this (without the -i flag) until you're satisfied it does what you want it to.

¹ Since we're using , we can use a process substitution:

sed -f <(sed -e 's!.*!s/install, element = &, at=/install, element = &, at= -0.001 +/g!' shiftLeft.txt) \ -i processedFiles/layoutDB.seq

In standard shell, we'd need to capture the transformed text as a string, and supply that as a command-line script:

sed -e "$(sed -e 's!.*!s/install, element = &, at=/install, element = &, at= -0.001 +/g!' shiftLeft.txt)" \ -i processedFiles/layoutDB.seq
Improve this answer
3
  • Thanks a lot! Since I have multiple loops, I output sed -e result to a file for all the sed replacements I need to do. I then call this output file using sed -i -f tempSed.sed targetFile.txt . Currently the server is not under heavy load, yet, with my old approach it took 127.448 seconds, whereas with your suggestion it only takes 5.623 seconds. Running it locally the speed improvement was from 3.848s to 3.522s. Commented Aug 30, 2021 at 15:33
  • 1
    @a20, you can probably combine the multiple loops without needing a temporary file: (loop1; loop2; loop3) | sed -f -. (OTOH, if you like a file, then put it somewhere local, such as in "${TMPDIR:-/tmp}/"). Commented Aug 30, 2021 at 17:33
  • 1
    The best way to find a good local location for your file is to use tmpfile, of course (don't forget to set a trap so it's removed even if your script exits early). Commented Aug 31, 2021 at 7:04
3

Don't call sed repeatedly in a shell loop, just call awk once, e.g. (untested since you didn't provide any sample input/output to test with) using GNU awk for "inplace" editing and the 3rd arg to match():

awk -i inplace ' NR==FNR { lines[$0] } (FNR>NR) && match($0,/(.*install, element = )([^,]+)(, at=)/,a) && (a[2] in lines) { $0 = a[0] " -0.001 +" } { print } ' shiftLeft.txt processedFiles/layoutDB.seq

There may be a better way to do that depending on what your input/output looks like.

Improve this answer
1
  • 1
    Thank you! The above answer worked for me, but it is very nice to know of alternative methods. Commented Aug 31, 2021 at 7:42

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.