1

I am fighting with sed to do a substitution where the substitute string contains slashes. This general topic has been discussed on stack overflow before. But, AFAICT, I have anew wrinkle that hasn't been addressed in previous questions.

Let's say I have a file, ENVIRO.tpml, which has several lines, one of which is

Loaded modules: SUPPLY_MODULES_HERE

I want to replace SUPPLY_MODULES_HERE in an automated fashion with a list of loaded modules. (At this point, if anyone has a better way to do this than sed, please let me know!) My first effort here is to define an environment variable and use sed to put it into the file:

> modules=$(module list 2>&1) > sed "s/SUPPLY_MODULES_HERE/${modules}/" ENVIRO.tmpl > ENVIRO.txt

(The 2>&1 being needed because module list sends its output to STDERR, for reasons I can't begin to understand.) However, as is often the case, the modules have slashes in them. For example

> echo ${modules} gcc/9.2.0 mpt/2.20

The slashes kill my command because sed can't understand the expression and thinks my substitution command is "unterminated".

So I do the usual thing and use some other character for the command delimiter:

> modules=$(module list 2>&1) > sed "s|SUPPLY_MODULES_HERE|${modules}|" ENVIRO.tmpl > ENVIRO.txt

and I still get an "unterminated 's'" error.

So I replace double quotes with single quotes:

> sed 's|SUPPLY_MODULES_HERE|${modules}|' ENVIRO.tmpl > ENVIRO.txt

and now I get no error, but the line in ENVIRO.txt looks like

Loaded modules: ${modules}

Not what I was hoping for.

So, AFAICT, I need double quotes to expand the variable, but I need single quotes to make the alternative delimiters work. But I need both at the same time. How do I get this?

UPDATE: Gordon Davisson's comment below got to the root of the matter: "echo ${modules} can be highly misleading". Examining $modules with declare -p shows that it actually has a newline (or, more generally, some kind of line break) in it. What I did was add an extra step to extract newlines out of the variable. With that change, everything worked fine. An alternative would be to convince sed to expand the variable with line breaks and substitute it as such into the text, but I haven't been able to make that work. Any takers?

Improve this question
7
  • I don't think you "need single quotes to make the alternative delimiters work". Your example seemed to work for me. Are you testing it with that actual example? modules="gcc/9.2.0 mpt/2.20";echo Loaded modules: SUPPLY_MODULES_HERE | sed "s|SUPPLY_MODULES_HERE|${modules}|" Or do you have a different scenario where the modules have a pipe in them? Commented Nov 3, 2021 at 17:10
  • 2
    What's the actual contents of $modules? echo ${modules} can be highly misleading, so try declare -p modules | LC_ALL=C cat -v instead. Commented Nov 3, 2021 at 17:24
  • There’s nothing obviously wrong with the version using double quotes and pipes. Prepend echo, removing redirection, and post that output, so we can see what sed is trying to execute. Commented Nov 3, 2021 at 17:26
  • If your templates have several substitutions to perform or more complex operations then you could directly use a templating language like PHP or ERB (Ruby). Commented Nov 3, 2021 at 21:19
  • @GordonDavisson You might be interested in my update. Commented Nov 3, 2021 at 22:22

1 Answer 1

Reset to default
4

sed is not the best tool here due to use of regex and delimiters.

Better to use awk command that doesn't require any regular expression.

awk -v kw='SUPPLY_MODULES_HERE' -v repl="$(module list 2>&1)" ' n = index($0, kw) { $0 = substr($0, 1, n-1) repl substr($0, n+length(kw)) } 1 ' file
  • index function uses plain string search in awk.
  • substr function is used to get substring before and after the search keyword.
Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.