37

In Linux, say I have the following file (e.g. conf.properties):

HOST_URL=http://$HOSTNAME STD_CONFIG=http://$HOSTNAME/config USER_CONFIG=http://$HOSTNAME/config/$unconfigured

I want to create another file with all the environment variables replaced...e.g. say the environment variable $HOSTNAME is 'myhost' and $unconfigured is not set, a script should produce the following output:

HOST_URL=http://myhost STD_CONFIG=http://myhost/config USER_CONFIG=http://myhost/config/

I was thinking this could be done in a simple one-liner with some sort of sed/awk magic, but I'm no expert and my searches have been in vein, so appreciate any help.

Edit:

I should mention that the file can really be any format text file, for example xml. I just want to replace anything that looks like an env variable with whatever is currently set in the environment.

Improve this question
1
  • You should consider accepting the correct answer Commented Aug 22, 2019 at 11:34

9 Answers 9

Reset to default
191

This is what envsubst is for.

echo 'Hello $USER' Hello $USER echo 'Hello $USER' | envsubst Hello malvineous

You would probably use it more like this though:

envsubst < input.txt > output.txt

envsubst seems to be part of GNU gettext.

Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

This seems to be the only really correct answer here. Should be marked up higher.
I would just like to say thanks for this answer. My friend and I were struggling with sed, awk and other "tricks" and this answer saved us. Knowledge is power. Thank you!
10 
sed 's/$HOSTNAME/myhost/g;s/$unconfigured//g' yourfile.txt > another_file.txt

update:

Based on updates in your question, this won't be a good solution.

update2 :

This is based on an answer to a related question. I've hacked at it (I'm unfamiliar with perl) to remove undefined vars.

perl -p -e 's/\$\{([^}]+)\}/defined $ENV{$1} ? $ENV{$1} : $&/eg; s/\$\{([^}]+)\}//eg' yourfile.txt

Should work for any input text file, however you will need to define vars using the ${...} format which simplifies the string matching.

(rant regarding the evilness of eval moved to a separate post so as not to confuse readers)

Improve this answer

5 Comments

Or more readable (IMHO): sed -e 's/$HOSTNAME/myhost/' -e 's/$unconfigured// ...
Assuming I know the env variables used within the config files I could use this technique, but I really want something more generic.
@Shawn-Chin Thanks! I'll probably use this as I like the ${} style of using properties.
I think the perl command can be simplified to perl -p -e 's/\$\{([^}]+)\}/defined $ENV{$1} ? $ENV{$1} : ""/eg' yourfile.txt based on comments on the related question, instead of having a 2nd regex to remove empty variables. You simply need to replace $& with ""
@JoelPearson's regex has the advantage of not removing backslash
8

"eval is evil"

This is not an answer, but a warning in response to using eval for this task. You really really really don't want to do that.

Exhibit 1: a malicious template file:

HOST_URL=http://$HOSTNAME STD_CONFIG=http://$HOSTNAME/config USER_CONFIG=http://$HOSTNAME/config/$unconfigured && cat /etc/redhat-release

An unsuspecting user:

[lsc@aphek]$ cat somefile | while read line; do echo $(eval echo `echo $line`); done HOST_URL=http://xyz STD_CONFIG=http://xyz/config USER_CONFIG=http://xyz/config/ Red Hat Enterprise Linux WS release 4 (Nahant Update 9)

Note the last line!

Now, imagine the possibilities....

Improve this answer

1 Comment

this came to mind too. i was surprised people considered that. there should be a substitution mechanism in bash, really! or else, we should write a standard tool for this.
6

I'd do it like this:

# Set the $HOSTNAME and other variables # Now evaluate the properties file as a shell script. . config.properties # Write the values cat >somefile <<EOF HOST_URL=$HOST_URL STD_CONFIG=$STD_CONFIG USER_CONFIG=$USER_CONFIG EOF

Edit: Or this very nasty thing (I'm sure there's a better way)

for name in HOST_URL STD_CONFIG USER_CONFIG echo "$name=$(eval echo `echo '$'$name`)" >>somefile end
Improve this answer

4 Comments

I can't assume the file is a valid shell script (just name=value) - it might be any format, for example xml.
@Andy Whitfield: Too bad :-) Then you're off to awk/sed.
Your edit gave my an idea: this little nugget: cat somefile | while read line; do echo $(eval echo echo $line); done
The formatting doesn't come out too well in the comment here, so I'll paste it as an answer for reference.
3

Thanks to @DarkDust I came up with this:

cat somefile | while read line; do echo $(eval echo `echo $line`); done > somefile.replaced
Improve this answer

5 Comments

And even a one-liner ;-) Now that looks like a solution, as long as you know that your input files will not have $ except for variables.
PLEASE DON'T USE THAT. Pardon the raised voice, but this is one situation where screaming is warranted. See explanation in my answer.
@Shawn No probs - thanks for the warning. I'm going with the perl line - works like a charm.
I was more worried about unsuspecting user that may come across it in the future. Posted warning as a separate item so as not to cause confusion.
This will omit spaces at the beginning of lines and also lines containing #.
2

I used this oneliner to replace ${VARIABLE} style variables in a file:

TARGET_FILE=/etc/apache2/apache2.conf; for VARNAME in $(grep -P -o -e '\$\{\S+\}' ${TARGET_FILE} | sed -e 's|^\${||g' -e 's|}$||g' | sort -u); do sed -i "s|\${$(echo $VARNAME)}|${!VARNAME}|g" ${TARGET_FILE}; done

I'm pretty sure someone can do this in 1/3rd of the length using awk… feel challenged! ;)

Improve this answer

Comments

1

Here is a snippet of Javascript that I like to have around for solving this exact problem:

// A Javascript version of envsubst for our builds // Purpose: replace all ocurrences of ${VAR} with the equivalent var from the environment from stdin var readline = require('readline'); var rl = readline.createInterface({ input: process.stdin, output: process.stdout, terminal: false }); const environment = process.env; rl.on('line', function(line) { const newLine = line.replace(/\$\{([a-zA-Z0-9_]+)\}/g, function(_match, variable) { const envVar = environment[variable]; return envVar ? envVar : ''; }); process.stdout.write(`${newLine}\n`); });

Hopefully this helps somebody else.

Improve this answer

Comments

0

Here's a short one-liner that uses python's curly brace formatting to safely do the magic:

contents=\"\"\"`cat $file`\"\"\"; python -c "import os;print $contents.format(**os.environ)"
  • avoids evil eval
  • allows outputting curly braces: use {{ instead of {
  • no need to specify vars explicitly when calling the script

For example, given properties file settings.properties:

# my properties file someVar = {MY_ENV_VAR} curlyBraceVar = has {{curly braces}}

Then, do the substitution with:

$ export MY_ENV_VAR="hello" $ file=settings.properties $ contents=\"\"\"`cat $file`\"\"\"; python -c "import os;print $contents.format(**os.environ)" # my properties file someVar = hello curlyBraceVar = has {curly braces}

A script is here: https://raw.githubusercontent.com/aneilbaboo/machome/master/bin/substenv

Improve this answer

1 Comment

This does not avoid evil eval, it just obfuscates it. echo '"""; print "arbitrary code execution"; """' > ~/myfile; creates a file that causes some arbitrary code to be evaluated.
0

if you have installed nodejs you can run

npx @utft/tt -e FOO=bar /path/to/input /path/to/output

or you can run it programmatically https://github.com/utftufutukgyftryidytftuv/tt

Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.