How to expand environment variables in python as bash does?

The following implementation maintain full compatibility with os.path.expandvars, yet allows a greater flexibility through optional parameters:

import os import re def expandvars(path, default=None, skip_escaped=False): """Expand environment variables of form $var and ${var}. If parameter 'skip_escaped' is True, all escaped variable references (i.e. preceded by backslashes) are skipped. Unknown variables are set to 'default'. If 'default' is None, they are left unchanged. """ def replace_var(m): return os.environ.get(m.group(2) or m.group(1), m.group(0) if default is None else default) reVar = (r'(?<!\\)' if skip_escaped else '') + r'\$(\w+|\{([^}]*)\})' return re.sub(reVar, replace_var, path) 

Below are some invocation examples:

>>> expandvars("$SHELL$unknown\$SHELL") '/bin/bash$unknown\\/bin/bash' >>> expandvars("$SHELL$unknown\$SHELL", '') '/bin/bash\\/bin/bash' >>> expandvars("$SHELL$unknown\$SHELL", '', True) '/bin/bash\\$SHELL' 

Try this:

re.sub('\$[A-Za-z_][A-Za-z0-9_]*', '', os.path.expandvars(path)) 

The regular expression should match any valid variable name, as per this answer, and every match will be substituted with the empty string.

Edit: if you don't want to replace escaped vars (i.e. \$VAR), use a negative lookbehind assertion in the regex:

re.sub(r'(?<!\\)\$[A-Za-z_][A-Za-z0-9_]*', '', os.path.expandvars(path)) 

(which says the match should not be preceded by \).

Edit 2: let's make this a function:

def expandvars2(path): return re.sub(r'(?<!\\)\$[A-Za-z_][A-Za-z0-9_]*', '', os.path.expandvars(path)) 

check the result:

>>> print(expandvars2('$TERM$FOO\$BAR')) xterm-256color\$BAR 

the variable $TERM gets expanded to its value, the nonexisting variable $FOO is expanded to the empty string, and \$BAR is not touched.

There is a pip package called expandvars which does exactly that.

pip3 install expandvars 

from expandvars import expandvars print(expandvars("$PATH:${HOME:?}/bin:${SOME_UNDEFINED_PATH:-/default/path}")) # /bin:/sbin:/usr/bin:/usr/sbin:/home/you/bin:/default/path 

It has the benefit of implementing default value syntax (i.e., ${VARNAME:-default}).

The alternative solution - as pointed out by @HuStmpHrrr - is that you let bash evaluate your string, so that you don't have to replicate all the wanted bash functionality in python.

Not as efficient as the other solution I gave, but it is very simple, which is also a nice feature :)

>>> from subprocess import check_output >>> s = '$TERM$FOO\$TERM' >>> check_output(["bash","-c","echo \"{}\"".format(s)]) b'xterm-256color$TERM\n' 

P.S. beware of escaping of " and \: you may want to replace \ with \\ and " with \" in s before calling check_output

Here's a solution that uses the original expandvars logic: Temporarily replace os.environ with a proxy object that makes unknown variables empty strings. Note that a defaultdict wouldn't work because os.environ

For your escape issue, you can replace r'\$' with some value that is guaranteed not to be in the string and will not be expanded, then replace it back.

class EnvironProxy(object): __slots__ = ('_original_environ',) def __init__(self): self._original_environ = os.environ def __enter__(self): self._original_environ = os.environ os.environ = self return self def __exit__(self, exc_type, exc_val, exc_tb): os.environ = self._original_environ def __getitem__(self, item): try: return self._original_environ[item] except KeyError: return '' def expandvars(path): replacer = '\0' # NUL shouldn't be in a file path anyways. while replacer in path: replacer *= 2 path = path.replace('\\$', replacer) with EnvironProxy(): return os.path.expandvars(path).replace(replacer, '$') 

I was unhappy with the various answers, needing a little more sophistication to handle more edge cases such as arbitrary numbers of backslashes and ${} style variables, but not wanting to pay the cost of a bash eval. Here is my regex based solution:

#!/bin/python import re import os def expandvars(data,environ=os.environ): out = "" regex = r''' ( (?:.*?(?<!\\)) # Match non-variable ending in non-slash (?:\\\\)* ) # Match 0 or even number of backslash (?:$|\$ (?: (\w+)|\{(\w+)\} ) ) # Match variable or END ''' for m in re.finditer(regex, data, re.VERBOSE|re.DOTALL): this = re.sub(r'\\(.)',lambda x: x.group(1),m.group(1)) v = m.group(2) if m.group(2) else m.group(3) if v and v in environ: this += environ[v] out += this return out # Replace with os.environ as desired envars = { "foo":"bar", "baz":"$Baz" } tests = { r"foo": r"foo", r"$foo": r"bar", r"$$": r"$$", # This could be considered a bug r"$$foo": r"$bar", # This could be considered a bug r"\n$foo\r": r"nbarr", # This could be considered a bug r"$bar": r"", r"$baz": r"$Baz", r"bar$foo": r"barbar", r"$foo$foo": r"barbar", r"$foobar": r"", r"$foo bar": r"bar bar", r"$foo-Bar": r"bar-Bar", r"$foo_Bar": r"", r"${foo}bar": r"barbar", r"baz${foo}bar": r"bazbarbar", r"foo\$baz": r"foo$baz", r"foo\\$baz": r"foo\$Baz", r"\$baz": r"$baz", r"\\$foo": r"\bar", r"\\\$foo": r"\$foo", r"\\\\$foo": r"\\bar", r"\\\\\$foo": r"\\$foo" } for t,v in tests.iteritems(): g = expandvars(t,envars) if v != g: print "%s -> '%s' != '%s'"%(t,g,v) print "\n\n" 

I have run across the same issue, but I would propose a different and very simple approach.

If we look at the basic meaning of "escape character" (as they started in printer devices), the purpose is to tell the device "do something different with whatever comes next". It is a sort of clutch. In our particular case, the only problem we have is when we have the two characters '\' and '$' in a sequence.

Unfortunately, we do not have control of the standard os.path.expandvars, so that the string is passed lock, stock and barrel. What we can do, however, is to fool the function so that it fails to recognize the '$' in that case! The best way is to replace the $ with some arbitrary "entity" and then to transform it back.

def expandvars(value): """ Expand the env variables in a string, respecting the escape sequence \$ """ DOLLAR = r"\$" escaped = value.replace(r"\$", r"\%s" % DOLLAR) return os.path.expandvars(escaped).replace(DOLLAR, "$") 

I used the HTML entity, but any reasonably improbable sequence would do (a random sequence might be even better). We might imagine cases where this method would have an unwanted side effect, but they should be so unlikely as to be negligible.

You can try this. Inspired from Seth Robertson's answer

def expand_vars(input_str: str, env_dict: Optional[dict[str, str]] = None) -> str: """ Method to expand variables in given string data. Args: input_str (str): String data in which variables need to be expanded. env_dict (dict, optional): Environment variables dict. Defaults to os.environ. Returns: str: Expanded string data. """ if env_dict is None: env_dict = dict(os.environ) def replacer(match: re.Match) -> str: prefix = match.group(1) var_name = match.group(2) or match.group(3) if var_name in env_dict: return prefix + env_dict[var_name] return prefix + (f"${{{var_name}}}" if match.group(3) else f"${var_name}") pattern = r"(.*?)(?:\$(\w+)|\$\{(\w+)\})" # Expand variables until no more changes occur previous_str = None while previous_str != input_str: previous_str = input_str input_str = re.sub(pattern, replacer, input_str) return input_str