Python Subprocess: Unable to Escape Quotes

A mental model of process and shells that I found very helpful:

This mental model has helped me a lot through the years.

Processes in your operating system receive an array of strings representing the arguments. In Python, this array can be accessed from sys.argv. In C, this is the argv array passed to the main function. And so on.

When you open a terminal, you are running a shell inside that terminal, for example bash or zsh. What happens if you run a command like this one?

$ /usr/bin/touch one two 

What happens is that the shell interprets the command that you wrote and splits it by whitespace to create the array ["/usr/bin/touch", "one", "two"]. It then launches a new process using that list of arguments, in this case creating two files named one and two.

What if you wanted one file named one two with a space? You can't pass the shell a list of arguments as you might want to do, you can only pass it a string. Shells like Bash and Zsh use single quotes to workaround this:

$ /usr/bin/touch 'one two' 

The shell will create a new process with the arguments ["/usr/bin/touch", "one two"], which in this case create a file named one two.

Shells have special features like piping. With a shell, you can do something like this:

$ /usr/bin/echo 'This is an example' | /usr/bin/tr a-z A-Z THIS IS AN EXAMPLE 

In this case, the shell interprets the | character differently. In creates a process with the arguments ["/usr/bin/echo", "This is an example"] and another process with the arguments ["/usr/bin/tr", "a-z", "A-Z"], and will pipe the output of the former to the input of the latter.

How this applies to subprocess in Python

Now, in Python, you can use subprocess with shell=False (which is the default, or with shell=True. If you use the default behaviour shell=False, then subprocess expects you to pass it a list of arguments. You cannot use special shell features like shell piping. On the plus side, you don't have to worry about escaping special characters for the shell:

import subprocess # create a file named "one two" subprocess.call(["/usr/bin/touch", "one two"]) 

If you do want to use shell features, you can do something like:

subprocess.call( "/usr/bin/echo 'This is an example' | /usr/bin/tr a-z A-Z", shell=True, ) 

If you are using variables with no particular guarantees, remember to escape the command:

import shlex import subprocess subprocess.call( "/usr/bin/echo " + shlex.quote(variable) + " | /usr/bin/tr a-z A-Z", shell=True, ) 

(Note that shlex.quote is only designed for UNIX shells, and not for DOS on Windows.)