5

I went through post after post on SO looking for a method to use quotation marks inside of arguments using subprocess.popen and I cannot seem to find a way.

This works fine from the commandline

runme.bat --include="check|check2"

Python

#!/usr/bin/python import sys import subprocess import shlex #command_line = "./runme.sh --include=\"check|check2\"" command_line = "runme.bat --include=\"check|check2\"" arg = shlex.shlex(command_line) arg.quotes = '"' arg.whitespace_split = True arg.commenters = '' command_line_args = list(arg) print command_line_args command_line_process = subprocess.Popen( command_line_args, universal_newlines=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE ) line = "" while True: line = command_line_process.stdout.readline() if line: print line break

runme.bat

echo %* >> someargs.txt

runme.sh

#!/bin/bash echo $@

I heard that subprocess.call() is a way around this but I'd like to be able to iterate line by line through the subprocess' output while the program is running.

Edit:

This seems to be a bug in Python because running runme.bat in cmd works correctly, running runme.py in linux works correctly, it's only when running runme.py on Windows where it doesn't work correctly. I created a ticket here.

Edit2:

It's not a python bug apparently lol. Look at chosen answer.

Improve this question
1
  • The documentation is a bit unclear, but it looks as if using a string (rather than a sequence) for args might resolve the problem. Have you tried that already? If that doesn't work either, you might want to try filing that as a bug. Python should give you some way of passing an arbitrary command string without it trying to second-guess you. Commented Apr 3, 2015 at 23:23

3 Answers 3

Reset to default
4

On Windows, a string is a native API. To avoid unnecessary conversions, pass the command as a string:

#!/usr/bin/env python from __future__ import print_function import subprocess command = 'runme.bat --include="check|check2"' process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True, bufsize=1) for line in iter(process.stdout.readline, ''): print(line, end='')

stderr=subprocess.STDOUT merges stderr into stdout. If you set stderr=PIPE then you should read from process.stderr in parallel with reading from process.stdout otherwise your program may deadlock.

Popen() passes the string to CreateProcess() Windows function. If the child process is actually a batch-file; you should probably pass shell=True explicitly to make it clear that the command is interpreted using cmd.exe rules (^, |, etc are meta-characters, for more details read the links in this answer).

If you want to pass the argument using %1 instead of %* so that it includes
the whole --include="check|check2" (not only --include) then you could use additional quotes around the argument as @eryksun suggested in the comments:

command = '"runme.bat" "--include="check^^^|check2""'

Notice: triple ^ to escape | here.

Improve this answer
Sign up to request clarification or add additional context in comments.

4 Comments

(1) the command in the answer has no shell=True i.e., it is already assumes that Popen() (via CreateProcess()) can run .bat files. The command line is interpreted differently if it is eventually given to cmd.exe that is why I said to include shell=True for documentation purposes (it makes no difference otherwise as far as I can tell in this case). (2) I don't see any point to include quotes around runme.bat or to include ^^^ here (^ is not special inside quotes). I'm sure there are many quirks accumulated over the years in batch-file syntax but the example works as is.
(1) "doesn't work" is not informative. I've used py -c "import sys; print(sys.argv)" %1 %* (in runme.bat) for testing and it works as expected. I'm sure; it may fail for other examples but I expect it to work for the case in the question. (2) It documents the syntax of the command line (cmd.exe rules) e.g., shell=True tells you whether ^ may be a meta-character.
@eryksun: I've included your more general command in the answer. Thank you for your comments. They are insightful as usual.
@Sebastian, brilliant!
3

You should not use shell=True to run a bat file. Use it only if you have to run some built-in shell command. In other words the use your are making is useless and the only effect is to increase security vulnerability of your program.

Also, note that the documentation clearly states that, when using shell=True it's recommended to pass the command line as a string:

If shell is True, it is recommended to pass args as a string rather than as a sequence.

So you should do:

subprocess.check_output('runme.bat --include="check|check2"', shell=True)

The check_output function should be used if you only care for the output. It's way simper than creating a Popen object and then reading the output manually.

See also my answer regarding how shell=True changes the meaning of the arguments.

Improve this answer

2 Comments

Thanks for the response. I just want to see the output line by line from the program while it's running which is possible with popen just not with arguments in double quotes. I removed the shell=True from the code above. You never mentioned the acknowledged the quotation issue in your answer... ?
@Some note that using shell=True is also much slower than not using it, because it had to spawn a whole new shell and then execute the program. I don't get what your are saying about acknowledgements and I don't see any issue with that. I did disclose that I'm the author of the links answer.
0

Another way to get the output is subprocess.check_output():

import subprocess command_line = "runme.bat --include=\"check|check2\"" output = subprocess.check_output( command_line, shell=True ) lines = output.splitlines(True) print lines

To view the process's output in real time, see: Getting realtime output using subprocess.

Edit: here's code with Popen to handle double quotes:

from subprocess import Popen, PIPE, STDOUT command_line = 'ls -la | grep "wheel"' p = Popen(command_line, stdout=PIPE, stderr=STDOUT, shell=True) while True: line = p.stdout.readline() if not line: break else: print line
Improve this answer

8 Comments

Thank you for your response but I'm looking for a way to run a command and capture output in realtime. I can do this with Popen but I also need to be able to use double quotes for a parameter...
I've added a link to an answer about getting real time ouput.
The answer you linked uses Popen with real time output (which is what I'm using above) and does not use double quotes in their arguments. The answer you gave uses check_output with double quotes but doesn't output line by line in real time. Unless I misread the link?
To handle the double quotes, you can wrap the command in single quotes, like this: command_line = 'runme.bat --include="check|check2"'.
I appreciate your suggestions but that does not work either. Please try the code I supplied above and you'll see why I posted about it. :)
|

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.