I have a program script.out which I need to run against 10 files and redirect their output to 10 files. I am using subprocess.Popen() for this work but it seems to catch some error when redirecting stdout stream.
... for i in range(10): data = subprocess.Popen([ script.out, "<", input_file[i], ">", output_file[i]]) For the workaround, I use os.system() to run directly. But I want to know why the subprocess.Popen() process failed there?
I/O redirection with < and > is done by the shell. When you call subprocess.Popen() with a list as the first argument or without shell=True, the program is executed directly, not using the shell to parse the command line. So you're executing the program and passing literal arguments < and > to it. It's as if you executed the shell command and quoted the < and > characters:
scriptname '<' infile.txt '>' outfile.txt If you want to use the shell you have to send a single string (just like using os.system().
data = subprocess.Popen(" ".join([ shlex.quote(script.out), "<", shlex.quote(input_file[i]), ">", shlex.quote(output_file[i])]), shell=True) Use shlex.quote() to escape arguments that shouldn't be treated as shell metacharacters.
Using '<' or '>' in subprocess.Popen arguments without passing shell=True will not have any effect, because input redirection is a feature of shells. Passing shell=True does have some security considerations if you deal with untrusted data though (namely shell injection), so I don't advise going that route for this particular task of yours. You can read more here: https://docs.python.org/3/library/subprocess.html#security-considerations
The simple solution for your task would be to use a stdout parameter on creation of subprocess.Popen object:
import subprocess with open('output.txt', 'w') as f: p = subprocess.Popen(['/path/to/executable', 'arg1'], stdout=f) Reference from the documentation
stdin, stdout and stderr specify the executed program’s standard input, standard output and standard error file handles, respectively. Valid values are PIPE, DEVNULL, an existing file descriptor (a positive integer), an existing file object, and None. PIPE indicates that a new pipe to the child should be created. DEVNULL indicates that the special file os.devnull will be used. With the default settings of None, no redirection will occur; the child’s file handles will be inherited from the parent. Additionally, stderr can be STDOUT, which indicates that the stderr data from the applications should be captured into the same file handle as for stdout.
terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc terminate called after throwing an instance of 'std::length_error' what(): cannot create std::vector larger than max_size()and I don't catch any error while running that command from shell manually. The script.out file is working finewith open(some_file_path, "w") as file_handle: file_handle.write(some_data)