350

If I do the following:

import subprocess from cStringIO import StringIO subprocess.Popen(['grep','f'],stdout=subprocess.PIPE,stdin=StringIO('one\ntwo\nthree\nfour\nfive\nsix\n')).communicate()[0]

I get:

Traceback (most recent call last): File "<stdin>", line 1, in ? File "/build/toolchain/mac32/python-2.4.3/lib/python2.4/subprocess.py", line 533, in __init__ (p2cread, p2cwrite, File "/build/toolchain/mac32/python-2.4.3/lib/python2.4/subprocess.py", line 830, in _get_handles p2cread = stdin.fileno() AttributeError: 'cStringIO.StringI' object has no attribute 'fileno'

Apparently a cStringIO.StringIO object doesn't quack close enough to a file duck to suit subprocess.Popen. How do I work around this?

Improve this question
3

12 Answers 12

Reset to default
399

Popen.communicate() documentation:

Note that if you want to send data to the process’s stdin, you need to create the Popen object with stdin=PIPE. Similarly, to get anything other than None in the result tuple, you need to give stdout=PIPE and/or stderr=PIPE too.

Replacing os.popen*

 pipe = os.popen(cmd, 'w', bufsize) # ==> pipe = Popen(cmd, shell=True, bufsize=bufsize, stdin=PIPE).stdin

Warning Use communicate() rather than stdin.write(), stdout.read() or stderr.read() to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process.

So your example could be written as follows:

from subprocess import Popen, PIPE, STDOUT p = Popen(['grep', 'f'], stdout=PIPE, stdin=PIPE, stderr=STDOUT) grep_stdout = p.communicate(input=b'one\ntwo\nthree\nfour\nfive\nsix\n')[0] print(grep_stdout.decode()) # -> four # -> five # ->

On Python 3.5+ (3.6+ for encoding), you could use subprocess.run, to pass input as a string to an external command and get its exit status, and its output as a string back in one call:

#!/usr/bin/env python3 from subprocess import run, PIPE p = run(['grep', 'f'], stdout=PIPE, input='one\ntwo\nthree\nfour\nfive\nsix\n', encoding='ascii') print(p.returncode) # -> 0 print(p.stdout) # -> four # -> five # ->
Improve this answer
Sign up to request clarification or add additional context in comments.

14 Comments

This is NOT a good solution. In particular, you cannot asynchronously process p.stdout.readline output if you do this since you'd have to wait for the entire stdout to arrive. It's is also memory-inefficient.
@OTZ What's a better solution?
@Nick T: "better" depends on context. Newton's laws are good for the domain they are applicable but you need special relativity to design GPS. See Non-blocking read on a subprocess.PIPE in python.
But note the NOTE for communicate: "do not use this method if the data size is large or unlimited"
You need python 3.6 to use the input arg with subprocess.run(). Older versions of python3 work if you do this: p = run(['grep', 'f'], stdout=PIPE, input=some_string.encode('ascii'))
|
50

I figured out this workaround:

>>> p = subprocess.Popen(['grep','f'],stdout=subprocess.PIPE,stdin=subprocess.PIPE) >>> p.stdin.write(b'one\ntwo\nthree\nfour\nfive\nsix\n') #expects a bytes type object >>> p.communicate()[0] 'four\nfive\n' >>> p.stdin.close()

Is there a better one?

Improve this answer

6 Comments

@Moe: stdin.write() usage is discouraged, p.communicate() should be used. See my answer.
Per the subprocess documentation: Warning - Use communicate() rather than .stdin.write, .stdout.read or .stderr.read to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process.
I think this is good way to do it if you're confident that your stdout/err won't ever fill up (for instance, it's going to a file, or another thread is eating it) and you have an unbounded amount of data to be sent to stdin.
In particular, doing it this way still ensures that stdin is closed, so that if the subprocesses is one that consumes input forever, the communicate will close the pipe and allow the process to end gracefully.
@Lucretiel, if the process consumes stdin forever, then presumably it can still write stdout forever, so we'd need completely different techniques all-round (can't read() from it, as communicate() does even with no arguments).
|
45

There's a beautiful solution if you're using Python 3.4 or better. Use the input argument instead of the stdin argument, which accepts a bytes argument:

output_bytes = subprocess.check_output( ["sed", "s/foo/bar/"], input=b"foo", )

This works for check_output and run, but not call or check_call for some reason.

In Python 3.7+, you can also add text=True to make check_output take a string as input and return a string (instead of bytes):

output_string = subprocess.check_output( ["sed", "s/foo/bar/"], input="foo", text=True, )
Improve this answer

2 Comments

This is the best answer for Python 3.4+ (using it in Python 3.6). It indeed does not work with check_call but it works for run. It also works with input=string as long as you pass an encoding argument too according to the documentation.
@Flimm the reason is obvious: run and check_output use communicate under the hood, while call and check_call do not. communicate is much more heavy, since it involves select to work with streams, call and check_call are much simplier and faster.
33

I'm a bit surprised nobody suggested creating a pipe, which is in my opinion the far simplest way to pass a string to stdin of a subprocess:

read, write = os.pipe() os.write(write, "stdin input here") os.close(write) subprocess.check_call(['your-command'], stdin=read)
Improve this answer

6 Comments

The os and the subprocess documentation both agree that you should prefer the latter over the former. This is a legacy solution which has a (slightly less concise) standard replacement; the accepted answer quotes the pertinent documentation.
I'm not sure that's correct, tripleee. The quoted documentation says why it is hard to use the pipes created by the process, but in this solution it creates a pipe and passes it in. I believe it avoids the potential deadlock problems of managing the pipes after the process has already started.
os.popen is deprecated in favour of subprocess
-1: it leads to the deadlock, it may loose data. This functionality is already provided by the subprocess module. Use it instead of reimplementing it poorly (try to write a value that is larger than an OS pipe buffer)
@tripleee the implementation of pipes in subprocess module is laughably bad, and is impossible to control. You cannot even get the information about the size of the built-in buffer, not to mention, you cannot tell it what are the read and write ends of the pipe, nor can you change the built-in buffer. In not so many words: subprocess pipes are trash. Don't use them.
|
15

I am using python3 and found out that you need to encode your string before you can pass it into stdin:

p = Popen(['grep', 'f'], stdout=PIPE, stdin=PIPE, stderr=PIPE) out, err = p.communicate(input='one\ntwo\nthree\nfour\nfive\nsix\n'.encode()) print(out)
Improve this answer

3 Comments

You don't specifically need to encode the input, it just wants a bytes-like object (e.g. b'something'). It will return err and out as bytes also. If you want to avoid this, you can pass universal_newlines=True to Popen. Then it will accept input as str and will return err/out as str also.
But beware, universal_newlines=True will also convert your newlines to match your system
If you're using Python 3, see my answer for an even more convenient solution.
12

Apparently a cStringIO.StringIO object doesn't quack close enough to a file duck to suit subprocess.Popen

I'm afraid not. The pipe is a low-level OS concept, so it absolutely requires a file object that is represented by an OS-level file descriptor. Your workaround is the right one.

Improve this answer

Comments

11 
from subprocess import Popen, PIPE from tempfile import SpooledTemporaryFile as tempfile f = tempfile() f.write('one\ntwo\nthree\nfour\nfive\nsix\n') f.seek(0) print Popen(['/bin/grep','f'],stdout=PIPE,stdin=f).stdout.read() f.close()
Improve this answer

1 Comment

fyi, tempfile.SpooledTemporaryFile.__doc__ says: Temporary file wrapper, specialized to switch from StringIO to a real file when it exceeds a certain size or when a fileno is needed.
7 
""" Ex: Dialog (2-way) with a Popen() """ p = subprocess.Popen('Your Command Here', stdout=subprocess.PIPE, stderr=subprocess.STDOUT, stdin=PIPE, shell=True, bufsize=0) p.stdin.write('START\n') out = p.stdout.readline() while out: line = out line = line.rstrip("\n") if "WHATEVER1" in line: pr = 1 p.stdin.write('DO 1\n') out = p.stdout.readline() continue if "WHATEVER2" in line: pr = 2 p.stdin.write('DO 2\n') out = p.stdout.readline() continue """ .......... """ out = p.stdout.readline() p.wait()
Improve this answer

1 Comment

Because shell=True is so commonly used for no good reason, and this is a popular question, let me point out that there are a lot of situations where Popen(['cmd', 'with', 'args']) is decidedly better than Popen('cmd with args', shell=True) and having the shell break the command and arguments into tokens, but not otherwise providing anything useful, while adding a significant amount of complexity and thus also attack surface.
7

On Python 3.7+ do this:

my_data = "whatever you want\nshould match this f" subprocess.run(["grep", "f"], text=True, input=my_data)

and you'll probably want to add capture_output=True to get the output of running the command as a string.

On older versions of Python, replace text=True with universal_newlines=True:

subprocess.run(["grep", "f"], universal_newlines=True, input=my_data)
Improve this answer

Comments

6

Beware that Popen.communicate(input=s)may give you trouble ifsis too big, because apparently the parent process will buffer it before forking the child subprocess, meaning it needs "twice as much" used memory at that point (at least according to the "under the hood" explanation and linked documentation found here). In my particular case,swas a generator that was first fully expanded and only then written tostdin so the parent process was huge right before the child was spawned, and no memory was left to fork it:

File "/opt/local/stow/python-2.7.2/lib/python2.7/subprocess.py", line 1130, in _execute_child self.pid = os.fork() OSError: [Errno 12] Cannot allocate memory

Improve this answer

Comments

3

This is overkill for grep, but through my journeys I've learned about the Linux command expect, and the python library pexpect

  • expect: dialogue with interactive programs
  • pexpect: Python module for spawning child applications; controlling them; and responding to expected patterns in their output.
import pexpect child = pexpect.spawn('grep f', timeout=10) child.sendline('text to match') print(child.before)

Working with interactive shell applications like ftp is trivial with pexpect

import pexpect child = pexpect.spawn ('ftp ftp.openbsd.org') child.expect ('Name .*: ') child.sendline ('anonymous') child.expect ('Password:') child.sendline ('[email protected]') child.expect ('ftp> ') child.sendline ('ls /pub/OpenBSD/') child.expect ('ftp> ') print child.before # Print the result of the ls command. child.interact() # Give control of the child to the user.
Improve this answer

Comments

2 
p = Popen(['grep', 'f'], stdout=PIPE, stdin=PIPE, stderr=STDOUT) p.stdin.write('one\n') time.sleep(0.5) p.stdin.write('two\n') time.sleep(0.5) p.stdin.write('three\n') time.sleep(0.5) testresult = p.communicate()[0] time.sleep(0.5) print(testresult)
Improve this answer

1 Comment

NameError: global name 'PIPE' is not defined

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.