How can I stream input into a python Popen subprocess and stream output from it?

Question

I'm attempting to basically create a shell in python using subprocess.Popen. As boilerplate code for testing purposes, this is what I have:

if __name__ == '__main__': ps = subprocess.Popen( input('command? '), shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True ) print(ps.stdout.read()) time.sleep(1) # provide enough time to register completion if the command is a one-off, like dir or echo while ps.poll() == None: # ps.poll() returns None if still running try: # method one ps.stdin.write(input()) ps.communicate() print(ps.stdout.read()) # method two ps.stdin.write(input()) print(ps.communicate()[0]) # method three print(ps.communicate(input())[0]) except: continue print('Process Finished')

Each method is a different (failed) attempt.

For commands like python which should open the python CLI interpreter, this completely fails. However, for one-off commands like dir or ls or even running python -c "<some python>" it works just fine.

CLI log:

C:\Users\nj2u2\Desktop\test>python test.py command? echo hello hello Process Finished C:\Users\nj2u2\Desktop\test>python test.py command? dir Volume in drive C has no label. Volume Serial Number is D6B7-6B8D Directory of C:\Users\nj2u2\Desktop\test 07/03/2020 12:26 AM <DIR> . 07/03/2020 12:26 AM <DIR> .. 07/03/2020 08:20 PM 6,811 subprocess_plus.py 07/04/2020 12:55 PM 580 test.py 07/03/2020 08:25 PM <DIR> __pycache__ 2 File(s) 7,391 bytes 3 Dir(s) 1,487,446,302,720 bytes free Process Finished C:\Users\nj2u2\Desktop\test>python test.py command? python

After that last command, python, it just hangs at print(ps.stdout.read()).

I'd like to know why it's hanging, and how I can fix it.

If your script works with some commands and not others, perhaps the issue is not related to your script but to how those "failing" commands are implemented. Have you tried simply opening the python CLI as a subprocess, and sending it a short, hard-coded command? Does that work? If it doesn't then your script is not going to work, and there is nothing you can do about it. Programs are not required to support the customary stdin/stdout protocol, so expect that some of them will not work with your approach. — Paul Cornelius
– Paul Cornelius, Commented Jul 4, 2020 at 3:01

Serge Ballesta · Accepted Answer · 2020-07-15 09:35:32Z

The communicate method can only be used for non interactive commands because it sends a one shot input and then waits for the sub process to terminate. That is the reason that it can only work with what you call one-off commands.

But beware, pipes do not support the terminal discipline in which a read call automatically flushes the output, and where a new line causes a read to return immediately. So nothing guarantees that you can use that to provide interactive IO to the python CLI interpretor. Pipes are just streams, and can be buffered so you can just be sure that all the input will be delivered, and that when the output channels will be closed (i.e. when the sub-process command will terminate) you will receive all the output. The way input and output are intermixed depends on the way the sub-process command is programmed. If it willingly flushes its output before reading, and you do the same in your code, then interactive processing is possible, but if it does not, there is no way to expect to receive something after each end of line.

Collectives™ on Stack Overflow

How can I stream input into a python Popen subprocess and stream output from it?

1 Answer 1

Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Related