Simultaneously reading stdin and writing to stdout in python

Question

I am new to threading so I feel as if I am missing an obvious point, but I couldn't find an previous question that pertained to this subject.

I want to make a program that writes to stdin and reads stdout of a c program. This is the code in the main program.

from subprocess import Popen, PIPE from threading import Thread from Queue import Queue, Empty from os import getcwd import time import random chatter = Queue(maxsize=10) # Queue of strings to be sent to the program class Chatter(): def stream_talker(self, identifier, stream): while True: if not chatter.empty(): self.proc.stdin.write(chatter.get(True, 1)) def stream_watcher(self, identifier, stream): while True: for line in stream: print line def main(self): self.proc = Popen(getcwd() + '/main', stdout=PIPE, stdin=PIPE) Thread(target=self.stream_talker, name='stdin-talker', args=('STDIN', self.proc.stdin)).start() Thread(target=self.stream_watcher, name='stdout-listening', args=('STDOUT', self.proc.stdout)).start() while True: chat = raw_input('Enter chatter: ') if len(chat) > 0: chatter.put(chat) if __name__ == '__main__': chatt = Chatter() chatt.main()

And here is the main.c program it invokes.

#include <stdio.h> #include <stdlib.h> int main(){ while (1){ int bytes_read; size_t nbytes = 100; char *my_string; my_string = (char *)malloc(nbytes + 1); bytes_read = getline (&my_string, &nbytes, stdin); if (bytes_read == -1) { puts ("ERROR!"); } else{ puts (my_string); } free(my_string); } return 0; }

The current issue is that while it will run, stdout is never printed.

1. you use stream_talker twice. You want to run stream_watcher too. 2. You should force line-buffering 3. It is enough to start a single thread to feed input and read output at the same time — jfs
– jfs, Commented Feb 11, 2015 at 18:16
@J.F.Sebastian I am looking into both of those answers now thank you for the links. — Liam Pieri
– Liam Pieri, Commented Feb 11, 2015 at 18:32

the paul · Accepted Answer · 2015-02-11 17:58:11Z

3

It looks like your main problem is that you invoke two stream_talker()s and no stream_watcher().

In addition, you probably don't want to busy-wait on your Queue (because that defies the whole point of using a Queue). Your code polls chatter.empty() as fast as it can until the queue has something in it. Use chatter.get() directly instead; it will block until something is available or until it hits your timeout.

Finally, you might save yourself some future confusion if you write to the stream argument in stream_talker(), and not hardcode self.proc.stdin.

edited Feb 11, 2015 at 17:58

answered Feb 11, 2015 at 17:52

the paul

9,1991 gold badge40 silver badges56 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

jfs Over a year ago

there is also the block-buffering issue and the read-ahead bug

Liam Pieri Over a year ago

Thank you for bringing that obvious stream_talker bug to my attention. However omitting the chatter.empty() results in the empty exception being raised, I put that in place to make sure that doesn't happen. Thanks for pointing out that I don't have to hardcode it as well.

the paul Over a year ago

The exception is raised after the timeout is hit (you've specified a timeout of 1 second) if the queue is still empty. If that's not what you want, just call .get() without parameters.

jfs Over a year ago

@LiamPieri: you could replace the while True loop with: for line in iter(chatter.get, None): stream.write(line)

Liam Pieri Over a year ago

@thepaul I see thank you for the clarification, I had forgotten what that argument was for.

|

Collectives™ on Stack Overflow

Simultaneously reading stdin and writing to stdout in python

1 Answer 1

6 Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Linked

Related