18

I'm trying to create a script which is using multiprocessing module with python. The script (lets call it myscript.py) will get the input from another script with pipe.

Assume that I call the scripts like this;

$ python writer.py | python myscript.py

And here is the codes; 

// writer.py import time, sys def main(): while True: print "test" sys.stdout.flush() time.sleep(1) main() //myscript.py def get_input(): while True: text = sys.stdin.readline() print "hello " + text time.sleep(3) if __name__ == '__main__': p1 = Process(target=get_input, args=()) p1.start()

this is clearly not working, since the sys.stdin objects are different for main process and p1. So I have tried this to solve it, 

//myscript.py def get_input(temp): while True: text = temp.readline() print "hello " + text time.sleep(3) if __name__ == '__main__': p1 = Process(target=get_input, args=(sys.stdin,)) p1.start()

but I come across with this error; 

Process Process-1: Traceback (most recent call last): File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run self._target(*self._args, **self._kwargs) File "in.py", line 12, in get_input text = temp.readline() ValueError: I/O operation on closed file

So, I guess that main's stdin file closed and I can't read from it. At this conjunction, how can I pass main's stdin file to another process? If passing stdin is not possible, how can I use main's stdin from another process?

update: Okay, I need to clarify my question since people think using multiprocessing is not really necessary. consider myscript.py like this;

//myscript.py def get_input(): while True: text = sys.stdin.readline() print "hello " + text time.sleep(3) def do_more_things(): while True: #// some code here time.sleep(60*5) if __name__ == '__main__': p1 = Process(target=get_input, args=()) p1.start() do_more_things()

so, I really need to run get_input() function parallelly with main function (or other sub processes). Sorry for the conflicts, I have a decent English, and I guess I couldn't be clear on this question. I would appreciate if you guys can tell me if i can use the main processes STDIN object in another process.

thanks in advance.

Improve this question
10
  • 1
    you can write to stdin using the syntax found here stackoverflow.com/questions/8880461/… There it is used for reading but you can use the same objects Commented Jan 23, 2012 at 19:05
  • Why does myscript.py create a subprocess? Why doesn't it simply read from sys.stdin? If it read from sys.stdin the shell script would work perfectly. Why create a subprocess? Commented Jan 23, 2012 at 19:13
  • Hi S.Lott, This is just a prototype of my problem, myscript.py contains another process which has another infinite loop and runs like a daemon, so I just can't read it in myscript main process since writer.py is also infinite and needs to be run as a daemon. @JohanLundberg, thanks for your advice, i'll check it. Commented Jan 23, 2012 at 19:37
  • 1
    Isn't this just ( python writer.py | python myscript.py ) & python do_more_things.py? I don't see why this isn't simply three separate Python programs. Two form a trivial pipeline, and the third is totally unrelated. Commented Jan 23, 2012 at 21:49
  • 1
    similar to: stackoverflow.com/questions/1450393/… Commented Jan 24, 2012 at 3:03

4 Answers 4

Reset to default
12

The simplest thing is to swap get_input() and do_more_things() i.e., read sys.stdin in the parent process:

def get_input(stdin): for line in iter(stdin.readline, ''): print("hello", line, end='') stdin.close() if __name__ == '__main__': p1 = mp.Process(target=do_more_things) p1.start() get_input(sys.stdin)

The next best thing is to use a Thread() instead of a Process() for get_input():

if __name__ == '__main__': t = Thread(target=get_input, args=(sys.stdin,)) t.start() do_more_things()

If the above doesn't help you could try os.dup():

newstdin = os.fdopen(os.dup(sys.stdin.fileno())) try: p = Process(target=get_input, args=(newstdin,)) p.start() finally: newstdin.close() # close in the parent do_more_things()
Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

although I just needed to third one, the other solutions are pretty working. thank's for taking time and helping comprehensively.
3

Each new process created with the multiprocessing module gets its own PID, and therefore it's own standard input device and output devices, even if they're both writing to the same terminal, hence the need for locks.

You're already creating two processes by separating the content into two scripts, and creating a third process with get_input(). get_input could read the standard input if it was a thread instead of a process. Then, no need to have a sleep function in the reader.

## reader.py from threading import Thread import sys def get_input(): text = sys.stdin.readline() while len(text) != 0: print 'hello ' + text text = sys.stdin.readline() if __name__ == '__main__': thread = Thread(target=get_input) thread.start() thread.join()
Improve this answer

Comments

2

This will only be a partial answer - as I'm unclear about subsequent parts of the question.

You start by saying that you anticipate calling your scripts:

$ python writer.py | python myscript.py

If you're going to do that, writer needs to write to standard out and myscript needs to read from standard input. The second script would look like this:

def get_input(): while True: text = sys.stdin.readline() print "hello " + text time.sleep(3) if __name__ == '__main__': get_input()

There's no need for the multiprocessing.Process object... you're firing off two processes from the command line already - and you're using the shell to connect them with an (anonymous) pipe (the "|" character) that connects standard output from the first script to standard input from the second script.

The point of the Process object is to manage launch of a second process from the first. You'd need to define a process; then start it - then you'd probably want to wait until it has terminated before exiting the first process... (calling p1.join() after p1.start() would suffice for this).

If you want to communicate between a pair of processes under python control, you'll probably want to use the multiprocess.Pipe object to do so. You can then easily communicate between the inital and the subordinate spawned process by reading and writing to/from the Pipe object rather than standard input and standard output. If you really want to re-direct standard input and standard output, this is probably possible by messing with low-level file-descriptors and/or by overriding/replacing the sys.stdin and sys.stdout objects... but, I suspect, you probably don't want (or need) to do this.

Improve this answer

Comments

1

To read the piped in input use fileinput:

myscript.py 

import fileinput if __name__ == '__main__': for line in fileinput.input(): #do stuff here process_line(line)
Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.