python stdout flush and tee

Question

The following code ends with broken pipe when piped into tee, but behave correctly when not piped :

#!/usr/bin/python import sys def testfun(): while 1: try : s = sys.stdin.readline() except(KeyboardInterrupt) : print('Ctrl-C pressed') sys.stdout.flush() return print s if __name__ == "__main__": testfun() sys.exit()

Expected output :

./bug.py Ctrl-C pressed

What is observed when piped into tee is either a broken pipe or no output at all, ie nothing on tee stdout, and nothing in bug.log :

./bug.py | tee bug.log Traceback (most recent call last): File "./bug.py", line 14, in <module> sys.stdout.flush() IOError: [Errno 32] Broken pipe

What can be the reason for this ?

GED · Accepted Answer · 2011-08-31 00:52:17Z

Nope, hitting Ctrl-C does NOT terminate both processes. It terminates the tee process only, the end of the tee process close the pipe between your script and tee, and hence your script dies with the broken pipe message.

To fix that, tee has an option to pass the Ctrl-C to its previous process in the pipe: -i

try: man tee

./bug.py ^CCtrl-C pressed ./bug.py | tee log ^CTraceback (most recent call last): File "./bug.py", line 14, in <module> testfun() File "./bug.py", line 9, in testfun sys.stdout.flush() IOError: [Errno 32] Broken pipe ./bug.py | tee -i log ^CCtrl-C pressed

This fixed the problem for me, but I did have to add some sys.stdout.flush() calls, which were not necessary when going to stdout only.
"It terminates the tee process only" -- this doesn't seem correct. When I did the experiment ./script | tee out, the script told me it dies due to keyboard interrupt instead of broken pipe. According to this thread, OS does send SIGINT to all process in the group. Besides that, the proposed -i argument does solve what I want. Thanks!

shodanex · Accepted Answer · 2009-07-28 09:25:35Z

This is not a python problem but a shell problem, as pointed by Brian, hitting Ctrl-C will terminate both process. Workaround is to use named pipes :

mknod mypipe p cat mypipe | tee somefile.log & ./bug.py > mypipe

Brian Agnew · Accepted Answer · 2009-07-28 09:10:07Z

When you hit Ctrl-C, the shell will terminate both processes (python and tee) and tear down the pipe connecting them.

So when you handle the Ctrl-C in your python process and flush, it'll find that tee has been terminated and the pipe is no more. Hence the error message.

(As an aside, would you expect anything in the log ? I don't see your process outputting anything other than the flush on exit)

Of course. Unfortunately the 'tee' process will probably have gone by then. I would simply redirect the python std out to a file, and then tail that file. Or, depending on your shell, you can probably do something fancy to redirect and still write to the console.

Collectives™ on Stack Overflow

python stdout flush and tee

3 Answers 3

2 Comments

Comments

3 Comments

Linked

Hot Network Questions