2

I am writing a large python script with various subprocess.call to execute commands available in the system, and I have an issue because the output differs if it is printed into the terminal or if is redirected to a file.

For reproduce the problem, this is a small part of the script:

#!/usr/bin/env python # -*- coding: utf-8 -*- from subprocess import call print "Hello at first" call(["rsync", "-aHvz", "root@server1:/tmp/a", '.']) print "Hello at the end"

Executing it from terminal return that in the correct order, print + rsync + print:

$ python keepconf.py Hello at the first receiving incremental file list sent 12 bytes received 123 bytes 270.00 bytes/sec total size is 55858143 speedup is 413764.02 Hello at the end

Executing the same, but redirecting the output to a file:

$ python keepconf.py > /tmp/output $ cat /tmp/output receiving incremental file list sent 12 bytes received 123 bytes 270.00 bytes/sec total size is 55858143 speedup is 413764.02 Hello at the first Hello at the end

The order now is rsync + print + print. Why is this behaviour?

Improve this question
1
  • And with sys.stdout.flush() after your prints ? Commented Jan 15, 2015 at 16:03

1 Answer 1

Reset to default
4

Output for a terminal (or more precisely, a tty) is typically opened in line buffer mode in Python. When you use a pipe, Python'll use a different buffer, of a fixed size.

This means that when you write text with a newline, the buffer is automatically flushed when printing to a terminal, but to a pipe it'll only be flushed when the buffer is full or a flush is forced (such as when the Python script exits).

In other words, when writing to the terminal the first print line is flushed out to the terminal before the rsync command is run. When redirecting to a pipe, the text is kept in a buffer, the rsync command output is run (writing to the pipe when it flushes, at least once at the end but possibly more often), after which you write some more to the buffer and that buffer is flushed to the pipe when Python exists.

You can force the flush manually:

import sys # .... print "Hello at first" sys.stdout.flush() call(["rsync", "-aHvz", "root@server1:/tmp/a", '.']) print "Hello at the end" sys.stdout.flush()
Improve this answer
Sign up to request clarification or add additional context in comments.

4 Comments

upvote. Correction: sys.stdout (and C stdout stream where the subprocess writes) may be line-buffered if it refers to a tty (common for stdio-based programs such as python2). Terminal editing mode (raw, line) is unrelated here.
@J.F.Sebastian: Qualified it; I habitually refer to it a a terminal for short, but having actually used physical terminals, you are correct that that is not precise enough.
no. I meant that sys.stdout Python object, C stdout (FILE*) (program properties -- inside) are different from terminal (stty may change its parameters). The latter exists before/during/after the program (outside). The buffers are inside the processes.
@J.F.Sebastian: right, I am talking about the buffers in the Python process here.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.