bash: force exec'd process to have unbuffered stdout

Question

I've got a script like:

#!/bin/bash exec /usr/bin/some_binary > /tmp/my.log 2>&1

Problem is that some_binary sends all of its logging to stdout, and buffering makes it so that I only see output in chunks of a few lines. This is annoying when something gets stuck and I need to see what the last line says.

Is there any way to make stdout unbuffered before I do the exec that will affect some_binary so it has more useful logging?

(The wrapper script is only setting a few environment variables before the exec, so a solution in perl or python would also be feasible.)

Dennis's answer is exactly right. Here's why. Programs on Unix that use C stdio for output will buffer when the output is not going to a terminal (boosts performance a lot) unless told not to with setvbuf; hardly anyone bothers with that. The expect system (of which unbuffer is a trivial application) uses black magic with Unix pseudo-terminal devices (ptys) to run programs with a terminal that isn't a real terminal so that the wrapped program doesn't buffer its output. It's clever, but be warned: most systems only have a limited number of ptys; a naturally unbuffered program is better. — Donal Fellows
– Donal Fellows, Commented Jul 26, 2010 at 8:32
BTW, this doesn't work so well on Windows. That's because expect on that platform uses debugging magic instead of terminal magic (because this is an area where Windows is very different). — Donal Fellows
– Donal Fellows, Commented Jul 26, 2010 at 8:34
Thanks. I decided to opt for modifying some_binary to do the right thing instead of relying on the black magic. expect is great, but I don't want it here... — bstpierre
– bstpierre, Commented Jul 26, 2010 at 14:09

Schlueter · Accepted Answer · 2018-09-07 19:47:44Z

GNU coreutils-8.5 also has the stdbuf command to modify I/O stream buffering:

http://www.pixelbeat.org/programming/stdio_buffering/

So, in your example case, simply invoke:

stdbuf -oL /usr/bin/some_binary > /tmp/my.log 2>&1

This will allow text to appear immediately line-by-line (once a line is completed with the end-of-line "\n" character in C). If you really want immediate output, use -o0 instead.

This way could be more desirable if you do not want to introduce dependency to expect via unbuffer command. The unbuffer way, on the other hand, is needed if you have to fool some_binary into thinking that it is facing a real tty standard output.

Dennis Williamson · Accepted Answer · 2010-07-26 08:23:17Z

14

You might find that the unbuffer script that comes with expect may help.

edited Jul 26, 2010 at 8:23

answered Jul 26, 2010 at 3:35

Dennis Williamson

364k95 gold badges386 silver badges446 bronze badges

2 Comments

Davide Over a year ago

Does not work with the echo -n . or prinff "." which I need in my case to show a progressbar in bash

Mihai Danila Over a year ago

See also: stackoverflow.com/questions/1401002/…

Community · Accepted Answer · 2017-05-23 12:33:39Z

Some command line programs have an option to modify their stdout stream buffering behaviour. So that's the way to go if the C source is available ...

# two command options ... man file | less -p '--no-buffer' man grep | less -p '--line-buffered' # ... and their respective source code # from: http://www.opensource.apple.com/source/file/file-6.2.1/file/src/file.c if(nobuffer) (void) fflush(stdout); # from: http://www.opensource.apple.com/source/grep/grep-28/grep/src/grep.c if (line_buffered) fflush (stdout);

As an alternative to using expect's unbuffer script or modifying the program's source code, you may also try to use script(1) to avoid stdout hiccups caused by a pipe:

See: Trick an application into thinking its stdin is interactive, not a pipe

# Linux script -c "[executable string]" /dev/null # FreeBSD, Mac OS X script -q /dev/null "[executable string]"

On Mac OS X, because "script" may change return code from "\n" to "\r\n", I needed to run like this: script -q /dev/null commands... | perl -pe 's/\n/\r\n/g'

Dexter Legaspi · Accepted Answer · 2016-02-10 18:50:57Z

I scoured the internets for an answer, and none of this worked for uniq which is too stubborn to buffer everything except for stdbuf:

{piped_command_here} | stdbuf -oL uniq | {more_piped_command_here}

Otheus · Accepted Answer · 2013-12-23 10:57:28Z

GNU Coreutils-8 includes a program called stdbuf which essentially does the LD_PRELOAD trick. It works on Linux and reportedly works on BSD systems.

EKM · Accepted Answer · 2016-07-19 23:00:56Z

An environment variable can set the terminal IO mode to unbuffered.

export NSUnbufferedIO=YES

This will set the terminal unbuffered for both C and Ojective-C terminal output commands.

Collectives™ on Stack Overflow

bash: force exec'd process to have unbuffered stdout

6 Answers 6

Comments

2 Comments

1 Comment

Comments

Comments

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

6 Answers 6

Comments

2 Comments

1 Comment

Comments

Comments

Comments

Linked

Related