Is there a way to run shell commands without output buffering?
For example, hexdump file | ./my_script will only pass input from hexdump to my_script in buffered chunks, not line by line.
What is a general solution to make the output for any command unbuffered?
Try stdbuf, included in GNU coreutils and thus virtually any Linux distro. This sets the buffer length for input, output and error to zero:
stdbuf -i0 -o0 -e0 command stdbuf works by modifying the environment variable LD_PRELOAD (see man 8 ld.so). This means it only works for executables that dynamically load libraries using ld.so without setuid.
unbuffer for me. stdbuf passed any signals (SIGUSR2 in my case) I sent to it to the command (which is what I wanted to happen), while unbuffer didn't seem to want to.
stdbuf -o0 bash runs a complete session with all my scripts and aliases available, and with no output buffering on any of the commands, which is exactly what I wanted. Buffering was of course restored on exit from this bash instance. I'm using Ubuntu 16.04.
if [[ "$0" == "-bash" ]]; then exec /usr/bin/stdbuf -i0 -oL -eL /bin/bash -l; fi
exec pattern for sending output to a logfile, with all output line buffered rather than buffering fully disabled: exec > >(stdbuf -i0 -oL -eL awk '{print strftime("%Y-%m-%d %H:%M:%S"), $0 }' | stdbuf -i0 -oL -eL tee "$LOGFILE") 2>&1The command unbuffer from the expect package disables the output buffering:
Ubuntu Manpage: unbuffer - unbuffer output
Example usage:
unbuffer hexdump file | ./my_script
AFAIK, you can't do it without ugly hacks. Writing to a pipe (or reading from it) automatically turns on full buffering and there is nothing you can do about it :-(. "Line buffering" (which is what you want) is only used when reading/writing a terminal. The ugly hacks exactly do this: They connect a program to a pseudo-terminal, so that the other tools in the pipe read/write from that terminal in line buffering mode. The whole problem is described here:
The page has also some suggestions (the aforementioned "ugly hacks") what to do, i.e. using unbuffer or pulling some tricks with LD_PRELOAD.
You could also use the script command to make the output of hexdump line-buffered (hexdump will be run in a pseudo terminal which tricks hexdump into thinking its writing its stdout to a terminal, and not to a pipe).
# cf. http://unix.stackexchange.com/questions/25372/turn-off-buffering-in-pipe/ stty -echo -onlcr script -q /dev/null hexdump file | ./my_script # FreeBSD, Mac OS X script -q -c "hexdump file" /dev/null | ./my_script # Linux stty echo onlcr 
stdbuf failed, but script works like a charm. The program buffers the output when it is being piped to tee, but not when running in terminal. So, script works! Thanks!
script's man page as referenced by this, "Certain interactive commands, such as vi(1), create garbage in the typescript file. Script works best with commands that do not manipulate the screen, the results are meant to emulate a hardcopy terminal."One should use grep or egrep "--line-buffered" options to solve this. no other tools needed.
Note that some applications have application-specific ways to disable buffering. For example, to disable normal buffering in grep, use the --line-buffering command-line argument:
grep --line-buffering Likewise, the --unbuffered argument for sed disables normal buffering and enables line buffering:
sed --unbuffered The -u argument for Python enables line buffering:
python -u The --unbuffered argument for jq flushes the output when each JSON object is printed:
jq --unbuffered