2

I read that xargs was good for processing the output of a command one line at a time (and it is). I have the following line in my script.

./gen-data | awk '{printf $2 " "; printf $1=$2=$3=""; gsub (" ", "", $0);if(length($0) == 0){ print "0000"} else{print $0}}' | xargs -t -n2 -P1 bash -c 'datatojson "$@"' _ 

It produces the right output, there is no question of that. However, gen-data produces something like 1000 lines, and what I really would like is for this command to execute after each line, not after 1000 lines (It's clearly stopping regularly to get more input).

Here is what gen-data looks like:

candump $interface & while true; do while read p; do cansend $interface $(echo $p | awk 'NF>1{print $NF}'); done < <(shuf $indoc) done 

(cansend sends data to an interface and candump reads from that interface and outputs it onto the screen, but I wager that's not too relevant). In any case candump seems to be continuously streaming output, but when I pipe that into awk and xargs, it becomes chunked. Is it just because I used shuf? I would think that since it's going through the interface, and being read on the other side, it would be less chunked than shuf provides.

3
  • Wy are you using shuf? Does this script work without using it? (I know it should, but it is necessary to rule it out) Commented Jul 27, 2015 at 14:33
  • @FelipeLema I just tried it with cat instead of shuf and it's doing the same thing. Not sure how else I can feed it into that loop, I'm not very experienced in scripting. Commented Jul 27, 2015 at 15:09
  • See unix.stackexchange.com/questions/117501/… Commented May 9, 2019 at 8:22

1 Answer 1

3

You can try the same command, this time using multiple hacks to avoid buffering:

./gen-data | gawk '{printf $2 " "; printf $1=$2=$3=""; gsub (" ", "", $0);if(length($0) == 0){ print "0000"} else{print $0}; fflush(stdout)}' | stdbuf -o0 xargs -t -n2 -P1 bash -c 'datatojson "$@"' _ 

Mind the change from awk to gawk and the use of fflush. You can also try mawk -Winteractive. Also mind that I added stdbuf -o0 before xargs. You can also try the latest at the beginning with ./gen-data

2
  • Worked perfectly. What does the -o0 do on the stdbuf command? Commented Jul 27, 2015 at 15:28
  • 1
    As stated in the stdbuf man page: sets the buffering of the command as "unbuffered" Commented Jul 27, 2015 at 18:52

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.