The issue of jq needing an explicit filter when the output is redirected is discussed all over the web. But I'm unable to redirect output if jq is part of a pipe chain, even when an explicit filter is in use.
Consider:
touch in.txt tail -f in.txt | jq '.f1' # in a different terminal: echo '{"f1":1,"f2":2}' >> in.txt echo '{"f1":3,"f2":2}' >> in.txt As expected, the output in the original terminal from the jq command is:
1 3 But if I add any sort of redirection or piping to the end of the jq command, the output goes silent:
rm in.txt touch in.txt tail -f in.txt | jq '.f1' | tee out.txt # in a different terminal: echo '{"f1":1,"f2":2}' >> in.txt echo '{"f1":3,"f2":2}' >> in.txt No output appears in the first terminal and out.txt is empty.
I've tried hundreds of variations but it's an elusive issue. The only workaround I've found, as discovered through mosquitto_sub and The Things Network (which was where I also discovered the issue), is to wrap the tail and jq functions in a shell script:
#!/bin/bash tail -f $1 | while IFS='' read line; do echo $line | jq '.f1' done Then:
./tail_and_jq.sh | tee out.txt # in a different terminal: echo '{"f1":1,"f2":2}' >> in.txt echo '{"f1":3,"f2":2}' >> in.txt And sure enough, the output appears:
1 3 This is with the latest jq installed via Homebrew:
$ echo $SHELL /bin/bash $ jq --version jq-1.5 $ brew install jq Warning: jq 1.5_3 is already installed and up-to-date Is this a (largely undocumented) bug in jq or with my understanding of pipe chains?
tail -fto provide continuous input to a program andteeto process the output. If you were still in need of an answer, I would have suggested simplifying the chain to<in.json jq '.f1' >out.jsonso that you could narrow down what's causing it.tail -f logfile | grep 'foo bar' | awk ...tailbit came about from efforts to break the pipe down (run the first command, tee and redirect to file, tail that, pipe to next command, redirect to file, etc) and run it continuously in sections. The<is a good tool to keep in mind though.