Multiple grep piping (include+exclude) results in not showing anything

Question

Currently I'm trying to tail a log but only showing the lines that has some keywords. Currently I'm using

tail -F file.log | grep -ie 'error\|fatal\|exception\|shutdown\|started'

and I'm getting the expected results: (for example)

10:22 This is an error 10:23 RuntimeException: uncaught problem

I also want to exclude lines that contain a <DATATAG>, even if the keywords slipped into it, because it contains a lot of binary data that clutters my log. I'm then trying to add to the pipe another grep that excludes the tag:

tail -F file.log | grep -ie 'error\|fatal\|exception\|shutdown\|started' | grep -vF '<DATATAG>'

However, this time no lines appear, not even the previous ones that has 'error'/'exception' but not <DATATAG>. When I tried the excluding grep alone:

tail -F file.log | grep -vF '<DATATAG>'

all lines appear, including those that have 'error'/'exception'.

Am I doing something wrong?

It's buffering as @jzer7 says but instead of multiple greps and pipes, consider just one awk: tail -F file.log | awk '!/<DATATAG>/ && tolower($0)~/error|fatal|exception|shutdown|started/'. — Ed Morton
– Ed Morton, Commented May 22, 2016 at 13:10

jzer7 · Accepted Answer · 2016-05-22 08:28:10Z

Your problem is one of buffering. grep is a tricky tool when it comes to that. From the man page:

By default, output is line buffered when standard output is a terminal and block buffered otherwise.

In your example, the first grep is buffering at the block level, so it will not turn an output to the 2nd grep for a while. The solution is to use the --line-buffered option to look like:

tail -F file.log | grep --line-buffered -ie 'error\|fatal\|exception\|shutdown\|started' | grep -vF '<DATATAG>'

Collectives™ on Stack Overflow

Multiple grep piping (include+exclude) results in not showing anything

1 Answer 1

Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Related